Learning interpretable representation for context-specific transcription regulatory networks using a foundation model

Gene expression is shaped by transcription regulatory networks (TRNs), where transcription regulators interact within regulatory elements in a context-specific manner. Despite significant efforts, understanding the intricate interactions of transcription regulators across different genomic regions and cell types remains a major challenge, largely due to data sparsity. Here, we introduce ChromBERT, a foundation model pre-trained on large-scale human ChIP-seq datasets. ChromBERT effectively captures the interaction syntax of approximately one thousand transcription regulators across diverse genomic contexts, generating interpretable representations of context-specific TRNs and their constituent regulators. Fine-tuned for various transcription regulation tasks, ChromBERT demonstrates superior performance in imputing previously unseen cistromes and modeling regulatory effects and dynamics in specific cell types. Notably, ChromBERT adapts its TRN representations for cell-type-specific tasks, providing deep insights into the roles of transcription regulators underlying observed regulatory effects or dynamics without requiring cell-type-specific genomic data for each regulator. By overcoming the limitations of sparse transcription regulator data, ChromBERT significantly enhances our ability to model, interpret, and predict transcriptional regulation across diverse biological contexts.