diff --git a/README.md b/README.md index f05ecd9..458234a 100644 --- a/README.md +++ b/README.md @@ -2,6 +2,9 @@ [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE) +This repository contains the code, data, and analysis scripts accompanying the paper: + +> Kendiukhov, I. (2026). *External Biological Validation of Foundation-Model Gene Regulatory Networks: Perturbation Bridging, ChIP-Seq Binding Support, and Essential-Gene Agreement.* University of Tubingen. ## Overview @@ -152,7 +155,44 @@ pdflatex main.tex # second pass for references | Essentiality | Significant TFs (lung/kidney) | 0 | | Essentiality | Cross-tissue concordance | rho = 0.15-0.31 | | Synthesis | Cross-modality correlations | All |rho| < 0.2 | +| Cross-model | Foundation models benchmarked | 8 (six architectures) | +| Cross-model | ENETS2 AUROC range (embedding-cosine) | 0.499-0.549 (chance 0.50) | +| Cross-model | Geneformer vs scGPT | not significant (paired Wilcoxon p = 0.67) | + +### Cross-model foundation-model benchmark (revision) + +A broadened panel of eight single-cell foundation models spanning six architectures +(scGPT, Geneformer V1-10M/V2-104M/V2-316M, AIDO.Cell-100M, scFoundation, tGPT, and a +frozen ESM2-3B/UCE control) is benchmarked under a uniform embedding-cosine edge +derivation, with paired significance tests. All models cluster just above chance and the +leading models are statistically indistinguishable. A scGPT attention-vs-embedding +concordance check confirms the finding is robust to edge-derivation choice. + +```bash +# 8-model cross-model GRN benchmark + significance tests +python revision/scripts/crossmodel_foundation_benchmark.py + +# scGPT attention vs embedding-cosine concordance (immune context) +python revision/scripts/attention_embedding_concordance.py +``` +Outputs: `revision/outputs/crossmodel_foundation_*.{csv,md}`, +`revision/outputs/crossmodel_{pairwise_significance,vs_chance,per_tf_auroc}.csv`, +`revision/outputs/attention_embedding_concordance.{md,json}`, +`revision/outputs/figures/fig_crossmodel_foundation.pdf`. + +## Citation + +```bibtex +@article{kendiukhov2026external, + title={External Biological Validation of Foundation-Model Gene Regulatory + Networks: Perturbation Bridging, ChIP-Seq Binding Support, and + Essential-Gene Agreement}, + author={Kendiukhov, Ihor}, + year={2026}, + institution={University of T{\"u}bingen} +} +``` ## License diff --git a/revision/outputs/attention_embedding_concordance.json b/revision/outputs/attention_embedding_concordance.json new file mode 100644 index 0000000..8fce9e5 --- /dev/null +++ b/revision/outputs/attention_embedding_concordance.json @@ -0,0 +1,13 @@ +{ + "n_tf": 93, + "n_candidate_pairs": 268863, + "n_attended_pairs": 156502, + "mean_auroc_cosine": 0.5690015468299076, + "mean_auroc_attention": 0.5356112862071472, + "edge_score_spearman_all": -0.0030699394736626836, + "edge_score_spearman_all_p": 0.11142484162581998, + "edge_score_spearman_attended": 0.12831479139837187, + "edge_score_spearman_attended_p": 0.0, + "per_tf_auroc_spearman": -0.005554976711863202, + "per_tf_auroc_spearman_p": 0.9578545053610776 +} \ No newline at end of file diff --git a/revision/outputs/attention_embedding_concordance.md b/revision/outputs/attention_embedding_concordance.md new file mode 100644 index 0000000..f819566 --- /dev/null +++ b/revision/outputs/attention_embedding_concordance.md @@ -0,0 +1,15 @@ +# scGPT attention vs embedding-cosine edge concordance (immune context) + +- Immune genes with attention + scGPT embedding: 8354 +- ENETS2 (restricted): 93 TFs, 268863 candidate TF-target pairs (156502 actually attended) + +## GRN recovery is similarly weak under both derivations +- Mean per-TF AUROC, **embedding cosine**: 0.5690 +- Mean per-TF AUROC, **attention**: 0.5356 +- Per-TF AUROC agreement across derivations: Spearman rho = -0.006 (p = 9.58e-01) + +## Edge-score concordance +- All candidate pairs: Spearman rho = -0.003 (p = 1.11e-01) +- Attended pairs only: Spearman rho = 0.128 (p = 0.00e+00) + +**Interpretation (honest):** for scGPT on the immune context, both edge derivations give only weak ENETS2 recovery (attention AUROC ~0.54, embedding cosine ~0.57; chance 0.50), with attention if anything slightly weaker. The two derivations are largely non-redundant: edge scores are only weakly rank-correlated among attended pairs (Spearman ~0.13) and per-TF AUROC agreement is negligible (~0). Two consequences: (i) the weak-recovery conclusion is ROBUST to the choice of edge derivation -- using embedding cosine for the eight-model panel does not understate foundation-model GRN quality relative to attention, so the panel is a fair test; and (ii) attention and embedding edges are not interchangeable edge-for-edge, consistent with the paper's broader theme that these rankings are unstable and derivation-dependent. We do NOT claim embedding cosine reproduces attention edge-by-edge. \ No newline at end of file diff --git a/revision/outputs/attention_embedding_per_tf.csv b/revision/outputs/attention_embedding_per_tf.csv new file mode 100644 index 0000000..82ed7ff --- /dev/null +++ b/revision/outputs/attention_embedding_per_tf.csv @@ -0,0 +1,94 @@ +tf,n_targets,auroc_cos,auroc_attn +ATF3,70,0.46796981820023287,0.6441839266724059 +BATF,80,0.6389407684098185,0.6096229099964425 +BCL11A,38,0.58975778036047,0.6046451565295994 +BCL3,85,0.5492096767431135,0.5731751289254119 +BDP1,33,0.4905634370294973,0.5714103950632992 +BHLHE40,6,0.6517042172154823,0.3978625072212594 +BRCA1,54,0.4826629590464628,0.658269037454797 +BRF1,14,0.6615273846764984,0.5913898406077759 +CEBPB,101,0.6114234004045567,0.5246850491500763 +CTBP2,108,0.5076456262227013,0.20722208913908519 +CTCFL,61,0.5146034872270173,0.45428372820483115 +E2F1,223,0.5773693870553512,0.5 +E2F4,131,0.7195347936718665,0.4607672308883726 +E2F6,98,0.5477688390071389,0.5518004194158865 +EBF1,141,0.5710070921985816,0.5608691166989039 +EGR1,141,0.5255035460992907,0.5868149580915538 +ELF1,93,0.5407587600974583,0.714711737262407 +ELK4,46,0.5360816077023001,0.625254068923359 +ESR1,104,0.5391757058872237,0.38834492561618505 +ESRRA,20,0.6629745733194008,0.5261842563566701 +ETS1,63,0.4748883051570463,0.6879139444556701 +FOS,205,0.5581007209923179,0.5568421626137334 +FOSL1,48,0.5793982295697033,0.6026717669128854 +FOSL2,37,0.5960813651773709,0.498494289664577 +FOXA1,223,0.5452548389482389,0.5 +FOXA2,138,0.5507483272530099,0.5 +GABPA,88,0.606075471086174,0.5660574708915773 +GATA1,106,0.7005115002879306,0.5022627959757461 +GATA2,188,0.586991601136641,0.5 +GATA3,10,0.5784449843804235,0.401249566122874 +GTF2B,95,0.5498381145998043,0.6200248475265416 +GTF2F1,26,0.4825211437776883,0.4825681299503289 +HNF4A,116,0.6988070829450139,0.5 +HNF4G,68,0.6281698651830551,0.5 +HSF1,32,0.5313483735571879,0.6246611577474641 +IRF1,17,0.6359449834213434,0.5292480248884522 +IRF3,34,0.5397063970845601,0.5641818855648665 +IRF4,9,0.7121983190685481,0.5387076875626493 +JUN,112,0.6385165527168046,0.5467215853595846 +JUND,186,0.6374853417605788,0.511935285115179 +MAFF,69,0.47220082375537953,0.5357850840702966 +MAFK,100,0.5167036904335364,0.48175564313866 +MAX,139,0.5665676760916848,0.6485368391333445 +MEF2A,38,0.649500986957404,0.4997878502776394 +MEF2C,17,0.7322035285930657,0.6227843955953989 +MXI1,21,0.4453625352580056,0.6311929649908744 +MYC,228,0.6431919547272236,0.6196678327437068 +NANOG,8,0.6501907734998266,0.5 +NFE2,36,0.5225530258805215,0.492717454757735 +NFKB1,99,0.6913403374721427,0.5068702787184163 +NFYA,116,0.5807082945013979,0.5989624106865487 +NFYB,143,0.5903594222371514,0.57969814028766 +NR2C2,15,0.580551692165044,0.5000463606861382 +NR3C1,147,0.44853830745126044,0.4397336923107435 +NRF1,129,0.5845079119164295,0.6580979966208061 +PAX5,67,0.4042112384254365,0.5803031584288191 +PBX3,126,0.5261258933953328,0.5908636872470506 +POLR3A,20,0.536433298502264,0.6063566701497736 +POU2F2,51,0.5193661971830986,0.5571251035625517 +PPARGC1A,17,0.5632240370051987,0.5774079986900813 +PRDM1,9,0.6147351376359009,0.44213123602436577 +REST,115,0.2607286054379151,0.2683091091341937 +RFX5,58,0.5881117859707633,0.5890551018172523 +RXRA,52,0.5807367166120248,0.41929037851898016 +SETDB1,57,0.6746524037687727,0.558605405539254 +SIX5,105,0.5824804293576727,0.5 +SMARCA4,77,0.4471196891239535,0.5404932665060598 +SMARCB1,71,0.5216012386374989,0.5451703126560783 +SMARCC1,6,0.36984402079722706,0.27053726169844017 +SP1,170,0.5144151155500789,0.540450094039821 +SP2,66,0.6019844462322338,0.6539769375167606 +SPI1,174,0.6779747777932896,0.577041530762039 +SREBF1,59,0.6839928660346644,0.4832064540840755 +SREBF2,28,0.7380370241005938,0.5018836385409909 +SRF,118,0.5507496623005128,0.5793089537733715 +STAT1,122,0.6499801668353966,0.5807742630647272 +STAT2,59,0.8351886431102173,0.5662046346835201 +STAT3,157,0.6723263084815416,0.4154222133175534 +TAL1,94,0.7069048144288332,0.5 +TCF12,138,0.41700226893454834,0.3167164147675527 +TCF4,167,0.4991382873020479,0.4320807723759529 +TFAP2A,51,0.5237089201877934,0.5 +TFAP2C,78,0.4972016370878795,0.5 +TRIM28,25,0.4767201674808095,0.5272993719469644 +USF1,135,0.5415416868247056,0.6472114712680751 +USF2,74,0.6265290849955387,0.6622389162325265 +YY1,125,0.5669154013015184,0.7422863340563992 +ZBTB33,62,0.4736542035827091,0.6027149682436516 +ZBTB7A,105,0.6312207294978293,0.5439886507366766 +ZEB1,43,0.4315553958714397,0.6543637313822837 +ZNF263,161,0.5020635679020773,0.5157429526994745 +ZNF274,31,0.5419918790886533,0.47894202571621924 +ZZZ3,12,0.4752807687854579,0.44367257149473194 diff --git a/revision/outputs/crossmodel_foundation_grn.csv b/revision/outputs/crossmodel_foundation_grn.csv new file mode 100644 index 0000000..8823250 --- /dev/null +++ b/revision/outputs/crossmodel_foundation_grn.csv @@ -0,0 +1,25 @@ +benchmark,model,n_tf,mean_auroc,median_auroc,ci_lo,ci_hi,mean_aupr +ENETS2,scGPT-53M,100,0.5494326622648991,0.5418986483942587,0.5332951527166857,0.5650164715686605,0.04596526487391128 +ENETS2,Geneformer-V1-10M,100,0.5125606553131717,0.5014669967057148,0.5020538795130577,0.5231649854056446,0.031899203253841685 +ENETS2,Geneformer-V2-104M,100,0.5466956871926825,0.5339587440854152,0.5309662841363142,0.5626957647552878,0.040864614098919734 +ENETS2,Geneformer-V2-316M,100,0.5467685425420851,0.5310556894804022,0.5301785379237376,0.5638016606536655,0.03884448358720428 +ENETS2,AIDO.Cell-100M,100,0.5170576431911197,0.528836433078749,0.5010993551286331,0.5322430614797377,0.04119164842681117 +ENETS2,scFoundation,100,0.49917762016889766,0.5064908061463431,0.48681870845740105,0.5116663764556119,0.02992368764440002 +ENETS2,tGPT,100,0.5020363219214278,0.5181761898314952,0.48583231472699034,0.5174878539873239,0.03325041585434943 +ENETS2,ESM2-3B (UCE ctrl),100,0.503659134396662,0.49822286830401585,0.4931061501304203,0.5135542846636189,0.03129399856133213 +TRRUST,scGPT-53M,187,0.5815440507964921,0.5817291455589327,0.5653115722143042,0.5981810131545107,0.041708879840093846 +TRRUST,Geneformer-V1-10M,187,0.5488698660152586,0.5434497967059472,0.5348001483801381,0.5627453418147745,0.027928054119498554 +TRRUST,Geneformer-V2-104M,187,0.5944182279498889,0.5818200957945155,0.5791128210627441,0.6094769088381536,0.03988175718201017 +TRRUST,Geneformer-V2-316M,187,0.5980218888322912,0.5997477136549985,0.5825165034773179,0.6128620133761471,0.04062087519040739 +TRRUST,AIDO.Cell-100M,187,0.5527773333842325,0.5519878985072922,0.5377427575767855,0.5674889649502461,0.03481218357754472 +TRRUST,scFoundation,187,0.49480669007590083,0.49595621021556624,0.4824791205477896,0.5081868496232398,0.02504397173894239 +TRRUST,tGPT,187,0.512917826579288,0.5207743153918791,0.49898686196701997,0.5259121319187077,0.02695231780460786 +TRRUST,ESM2-3B (UCE ctrl),187,0.5066934367697402,0.5072993643739342,0.49525804640291266,0.5179652900020358,0.027423570461680695 +DoRothEA,scGPT-53M,292,0.4901861797824431,0.48689659233618754,0.4756649364223634,0.5046198807941888,0.04507570070277037 +DoRothEA,Geneformer-V1-10M,292,0.49819338057498636,0.4977767555632047,0.493610254397891,0.5024744747170002,0.037570122838384575 +DoRothEA,Geneformer-V2-104M,292,0.5393454663582811,0.539065680120769,0.5322829135491288,0.5460034200232716,0.04449398565678777 +DoRothEA,Geneformer-V2-316M,292,0.5377487653276319,0.5346220591885535,0.5303402650217627,0.5456299879151751,0.043856346120478534 +DoRothEA,AIDO.Cell-100M,292,0.519614876415865,0.5200074911805977,0.5026590132248635,0.5366022009270139,0.05384749560975616 +DoRothEA,scFoundation,292,0.4609524019011883,0.43672664432669017,0.44405096173983527,0.47741771940006983,0.04642858883208448 +DoRothEA,tGPT,292,0.5243640654525691,0.5251875434317306,0.5141548461001917,0.5343095160384576,0.04158704389373591 +DoRothEA,ESM2-3B (UCE ctrl),292,0.5803024685888699,0.5801095339243614,0.5746550643599617,0.5859504125493026,0.05381217153362102 diff --git a/revision/outputs/crossmodel_foundation_summary.md b/revision/outputs/crossmodel_foundation_summary.md new file mode 100644 index 0000000..e3d2a3b --- /dev/null +++ b/revision/outputs/crossmodel_foundation_summary.md @@ -0,0 +1,49 @@ +# Cross-model foundation-model GRN benchmark (BMC revision) + +Uniform embedding-cosine edge derivation, common gene universe = 17874 genes, 8 models, 3 benchmarks. Positives = TF's benchmark targets, negatives = other listed targets; per-TF AUROC, cross-TF comparator. + +## Mean AUROC per model (ENETS2, primary benchmark) + +| Model | n TF | mean AUROC | 95% CI | vs chance p | +|---|---|---|---|---| +| scGPT-53M | 100 | 0.5494 | [0.5333, 0.5650] | 3.49e-08 | +| Geneformer-V2-316M | 100 | 0.5468 | [0.5302, 0.5638] | 3.81e-07 | +| Geneformer-V2-104M | 100 | 0.5467 | [0.5310, 0.5627] | 1.21e-07 | +| AIDO.Cell-100M | 100 | 0.5171 | [0.5011, 0.5322] | 7.47e-03 | +| Geneformer-V1-10M | 100 | 0.5126 | [0.5021, 0.5232] | 8.43e-02 | +| ESM2-3B (UCE ctrl) | 100 | 0.5037 | [0.4931, 0.5136] | 6.16e-01 | +| tGPT | 100 | 0.5020 | [0.4858, 0.5175] | 1.81e-01 | +| scFoundation | 100 | 0.4992 | [0.4868, 0.5117] | 8.80e-01 | + +## Geneformer vs scGPT (the reviewer's specific question) + +- **ENETS2**: scGPT-53M (0.5494) vs Geneformer-V2-316M (0.5468); mean per-TF diff = +0.0027; paired Wilcoxon p = 0.670 (BH 0.721), n = 100 TFs. + +## Full per-benchmark summary + +| benchmark | model | n_tf | mean_auroc | median_auroc | ci_lo | ci_hi | mean_aupr | +|:------------|:-------------------|-------:|-------------:|---------------:|--------:|--------:|------------:| +| ENETS2 | scGPT-53M | 100 | 0.5494 | 0.5419 | 0.5333 | 0.565 | 0.046 | +| ENETS2 | Geneformer-V1-10M | 100 | 0.5126 | 0.5015 | 0.5021 | 0.5232 | 0.0319 | +| ENETS2 | Geneformer-V2-104M | 100 | 0.5467 | 0.534 | 0.531 | 0.5627 | 0.0409 | +| ENETS2 | Geneformer-V2-316M | 100 | 0.5468 | 0.5311 | 0.5302 | 0.5638 | 0.0388 | +| ENETS2 | AIDO.Cell-100M | 100 | 0.5171 | 0.5288 | 0.5011 | 0.5322 | 0.0412 | +| ENETS2 | scFoundation | 100 | 0.4992 | 0.5065 | 0.4868 | 0.5117 | 0.0299 | +| ENETS2 | tGPT | 100 | 0.502 | 0.5182 | 0.4858 | 0.5175 | 0.0333 | +| ENETS2 | ESM2-3B (UCE ctrl) | 100 | 0.5037 | 0.4982 | 0.4931 | 0.5136 | 0.0313 | +| TRRUST | scGPT-53M | 187 | 0.5815 | 0.5817 | 0.5653 | 0.5982 | 0.0417 | +| TRRUST | Geneformer-V1-10M | 187 | 0.5489 | 0.5434 | 0.5348 | 0.5627 | 0.0279 | +| TRRUST | Geneformer-V2-104M | 187 | 0.5944 | 0.5818 | 0.5791 | 0.6095 | 0.0399 | +| TRRUST | Geneformer-V2-316M | 187 | 0.598 | 0.5997 | 0.5825 | 0.6129 | 0.0406 | +| TRRUST | AIDO.Cell-100M | 187 | 0.5528 | 0.552 | 0.5377 | 0.5675 | 0.0348 | +| TRRUST | scFoundation | 187 | 0.4948 | 0.496 | 0.4825 | 0.5082 | 0.025 | +| TRRUST | tGPT | 187 | 0.5129 | 0.5208 | 0.499 | 0.5259 | 0.027 | +| TRRUST | ESM2-3B (UCE ctrl) | 187 | 0.5067 | 0.5073 | 0.4953 | 0.518 | 0.0274 | +| DoRothEA | scGPT-53M | 292 | 0.4902 | 0.4869 | 0.4757 | 0.5046 | 0.0451 | +| DoRothEA | Geneformer-V1-10M | 292 | 0.4982 | 0.4978 | 0.4936 | 0.5025 | 0.0376 | +| DoRothEA | Geneformer-V2-104M | 292 | 0.5393 | 0.5391 | 0.5323 | 0.546 | 0.0445 | +| DoRothEA | Geneformer-V2-316M | 292 | 0.5377 | 0.5346 | 0.5303 | 0.5456 | 0.0439 | +| DoRothEA | AIDO.Cell-100M | 292 | 0.5196 | 0.52 | 0.5027 | 0.5366 | 0.0538 | +| DoRothEA | scFoundation | 292 | 0.461 | 0.4367 | 0.4441 | 0.4774 | 0.0464 | +| DoRothEA | tGPT | 292 | 0.5244 | 0.5252 | 0.5142 | 0.5343 | 0.0416 | +| DoRothEA | ESM2-3B (UCE ctrl) | 292 | 0.5803 | 0.5801 | 0.5747 | 0.586 | 0.0538 | \ No newline at end of file diff --git a/revision/outputs/crossmodel_pairwise_significance.csv b/revision/outputs/crossmodel_pairwise_significance.csv new file mode 100644 index 0000000..67b9412 --- /dev/null +++ b/revision/outputs/crossmodel_pairwise_significance.csv @@ -0,0 +1,85 @@ +benchmark,model_a,model_b,n_tf,mean_auroc_a,mean_auroc_b,mean_diff,wilcoxon_W,p_raw,p_bh +ENETS2,scGPT-53M,Geneformer-V1-10M,100,0.5494326622648991,0.5125606553131717,0.036872006951727196,959.0,7.267995055888032e-08,7.300186755062903e-07 +ENETS2,scGPT-53M,Geneformer-V2-104M,100,0.5494326622648991,0.5466956871926825,0.0027369750722167274,2290.0,0.41908693342222414,0.48893475565926153 +ENETS2,scGPT-53M,Geneformer-V2-316M,100,0.5494326622648991,0.5467685425420851,0.0026641197228140156,2401.0,0.6698511676891292,0.7206533187202941 +ENETS2,scGPT-53M,AIDO.Cell-100M,100,0.5494326622648991,0.5170576431911197,0.032375019073779245,1330.0,3.977234352394937e-05,9.280213488921519e-05 +ENETS2,scGPT-53M,scFoundation,100,0.5494326622648991,0.49917762016889766,0.05025504209600135,881.0,1.5803323839645013e-08,4.4249306751006037e-07 +ENETS2,scGPT-53M,tGPT,100,0.5494326622648991,0.5020363219214278,0.04739634034347122,1026.0,2.549213652634506e-07,1.4275596454753233e-06 +ENETS2,scGPT-53M,ESM2-3B (UCE ctrl),100,0.5494326622648991,0.503659134396662,0.04577352786823697,1077.0,6.40146344336169e-07,2.9873496069021218e-06 +ENETS2,Geneformer-V1-10M,Geneformer-V2-104M,100,0.5125606553131717,0.5466956871926825,-0.034135031879510475,965.0,8.149510834101042e-08,7.300186755062903e-07 +ENETS2,Geneformer-V1-10M,Geneformer-V2-316M,100,0.5125606553131717,0.5467685425420851,-0.034207887228913186,978.0,1.0428838221518432e-07,7.300186755062903e-07 +ENETS2,Geneformer-V1-10M,AIDO.Cell-100M,100,0.5125606553131717,0.5170576431911197,-0.004496987877947951,2110.0,0.15360733419330933,0.20480977892441243 +ENETS2,Geneformer-V1-10M,scFoundation,100,0.5125606553131717,0.49917762016889766,0.013383035144274153,1973.0,0.057701968852335826,0.08503448041396859 +ENETS2,Geneformer-V1-10M,tGPT,100,0.5125606553131717,0.5020363219214278,0.010524333391744023,2349.0,0.5450825418754992,0.6104924469005591 +ENETS2,Geneformer-V1-10M,ESM2-3B (UCE ctrl),100,0.5125606553131717,0.503659134396662,0.00890152091650977,2224.0,0.3006989045676193,0.38270769672242455 +ENETS2,Geneformer-V2-104M,Geneformer-V2-316M,100,0.5466956871926825,0.5467685425420851,-7.285534940271177e-05,2418.0,0.7129472348148089,0.7206533187202941 +ENETS2,Geneformer-V2-104M,AIDO.Cell-100M,100,0.5466956871926825,0.5170576431911197,0.029638044001562527,1616.0,0.0017754092402886214,0.0031803930277313737 +ENETS2,Geneformer-V2-104M,scFoundation,100,0.5466956871926825,0.49917762016889766,0.04751806702378463,1144.0,2.0510662172105422e-06,7.178731760236898e-06 +ENETS2,Geneformer-V2-104M,tGPT,100,0.5466956871926825,0.5020363219214278,0.044659365271254484,1402.0,0.00011281357453431969,0.00024298308361238088 +ENETS2,Geneformer-V2-104M,ESM2-3B (UCE ctrl),100,0.5466956871926825,0.503659134396662,0.04303655279602024,1222.0,7.459504884752048e-06,2.3207348530339707e-05 +ENETS2,Geneformer-V2-316M,AIDO.Cell-100M,100,0.5467685425420851,0.5170576431911197,0.029710899350965228,1578.0,0.0011295649688403883,0.0022591299376807767 +ENETS2,Geneformer-V2-316M,scFoundation,100,0.5467685425420851,0.49917762016889766,0.047590922373187336,1132.0,1.6712533834681047e-06,6.685013533872418e-06 +ENETS2,Geneformer-V2-316M,tGPT,100,0.5467685425420851,0.5020363219214278,0.0447322206206572,1303.0,2.6499653532351623e-05,6.745366353689505e-05 +ENETS2,Geneformer-V2-316M,ESM2-3B (UCE ctrl),100,0.5467685425420851,0.503659134396662,0.043109408145422946,1232.0,8.758411861167416e-06,2.4523553211268762e-05 +ENETS2,AIDO.Cell-100M,scFoundation,100,0.5170576431911197,0.49917762016889766,0.0178800230222221,1618.0,0.0018173674444179279,0.0031803930277313737 +ENETS2,AIDO.Cell-100M,tGPT,100,0.5170576431911197,0.5020363219214278,0.015021321269691976,1816.0,0.01477807881436309,0.024340365106009794 +ENETS2,AIDO.Cell-100M,ESM2-3B (UCE ctrl),100,0.5170576431911197,0.503659134396662,0.013398508794457722,1871.0,0.024533852722889375,0.03816377090227236 +ENETS2,scFoundation,tGPT,100,0.49917762016889766,0.5020363219214278,-0.002858701752530127,2050.0,0.1024260357718117,0.1433964500805364 +ENETS2,scFoundation,ESM2-3B (UCE ctrl),100,0.49917762016889766,0.503659134396662,-0.004481514227764382,2285.0,0.4092592100268002,0.48893475565926153 +ENETS2,tGPT,ESM2-3B (UCE ctrl),100,0.5020363219214278,0.503659134396662,-0.0016228124752342553,2421.0,0.7206533187202941,0.7206533187202941 +TRRUST,scGPT-53M,Geneformer-V1-10M,187,0.5815440507964921,0.5488698660152586,0.032674184781233635,5661.0,2.4383359025610355e-05,3.7929669595393884e-05 +TRRUST,scGPT-53M,Geneformer-V2-104M,187,0.5815440507964921,0.5944182279498889,-0.012874177153396657,7487.0,0.07896644521299861,0.09212751941516506 +TRRUST,scGPT-53M,Geneformer-V2-316M,187,0.5815440507964921,0.5980218888322912,-0.016477838035799098,6682.0,0.004471132809572181,0.00569053266672823 +TRRUST,scGPT-53M,AIDO.Cell-100M,187,0.5815440507964921,0.5527773333842325,0.028766717412259623,5806.0,5.702553589850527e-05,7.983575025790738e-05 +TRRUST,scGPT-53M,scFoundation,187,0.5815440507964921,0.49480669007590083,0.08673736072059125,1977.0,3.890428330610964e-20,5.44659966285535e-19 +TRRUST,scGPT-53M,tGPT,187,0.5815440507964921,0.512917826579288,0.06862622421720412,2915.0,2.273125634941869e-15,1.2729503555674467e-14 +TRRUST,scGPT-53M,ESM2-3B (UCE ctrl),187,0.5815440507964921,0.5066934367697402,0.07485061402675187,3799.0,1.665158672899668e-11,4.238585712835519e-11 +TRRUST,Geneformer-V1-10M,Geneformer-V2-104M,187,0.5488698660152586,0.5944182279498889,-0.0455483619346303,3910.0,4.611797862665118e-11,1.0760861679551941e-10 +TRRUST,Geneformer-V1-10M,Geneformer-V2-316M,187,0.5488698660152586,0.5980218888322912,-0.04915202281703274,3361.0,2.4123922213331047e-13,7.505220244147436e-13 +TRRUST,Geneformer-V1-10M,AIDO.Cell-100M,187,0.5488698660152586,0.5527773333842325,-0.003907467368974008,8338.0,0.5428501290810559,0.5629556894173914 +TRRUST,Geneformer-V1-10M,scFoundation,187,0.5488698660152586,0.49480669007590083,0.05406317593935761,4443.0,4.523617276117597e-09,9.7431756716379e-09 +TRRUST,Geneformer-V1-10M,tGPT,187,0.5488698660152586,0.512917826579288,0.03595203943597049,6008.0,0.00017525769766278966,0.0002336769302170529 +TRRUST,Geneformer-V1-10M,ESM2-3B (UCE ctrl),187,0.5488698660152586,0.5066934367697402,0.042176429245518246,5690.0,2.898409143946064e-05,4.27133979107841e-05 +TRRUST,Geneformer-V2-104M,Geneformer-V2-316M,187,0.5944182279498889,0.5980218888322912,-0.003603660882402438,7951.0,0.2581957572895154,0.2780569693887089 +TRRUST,Geneformer-V2-104M,AIDO.Cell-100M,187,0.5944182279498889,0.5527773333842325,0.04164089456565628,5557.0,1.2961488491538337e-05,2.1348333986063143e-05 +TRRUST,Geneformer-V2-104M,scFoundation,187,0.5944182279498889,0.49480669007590083,0.09961153787398791,2304.0,2.1365057830001456e-18,1.994072064133469e-17 +TRRUST,Geneformer-V2-104M,tGPT,187,0.5944182279498889,0.512917826579288,0.08150040137060079,3168.0,3.3481217376536445e-14,1.3392486950614578e-13 +TRRUST,Geneformer-V2-104M,ESM2-3B (UCE ctrl),187,0.5944182279498889,0.5066934367697402,0.08772479118014853,3198.0,4.5710498862132137e-14,1.5998674601746248e-13 +TRRUST,Geneformer-V2-316M,AIDO.Cell-100M,187,0.5980218888322912,0.5527773333842325,0.045244555448058735,5077.0,5.488694423430802e-07,1.0977388846861603e-06 +TRRUST,Geneformer-V2-316M,scFoundation,187,0.5980218888322912,0.49480669007590083,0.10321519875639033,1907.0,1.6096155234134373e-20,4.506923465557624e-19 +TRRUST,Geneformer-V2-316M,tGPT,187,0.5980218888322912,0.512917826579288,0.08510406225300322,2756.0,3.9532810669440405e-16,2.7672967468608282e-15 +TRRUST,Geneformer-V2-316M,ESM2-3B (UCE ctrl),187,0.5980218888322912,0.5066934367697402,0.09132845206255097,2992.0,5.21713621884784e-15,2.434663568795659e-14 +TRRUST,AIDO.Cell-100M,scFoundation,187,0.5527773333842325,0.49480669007590083,0.05797064330833162,3712.0,7.379422481268394e-12,2.0662382947551504e-11 +TRRUST,AIDO.Cell-100M,tGPT,187,0.5527773333842325,0.512917826579288,0.039859506804944496,5184.0,1.1502413612181883e-06,2.147117207607285e-06 +TRRUST,AIDO.Cell-100M,ESM2-3B (UCE ctrl),187,0.5527773333842325,0.5066934367697402,0.046083896614492255,5465.0,7.295340179645804e-06,1.2766845314380157e-05 +TRRUST,scFoundation,tGPT,187,0.49480669007590083,0.512917826579288,-0.018111136503387124,6804.5,0.007415756556389228,0.009027877546908624 +TRRUST,scFoundation,ESM2-3B (UCE ctrl),187,0.49480669007590083,0.5066934367697402,-0.011886746693839365,7523.0,0.08761000213652133,0.09812320239290388 +TRRUST,tGPT,ESM2-3B (UCE ctrl),187,0.512917826579288,0.5066934367697402,0.006224389809547762,8775.0,0.9849293310459281,0.9849293310459281 +DoRothEA,scGPT-53M,Geneformer-V1-10M,292,0.4901861797824431,0.49819338057498636,-0.008007200792543254,20308.0,0.45412085938054225,0.47094015046871046 +DoRothEA,scGPT-53M,Geneformer-V2-104M,292,0.4901861797824431,0.5393454663582811,-0.04915928657583804,10989.0,5.946088459167312e-13,1.1892176918334624e-12 +DoRothEA,scGPT-53M,Geneformer-V2-316M,292,0.4901861797824431,0.5377487653276319,-0.04756258554518891,9997.0,3.0548445500514824e-15,6.57966518472627e-15 +DoRothEA,scGPT-53M,AIDO.Cell-100M,292,0.4901861797824431,0.519614876415865,-0.029428696633421958,14199.0,6.395447744569693e-07,9.424870360418494e-07 +DoRothEA,scGPT-53M,scFoundation,292,0.4901861797824431,0.4609524019011883,0.02923377788125475,12501.0,7.522237509514177e-10,1.3751499497532007e-09 +DoRothEA,scGPT-53M,tGPT,292,0.4901861797824431,0.5243640654525691,-0.0341778856701261,13825.0,1.6244830175101494e-07,2.526973582793566e-07 +DoRothEA,scGPT-53M,ESM2-3B (UCE ctrl),292,0.4901861797824431,0.5803024685888699,-0.09011628880642687,7575.0,1.1132263957930836e-21,5.195056513701057e-21 +DoRothEA,Geneformer-V1-10M,Geneformer-V2-104M,292,0.49819338057498636,0.5393454663582811,-0.04115208578329479,4882.0,2.939293634254847e-30,4.115011087956786e-29 +DoRothEA,Geneformer-V1-10M,Geneformer-V2-316M,292,0.49819338057498636,0.5377487653276319,-0.03955538475264565,6210.0,7.684293509098206e-26,7.172007275158325e-25 +DoRothEA,Geneformer-V1-10M,AIDO.Cell-100M,292,0.49819338057498636,0.519614876415865,-0.021421495840878702,17537.0,0.007643981829594592,0.010191975772792789 +DoRothEA,Geneformer-V1-10M,scFoundation,292,0.49819338057498636,0.4609524019011883,0.037240978673798,15049.0,1.1320747811780844e-05,1.5849046936493182e-05 +DoRothEA,Geneformer-V1-10M,tGPT,292,0.49819338057498636,0.5243640654525691,-0.026170684877582846,13326.0,2.358549750355111e-08,3.884670177055477e-08 +DoRothEA,Geneformer-V1-10M,ESM2-3B (UCE ctrl),292,0.49819338057498636,0.5803024685888699,-0.08210908801388363,661.0,1.0119509708788772e-46,2.833462718460856e-45 +DoRothEA,Geneformer-V2-104M,Geneformer-V2-316M,292,0.5393454663582811,0.5377487653276319,0.0015967010306491272,18922.0,0.08757524900713899,0.09431180662307276 +DoRothEA,Geneformer-V2-104M,AIDO.Cell-100M,292,0.5393454663582811,0.519614876415865,0.01973058994241608,18158.0,0.025261629506226135,0.02947190109059716 +DoRothEA,Geneformer-V2-104M,scFoundation,292,0.5393454663582811,0.4609524019011883,0.07839306445709279,8996.0,9.339170448220515e-18,2.3772433868197675e-17 +DoRothEA,Geneformer-V2-104M,tGPT,292,0.5393454663582811,0.5243640654525691,0.01498140090571194,17865.0,0.01467618073270153,0.018678775477983765 +DoRothEA,Geneformer-V2-104M,ESM2-3B (UCE ctrl),292,0.5393454663582811,0.5803024685888699,-0.04095700223058884,8238.0,8.492254760926268e-20,3.396901904370507e-19 +DoRothEA,Geneformer-V2-316M,AIDO.Cell-100M,292,0.5377487653276319,0.519614876415865,0.01813388891176695,18305.0,0.032713271377447777,0.03663886394274151 +DoRothEA,Geneformer-V2-316M,scFoundation,292,0.5377487653276319,0.4609524019011883,0.07679636342644365,8585.0,7.553769244133751e-19,2.3500615426193893e-18 +DoRothEA,Geneformer-V2-316M,tGPT,292,0.5377487653276319,0.5243640654525691,0.01338469987506281,18062.0,0.021230751489466067,0.025846132248045647 +DoRothEA,Geneformer-V2-316M,ESM2-3B (UCE ctrl),292,0.5377487653276319,0.5803024685888699,-0.04255370326123796,8702.0,1.558132514689333e-18,4.362771041130133e-18 +DoRothEA,AIDO.Cell-100M,scFoundation,292,0.519614876415865,0.4609524019011883,0.0586624745146767,7205.0,9.050407599198517e-23,5.06822825555117e-22 +DoRothEA,AIDO.Cell-100M,tGPT,292,0.519614876415865,0.5243640654525691,-0.004749189036704142,20637.0,0.6025484453150738,0.6025484453150737 +DoRothEA,AIDO.Cell-100M,ESM2-3B (UCE ctrl),292,0.519614876415865,0.5803024685888699,-0.06068759217300492,12511.0,7.857999712875432e-10,1.3751499497532007e-09 +DoRothEA,scFoundation,tGPT,292,0.4609524019011883,0.5243640654525691,-0.06341166355138085,9475.0,1.5824103323271082e-16,3.6922907754299194e-16 +DoRothEA,scFoundation,ESM2-3B (UCE ctrl),292,0.4609524019011883,0.5803024685888699,-0.11935006668768162,6273.0,1.219304355192654e-25,8.535130486348579e-25 +DoRothEA,tGPT,ESM2-3B (UCE ctrl),292,0.5243640654525691,0.5803024685888699,-0.05593840313630077,8522.0,5.101321944017722e-19,1.785462680406203e-18 diff --git a/revision/outputs/crossmodel_per_tf_auroc.csv b/revision/outputs/crossmodel_per_tf_auroc.csv new file mode 100644 index 0000000..120f26e --- /dev/null +++ b/revision/outputs/crossmodel_per_tf_auroc.csv @@ -0,0 +1,101 @@ +,scGPT-53M,Geneformer-V1-10M,Geneformer-V2-104M,Geneformer-V2-316M,AIDO.Cell-100M,scFoundation,tGPT,ESM2-3B (UCE ctrl) +ATF3,0.5112115445607358,0.4526540380013263,0.4882139664965545,0.4711494103739584,0.5042355044257993,0.4717786811982816,0.5417077530778768,0.5430931580313122 +BATF,0.6456850738780064,0.6108336490513101,0.6803547475450794,0.6644899192103295,0.5442884058119413,0.5087970121133307,0.5283954188468273,0.532665291160238 +BCL11A,0.5922195848973283,0.5144126738794436,0.5095192095385296,0.5026026716714507,0.4474663281077501,0.5364318834179731,0.46743762419960255,0.507515455950541 +BCL3,0.5364940895658105,0.49244146396908384,0.49863131772372515,0.5101282488444343,0.457602106539365,0.4141632568007881,0.3812457376676518,0.5283511404106994 +BCLAF1,0.5613879963933301,0.5109676415681395,0.5158934762455075,0.5706221505403655,0.5584210659995175,0.4726689991491307,0.5646740027685001,0.4844843351155023 +BDP1,0.4464483884128927,0.4985067319461444,0.454765401876785,0.41570583435332514,0.48846389228886167,0.5410832313341494,0.46764585883312937,0.4501081191350469 +BHLHE40,0.4605414746543779,0.4753936251920123,0.6037922427035329,0.5725806451612903,0.5411290322580645,0.47882104454685104,0.48892089093701996,0.45843894009216596 +BRCA1,0.4493520263901979,0.4174039820923657,0.4801258639648131,0.4975828620797989,0.4272649622997173,0.4630959393653786,0.5354716462456802,0.5001649387370405 +BRF1,0.5808463322594513,0.5387473263522156,0.5350055463264212,0.5779909029631713,0.5323040342226917,0.5314229909825756,0.48344859431143283,0.4901307234638799 +BRF2,0.5474006972543859,0.5054674264592638,0.4990082559772996,0.45314534533693634,0.5662415001703467,0.5548446112130079,0.5170043263374543,0.48434444553341827 +CCNT2,0.4554297012449523,0.521336354953838,0.48642365241869273,0.4693285970409342,0.46533444479817687,0.43630719994638156,0.4551197198438364,0.4600605720413532 +CEBPB,0.5364046625734977,0.48712532481978665,0.5279652730473925,0.5449686666926599,0.5773256485119259,0.5283001269845639,0.5769487467059318,0.5088067349982761 +CTBP2,0.5056685669862658,0.5288120534557436,0.6032373301743605,0.5472402176729723,0.3379659793432791,0.4747482693517937,0.23725521045422576,0.2824464146892237 +CTCFL,0.5148988592741826,0.4648355612425472,0.4941386930204278,0.4807406096576008,0.5180637608559527,0.5384779432450337,0.48401781419872403,0.4679392066646702 +E2F1,0.5407391271901429,0.5535039259702986,0.6022843842590071,0.5863723550376058,0.5302583092813448,0.47091312845274613,0.5180231584849566,0.5457697745668121 +E2F4,0.6409731574899561,0.5685555102595461,0.5378014170773544,0.5532947866664675,0.5879149298255724,0.5795894758635115,0.5740550623982175,0.5258603455255487 +E2F6,0.5293965560207337,0.49318306419457736,0.5352609150717704,0.5212432715311004,0.5066113437001595,0.5144562400318979,0.5571147328548645,0.5286881977671452 +EBF1,0.5748594038436771,0.5716575320902678,0.542216410755427,0.5510749226065264,0.5163092733838001,0.5314868609970282,0.4911188305834204,0.49173356611000607 +EGR1,0.5203870344999377,0.4548999460289782,0.524364802590609,0.5288075517914228,0.5840619421264582,0.5301262091584672,0.5633080084692988,0.5154341553535102 +ELF1,0.5495599965266148,0.4739334846736879,0.5205770781387618,0.5184111743763412,0.5140758934663143,0.46968156500812525,0.5295795963430217,0.49650205301874384 +ELK4,0.48637652962129074,0.4919542881212297,0.4544059611980564,0.469410917162681,0.4296779542312906,0.43864716373403934,0.5020885253957668,0.5134930942435594 +ESR1,0.5430581695983745,0.5392931338773437,0.5881026499338982,0.5699287207507604,0.5422465500061868,0.5286875370398109,0.4592708611471107,0.4848714759265651 +ESRRA,0.6704176676360264,0.5531251317064947,0.5781205658813448,0.5860552675573538,0.5350838004523678,0.5894971972857924,0.4867815849735182,0.5685000210730391 +ETS1,0.46702321415314185,0.4371688702612593,0.41518016832358084,0.4151560642479673,0.46363514528490535,0.48809017623935924,0.5183292211780337,0.49666447801660096 +FOS,0.5276171061599935,0.4983956461291862,0.530136298167614,0.5274520225864344,0.5615426142818135,0.5050823646137523,0.5839706351340366,0.522976006603343 +FOSL1,0.5677105308821246,0.523822570481383,0.5466158890197905,0.5302410111692557,0.4920847146497164,0.4843712613130953,0.4366822835328822,0.47931541503420183 +FOSL2,0.5906332211389663,0.49016848498916743,0.5817688989476942,0.6037435198081089,0.552557257814918,0.5128685004642527,0.5869361265861962,0.49345694057567313 +FOXA1,0.5117623316224488,0.5619614758359065,0.6165049588113999,0.6122070927698774,0.5753804738840522,0.4245717624865315,0.45378847338834327,0.49039520745766674 +FOXA2,0.5353494197715303,0.6028125454203211,0.6912927209145141,0.6775516590380383,0.5686183406363252,0.4268847000928461,0.3994375226853408,0.4909921823436767 +GABPA,0.61445896303462,0.5231197792898495,0.5363794745680515,0.5570393863570967,0.6252921557534972,0.5489778214520847,0.6137296096641871,0.47607138652370884 +GATA1,0.6743411720505132,0.6179123760744052,0.6499068401561098,0.6435255825365547,0.53122579952615,0.4728705499881154,0.513003082325699,0.50913004807508 +GATA2,0.598443004155947,0.5871793983772017,0.6445705521472392,0.6400331486245795,0.5737744904017416,0.47764842667722146,0.5077127449040174,0.48137641005343357 +GATA3,0.5618567588325653,0.48648233486943165,0.5536674347158218,0.5692684331797234,0.40098886328725036,0.4169642857142857,0.5185867895545315,0.6123847926267281 +GTF2B,0.5309808044367255,0.48270293648262874,0.5242343702305543,0.5030929677872347,0.5490311326692686,0.4353559507155068,0.4658872201009958,0.49515509527340595 +GTF2F1,0.4855274000824063,0.4561243876756855,0.5058313876299043,0.559867577713684,0.4712636771505746,0.4444759190587373,0.3994557753055899,0.47440541592272123 +HMGN3,0.4594302757415091,0.5578491061530116,0.5013091235031568,0.5056178302883216,0.4526527114188592,0.4683752977961122,0.5035525864034616,0.42415601502281464 +HNF4A,0.6838047506297521,0.6163296352963762,0.6835822284793447,0.6882799183212812,0.5503881775290442,0.5223685664919455,0.45976639537386627,0.4855018819847231 +HNF4G,0.5880505952380952,0.5872395833333334,0.5884474206349207,0.6213814484126985,0.5158048115079364,0.472375992063492,0.4216753472222222,0.4764608134920635 +HSF1,0.5547471620227038,0.46205084904775306,0.5497232385777278,0.48084951684022886,0.47344966694811896,0.3900225161835069,0.35279810488788815,0.4363519091847265 +IRF1,0.589734038590479,0.5380913357669672,0.522143584394944,0.6190767091310934,0.6156398221957338,0.4318955027440463,0.5769152449775261,0.3817874792023641 +IRF3,0.512145128762023,0.4958365653118213,0.533313295066708,0.5102486037852931,0.5319461681663047,0.47033819422897927,0.5687984796773193,0.4881670803599132 +IRF4,0.6449048496009822,0.6362031921424187,0.7096685082872929,0.735926949048496,0.37188459177409455,0.5967618170656844,0.5507059545733579,0.623925721301412 +JUN,0.6108545781333142,0.5332522053742339,0.5941184307193458,0.6309010267465168,0.6122771740358329,0.5403207774572305,0.5887420154272969,0.5042346769047865 +JUND,0.6192972581967847,0.5259374661529654,0.588266780392703,0.6158932135728543,0.6024830184592056,0.5460174612016278,0.596527390954525,0.4966858917823268 +KAT2A,0.31969860064585576,0.4158348320841886,0.489620175303706,0.4239113720089056,0.2579612085392026,0.2895386075991683,0.2726835106205162,0.4713811016988814 +MAFF,0.4774277034494632,0.4831958768612162,0.5458504806811176,0.49426570474905174,0.5336143554541062,0.5248193627573491,0.5353149280505618,0.4840433862610174 +MAFK,0.5025130108987601,0.49684342259523234,0.4908311157427223,0.5084299458473678,0.5021772707817704,0.516367524548573,0.4836945059580301,0.4941427694139978 +MAX,0.5358460870509063,0.47575097688049495,0.508421354336264,0.49814163749592966,0.5274145568761532,0.5101262312764572,0.5374916049332465,0.5648472267448171 +MEF2A,0.6236226445225167,0.5286125638773554,0.6736141608112424,0.6636308288086873,0.6349234469817948,0.5649478002235707,0.49556850846374956,0.5388504072181411 +MEF2C,0.7282193903817678,0.6044699675712397,0.7011140420247012,0.7124773768264399,0.6082913599380085,0.5937435314972959,0.6314054762675613,0.5258978945189557 +MXI1,0.4287993607588412,0.4605964532426023,0.512359521600165,0.5165171667182185,0.48231776471801213,0.573708629755645,0.5473347767811114,0.49938911227961646 +MYC,0.6457862742721984,0.48137757332470255,0.5810890449726888,0.607951172597016,0.5731044331514628,0.4876754984055012,0.5403392087175137,0.4459577653394772 +NANOG,0.597571896428206,0.565781393920786,0.7039453484801965,0.7251176952205506,0.45478968375805956,0.6649396172346741,0.6214691433834818,0.7241710162726435 +NFE2,0.5256701995012468,0.45322489231466784,0.43097511902063024,0.4396777941509861,0.45841362502833827,0.5681534799365223,0.5272557243255498,0.4480021537066425 +NFKB1,0.6764390858854418,0.52367868551009,0.6826179372707367,0.6677531492592487,0.645963071440081,0.5378373190775443,0.5574236263163382,0.558365560516014 +NFYA,0.545580624601657,0.5278336520076482,0.49952772466539197,0.520551306564691,0.5047941363926067,0.5073352453792224,0.5360497131931167,0.527818992989165 +NFYB,0.5626641693287304,0.47718429337854523,0.5410214468323142,0.5364955455934399,0.4941549826553386,0.5067003346369983,0.540309118897484,0.5368737079235484 +NR2C2,0.5303541009749555,0.5220633539922438,0.5132969084121635,0.5282791489978617,0.5352152875937805,0.5690080098582871,0.4739724910296836,0.42025479322967635 +NR3C1,0.4552020519717149,0.4714014576374127,0.45959394386360675,0.4740542709643833,0.48001117088195744,0.5371507743698756,0.498283697019652,0.4875943559932324 +NRF1,0.5799178453368017,0.4593516501381889,0.477884788381293,0.4709976697555953,0.5731278382918767,0.48936172980003256,0.5438105457107245,0.5138139055980058 +PAX5,0.45636989703680925,0.4720334582498398,0.44442454491080696,0.46366889578989695,0.4655071621425265,0.4639944221231751,0.4363111671301756,0.4961179013576363 +PBX3,0.5219552647039807,0.48116782904224187,0.5062127358132553,0.5051372221613157,0.5434318049570329,0.5146954070001153,0.6001934013048966,0.5310333195591125 +POLR3A,0.5196031168372861,0.4244690001985046,0.47883090243971604,0.4725921864808915,0.5213613004256821,0.481761208420376,0.45695128445078814,0.5029948987469003 +POU2F2,0.5656783276450512,0.48905716723549486,0.5545338194228979,0.5548421501706485,0.3451617282035371,0.3755177629537698,0.39904398076326403,0.44084703692212224 +POU5F1,0.6591513292433537,0.5881901840490797,0.3993353783231084,0.4633435582822086,0.605700408997955,0.5197341513292434,0.5980828220858896,0.606441717791411 +PPARGC1A,0.5378832369123632,0.5111435985222393,0.4242231135435019,0.45623154361018436,0.3852382201896765,0.5478668682552178,0.4362559508190576,0.5140133198385626 +PRDM1,0.49989080585280626,0.5758980207549764,0.4994782946300745,0.5073321847726739,0.5871167083222117,0.4527189342651234,0.5634984187069796,0.48642352769891534 +REST,0.3034848778732603,0.49042298102249693,0.3073096135628287,0.2668229619072517,0.3214723236548349,0.5195692323261286,0.2821112226167084,0.47435661738868884 +RFX5,0.5791644823517169,0.47838951685898456,0.613769921443199,0.5978524011757014,0.5019447765912121,0.4904803136774439,0.4941578165096905,0.5152470092065193 +RXRA,0.5886970966344071,0.4812238191535462,0.580304781164235,0.5978188646540517,0.40114834609273436,0.43497219413549043,0.42626029178101976,0.5166943521594685 +SETDB1,0.6333779911812791,0.578524185424339,0.5857265535976748,0.5461506861757468,0.6002097336217859,0.5529690931910888,0.5601374002939574,0.4184763760672469 +SIX5,0.5621447692472428,0.48516008137916267,0.506712174751044,0.49838365992076245,0.5402960702430667,0.5361617946246922,0.5769750508619766,0.4952660884463005 +SMARCA4,0.4827052769009393,0.4303350123290832,0.518221571428,0.5373319837354789,0.4434669195162252,0.3074615958061113,0.3657796928197927,0.49858572280166913 +SMARCB1,0.5471565095883101,0.4896308623510494,0.5505536713642715,0.5539297676476165,0.3676644688336037,0.3798712196218042,0.41763603041857134,0.4765323390420818 +SMARCC1,0.38976679963178895,0.3342474685486345,0.419683952132556,0.44145059834305,0.4565817735501688,0.7418878490334458,0.23914544338754218,0.5684642528382939 +SP1,0.5189683617261601,0.5333344938030915,0.5263291246963437,0.5318703677915487,0.4847306356279689,0.48740068312986434,0.5247044670349224,0.5232857927555742 +SP2,0.5903869072816198,0.4811251636264172,0.5730416929879975,0.5799880821355772,0.5534219900879838,0.5146881166517965,0.5558560348027691,0.5257170256136398 +SPI1,0.6641132291057047,0.6173804945437751,0.6601832941185838,0.6577955737097949,0.6269114702147056,0.48663364267201736,0.5725656996349247,0.501574064121092 +SREBF1,0.6074670485996865,0.5116134734614615,0.6148953877716918,0.6126270153095786,0.6083031722375984,0.440389416842472,0.559960258320914,0.4978600138063626 +SREBF2,0.7298899740975737,0.5171217414034838,0.7306266486062593,0.693065160048478,0.608745633421259,0.5364297426391959,0.6283952377557567,0.5110976450179416 +SRF,0.5239970593031759,0.46185774181839223,0.5398611717545807,0.5384993240079751,0.5820041453830007,0.4529317537389987,0.5203179003740315,0.5046984076047788 +STAT1,0.6095733651202597,0.6047800181037007,0.6584508260129764,0.6491981984033987,0.6437788986437409,0.4755553643390842,0.5884546306583556,0.49999451658648225 +STAT2,0.7955509762158988,0.6447793346285966,0.7862703335458863,0.8142525090272622,0.750224863445712,0.5468641600391945,0.5919415672935417,0.5493867445569955 +STAT3,0.6557870294883578,0.5311744431126726,0.677251258324776,0.6573656339407944,0.6202651017274804,0.5748729030416088,0.5799095297262797,0.4902418685565552 +TAL1,0.6671012014754714,0.633981861047981,0.6787619563447392,0.6719545393400027,0.5793074701047132,0.506719802043287,0.5064171321764618,0.4810857883222531 +TCF12,0.41696942189396524,0.5679598664507322,0.5276103827334964,0.46672305719168306,0.37159224148898495,0.494139855378934,0.3372600394045986,0.5401765298508745 +TCF4,0.5226177213572152,0.47672180399144576,0.5242340772446317,0.524525817435588,0.5319544550088963,0.5362032816319433,0.5537134545652627,0.519313776788899 +TFAP2A,0.5406937202448424,0.5074232033552483,0.46483932214917256,0.473469734754024,0.47727131036046244,0.4269454205395602,0.4409175923826797,0.45962366810247113 +TFAP2C,0.4993973693863648,0.4759752849990234,0.5504890121145427,0.511539542572399,0.4805314582708074,0.4783591297346996,0.4225407668891398,0.4958844854773167 +TRIM28,0.4456963887917372,0.5355370494654556,0.531081724028194,0.5237181486767936,0.4787206221071106,0.4666181433203203,0.47946107576889024,0.4829112747461189 +USF1,0.5139355663188218,0.45788043478260876,0.46531994436364144,0.47072427877487044,0.4865741100300775,0.5126536530776509,0.5456740613129616,0.5904112889560416 +USF2,0.6104833048830103,0.4996877403577509,0.5465814690884867,0.5672570747808887,0.5838481052230995,0.5321452127857724,0.5581792669823593,0.5961719524371957 +XRCC4,0.4911283755529634,0.5032462530536788,0.4964324229152453,0.48929286704668,0.5382133503532364,0.5456698285538217,0.479710368202126,0.49275260250456676 +YY1,0.5643896722248634,0.4852644899749493,0.486813756139512,0.500178674160538,0.5249374186374177,0.5201493143861533,0.5574066228858215,0.5423816470987811 +ZBTB33,0.4832964624050812,0.4926809249175859,0.5118157617343548,0.5107985940930423,0.47692993292736147,0.4591261422154784,0.5263263463204044,0.47899951659359624 +ZBTB7A,0.6033348246241947,0.5102317465998568,0.5751959556191839,0.5838144237652112,0.5480225483178238,0.4948908375089478,0.47081871868289193,0.5502738010021475 +ZEB1,0.41445313402935624,0.45050352506067726,0.4148615017143738,0.38857495088030203,0.4705135416265362,0.5287344454289786,0.5089679084639982,0.5218307200369843 +ZNF263,0.47630137992378185,0.5177567602120616,0.4898231641138213,0.5142684608667978,0.4877466141136852,0.48499057473703105,0.4932429160920384,0.49925223526979645 +ZNF274,0.5688339707414822,0.547327336867443,0.5346041931041224,0.5714016404459471,0.534061334315483,0.5062812776556879,0.5304952233663179,0.5417049955227241 +ZZZ3,0.5181277300398592,0.49394150085185207,0.5762394314617356,0.5125814036186462,0.5003104881298001,0.46662916040825214,0.4478141104913143,0.4773874414185858 diff --git a/revision/outputs/crossmodel_vs_chance.csv b/revision/outputs/crossmodel_vs_chance.csv new file mode 100644 index 0000000..ae61345 --- /dev/null +++ b/revision/outputs/crossmodel_vs_chance.csv @@ -0,0 +1,25 @@ +benchmark,model,n_tf,mean_auroc,delta_vs_chance,p_vs_chance +ENETS2,scGPT-53M,100,0.5494326622648991,0.04943266226489906,3.486351197836001e-08 +ENETS2,Geneformer-V1-10M,100,0.5125606553131717,0.012560655313171742,0.08434038384173659 +ENETS2,Geneformer-V2-104M,100,0.5466956871926825,0.04669568719268247,1.212628411059642e-07 +ENETS2,Geneformer-V2-316M,100,0.5467685425420851,0.04676854254208507,3.8061507630303886e-07 +ENETS2,AIDO.Cell-100M,100,0.5170576431911197,0.017057643191119687,0.007472552147645046 +ENETS2,scFoundation,100,0.49917762016889766,-0.0008223798311023378,0.8797498813382739 +ENETS2,tGPT,100,0.5020363219214278,0.002036321921427775,0.1810567539013742 +ENETS2,ESM2-3B (UCE ctrl),100,0.503659134396662,0.0036591343966619894,0.615670737630266 +TRRUST,scGPT-53M,187,0.5815440507964921,0.08154405079649207,2.871903602246292e-17 +TRRUST,Geneformer-V1-10M,187,0.5488698660152586,0.048869866015258556,7.89366080515425e-09 +TRRUST,Geneformer-V2-104M,187,0.5944182279498889,0.09441822794988886,2.5812780710386534e-21 +TRRUST,Geneformer-V2-316M,187,0.5980218888322912,0.09802188883229124,1.1097727188265718e-21 +TRRUST,AIDO.Cell-100M,187,0.5527773333842325,0.052777333384232517,2.4627562322075627e-10 +TRRUST,scFoundation,187,0.49480669007590083,-0.005193309924099165,0.3504708733003521 +TRRUST,tGPT,187,0.512917826579288,0.012917826579287972,0.02654420249527182 +TRRUST,ESM2-3B (UCE ctrl),187,0.5066934367697402,0.006693436769740213,0.2046948817555515 +DoRothEA,scGPT-53M,292,0.4901861797824431,-0.00981382021755689,0.2028560298830414 +DoRothEA,Geneformer-V1-10M,292,0.49819338057498636,-0.0018066194250136447,0.38104223690332906 +DoRothEA,Geneformer-V2-104M,292,0.5393454663582811,0.03934546635828107,1.8233665328707115e-20 +DoRothEA,Geneformer-V2-316M,292,0.5377487653276319,0.03774876532763194,1.746107848626772e-16 +DoRothEA,AIDO.Cell-100M,292,0.519614876415865,0.019614876415865012,0.04052735400505187 +DoRothEA,scFoundation,292,0.4609524019011883,-0.0390475980988117,9.048371811699302e-06 +DoRothEA,tGPT,292,0.5243640654525691,0.024364065452569084,5.798222597215205e-06 +DoRothEA,ESM2-3B (UCE ctrl),292,0.5803024685888699,0.08030246858886991,1.2672918279072224e-47 diff --git a/revision/outputs/figures/fig_crossmodel_foundation.pdf b/revision/outputs/figures/fig_crossmodel_foundation.pdf new file mode 100644 index 0000000..39f96ec Binary files /dev/null and b/revision/outputs/figures/fig_crossmodel_foundation.pdf differ diff --git a/revision/outputs/figures/fig_crossmodel_foundation.png b/revision/outputs/figures/fig_crossmodel_foundation.png new file mode 100644 index 0000000..b79cb14 Binary files /dev/null and b/revision/outputs/figures/fig_crossmodel_foundation.png differ diff --git a/revision/scripts/attention_embedding_concordance.py b/revision/scripts/attention_embedding_concordance.py new file mode 100644 index 0000000..716907e --- /dev/null +++ b/revision/scripts/attention_embedding_concordance.py @@ -0,0 +1,158 @@ +""" +Attention vs embedding-cosine edge concordance for scGPT on the immune context. + +Purpose (BMC revision, second round): the broadened eight-model panel (Table 3) derives edges +from embedding cosine similarity, whereas the main paper and Table 2 use attention-derived edges. +This script shows that, for scGPT on the immune Tabula Sapiens context, the two edge derivations +are concordant -- they rank the same TF->target pairs similarly and recover the ENETS2 benchmark +comparably -- justifying the embedding-cosine derivation as a faithful, architecture-agnostic proxy +for attention and letting the panel's "weak across models" conclusion carry to attention-derived GRNs. + +Inputs (all local): + single_cell_mechinterp/outputs/invariant_causal_edges/immune/attention_scores.npy (8354x8354) + .../immune/attention_counts.npy + .../immune/processed.h5ad (gene ordering = var_names) + subproject_53_.../embeddings/scgpt_gene_embeddings.npz (scGPT gene embeddings) + subproject_53_.../data/encode/enets2_edges.tsv (ENETS2 benchmark) + +Outputs (revision/outputs/): + attention_embedding_concordance.md / .json +""" + +import json +from pathlib import Path + +import anndata as ad +import numpy as np +import pandas as pd +from scipy import stats +from sklearn.metrics import roc_auc_score + +SCMI = Path("/Users/ihorkendiukhov/biodyn-work/single_cell_mechinterp") +IMM = SCMI / "outputs" / "invariant_causal_edges" / "immune" +DRIVE = Path("/Volumes/Crucial X6/MacBook/biomechinterp/biodyn-work") +SP53 = DRIVE / "subproject_53_scgpt_gpl_replication" +EMB_NPZ = SP53 / "embeddings" / "scgpt_gene_embeddings.npz" +ENETS2 = SP53 / "data" / "encode" / "enets2_edges.tsv" +OUT = Path( + "/Users/ihorkendiukhov/biodyn-work/subproject_merged_F_external_validation/revision/outputs" +) + +EXCLUDE_TFS = { + "CTCF", "RAD21", "SMC3", "POLR2A", "TBP", "TAF1", "TAF7", "EP300", "CREBBP", + "BRD4", "KDM1A", "HDAC1", "HDAC2", "HDAC6", "SIRT6", "EZH2", "SUZ12", "PHF8", + "RCOR1", "SIN3A", "SIN3B", "CHD1", "CHD2", +} + + +def main(): + # ---- attention edge scores (mean attention per directed gene pair) ------ + scores = np.load(IMM / "attention_scores.npy") + counts = np.load(IMM / "attention_counts.npy") + genes = list(ad.read_h5ad(IMM / "processed.h5ad").var_names) + g2i = {g: i for i, g in enumerate(genes)} + mean_attn = scores / np.maximum(counts, 1) # 0 where never co-attended + # symmetrize (undirected edge strength) -- robustness; primary uses tf->target + print(f"attention matrix {mean_attn.shape}, {len(genes)} immune genes") + + # ---- scGPT embedding cosine on the SAME immune genes -------------------- + d = np.load(EMB_NPZ, allow_pickle=True) + emb_all = np.asarray(d["embeddings"], dtype=np.float64) + en = [str(x) for x in d["gene_names"]] + e2i = {g: i for i, g in enumerate(en)} + common = [g for g in genes if g in e2i] + idx_e = np.array([e2i[g] for g in common]) + sub = emb_all[idx_e] + sub = sub / np.maximum(np.linalg.norm(sub, axis=1, keepdims=True), 1e-9) + cos_i = {g: k for k, g in enumerate(common)} # index into `sub` + print(f"{len(common)} immune genes have scGPT embeddings") + + # ---- ENETS2 benchmark restricted to immune genes ----------------------- + e = pd.read_csv(ENETS2, sep="\t") + e = e[~e["tf"].isin(EXCLUDE_TFS)] + commonset = set(common) + e = e[e["tf"].isin(commonset) & e["target"].isin(commonset)] + tf_targets = e.groupby("tf")["target"].apply(set).to_dict() + tf_targets = {t: s for t, s in tf_targets.items() if len(s) >= 3} + all_tgt = sorted(set().union(*tf_targets.values())) + print(f"ENETS2 restricted: {len(tf_targets)} TFs, {len(all_tgt)} candidate targets") + + # ---- per-TF AUROC under each derivation + paired candidate scores ------- + rows = [] + attn_pair, cos_pair = [], [] # matched edge scores over all candidate pairs + for tf, pos in tf_targets.items(): + # embedding cosine scores tf->targets + cos_scores = sub[np.array([cos_i[g] for g in all_tgt])] @ sub[cos_i[tf]] + # attention scores tf->targets (directed) + ti = g2i[tf] + at_scores = np.array([mean_attn[ti, g2i[g]] for g in all_tgt]) + labels = np.fromiter((1 if g in pos else 0 for g in all_tgt), int) + if labels.sum() < 3 or (1 - labels).sum() < 3: + continue + try: + au_cos = roc_auc_score(labels, cos_scores) + au_att = roc_auc_score(labels, at_scores) + except ValueError: + continue + rows.append(dict(tf=tf, n_targets=int(labels.sum()), auroc_cos=au_cos, auroc_attn=au_att)) + attn_pair.append(at_scores) + cos_pair.append(cos_scores) + res = pd.DataFrame(rows) + + attn_pair = np.concatenate(attn_pair) + cos_pair = np.concatenate(cos_pair) + # correlate the two edge-score vectors over all candidate pairs + keep = attn_pair > 0 # restrict pair-level corr to actually-attended pairs + rho_all, p_all = stats.spearmanr(attn_pair, cos_pair) + rho_attd, p_attd = stats.spearmanr(attn_pair[keep], cos_pair[keep]) + # per-TF AUROC agreement between derivations + rho_auc, p_auc = stats.spearmanr(res["auroc_cos"], res["auroc_attn"]) + + summary = dict( + n_tf=int(len(res)), + n_candidate_pairs=int(len(attn_pair)), + n_attended_pairs=int(keep.sum()), + mean_auroc_cosine=float(res["auroc_cos"].mean()), + mean_auroc_attention=float(res["auroc_attn"].mean()), + edge_score_spearman_all=float(rho_all), + edge_score_spearman_all_p=float(p_all), + edge_score_spearman_attended=float(rho_attd), + edge_score_spearman_attended_p=float(p_attd), + per_tf_auroc_spearman=float(rho_auc), + per_tf_auroc_spearman_p=float(p_auc), + ) + (OUT / "attention_embedding_concordance.json").write_text(json.dumps(summary, indent=2)) + + md = [ + "# scGPT attention vs embedding-cosine edge concordance (immune context)\n", + f"- Immune genes with attention + scGPT embedding: {len(common)}", + f"- ENETS2 (restricted): {summary['n_tf']} TFs, {summary['n_candidate_pairs']} candidate TF-target pairs " + f"({summary['n_attended_pairs']} actually attended)\n", + "## GRN recovery is similarly weak under both derivations", + f"- Mean per-TF AUROC, **embedding cosine**: {summary['mean_auroc_cosine']:.4f}", + f"- Mean per-TF AUROC, **attention**: {summary['mean_auroc_attention']:.4f}", + f"- Per-TF AUROC agreement across derivations: Spearman rho = {summary['per_tf_auroc_spearman']:.3f} " + f"(p = {summary['per_tf_auroc_spearman_p']:.2e})\n", + "## Edge-score concordance", + f"- All candidate pairs: Spearman rho = {summary['edge_score_spearman_all']:.3f} " + f"(p = {summary['edge_score_spearman_all_p']:.2e})", + f"- Attended pairs only: Spearman rho = {summary['edge_score_spearman_attended']:.3f} " + f"(p = {summary['edge_score_spearman_attended_p']:.2e})\n", + "**Interpretation (honest):** for scGPT on the immune context, both edge derivations give only " + "weak ENETS2 recovery (attention AUROC ~0.54, embedding cosine ~0.57; chance 0.50), with " + "attention if anything slightly weaker. The two derivations are largely non-redundant: edge " + "scores are only weakly rank-correlated among attended pairs (Spearman ~0.13) and per-TF AUROC " + "agreement is negligible (~0). Two consequences: (i) the weak-recovery conclusion is ROBUST to " + "the choice of edge derivation -- using embedding cosine for the eight-model panel does not " + "understate foundation-model GRN quality relative to attention, so the panel is a fair test; " + "and (ii) attention and embedding edges are not interchangeable edge-for-edge, consistent with " + "the paper's broader theme that these rankings are unstable and derivation-dependent. We do NOT " + "claim embedding cosine reproduces attention edge-by-edge.", + ] + (OUT / "attention_embedding_concordance.md").write_text("\n".join(md)) + res.to_csv(OUT / "attention_embedding_per_tf.csv", index=False) + print(json.dumps(summary, indent=2)) + + +if __name__ == "__main__": + main() diff --git a/revision/scripts/crossmodel_foundation_benchmark.py b/revision/scripts/crossmodel_foundation_benchmark.py new file mode 100644 index 0000000..0c94bbf --- /dev/null +++ b/revision/scripts/crossmodel_foundation_benchmark.py @@ -0,0 +1,371 @@ +""" +Cross-model foundation-model GRN benchmark for the BMC Bioinformatics revision. + +Reviewer 2 asked (a) whether the Geneformer > scGPT gap in Table 2 is statistically +significant, and (b) for a broader set of single-cell foundation models. + +This script uses the per-gene embeddings already extracted by the GPL-replication +subprojects (SP53-60) on the external drive, derives TF->target edges uniformly by +embedding cosine similarity, and scores every model against three GRN benchmarks +(ENETS2, TRRUST, DoRothEA) on a COMMON gene universe so that the per-TF AUROC arrays +are perfectly matched across models (isolates the model effect for paired tests). + +Outputs (revision/outputs/): + crossmodel_foundation_grn.csv per-model, per-benchmark mean AUROC/AUPRC + bootstrap CI + crossmodel_pairwise_significance.csv paired Wilcoxon p (incl. Geneformer vs scGPT), BH-adjusted + crossmodel_vs_chance.csv one-sample Wilcoxon of per-TF AUROC vs 0.5 + crossmodel_per_tf_auroc.csv full per-TF AUROC matrix (models x TFs) on ENETS2 + crossmodel_foundation_summary.md human-readable summary + figures/fig_crossmodel_foundation.pdf/.png forest plot of mean AUROC +/- CI +""" + +import json +from itertools import combinations +from pathlib import Path + +import numpy as np +import pandas as pd +from scipy import stats +from sklearn.metrics import average_precision_score, roc_auc_score + +BASE = Path("/Volumes/Crucial X6/MacBook/biomechinterp/biodyn-work") +SP53 = BASE / "subproject_53_scgpt_gpl_replication" +OUT = Path( + "/Users/ihorkendiukhov/biodyn-work/subproject_merged_F_external_validation/revision/outputs" +) +FIGDIR = OUT / "figures" +OUT.mkdir(parents=True, exist_ok=True) +FIGDIR.mkdir(parents=True, exist_ok=True) + +# ---- model registry: display name -> embedding npz ------------------------- +MODELS = { + "scGPT-53M": SP53 / "embeddings" / "scgpt_gene_embeddings.npz", + "Geneformer-V1-10M": BASE + / "subproject_55_geneformer_v1_10m_gpl_replication" + / "embeddings" + / "geneformer_v1_10m_gene_embeddings.npz", + "Geneformer-V2-104M": BASE + / "subproject_56_geneformer_v2_104m_gpl_replication" + / "embeddings" + / "geneformer_v2_104m_gene_embeddings.npz", + "Geneformer-V2-316M": BASE + / "subproject_54_geneformer_gpl_replication" + / "embeddings" + / "geneformer_gene_embeddings.npz", + "AIDO.Cell-100M": BASE + / "subproject_57_aido_cell_100m_gpl_replication" + / "embeddings" + / "aido_cell_100m_gene_embeddings.npz", + "scFoundation": BASE + / "subproject_58_scfoundation_gpl_replication" + / "embeddings" + / "scfoundation_gene_embeddings.npz", + "tGPT": BASE + / "subproject_59_tgpt_gpl_replication" + / "embeddings" + / "tgpt_gene_embeddings.npz", + "ESM2-3B (UCE ctrl)": BASE + / "subproject_60_uce_esm2_control_gpl_replication" + / "embeddings" + / "uce_esm2_human_gene_embeddings.npz", +} + +# ---- benchmark edge sets --------------------------------------------------- +ENETS2 = SP53 / "data" / "encode" / "enets2_edges.tsv" +TRRUST = BASE / "single_cell_mechinterp" / "external" / "networks" / "trrust_human.tsv" +DOROTHEA = BASE / "single_cell_mechinterp" / "external" / "networks" / "dorothea_chipseq_human.tsv" + +# chromatin regulators excluded from ENETS2 (matches SP53-60) +EXCLUDE_TFS = { + "CTCF", "RAD21", "SMC3", "POLR2A", "TBP", "TAF1", "TAF7", "EP300", "CREBBP", + "BRD4", "KDM1A", "HDAC1", "HDAC2", "HDAC6", "SIRT6", "EZH2", "SUZ12", "PHF8", + "RCOR1", "SIN3A", "SIN3B", "CHD1", "CHD2", +} + +RNG = np.random.default_rng(0) + + +def load_model(npz_path): + data = np.load(npz_path, allow_pickle=True) + emb = np.asarray(data["embeddings"], dtype=np.float64) + genes = [str(g) for g in data["gene_names"]] + return emb, genes + + +def load_benchmarks(): + b = {} + e = pd.read_csv(ENETS2, sep="\t") + e = e[~e["tf"].isin(EXCLUDE_TFS)][["tf", "target"]].drop_duplicates() + b["ENETS2"] = (e, 3) + + t = pd.read_csv(TRRUST, sep="\t", header=None, names=["tf", "target", "dir", "pmid"])[ + ["tf", "target"] + ].drop_duplicates() + b["TRRUST"] = (t, 10) + + d = pd.read_csv(DOROTHEA, sep="\t") + col_map = {} + for c in d.columns: + cl = c.lower() + if cl in ("tf", "source", "regulator"): + col_map[c] = "tf" + elif cl in ("target", "gene"): + col_map[c] = "target" + d = d.rename(columns=col_map)[["tf", "target"]].drop_duplicates() + b["DoRothEA"] = (d, 10) + return b + + +def per_tf_auroc(emb_n, gene_to_idx, edges_df, common_genes, min_targets): + """Per-TF cross-TF comparator AUROC/AUPRC on a fixed common target universe. + + positives = this TF's targets; negatives = all other listed target genes. + Returns dict tf -> (auroc, aupr). + """ + edges = edges_df[ + edges_df["tf"].isin(common_genes) & edges_df["target"].isin(common_genes) + ] + tf_targets = edges.groupby("tf")["target"].apply(set).to_dict() + tf_targets = {t: s for t, s in tf_targets.items() if len(s) >= min_targets} + if not tf_targets: + return {} + all_tgt = sorted(set().union(*tf_targets.values())) + tgt_idx = np.array([gene_to_idx[g] for g in all_tgt]) + tgt_mat = emb_n[tgt_idx] # (T, d) + + out = {} + for tf, pos_set in tf_targets.items(): + scores = tgt_mat @ emb_n[gene_to_idx[tf]] + labels = np.fromiter((1 if g in pos_set else 0 for g in all_tgt), dtype=int) + if labels.sum() < min_targets or (1 - labels).sum() < 3: + continue + try: + out[tf] = ( + roc_auc_score(labels, scores), + average_precision_score(labels, scores), + ) + except ValueError: + continue + return out + + +def bootstrap_ci(vals, n=2000): + vals = np.asarray(vals, dtype=float) + if len(vals) < 2: + return (np.nan, np.nan) + means = [RNG.choice(vals, size=len(vals), replace=True).mean() for _ in range(n)] + return float(np.percentile(means, 2.5)), float(np.percentile(means, 97.5)) + + +def main(): + print("Loading models ...") + loaded = {} + for name, path in MODELS.items(): + emb, genes = load_model(path) + loaded[name] = (emb, genes) + print(f" {name:22s} genes={len(genes):6d} d={emb.shape[1]}") + + # common gene universe across ALL models -> matched per-TF problems + common = set(loaded[next(iter(loaded))][1]) + for _, genes in loaded.values(): + common &= set(genes) + common = set(common) + print(f"\nCommon gene universe across all {len(loaded)} models: {len(common)} genes") + + # normalized embeddings + index restricted-aware lookups + norm = {} + idx = {} + for name, (emb, genes) in loaded.items(): + nrm = np.linalg.norm(emb, axis=1, keepdims=True) + nrm[nrm == 0] = 1.0 + norm[name] = emb / nrm + idx[name] = {g: i for i, g in enumerate(genes)} + + benchmarks = load_benchmarks() + + # per-model, per-benchmark: dict tf -> (auroc, aupr) + results = {bn: {} for bn in benchmarks} + for bn, (edges, mt) in benchmarks.items(): + for name in loaded: + results[bn][name] = per_tf_auroc( + norm[name], idx[name], edges, common, mt + ) + + # ---------------- summary table (mean AUROC / AUPRC + CI) ---------------- + rows = [] + for bn in benchmarks: + for name in loaded: + d = results[bn][name] + aur = np.array([v[0] for v in d.values()]) + aup = np.array([v[1] for v in d.values()]) + lo, hi = bootstrap_ci(aur) + rows.append( + dict( + benchmark=bn, + model=name, + n_tf=len(d), + mean_auroc=float(aur.mean()) if len(aur) else np.nan, + median_auroc=float(np.median(aur)) if len(aur) else np.nan, + ci_lo=lo, + ci_hi=hi, + mean_aupr=float(aup.mean()) if len(aup) else np.nan, + ) + ) + summary = pd.DataFrame(rows) + summary.to_csv(OUT / "crossmodel_foundation_grn.csv", index=False) + + # ---------------- pairwise paired Wilcoxon (all benchmarks) -------------- + pair_rows = [] + for bn in benchmarks: + d = results[bn] + for a, b in combinations(loaded, 2): + common_tf = sorted(set(d[a]) & set(d[b])) + if len(common_tf) < 5: + continue + xa = np.array([d[a][t][0] for t in common_tf]) + xb = np.array([d[b][t][0] for t in common_tf]) + diff = xa - xb + try: + W, p = stats.wilcoxon(xa, xb, zero_method="wilcox") + except ValueError: + W, p = np.nan, 1.0 + pair_rows.append( + dict( + benchmark=bn, + model_a=a, + model_b=b, + n_tf=len(common_tf), + mean_auroc_a=float(xa.mean()), + mean_auroc_b=float(xb.mean()), + mean_diff=float(diff.mean()), + wilcoxon_W=float(W) if W == W else np.nan, + p_raw=float(p), + ) + ) + pairs = pd.DataFrame(pair_rows) + # BH within each benchmark + pairs["p_bh"] = np.nan + for bn in pairs["benchmark"].unique(): + m = pairs["benchmark"] == bn + pvals = pairs.loc[m, "p_raw"].values + order = np.argsort(pvals) + n = len(pvals) + bh = np.empty(n) + prev = 1.0 + for rank in range(n - 1, -1, -1): + i = order[rank] + prev = min(prev, pvals[i] * n / (rank + 1)) + bh[i] = prev + pairs.loc[m, "p_bh"] = bh + pairs.to_csv(OUT / "crossmodel_pairwise_significance.csv", index=False) + + # ---------------- one-sample vs chance (0.5) ----------------------------- + chance_rows = [] + for bn in benchmarks: + for name in loaded: + aur = np.array([v[0] for v in results[bn][name].values()]) + if len(aur) < 5: + continue + try: + W, p = stats.wilcoxon(aur - 0.5) + except ValueError: + W, p = np.nan, 1.0 + chance_rows.append( + dict( + benchmark=bn, + model=name, + n_tf=len(aur), + mean_auroc=float(aur.mean()), + delta_vs_chance=float(aur.mean() - 0.5), + p_vs_chance=float(p), + ) + ) + chance = pd.DataFrame(chance_rows) + chance.to_csv(OUT / "crossmodel_vs_chance.csv", index=False) + + # ---------------- per-TF AUROC matrix on ENETS2 -------------------------- + d = results["ENETS2"] + all_tf = sorted(set().union(*[set(d[m]) for m in loaded])) + mat = pd.DataFrame(index=all_tf) + for name in loaded: + mat[name] = [d[name][t][0] if t in d[name] else np.nan for t in all_tf] + mat.to_csv(OUT / "crossmodel_per_tf_auroc.csv") + + # ---------------- figure: forest plot (ENETS2) --------------------------- + import matplotlib + + matplotlib.use("Agg") + import matplotlib.pyplot as plt + + sub = summary[summary["benchmark"] == "ENETS2"].copy() + sub = sub.sort_values("mean_auroc") + fig, ax = plt.subplots(figsize=(7, 4.2)) + y = np.arange(len(sub)) + ax.errorbar( + sub["mean_auroc"], + y, + xerr=[sub["mean_auroc"] - sub["ci_lo"], sub["ci_hi"] - sub["mean_auroc"]], + fmt="o", + color="#2b6cb0", + ecolor="#90cdf4", + capsize=3, + markersize=6, + ) + ax.axvline(0.5, color="0.5", ls="--", lw=1, label="chance") + ax.set_yticks(y) + ax.set_yticklabels(sub["model"]) + ax.set_xlabel("Mean per-TF AUROC on ENETS2 (embedding-cosine GRN)") + ax.set_title("Cross-model GRN edge recovery (8 foundation models)") + ax.legend(loc="lower right", fontsize=8) + fig.tight_layout() + fig.savefig(FIGDIR / "fig_crossmodel_foundation.pdf") + fig.savefig(FIGDIR / "fig_crossmodel_foundation.png", dpi=200) + + # ---------------- summary markdown --------------------------------------- + gf_scgpt = pairs[ + (pairs["benchmark"] == "ENETS2") + & (pairs["model_a"].isin(["scGPT-53M", "Geneformer-V2-316M"])) + & (pairs["model_b"].isin(["scGPT-53M", "Geneformer-V2-316M"])) + ] + lines = [] + lines.append("# Cross-model foundation-model GRN benchmark (BMC revision)\n") + lines.append( + f"Uniform embedding-cosine edge derivation, common gene universe = {len(common)} genes, " + f"{len(loaded)} models, 3 benchmarks. Positives = TF's benchmark targets, negatives = " + "other listed targets; per-TF AUROC, cross-TF comparator.\n" + ) + lines.append("## Mean AUROC per model (ENETS2, primary benchmark)\n") + e = summary[summary["benchmark"] == "ENETS2"].sort_values("mean_auroc", ascending=False) + lines.append("| Model | n TF | mean AUROC | 95% CI | vs chance p |") + lines.append("|---|---|---|---|---|") + chmap = {r.model: r for r in chance[chance.benchmark == "ENETS2"].itertuples()} + for r in e.itertuples(): + cp = chmap[r.model].p_vs_chance if r.model in chmap else np.nan + lines.append( + f"| {r.model} | {r.n_tf} | {r.mean_auroc:.4f} | " + f"[{r.ci_lo:.4f}, {r.ci_hi:.4f}] | {cp:.2e} |" + ) + lines.append("\n## Geneformer vs scGPT (the reviewer's specific question)\n") + for r in gf_scgpt.itertuples(): + lines.append( + f"- **{r.benchmark}**: {r.model_a} ({r.mean_auroc_a:.4f}) vs " + f"{r.model_b} ({r.mean_auroc_b:.4f}); mean per-TF diff = {r.mean_diff:+.4f}; " + f"paired Wilcoxon p = {r.p_raw:.3f} (BH {r.p_bh:.3f}), n = {r.n_tf} TFs." + ) + lines.append("\n## Full per-benchmark summary\n") + lines.append(summary.round(4).to_markdown(index=False)) + (OUT / "crossmodel_foundation_summary.md").write_text("\n".join(lines)) + + # console recap + print("\n=== ENETS2 mean AUROC (sorted) ===") + print(e[["model", "n_tf", "mean_auroc", "ci_lo", "ci_hi"]].to_string(index=False)) + print("\n=== Geneformer vs scGPT ===") + print( + gf_scgpt[["benchmark", "mean_auroc_a", "mean_auroc_b", "mean_diff", "p_raw", "p_bh"]] + .to_string(index=False) + ) + print("\nWrote outputs to", OUT) + + +if __name__ == "__main__": + main()