A Single-Cell AI Model Passed the Simulation Benchmark. Tested Against Real CRISPR Data, It Got the Direction Right 40.9% of the Time — Worse Than a Coin Flip.

CellOracle was one of only two out of eight tested methods that reliably detected a known transcription-factor-to-pathway signal in a systematic benchmark spanning four single-cell datasets and six mathematical frameworks. It looked like the benchmark had a winner. Then researchers checked CellOracle's predictions against real CRISPR interference knockdowns in human K562 cells, where the true direction of the effect was already known from the experiment itself. CellOracle called the correct direction 40.9% of the time — statistically indistinguishable from a coin flip. A second finding from the same paper: switching between two other methods, DDIM and scTenifoldKnk, can reverse a study's biological conclusion outright — their gene rankings on the identical task are anti-correlated, rho = −0.811.

What the benchmark actually tested

"Method Choice, Not Biology, Determines In Silico Perturbation Results: A Systematic Evaluation of Eight Methods Across Four Datasets" (Ge, Wu, Hu, et al.; Wuxi Hospital of Traditional Chinese Medicine; bioRxiv, DOI 10.64898/2026.08.11.744106, posted August 11, 2026) asks a question most published perturbation-modeling papers never answer: if you take the methods that predict what a gene knockdown does to a single cell — the tools used to prioritize CRISPR targets and simulate perturbations computationally before running the wet-lab experiment — and test eight of them, spanning six different mathematical frameworks (VAE-based models, tensor decomposition, and others), across four independent datasets, do they agree with each other? And do their predictions hold up against ground truth?

The first test was a detection problem: can a method find a transcription factor's known regulatory relationship to a downstream pathway at all? Using PBMC monocytes as the test bed, the team looked for whether each method could recover TF-to-glycolysis directional regulation, and used cross-pathway checks to confirm the signal was real and specific — SPI1's regulation of glycolysis genes showed a 4.4x enrichment, FOS showed the same 4.4x enrichment for its AP-1 target genes, and SOX9, included as a specificity control with no expected regulatory relationship to either pathway, correctly showed no enrichment. The biology being tested for was real and detectable in the data. Six of the eight methods still failed to find it.

Two winners out of eight, then a harder question

Only CellOracle and DDIM consistently detected the TF-to-glycolysis signal. On a paper that stopped here, that would read as a clean result: two methods work, six do not, use the two that work. The team asked a second question instead — not just whether a method can detect that a relationship exists, but whether different methods agree on the specifics of it. DDIM and scTenifoldKnk, tested on the identical ranking task, produced gene-by-gene rankings that were significantly anti-correlated: rho = −0.811, p = 0.027. Two methods, same data, same target genes, and a ranking from one that runs almost exactly backward from the other. The paper's own framing is blunt about what that means: method choice alone could reverse a study's biological conclusion, independent of what the underlying biology actually is.

The real test: does the winner know cause from correlation

Detecting a TF-to-pathway signal in resting-state single-cell data is a correlational test — it asks whether a transcription factor's expression pattern statistically tracks a pathway's activity across cells. It does not test whether the method understands what happens when you actually intervene. To test that, the team ran CRISPRi Perturb-seq validation in K562 cells: real transcription-factor knockdowns, with the real, measured effect on glycolysis gene expression as ground truth. Every knockdown tested significantly suppressed glycolysis expression, confirming the causal relationship was real and in the expected direction.

CellOracle, the benchmark's own apparent winner from the correlational test, predicted the correct direction of the perturbation's effect 40.9% of the time across the full validation set — a result the paper reports as not statistically different from chance. The method that passed the test built to measure whether it had found a real regulatory relationship failed the test built to measure whether it understood what that relationship meant when you actually pulled the lever. That gap, between steady-state correlation and causal perturbation, is the paper's central finding, and it is a stronger claim than "this particular tool has a bug" — it is a claim about what passing this class of benchmark does and does not tell you.

Patching the winner didn't help

The team's next step is the part that separates a real ablation from a takedown: they tried to fix it. A natural hypothesis is that DDIM (and, by extension, tools like it) fail because they lack an explicit model of the underlying gene-regulatory network — so the team added a gene-regulatory-network prior directly into DDIM and re-ran the target recall test. The result: zero improvement, delta = 0 for every transcription factor tested. Whatever DDIM is missing, it is not a missing prior; the paper calls the gap "multi-factorial," and its own diagnostics back that up. Profiling the VAE-based methods' latent space found genuine signal-to-noise competition between transcription factors — STAT3 carried a signal-to-noise ratio of 0.44 against SPI1's 4.25 in the same latent space, meaning a real signal for one TF can get swamped by a stronger signal from another sharing the same representation. Correlation-based methods showed TF-to-glycolysis correlation of |r| = 0.038, indistinguishable from a measured background correlation of |r| = 0.047 — the signal was there, and it was buried in noise at the same magnitude. And graph-based methods showed non-specific enrichment of 0.84x, below even the SOX9 negative control's baseline. Three different failure modes, from three different mathematical frameworks, none fixed by adding the obvious missing ingredient to the one method that seemed to be working.

The honest limits

This is a single corresponding lab (Wuxi Hospital of Traditional Chinese Medicine) running a benchmark across public datasets and one CRISPRi validation cohort in one cell line, K562 — a workhorse line for perturbation biology, but one cell line, not a cross-tissue claim. The paper is explicit that its own guidance (cross-pathway validation, direction-aware benchmarking, a minimum of 500 cells and 1,000 highly variable genes before trusting a result) is preliminary, not a settled protocol, and it does not claim CellOracle or DDIM are unusable — it claims that the specific correlational benchmark six of eight methods were tested against does not predict causal-direction accuracy, which is a narrower and more useful claim than "these tools don't work." The paper also does not resolve which of the six failing methods might succeed on a different dataset or a different pathway; it establishes that failure is common and multi-factorial, not that it is universal or method-specific in a way that generalizes cleanly.

What this means for reproducible, local-first science

The uncomfortable version of this finding is not "six methods are bad." It is that a benchmark measuring the wrong thing can crown a winner, and the crowning looks exactly like a real result until someone runs the more expensive, harder-to-fake experimental check. CellOracle did not fail because it is a bad method in the abstract; it failed a test that a correlational benchmark was never built to administer, and the paper only caught that because the team went and generated real CRISPRi ground truth instead of stopping at the benchmark that already had a clean-looking winner. For anyone building a perturbation-prediction pipeline on top of a method that "won" a published benchmark, the finding here is a checklist item, not a verdict: passing a signal-detection test is not evidence of causal accuracy, and the only way to know the difference is to hold out real interventional data and check, exportable and re-runnable, rather than trusting the benchmark's own leaderboard.