65% of the Data Powering AI Virtual-Cell Models Is Statistically Unreliable. The Models Trained Without It Won Anyway.

A new ETH Zurich analysis of 7,170 single-cell perturbations across 29 public datasets found that 65% are statistically unreliable — noise sitting inside the exact ground truth every perturbation-prediction model in the field is trained and graded against. Cut it out, and training on only the reliable slice — 55% of the perturbation volume — still matches or beats training on everything, unreliable data included. Eight days later, Arc Institute opened its second annual Virtual Cell Challenge: a $100,000 global contest scoring entrants on exactly this kind of CRISPRi perturbation-response prediction.

What "the virtual cell" is actually trained on

A large share of AI-for-biology money right now is chasing one goal: a model that can predict what a cell will do — which genes turn on, which pathways activate, how a cell type shifts — if you perturb it. Knock out a gene, add a drug, edit a promoter. Get that prediction right and you can screen millions of interventions in software before running a single wet-lab experiment. Arc Institute, Chan Zuckerberg Initiative, and a growing list of biotech AI teams are all building toward some version of this "virtual cell." The training and evaluation data for nearly all of it comes from the same source: perturb-seq, a technique that reads out a cell's full transcriptional response to a genetic or chemical perturbation, one cell at a time, at scale.

The assumption underneath every model built this way is that the perturb-seq measurement, once standard preprocessing is applied, is a trustworthy record of what the perturbation actually did. "Reliable single-cell perturbations explain and improve model performance" (Wang, Kuipers, Hugi, Platt, Beerenwinkel; ETH Zurich, SIB Swiss Institute of Bioinformatics, and the University of Basel; bioRxiv, DOI 10.64898/2026.08.11.744177, posted August 12, 2026) tests that assumption directly, instead of taking it as given.

Three buckets, and two-thirds of the data lands in the wrong one

The team classified each of 7,170 perturbations, drawn from 29 public datasets, into one of three categories: unreliable, where the measured transcriptional effect does not separate from background noise; shared, where there is a real signal but it looks like the signal produced by many other, unrelated perturbations, suggesting an off-target or nonspecific stress response rather than the perturbation's intended effect; and specific, a real signal distinct to that perturbation — the kind of measurement a model is actually supposed to learn a causal relationship from.

Only 24% of the perturbations tested carry the kind of signal the field's models are nominally trying to learn from. The paper's next step is the one that matters for anyone citing a leaderboard: applying these quality labels back onto published perturbation-prediction benchmarks changes which method wins. Model comparisons, the authors report, depend on perturbation quality — a ranking computed across all 7,170 perturbations is not necessarily the same ranking you get once the 65% majority classified as unreliable is set aside. A benchmark result is not just a statement about the model. It is a statement about the model conditional on a data-quality assumption that, until now, nobody had directly measured at this scale.

The counterintuitive result: less data, same or better model

The obvious worry, once you show two-thirds of a dataset is unreliable, is that throwing it out starves the model of training signal. The paper tests this directly and finds the opposite. Training with reliable perturbations alone matches or outperforms full-data performance, while using 55% of all training perturbations — discarding a large share of the volume and losing nothing measurable in return. That is the ablation that makes this more than a data-hygiene footnote: the unreliable 65% was not neutral filler sitting harmlessly in the training set. It was, on net, actively unhelpful, and a model trained without it generalizes at least as well as one trained with it.

That result reframes what "more data" has meant in this field's scaling conversations. A bigger perturb-seq atlas has generally been treated as a straightforward win — more cells, more perturbations, more coverage. This paper is evidence that the number worth optimizing is not the perturbation count but the reliable-perturbation count, and that the two can diverge by a wide margin depending on the assay and cell type.

A cheaper way to know before you screen

The paper's most immediately practical finding is a prospective one: for most perturbations, a pilot experiment using just 28 cells accurately predicts how many cells a full screen will need to reach reliability. Perturb-seq screens are expensive per perturbation at the scale modern virtual-cell efforts want — tens of thousands of conditions, each needing enough cells to separate real signal from noise. A 28-cell check that flags, before the full screen runs, which perturbations are worth scaling up is a direct cost lever for any lab planning the next atlas, not just a statistical curiosity for people auditing the last one.

The contest that opened eight days later

On August 20, 2026, Arc Institute announced the second edition of its Virtual Cell Challenge at virtualcellchallenge.org. The task: zero-shot prediction of CRISPRi knockdown responses in six cell lines that entrants have never seen perturbed, with no training set provided for those cell lines and an evaluation dataset the organizers describe as significantly larger than last year's round. The grand prize is $100,000, sponsored by NVIDIA, 10x Genomics, and Ultima Genomics.

To be clear about what is and is not being claimed here: nothing in the ETH Zurich paper says anything about Arc's specific evaluation set, and Arc has not published the reliability composition of this year's held-out cell lines. It would be unfair, and unsupported by the evidence, to claim the Virtual Cell Challenge is scored on 65% noise — that is not a number anyone has published, and a competition organizer running a high-stakes public benchmark has every incentive to curate its ground truth carefully. What the ETH paper does establish is that this is exactly the kind of open question worth asking of any perturbation benchmark before trusting its leaderboard: what fraction of the held-out ground truth is itself reliable, and does the ranking of entrants change once that is accounted for? Before August 12, that question did not have a general answer for this field. Now there is a published method for answering it, and a concrete number — 65% — showing how large the gap between "passed preprocessing" and "trustworthy ground truth" can be across the 29 datasets already in public use.

That is also not a new question for this specific corner of AI-for-science. This same blog covered a related finding one week earlier, in "the causal gap": a systematic benchmark of eight perturbation-prediction methods found that switching between two of them can reverse a study's biological conclusion outright, even though both methods score similarly on the standard correlational benchmark. That paper showed method choice can flip a result. This one shows the ground truth those methods are judged against can, for the majority of its rows, fail to separate from noise in the first place. They are two independent audits of the same evaluation stack, arriving from different directions in the same two weeks, and they compound: a method-choice problem layered on top of a data-reliability problem means a published perturbation-prediction result can look decisive while resting on foundations that neither the reader nor, plausibly, the authors have fully characterized.

What this means for reproducible, local-first science

The instinct this paper argues against is the one that treats a public dataset as settled once it clears standard preprocessing and lands on a benchmark leaderboard. The more useful instinct, and the one the paper models directly, is to keep asking what fraction of any given ground truth is reliable before building a claim on top of it — and to make that check cheap and routine rather than a special, occasional audit. A 28-cell pilot that tells you whether a full screen is worth running is the kind of check that only gets used if it is fast enough to run locally, on the data you already have, before you commit budget to the full experiment. That is the same argument for exporting a reproducibility record with every result: not because any individual model or benchmark is untrustworthy, but because the field just got direct evidence that the question "how much of this ground truth can I actually trust" has a real, checkable answer, and that answer is not always the one the leaderboard assumed.