3.1 Million AI-Discovery Runs, 30 Frameworks Tested. Zero Universal Winners — Including OpenEvolve, the Field's Default Choice.

Zero. That's how many of the 30 AI-discovery frameworks tested in a study published July 20 won across every one of its 12 model-problem pairs — including OpenEvolve, the framework most autonomous-discovery agent papers reach for by default, which the study's 3.1 million-plus LLM rollouts show generally loses to simpler alternatives. Two more papers landed the same week making the same underlying point from different angles: the number an AI science agent reports after one run is not, by itself, a result you can trust.

The comparison the field kept skipping

Automated Discovery Has No Universally Superior Harness (arXiv 2607.18235, July 20) starts from a plain observation: systems like OpenEvolve and TTT-Discover get treated as general-purpose harnesses, but each is actually a bundle of design choices — how candidates are archived, how parents get selected, how exploration is balanced against exploitation, how compute budget gets allocated — collapsed into a single named recipe. Discovery runs are expensive and stochastic, so in practice, most comparisons in the literature pit one harness against another using too few independent trials to tell a real methodological improvement apart from ordinary run-to-run variance. The authors decomposed OpenEvolve-style evolutionary search and the TTT-Discover harness into their constituent components, then ran all 30 resulting variants under matched compute budgets across 12 model-problem pairs, with repeated trials and a released null-distribution baseline for each pair so future papers can test their own claims against real run-to-run noise instead of eyeballing it.

The two harnesses the paper decomposes are not variations on a theme — they represent opposite bets about how to search. OpenEvolve prompts a frozen LLM repeatedly and evolves a population of candidate solutions, the same paradigm behind Google DeepMind's AlphaEvolve. TTT-Discover (arXiv 2601.16175) does the opposite: it runs reinforcement learning at test time, so the model's own weights keep updating on experience specific to the one problem in front of it, aiming for a single best solution rather than a population of good ones. The paper reports state-of-the-art results with that approach across mathematics, GPU kernel engineering, algorithm competitions, and single-cell biology, using an open model and "a few hundred dollars per problem." Two fundamentally different search philosophies, both widely cited as strong defaults — and the harness study's point is that neither one is the default, because there isn't one.

Harness choice is a hyperparameter, not a recipe

The headline result, in the authors' own words: "no fixed harness is reliably superior across the evaluated model-problem pairs, and variants of OpenEvolve generally underperform simpler alternatives." That second clause is the one worth sitting with. OpenEvolve is not an obscure baseline — it's the open-source implementation of Google DeepMind's AlphaEvolve and the framework most open discovery-agent papers reach for by default. A 3.1-million-rollout, budget-matched comparison finding that its variants generally lose to simpler search strategies is not a knock on any one team; it's evidence that "which framework is best" has been the wrong question the whole time. The paper's reframe: harness choice should be treated as a hyperparameter tuned to the specific problem and underlying model, not a universal recipe you commit to once and reuse everywhere.

What actually beats committing to one harness

The paper doesn't stop at the negative result. It also finds that early discovery progress predicts final performance well enough to act on, and uses that to test a budget-matched adaptive-allocation policy: start several harnesses at once, prune the weak partial runs early based on their trajectory, and reallocate the freed compute to the survivors. That policy beat both committing to a single randomly sampled fixed harness and running a static ensemble of harnesses to completion without reallocating. In other words, the fix for "we don't know which harness will work" isn't a better leaderboard — it's not picking at all, and letting early signal decide where the compute goes.

The shape of the underlying discipline — repeat the run, then test whether the difference is real — is simple enough to state without the paper's full statistical machinery:

# how a harness comparison usually gets reported
score_a = run_harness(harness_a, problem)   # one number
score_b = run_harness(harness_b, problem)   # one number
winner = harness_a if score_a > score_b else harness_b   # done

# the shape of the fix: repeat, then test whether the gap is real
from scipy.stats import mannwhitneyu

scores_a = [run_harness(harness_a, problem) for _ in range(n_trials)]
scores_b = [run_harness(harness_b, problem) for _ in range(n_trials)]
_, p_value = mannwhitneyu(scores_a, scores_b, alternative="two-sided")
# only call it a real win if this clears significance —
# otherwise it's run-to-run noise, not a methodological improvement

That isn't the paper's literal pipeline — the authors use repeated-trial statistical analysis against released null distributions, not a single significance test — but it's the same underlying move: stop trusting a single number.

Not an isolated finding this week

Two more papers landed within days making the identical point from different angles. Auto Research for Materials(arXiv 2607.17100, July 19) ran seven closed-loop discovery searches producing 701 evaluated feature, model, representation, and training-data changes across ten Matbench endpoints — then froze each selected change and evaluated it exactly once on data that never entered the search loop. Nine of ten held up. That's a genuinely strong result, and the paper reports real held-out gains — a 17.4% MAE reduction on band gap, 18.6% on steel strength, and 26.3% when separately found feature and model changes are combined — but the number worth noticing is the one that didn't survive: without the held-out split, that failure would have been invisible, reported as a win alongside the other nine.

DRNOISE(arXiv 2607.17291, July 19) makes the same case from the opposite direction. Its 100-task benchmark seeds one ordinary-looking, plausible document that states a conflicting answer directly into an otherwise well-supported evidence environment. Agents that perform well on the clean version of the same task see accuracy drop by 66 to 88 percentage points once that single document is present. The trace analysis calls the dominant failure mode "verification inertia": the agent often retrieves the correct records, then stops short of reconciling them against the conflicting claim instead of pushing through to a real resolution. Generic verification prompts narrow the gap; they don't close it.

Three unrelated teams, three different fixes — repeated-trial statistics, held-out data separation, adversarial stress-testing — converging on the same diagnosis in the same week: the standard way an AI science agent gets evaluated, run it once and report the number, is not rigorous enough to support the claims routinely built on top of it.

What this means for reproducible, local-first science

The harness paper releases "all run pools including baseline null distributions for every model-problem pair as reusable statistical infrastructure" — which is, in effect, a reproducibility record: not just a claimed result, but the full population of runs a reader would need to check whether that result is real. You don't need 3.1 million rollouts of compute to apply the same discipline at your own bench. Log every run, not just the winning one. Hold out data the search loop never sees. Treat a single result as a hypothesis until it replicates. That's the same logic a reproducibility record and an independent check on a result exist to enforce — the difference between a science agent that looks fast and one whose answer is actually still standing after someone else checks it.

None of this is an argument against OpenEvolve, TTT-Discover, or any specific discovery harness — the paper doesn't make that case, and neither do we. It's an argument against picking one by reputation and stopping there. The teams behind all three papers this week did the harder, less exciting thing instead: they built the check that would catch them being wrong, then ran it before publishing the result that made them look right.