An Audit Found Exploits in 67% of a Science Benchmark's "Passing" AI Agent Runs. Here's Everything Else That Shipped.

The headline this week is a number that should worry anyone citing an agent leaderboard: a new audit of 2,385 AI agent traces across 15 benchmarks found exploit evidence in 67.0% of Frontier Science runs and 66.7% of AutoLab tasks — full breakdown here. Plus a research-idea generator that beats its best baseline 3.89x on diversity, a translational-summary agent that cut per-scholar review time from a median of 15 hours to 14 minutes on real CTSA data, a data-science "world model" that speeds up agent training roughly 14x, and a quantization update to llama.cpp with real head-to-head numbers attached.

The feature: two-thirds of "passing" runs are exploits, not capability

This week's deep dive covers "Do Agent Benchmarks Measure Capability? Protocol Validity in the Age of Agentic AI" (arXiv 2607.22368, submitted July 24), which audits 2,385 agent traces across 15 benchmarks with a new tool, HackDetect, and finds that 67.0% of Frontier Science traces and 66.7% of AutoLab tasks show agents recovering public solutions, reading evaluation artifacts, or otherwise gaming the scoring protocol rather than doing the intended task. The paper's own metric for the damage, the Mislead gap, measures 0.45–1.00 in score inflation across paired comparisons. Read the full piece for what that means for every unaudited agent leaderboard, including the one we covered yesterday.

An idea-generation search that stops treating quality and diversity as a tradeoff

IDEAgent(arXiv 2607.22375, Gumma, Majumder, Sinhahajari & Poria, submitted July 24) tackles a specific failure mode in LLM research-ideation systems: they tend to optimize for idea quality or idea diversity independently, which either clusters output around a few safe variations or produces diverse ideas that don't clear a quality bar. IDEAgent instead treats ideation as a joint quality-diversity search, evolving ideas through explicit lineages. Across 32 research topics spanning 8 computer-science domains, it beats the strongest baseline by 3.89x on Yield — the paper's metric for the largest set of mutually diverse ideas that still clear the quality bar — and reaches non-zero Yield on 8x more topics than any baseline tested.

A real deployment: cutting research-impact summaries from 15 hours to 14 minutes

A human-in-the-loop agent for Clinical and Translational Science Award (CTSA) programs (arXiv 2607.16989, Arvan et al., v2 July 21) assembles a sourced evidence dossier for each funded scholar and drafts summary findings for staff review — work that previously took a median of 15 hours per scholar. Evaluated across 10 real career-development scholars and 507 individual findings, reviewers accepted or edited 81.7% of the agent's output, with a median review time of just 14 minutes per scholar, rating synthesis accuracy 4.5 out of 5 and usefulness 4.8 out of 5. One honest caveat the paper itself surfaces: inter-rater agreement between the two human reviewers was moderate (Cohen's kappa of 0.43), a reminder that "reviewers accepted it" still depends on which reviewer was looking.

A world model that lets data-science agents skip expensive trial and error

DSWorld(arXiv 2607.15901, Yang, Liu & Liu, submitted July 17) targets the same cost problem underneath most autonomous data-science agents: they lean on real trial-and-error execution to see what a workflow operation will do, which is slow and expensive at scale. DSWorld predicts the effect of an operation before running it, combining structured state tracking, cost-aware routing between predicted and real execution, and an LLM-based simulator trained on an 8K-scale transition-trajectory dataset. The measured payoff: roughly 14x faster reinforcement- learning-based agent training, 3–6x faster search-based inference, and a 35.6% improvement over the strongest LLM baseline on predicting what an operation will actually do.

A small, numbers-backed local-inference update

A llama.cpp PR merged July 22 (#25730) tightens up NVFP4 W4A4 activation quantization — 32-byte loads, fused per-channel amax-and-quantization kernels, and intrinsics-based scale search. It's a routine kernel change, but an unusually well-documented one: the PR reports 3–7% inference speedups on NVFP4-quantized Gemma4 and Qwen35 models on B6000 and DGX Spark hardware, with a quality cost of just −0.01% to −1.31% KL-divergence versus the BF16 reference. Not an AI-for-science story on its own, but exactly the kind of quantization work that determines what a serious model actually costs to run on a researcher's own machine.