0.731 AUC on Malaria Drug Screening Required Fine-Tuning. Without It, the Same Model Scored 0.499, No Better Than Random Guessing.

No single feature this week: five separate papers instead, each one a clean measurement of where an AI method for the life sciences actually breaks. The sharpest number comes from a direct test of a claim this blog stays skeptical of by default: that a frontier model can be prompted directly into working on a specialized bio task. A gene-and-drug-tuned open model scores 0.731 AUC identifying real drug hits on malaria targets it has never seen. Strip the fine-tuning and evaluate the same base model few-shot instead, and it falls to 0.499, no better than random guessing. General-purpose frontier models tested the same way land in between and still lose to the fine-tuned specialist. Below that: a CRISPR off-target classifier's real ceiling against actual GUIDE-seq data, a Nature Methods paper naming manual spatial annotation as the benchmarking problem rather than the solution, an enzyme-function benchmark fooled by catalytically dead decoys, and a leakage audit across 210 published obesity-ML papers.

Cheminformatics & drug discovery: fine-tuning is the whole result, not a detail

"Malaria-Instruct" (arXiv 2608.20418, submitted August 18, 2026) evaluates virtual screening for antimalarial drug candidates on a genuinely out-of-distribution ChEMBL-derived test set. Fine-tuned, TxGemma-9B reaches an ROC-AUC of 0.731. The same base model, prompted few-shot with no fine-tuning, drops to 0.499 — statistically indistinguishable from chance. General-purpose frontier models tested the same way land in between and well below the fine-tuned specialist: Gemini 2.5 scores roughly 0.53, OpenAI o3 roughly 0.59. The paper's framing is blunt — fine-tuning on the actual task is "categorically indispensable" here, not an optimization on top of a frontier model that already mostly works.

Genomics: a real ceiling for CRISPR off-target prediction, measured honestly

A new benchmark (bioRxiv 10.64898/2026.08.19.745843, posted August 23, 2026) scores off-target classifiers against real GUIDE-seq data rather than simulated labels — and against the actual class imbalance a CRISPR-Cas9 screen produces: 54 true off-target sites out of 95,829 candidates. The best-performing model reaches a PR-AUC of only 0.355 and a best F1 of 0.50. The paper is explicit that ROC-AUC — the metric most off-target papers report — is misleadingly high on a problem this imbalanced, and argues PR-AUC is the number that should be leading these comparisons instead.

Single-cell / spatial biology: Nature Methods says the annotation is the benchmarking problem

"SACCELERATOR" (Nature Methods, s41592-026-03194-8, published August 24, 2026) is a consensus framework for evaluating spatial-clustering methods, and its stated reason for existing is direct: manual anatomical annotation, the field's default ground truth for scoring these methods, is "often biased, error prone and unsuitable for benchmarking." The paper proposes expert-in-the-loop disagreement-mapping in its place — treating where annotators disagree as signal, instead of forcing every method toward a single annotated answer assumed to be correct. A companion piece in the same issue, "Benchmark pitfalls expose need for expert-guided spatial clustering" (s41592-026-03193-9), covers the same ground.

Structural biology: two enzyme-function tools still pass a dead decoy

"EnzymARC" (bioRxiv 10.64898/2026.08.21.746242, posted August 22, 2026) builds decoy sequences by computationally destroying the catalytic residues of real enzymes, then checks whether function-prediction tools notice. DIAMOND and CLEAN — the latter a widely cited protein-language-model contrastive learning method — still assign the original EC number to more than 90% of these catalytically dead decoys. DeepEC does better but still struggles. The paper's diagnosis is a "phylogenetic shortcut": these tools are matching sequence homology, not verifying that the residues doing the actual chemistry are intact.

Clinical & translational AI: a leakage pattern nobody was checking for

A medRxiv audit (10.64898/2026.08.19.26360787, posted August 22, 2026) reviewed 210 published papers using NHANES survey data to predict obesity from phthalate exposure with machine learning. None — zero of 210 — reported checking for a specific proxy-mediated imputation-leakage pattern the authors identify, which can manufacture a spurious correlation above |ρ| = 0.86 out of a true association below 0.15. The audit isn't purely a takedown: it also confirms a real, much smaller effect once the leakage is controlled for — the exposure block adds ΔAUC = +0.016 (95% CI 0.010–0.023) over demographics alone. Separately, a new 750-task expert-authored benchmark for general life-science reasoning, LifeSciBench (bioRxiv 10.64898/2026.08.13.744657, posted August 22, 2026), found the best-scoring system in its evaluation reaches only a 36.1% pass rate, and 22.8% of its 750 tasks got zero passing responses from any of the five models tested — a benchmark with real headroom left, unlike most saturated general-purpose leaderboards.

What this means for reproducible, local-first science

Every item this week is the same category of finding wearing a different beat's clothes: a metric that looked adequate turns out to be measuring the wrong thing, once someone checks it against a harder, more honest standard — real GUIDE-seq counts instead of simulated labels, catalytically dead decoys instead of homology alone, expert disagreement instead of a single annotated map, a documented leakage pathway instead of an unchecked assumption. None of these are exotic checks. They're the kind of re-run-it-a-different-way step that's cheap when it's local and routine, and expensive — or skipped entirely — when every re-evaluation means another cloud job and another wait.