An AI Memory System Topped a Science-Agent Leaderboard by Retrieving 2.6 Million Characters a Query. Cap It at 30K and It Drops to Last.

2,552,000. That's how many characters a knowledge-graph memory system pulled per query to take the number one spot on a new scientific-memory leaderboard for AI research agents: over 410 times what its closest competitor used for the same questions. A new benchmark ran the obvious control, equalizing how much context every system is allowed to retrieve. The number one system's score collapsed from 8.04 to 5.27, dead last, while three architecturally ordinary retrieval systems it had been beating converged within 0.03 points of each other. The leaderboard wasn't measuring memory. It was measuring who was allowed to retrieve the most, and every vendor claiming a memory advantage should have to show their retrieval budget before anyone believes the score.

A leaderboard for a claim vendors keep making

Nearly every AI research-agent pitch that touts long-term memory over a corpus of papers — remembering what it read last week, synthesizing across a growing library, picking up a project where it left off — is implicitly claiming an architectural advantage in retrieval. Until now, almost nothing tested that claim under conditions a working scientist would recognize as fair. Beyond Memory Leaderboards(arXiv 2607.16848, Sheverev, Finkelstein, and Nikolenko, submitted July 18) built two new benchmarks to do exactly that: PAIM, 66 questions over 81 real papers, and PTr, 98 questions over 252 papers. Both test whether a memory or retrieval system can restore the specific evidence needed to answer a question about a paper it was given earlier — not whether it can chat convincingly about the topic in general. Eight memory and retrieval systems were evaluated head to head, from knowledge-graph architectures like Graphiti to plain BM25-hybrid Simple RAG.

The uncapped leaderboard: bigger context wins

Run each system natively — however much context it wants to pull, no limit imposed — and Graphiti tops the PAIM leaderboard at a score of 8.04. It gets there by retrieving between 2,552,000 and 2,596,000 characters per query. Mem0, tested on the same questions over the same papers, retrieves just 6,222 characters per query — a gap north of 410x. Nothing in that comparison distinguishes better recall from more recall. A system that returns roughly the equivalent of a small book's worth of raw text per question has an obvious structural advantage over one that returns a page, and the native leaderboard has no way to tell you which part of the score is architecture and which part is volume.

Cap the budget and the ranking inverts

The paper's intervention is blunt and effective: hold every system to the same retrieval budget and re-run the exact same questions. At roughly 30,000 characters per query on PAIM — still generous, just no longer unlimited — Graphiti's knowledge-graph variant drops from 8.04 to 5.27. That's a 34% fall, and it lands the system that led the uncapped leaderboard in last place among the eight tested. Two systems the native leaderboard made look mediocre move to the top under the same budget: Simple RAG at 7.25, Theoria at 7.19. Neither is exotic. Simple RAG is, per its name, not doing anything architecturally clever — it just doesn't need 2.6 million characters to find the right answer.

It isn't a fluke — the second benchmark shows the same collapse

A single inversion on one benchmark could be an artifact of PAIM's specific question set. The paper's second benchmark, PTr, tests the same idea on a different corpus — 252 papers, 98 questions — and finds the same pattern from a different angle. At a 50,000-character budget with BM25 hybrid retrieval, Mem0, Simple RAG, and Theoria land within 0.03 points of each other: 9.24, 9.22, and 9.21. The authors describe budget control as "the single most significant intervention" in the entire study — not a footnote finding, but the headline one. Once every system is forced to work with the same amount of retrieved evidence, the architectural differences that drove the uncapped rankings mostly disappear.

Is this just judge noise? The paper checks for that too

A result this clean invites a reasonable objection: maybe the scoring itself is noisy enough to produce a leaderboard flip that doesn't reflect anything real. Beyond Memory Leaderboards checks its own grading before drawing conclusions from it. Cross-frontier-model judge correlation lands at ρ = 0.90–0.97 — different LLM judges scoring the same answers largely agree with each other. Human blinded evaluators, checking the judges' calls against their own reading of the source papers, agree with the automated scores up to 77% of the time (with a one-point tie tolerance). That's calibration good enough to trust the direction of the effect: the leaderboard inversion is a property of the systems under test, not an artifact of how they were graded.

What "memory" actually has to mean for a research agent

The uncomfortable part for the AI-for-science field isn't that one knowledge-graph product looked worse than advertised. It's that "memory" as a marketed capability has been getting measured, in at least this case, without disclosing the one variable that decided the outcome. A research agent that has to work inside a fixed context window — which is every research agent, eventually, once a project's paper library grows past what fits in one prompt — needs a memory system that performs well under a budget, because a budget is the condition it will actually run under. An uncapped leaderboard measures something closer to "how much of the corpus can this system stuff into an answer," which is a real capability but a different one, and conflating the two lets a system win on volume while claiming to have won on retrieval intelligence.

The fix the paper argues for is straightforward and worth becoming a norm well beyond this one benchmark: report the retrieval budget next to the score, every time. A number like "8.04" without a disclosed character budget is not comparable to another system's "8.04" unless both operated under the same constraint — exactly the same reason a false-positive rate has to travel with a detection rate, which this blog covered from the citation-verification side earlier this week. Two different benchmarks, two different failure modes, one shared lesson: an AI-for-science metric that reports only the favorable half of a trade-off isn't measuring the thing it claims to measure.

What this means for reproducible, local-first science

Beyond Memory Leaderboards doesn't test MegaBrain BioScience, and none of its numbers transfer to any specific product — it's a benchmark of memory architectures as a category. But the generalizable finding lands on ground this blog keeps returning to: a capability claim in an AI research tool is only as trustworthy as the conditions disclosed alongside it. A reproducibility record that reports a result without the constraints it ran under — compute budget, retrieval budget, context window, tool access — invites exactly the kind of leaderboard-gaming this paper caught. The fix isn't a better memory architecture. It's reporting the budget.