83.6% at the Hospital That Trained It. 52.0% at a Different One. Same AI Coding Agent.

Clinico is an AI agent built to adjudicate hospital diagnosis codes — the administrative record that determines reimbursement, disease surveillance, and how a case gets counted. Evaluated inside the Shanghai hospital whose records shaped its workflow, it matched the correct principal diagnosis 83.6% of the time. Evaluated on a second Shanghai hospital it had never seen, that fell to 52.0%. It still beat every other method tested, at both sites.

Why a coding agent is a reproducibility story

International Classification of Diseases (ICD) codes are what a hospital stay becomes after the fact: the record used for reimbursement, health-service planning, and disease surveillance. But a diagnosis is rarely settled on day one. Provisional, superseded, and conflicting diagnoses accumulate across an admission as new evidence arrives, and an automated system built to extract diagnoses from isolated notes can lock in an early guess that the rest of the chart later overturns — a wrong principal diagnosis doesn't just miscount a statistic, it can shift what a hospital is reimbursed for the stay. A team at Zhongshan Hospital, Fudan University, built Clinico to solve that specific problem: an agent that reads a stay chronologically, links each diagnosis to the evidence behind it, and revises its own calls as the clinical picture evolves — rather than a single pass over the discharge summary that most automated coding tools rely on.

What Clinico does differently

Clinico keeps what the authors call an evidence-linked diagnostic ledger across each admission: every diagnosis is tied to the specific note that supports it, timestamped, and reconciled against related or conflicting claims as the ledger updates — a diagnosis raised on day one can be confirmed, superseded, or ruled out by day four, and Clinico is built to track that status change rather than freeze on its first read. Only once the admission closes does it finalize the code set and select a principal diagnosis. The team refined Clinico's prompts and workflow using 5,000 development stays — without fine-tuning any model parameters — then tested it against three prespecified comparator systems, two of which were trained on 80,340 stays, an order of magnitude more data than Clinico ever saw directly. The primary outcome was exact-match agreement on the principal diagnosis against routine, coder-finalized codes; the key secondary outcome was micro-F1 across the complete code set, which credits or penalizes every code the system assigns, not just the headline diagnosis.

Inside the hospital that shaped it, Clinico won clearly

On 5,000 internal test stays — drawn from the same hospital whose 5,000 development stays shaped Clinico's prompts, though not overlapping with them — Clinico reached 83.6% exact-match agreement on the principal diagnosis and 90.6% micro-F1 across the full code set. Both were the highest observed estimates among every method the study evaluated, ahead of a next-best comparator at 74.7% and 90.3% respectively. An agent tuned on 5,000 stays and no fine-tuned parameters outperformed systems trained on 80,340 labeled stays, a sixteen-fold data advantage that bought the comparators nothing at this site. Taken on its own, that result is a clean methods paper: longitudinal, evidence-linked reasoning beats scale.

Move it one hospital over, and the number nearly halves

The external test used 4,109 stays from a second Shanghai hospital that contributed none of Clinico's development data. Principal-diagnosis accuracy fell from 83.6% to 52.0%. Complete code-set micro-F1 fell further in relative terms, from 90.6% to 42.6% — a number that says Clinico is not just occasionally picking the wrong headline diagnosis externally, it is reconstructing a meaningfully different code set for the same admissions. The comparators fell too, from 74.7% to 31.5% on principal diagnosis and from 90.3% to 25.5% on micro-F1 — proportionally similar drops. Every method the study evaluated got worse leaving its home hospital. Clinico still finished first at the external site. It just isn't the same system there that it was at home.

What held, and what didn't

One number in this study did survive the move: relative rank. Clinico finished first at both hospitals, on both metrics, by a wide margin over the next-best comparator both times. If the study had reported only ordinal rankings — which system won at each site — the paper would read as an unambiguous validation of Clinico's design. It is only because the authors reported absolute accuracy alongside rank that the second story becomes visible: winning by more at home (83.6% against a 74.7% runner-up, an 8.9 point margin) than away (52.0% against 31.5%, a 20.5 point margin, proportionally larger but off a much lower base) is not the same claim as being reliably usable at either site. A hospital deciding whether to adopt Clinico from a leaderboard position alone would see only the part of this result that held.

What the study does and does not explain

The preprint reports the size of the drop with tight confidence intervals; it does not isolate why the drop happens. That is worth stating plainly rather than filling in a plausible-sounding mechanism the paper itself doesn't test. Two Shanghai hospitals share a language, a national coding standard, and likely broad similarities in how medicine gets practiced — this is not a cross-country or cross-language transfer test, which makes the size of the collapse more notable, not less. Clinical NLP systems built on evidence extracted from free-text notes are known in the broader literature to be sensitive to site-specific documentation habits — template structure, abbreviation conventions, which findings a given hospital's clinicians bother to write down explicitly versus leave implicit — and every comparator in this study degraded on the move too, which is consistent with a shared, site-level cause rather than something specific to Clinico's architecture. But this study was not designed to isolate which of those mechanisms did it, or in what proportion, and neither are we going to assign one for them.

The honest limits

This is a preprint, not yet peer-reviewed, funded by no declared source, evaluated retrospectively against routine coder-finalized codes rather than an independently adjudicated gold standard, and run at two hospitals in one city rather than across health systems or countries — a harder transfer test would plausibly show a larger drop, not a smaller one. The routine codes used as ground truth are themselves produced by human coders working under the same time pressure and incentive structure that motivated building an automated system in the first place, so some of what looks like Clinico "disagreeing with the correct answer" may be Clinico disagreeing with a routine code that was itself rushed. The authors are explicit that prospective studies should test Clinico's effect inside a real clinical or billing workflow, using independently adjudicated references rather than the routine codes used here. None of that erases what the numbers show: an internal validation result strong enough to justify deployment, sitting next to an external result that would not survive a second look before deployment at a new site.

What this means for reproducible clinical AI

The uncomfortable finding here is not that Clinico has a generalization gap — most clinical NLP systems do, to some degree, when they meet a hospital's specific documentation habits. It is that the gap was invisible from inside the hospital that built the system. An 83.6%-accurate, 90.6%-F1 internal validation looks, by every normal standard, like a finished result: the kind of number that clears an internal review, gets written into a product page, and moves a tool toward a purchasing decision at the next hospital in line. The only way this study caught the collapse before that happened was by deliberately testing on a second site, with its own independent test set, before publishing the first number at all. Most deployed clinical AI systems never get that second test run before their vendor markets the first one.

That is the exact discipline a reproducibility record is supposed to enforce by default: not a claim checked once against the data that produced it, but a result re-run against data the system has never seen, with both numbers reported side by side rather than the stronger one leading a press release. Clinico's authors did that work themselves, on their own result, before anyone asked them to — which is a higher bar than most clinical AI reporting clears, and the reason this is a study worth taking seriously rather than a takedown. The gap they found is exactly the kind of gap that stays invisible when a lab only ever reports the number computed on the data it started with.