AI's Best Model Scores 66% Once. Ask It to Repeat the Answer 5 Times, Every Model Drops Below 55%.

The metric almost nobody publishes

Every model launch comes with a benchmark score, and almost every one of those scores answers the same question: can the model get this right? Usually that means pass@1 (did it get the right answer on a single try) or, generously, pass@k (did it get the right answer at least once across k tries, taking the best case). Both numbers describe a ceiling — the model's best day.

AA-AnalystAgent, a new benchmark Artificial Analysis released on August 10, asks a different, less flattering question. It runs 80 real-world business and scientific analysis tasks — drawn from 14 domains, the kind of work that involves reading a source document, applying professional judgment about which caveats and exceptions apply, and landing on a defensible number — through 26 models, 5 independent attempts per task, 10,400 graded attempts in total. The headline metric isn't pass@1 or pass@5. It's pass^5: a task counts as solved only if the model answers it correctly on all 5 attempts.

# pass_n.py — the reliability metric AA-AnalystAgent reports instead of pass@1
def solved_once(attempts: list[bool]) -> bool:
    """pass@k style: True if ANY attempt succeeded."""
    return any(attempts)

def solved_every_time(attempts: list[bool]) -> bool:
    """pass^5 style: True only if ALL attempts succeeded."""
    return all(attempts)

# one task, run 5 independent times
task = [True, True, False, True, True]  # 4 of 5 correct

print("pass@k (best of 5):   ", solved_once(task))       # True
print("pass^5 (all 5 of 5):  ", solved_every_time(task))  # False

$ python pass_n.py
pass@k (best of 5):    True
pass^5 (all 5 of 5):   False

✓ OK: this task would count as "solved" on almost every public leaderboard
⚠ FIX: it does not count as solved on the metric that predicts whether you
  can trust the model's output without checking it yourself

That distinction sounds pedantic until you see what it does to the leaderboard. Once you stop measuring a model's best day and start measuring whether today looks like every other day, the ranking of “best AI model” changes.

Two leaderboards, two different winners

On pass@1, OpenAI's GPT-5.5 (xhigh reasoning effort) posts the highest single-shot score in the field: 66%. Gemini 3.1 Pro Preview and Claude Opus 5 (Adaptive Reasoning, Max Effort) sit close behind at 64%. If you stopped reading a launch announcement at the first number, GPT-5.5 looks like the benchmark's clear winner.

Switch to pass^5 and Anthropic takes 3 of the top 5 spots. Claude Opus 5 leads at 54%, GPT-5.5 follows at 50%, and Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback) takes third at 49%. The top 3 models are separated by just 4 solved tasks out of 80.

Artificial Analysis's own read of the gap: GPT-5.5 has the highest pass@1, but Opus 5 leads pass^5 “because it repeats what it gets right.” That single sentence is the whole story of this benchmark. Getting an answer right once and getting it right reliably are not the same skill, and the model that's best at one is not automatically best at the other.

Gemini's 40-point collapse

The clearest illustration is Gemini 3.1 Pro Preview. Give it a task and let it try 5 times, counting a win if any single attempt lands: it solves 81% of the 80 tasks — near the top of the field on pass@5. Require all 5 attempts to agree, and that collapses to 41%. A 40-point drop, and a finish of 9th place on the metric that actually predicts whether you can trust its output unsupervised.

Artificial Analysis's failure-mode breakdown offers a partial explanation of why: Gemini 3.1 Pro Preview “takes sources at their word but fumbles the execution.” It sits below the ten-model median on misreading domain terms and on overriding evidence in favor of a generic prior, but posts above-median rates of modeling, scaling, and aggregation errors — 51% of its failures — and skipped verification, 39%. In other words: it usually understands the question correctly. It just doesn't consistently execute the arithmetic and verification steps that turn understanding into a repeatable right answer.

27x the price, the identical score

Cost per task varies enormously across the field, and it does not track reliability the way you'd expect. Claude Sonnet 4.6 (Adaptive Reasoning, Max Effort) and the much smaller open-weight model MiMo-V2.5-Pro land on the exact same pass^5 score — 20% — at wildly different prices: $1.34 per task for Sonnet 4.6, versus $0.05 per task for MiMo-V2.5-Pro. That's 27 times the cost for an identical reliability score.

The pattern holds at the top of the price range too. Claude Opus 4.7 (Adaptive Reasoning, Max Effort) is the single most expensive model in the entire test at $1.98 per task, and it still only places 8th on pass^5. GPT-5.5 (xhigh) runs the same benchmark for $1.15 per task and takes 2nd place. Price and reliability are, at best, loosely correlated here — and at this specific price point, not correlated at all.

Open weights: catching up, not caught up

Open-weight models are closing the gap with the closed frontier, but they haven't closed it. Kimi K3 (max) leads the open-weight field at 39% pass^5 — 15 points behind Claude Opus 5's 54% frontier score, and still the best open model by a wide margin. The next-best open-weight model, DeepSeek V4 Flash (Reasoning, Max Effort), trails at 25%. The 14-point gap between the two best open models is wider than the 15-point gap separating the open-weight leader from the closed frontier above it — the spread inside the open-weight field is currently the bigger story than the distance to the top.

Why models fail: anchoring is the tell

Artificial Analysis didn't stop at scores. It tagged 1,567 failed attempts across 10 leading models by failure type, letting each attempt carry more than one tag where multiple failure modes applied. The single most common cause, present in 57% of every failureacross the board, was anchoring: the model commits to an early interpretation of a source, a number, or a methodology, and defends that interpretation for the rest of the trajectory — including, notably, on the 2nd, 3rd, 4th, and 5th independent attempts at the exact same question.

That last part is the uncomfortable detail. Anchoring isn't just a within-attempt failure — a model that anchors incorrectly on attempt 1 doesn't reliably self-correct on attempts 2 through 5, either. It tends to anchor the same way again, which is exactly the mechanism that turns a decent pass@1 score into a mediocre pass^5 score: the error isn't random noise that repetition averages out, it's a systematic bias that repetition reproduces.

Two individual models illustrate the same failure surfacing in opposite directions. Grok 4.5 (high) “understands the field's language, then substitutes its own assumptions for what the documents say” — misreading domain terms on 23% of its failures against a 38% ten-model median, but overriding evidence on 54% of its failures against a 43% median. Kimi K3 (max) sits at the opposite end on domain-term misreads, at 48% — 25 points clear of Grok 4.5 on that specific failure mode, even while placing well ahead of it on the overall pass^5 leaderboard.

What this means if your agent doesn't have a human reading every answer

Put the whole benchmark together and the uncomfortable conclusion is this: the single best model available today, run on real business and scientific questions, still fails at least 1 out of every 2 of them when you require it to be right every time you ask. Not because it doesn't know the answer — the pass@1 and pass@5 numbers show it demonstrably does, often. It fails the reliability bar because it won't consistently let go of its first guess, and that specific failure mode doesn't show up in a benchmark score that only asks “can it get this right,” because that question only ever needed one right answer to say yes.

That gap is exactly the one that matters for anything you intend to run unsupervised — an agent monitoring a pipeline, triaging tickets, or making a call on a repeating schedule without a human reviewing each output. A pass@1 or pass@5 number tells you what a model can do on its best attempt. It tells you almost nothing about whether attempt number 40 in a month of unattended runs will look like attempt number 1. Until a vendor publishes its pass^5, the honest assumption is that the gap between “looked right in the demo” and “right enough to leave alone” is exactly this wide: 15 to 40 points, depending on the model.

If you're building or running agents that operate without a human checking every call, this is the number to ask vendors for before you ask about pricing. A mhermesagent — MegaBrain's always-on runtime, with shell access, scheduling, and persistent memory across sessions — is designed for exactly the unattended, repeating workloads where a pass@1 headline score is the wrong thing to optimize for. And because MegaBrainroutes to any of 500+ models through one API at transparent, zero-markup pricing, swapping in whichever model actually clears your reliability bar — not whichever one has the loudest launch tweet — is a config change, not a migration.

Sign up at getmegabrain.com to route your agents at cost, or spin up a mhermes agent to track this benchmark the next time Artificial Analysis refreshes it.