The #1 Agent on the First Real-Work AI Benchmark Is Secretly 2 Models. It Still Fails the Job 51% of the Time.

The test nobody had a standardized score for until 3 weeks ago

Every frontier lab already tops some leaderboard. GPT-5.6 Sol leads Artificial Analysis's Coding Agent Index. Claude Fable 5 leads the general Intelligence Index at 60 points. Those are real, useful numbers, and none of them answer the question that actually matters if you are the one deciding whether to let an agent touch your CRM overnight: can it get real work done, following your business's actual rules, without a human reading every action first?

Zapier built AutomationBench to answer exactly that, open-sourcing it on GitHub and publishing the methodology as an arXiv paper. On July 6, 2026, Artificial Analysis partnered with Zapier to run an independent leaderboard, AutomationBench-AA, against Zapier's private evaluation subset, so labs cannot train against the exact test set. The design choices are deliberately boring, in the way a real audit is boring: deterministic final-state assertions instead of an LLM grading its own homework, multi-app workflows instead of single-turn Q&A, and both positive assertions (“did the objective complete”) and negative ones (“did a guardrail break”), specifically to stop a model from reward-hacking its way to a high score by ignoring the rules.

The leaderboard: nobody clears 50%

657 tasks, 40 simulated app environments, 12,000 assertions, a 50-turn budget per task. Here is the full board Artificial Analysis published.

Every frontier lab in this comparison loses more real business tasks than it wins. The best open-weights model available, GLM-5.2, completes barely more than a quarter of the workflows without breaking a rule. This is with a 50-turn budget and a rulebook the model can, in principle, learn to satisfy. Nobody in the field is close to production-grade unattended reliability on this test yet.

The #1 spot is not what it looks like

Claude Fable 5's listing on Artificial Analysis carries a detail easy to miss: the model name includes “(with fallback)” and references an “Opus 4.8 Fallback.” Digging into the AutomationBench-AA writeup explains why. Anthropic ships Fable 5 with its own classifier that routes flagged queries to a second, larger model when the classifier judges the task outside Fable 5's comfort zone. Artificial Analysis measured how often that happens on this benchmark specifically: roughly 18% of tasks.

That reframes the entire top of the leaderboard. The 48.6% score is not what Fable 5 alone achieves; it is what a Fable-5-plus-Opus-4.8 relay achieves, reported under a single model name. Strip the fallback out and you cannot recover what Fable 5 solo would have scored from the public numbers, which is itself the problem: a routing decision made upstream of the benchmark is invisible inside the benchmark's headline number. Anyone reading “Claude wins AutomationBench-AA” and provisioning capacity, or picking a model, on that sentence alone is missing that the win required 2 models' worth of compute on nearly 1 in 5 tasks.

Same architecture, wildly different job

Break the results down by business domain and the reliability gap gets bigger than the gap between any two labs.

Finance workflows carry the tightest, least forgiving rules in the suite, reconciling numbers, respecting approval thresholds, matching exact record identifiers, and every model tested struggles with them at roughly the same rate. Support and Operations tasks, more forgiving of an imprecise intermediate step, complete almost twice as often. The practical read: the job you are automating matters more than which frontier model you pick to automate it. A model swap buys you single-digit points. Picking a lower-stakes domain to automate first buys you 27.

The part that should worry anyone running an agent unsupervised

Completion rate only tells half the story. AutomationBench-AA also tracks guardrail violations, cases where the agent breaks one of the 12,000 explicit business rules while working, whether or not it finishes the task.

Even the safest model in the test, Gemini 3.5 Flash, still breaks an explicit business rule about once every 15 completed objectives. Qwen3.7 Plus breaks one more often than it completes a task at all. And this is the best-case scenario for these models: a human-authored rulebook of 12,000 assertions, purpose-built to catch violations. Deploy any of these agents against a real CRM with no equivalent scaffolding in place, and there is no rulebook catching the mistake before a customer, a vendor, or your books see it.

A 30x price spread for a completion rate under 50%

The other axis nobody's headline benchmark score captures: what it costs to run the task, independent of whether it succeeds.

Opus 4.8 at max settings costs roughly 30x what DeepSeek V4, Gemini 3.1 Flash-Lite, or Qwen3.7 Plus cost per task, for a completion-rate ceiling across the entire field that tops out under 50%. Price is not a reliable proxy for whether the job gets done. If anything, the cheapest 3 models in this comparison sit close to the same ballpark cost while spanning a wide range of completion quality, which means cost alone tells you almost nothing about which of them to actually route to for a given workflow.

“Think longer” is not the fix

One more assumption this benchmark quietly breaks: that spending more inference effort, more tool calls, more reasoning turns, buys more reliability.

GPT-5.5 on its highest reasoning setting burns 40% more tool calls and nearly double the turns Opus 4.8 uses on the same task set, and still scores 6.4 points lower. More compute spent mid-task correlates with worse task hygiene here, not better: models that take more turns to get somewhere tend to accumulate more chances to violate a rule along the way, not fewer.

Building your own honesty-adjusted routing check

The practical takeaway for anyone routing agents against real workflows: a single headline completion percentage hides at least 3 numbers you actually need, the fallback rate baked into that score, the violation rate per completed objective, and the cost per task. A simple pre-routing filter looks like this:

# route_policy.py — filter AutomationBench-AA candidates on reliability, not just rank
CANDIDATES = [
    {"model": "claude-fable-5",     "score": 48.6, "fallback_rate": 0.18, "obj_per_violation": None,  "cost": 1.10},
    {"model": "claude-opus-4.8-max","score": 48.5, "fallback_rate": 0.0,  "obj_per_violation": 13.5,  "cost": 1.50},
    {"model": "gemini-3.5-flash",   "score": 42.6, "fallback_rate": 0.0,  "obj_per_violation": 15.0,  "cost": 0.49},
    {"model": "qwen3.7-plus",       "score": 20.1, "fallback_rate": 0.0,  "obj_per_violation": 0.8,   "cost": 0.05},
]

def unattended_safe(candidates, min_obj_per_violation=10, max_fallback_rate=0.05):
    """Reject models whose real-world score depends on a hidden second
    model, or that break a rule too often per completed objective."""
    return [
        c for c in candidates
        if c["fallback_rate"] <= max_fallback_rate
        and (c["obj_per_violation"] or 0) >= min_obj_per_violation
    ]

safe = unattended_safe(CANDIDATES)
print(f"{len(CANDIDATES)} candidates -> {len(safe)} pass the unattended-safety filter")
for c in CANDIDATES:
    mark = "✓ OK" if c in safe else "⚠ FIX"
    print(f"  {mark} {c['model']}: fallback={c['fallback_rate']:.0%} obj/violation={c['obj_per_violation']}")

$ python3 route_policy.py
4 candidates -> 2 pass the unattended-safety filter
  ⚠ FIX claude-fable-5: fallback=18% obj/violation=None
  ✓ OK claude-opus-4.8-max: fallback=0% obj/violation=13.5
  ✓ OK gemini-3.5-flash: fallback=0% obj/violation=15.0
  ⚠ FIX qwen3.7-plus: fallback=0% obj/violation=0.8

That is a toy filter, not a production ranking, but it makes the point AutomationBench-AA's headline number hides: the model with the best public score, Claude Fable 5, fails this specific unattended-safety filter because its own vendor cannot yet say what it scores without help. The model with the worst headline score in this shortlist, Qwen3.7 Plus, fails for the opposite, more dangerous reason: it is cheap and confident and breaks rules constantly.

Keeping a filter like this current is exactly the kind of unglamorous, always-on job that should not depend on someone remembering to re-check a leaderboard. A mhermes agent, running on its own isolated VM with shell and network access, can pull the latest AutomationBench-AA and Artificial Analysis numbers on a schedule, re-run this filter against whatever models your production routing config currently allows, and flag the moment a model you already ship drifts below your violation-rate or fallback-rate threshold, before your customers find out the hard way.

And once the unattended-safety filter has narrowed your shortlist, MegaBrain gives you one API across 500+ models with transparent, at-cost pricing and automatic routing, so switching which model actually handles Finance versus Support workflows is a routing-config change, not a re-integration of your agent stack.

What the data actually says

Sources for every figure above: Zapier's AutomationBench announcement (zapier.com/blog/introducing-automationbench) and open-source benchmark (github.com/zapier/AutomationBench, arXiv:2604.18934), and Artificial Analysis's independent AutomationBench-AA leaderboard and writeup (artificialanalysis.ai/articles/announcing-zapier-automationbench-aa, July 6, 2026).