The Best AI Agent Completes 89% of a Business Task. It Finishes Clean 42% of the Time.

The benchmark built to catch the gap between attempted and trusted

Most AI benchmarks score a single exchange: a question in, an answer out. AutomationBench, published by Zapier as an arXiv paper on April 21, 2026, scores something harder: whether an agent can run an entire real business workflow, alone, across multiple applications, and get the correct data into the correct system at the end. 657 tasks span six domains, Sales, Marketing, Operations, Support, Finance, and HR, built on simulated versions of Gmail, Google Sheets, Slack, Salesforce, Zendesk, Jira, and HubSpot. Agents have to discover their own API endpoints, follow layered business rules buried in policy documents, and navigate environments deliberately salted with irrelevant and misleading records. Grading is programmatic and end-state only: is the correct data sitting in the correct system, yes or no.

Zapier's own paper is blunt about where that left the industry at launch: “even the best frontier models currently score below 10%.” Artificial Analysis independently adopted the benchmark two and a half months later, and MegaBrain covered that first run in our July 17 piece on Claude Fable 5's silent fallback, when the top score on Artificial Analysis's version sat at 48.6%. On September 7, Artificial Analysis folded a harder implementation, AutomationBench-AA, into Intelligence Index v4.3, alongside an upgrade to Terminal-Bench 4.0. That update is where the number everyone should be quoting, and almost nobody is, comes from.

One model, one run, three scores

AutomationBench-AA doesn't report a single pass rate. It reports three, and they describe three different questions. “Objectives completed” measures raw progress: how much of the task got done, full stop. “Score” adds a hard rule, any task with a guardrail violation counts as zero, then reports the objectives completed on top of that constraint. “Tasks completed” is the strictest cut: every single objective finished, with zero violations anywhere in the run. For GPT-6 Astra at max effort, the current leader, those three questions return three very different answers.

Every one of those numbers is real, measured on the same underlying run. The gap between them is not noise, it's the entire question of whether you can leave an agent alone with your CRM overnight. An agent that finishes 89% of a task and quietly mishandles the rest isn't 89% trustworthy. On this benchmark's own logic, it's 41.6% trustworthy, because the other 47.4 percentage points of “progress” belong to runs that broke a rule somewhere along the way.

# What "89% done" actually means once violations count
objectives_completed = 0.89   # raw progress, ignores rule-breaking
score = 0.685                 # objectives completed, but any violation -> task scores 0
tasks_completed = 0.416       # every objective done AND zero violations, strict

# The headline number in most demos and press releases:
headline = objectives_completed
# The number that matters if nobody is watching the agent run:
trust_number = tasks_completed

gap = headline - trust_number
print(f"Gap between the number that gets quoted and the number that matters: {gap:.1%}")
# Gap between the number that gets quoted and the number that matters: 47.4%

Under 10% in April, over 40% by September

Read the strict metric on its own timeline and the story isn't stagnation, it's one of the faster capability curves in agentic AI this year. Zapier's April 21 paper measured frontier models below 10% on the full private test set. By July 6, when Artificial Analysis ran its first pass and MegaBrain covered it, the top score (a partial-credit “Score”-style metric, not the strict full-completion cut) sat at Claude Fable 5's 48.6%. Ten days later, Moonshot AI's Kimi K3 took the top spot at 53% on that same metric family. By September 7, GPT-6 Astra's Score sits at 68.5%, and its strict full-task-completion number, the hardest cut of all, is already past 41%, a figure that didn't clearly exist as a separately tracked leaderboard metric back in April.

That's better than a 4x jump on the strictest metric this benchmark has, in under five months. Whatever else this data says about how far agents have to go before they can run a business function unattended, the slope underneath it is not flat.

What a guardrail violation actually looks like

“Guardrail violation” sounds abstract until you read the task set. AutomationBench-AA's published example scenarios describe concrete, specific failure modes, not vague rule infractions:

None of those are reasoning failures in the sense of getting a fact wrong. The agent found the right candidate, built a real signing chain, allocated a real expense line. It just didn't check one more constraint before acting, and on a benchmark with zero partial credit for guardrails, that one miss is worth the same as not attempting the task at all. In a live business, any one of those three specific mistakes is the kind that reaches a person's inbox, a compliance officer, or a payroll error, not a footnote on a leaderboard.

The rest of the field is bunched a single lab apart

GPT-6 Astra leads, but not by the kind of margin that suggests one company solved something the rest missed. Every other frontier model measured sits inside a roughly 10-point band, and all of them are still south of half on full-task completion.

* 31.4% on Zapier's board, 32.1% on Artificial Analysis's “tasks completed” metric, two labs, a month apart, within a point of each other.

Claude Fable 5.1 running with an Opus 5 fallback at max effort lands within a point of itself across two independent evaluations run by two different organizations. That kind of agreement is worth noting on its own: it means the benchmark is measuring something reasonably stable, not an artifact of one lab's test harness. It also means the entire frontier, not just one lagging model, is still failing more than half of real business workflows end to end.

The effort dial moves the number more than the model does

The single biggest lever inside this data isn't which lab you pick, it's how much thinking time you let the same model spend. Zapier's leaderboard reports GPT-6 Astra at five separate effort settings, same weights, same architecture, nothing changed except how long it's allowed to reason before acting.

Turning the dial from low to max is worth 11.1 points of full-task completion, more than the entire gap separating GPT-6 Astra's low-effort setting from last generation's GPT-5.6 Sol. On a benchmark this unforgiving, effort isn't a footnote in the model card, it's most of the distance between an average result and the leading one, and it is the one variable a team building on top of these models can actually control without waiting on a lab's next release.

Zero violations isn't the same as safe

The most easily misread number on the whole leaderboard sits at the bottom, not the top. Three of the lowest-violation agents measured, gpt-oss-20b, Magistral Small 1.2, and Llama 4 Scout, post exactly 0.00 guardrail violations per task. Read in isolation, that looks like the safest tier on the chart. Read against those same models' much lower objective-completion rates, it reads differently: a model that attempts very little of a multi-step, multi-app workflow has very little surface area left on which to trip a guardrail. Zero violations and zero attempts produce the identical safety score on this benchmark. That's not a flaw unique to AutomationBench-AA, it's a reminder that any “violation rate” metric needs to be read next to a completion rate, never alone, or it rewards agents for doing the least.

What this means if you're planning to run an agent unattended

Put the three numbers back together and the honest state of agentic AI in real business software, as of this week, is not a story about intelligence. GPT-6 Astra, Claude Fable 5.1, and the rest are all clearly capable of parsing a policy document, finding an API endpoint, and taking the right first, second, and third action in a workflow, 89% of the time on the leading model. What they aren't yet reliably capable of is finishing that same workflow with nothing broken, every single time, at a rate above roughly two in five. That's the actual last mile between “impressive agent demo” and “safe to leave running overnight,” and it moved from under 10% to over 40% in less than five months. Hold that slope, and the gap keeps closing on a timeline measured in quarters, not years, but it hasn't closed yet, and every lab shipping an agent demo this month is still leading with the 89, not the 41.6.

If you're deciding today whether a given workflow is safe to hand to an agent with no human in the loop, the objectives-completed number in a vendor's pitch deck is close to useless for that decision. The number worth asking for is the strict one: on a representative task set, with real business rules, what share of runs finish completely clean. Right now, across every frontier model measured, that answer is still south of 42%.

This is exactly the reliability question mhermes, MegaBrain's always-on agent runtime, is built around: shell access, a headless browser, persistent storage, and cross-session memory so an agent's work is inspectable and resumable instead of a black box you find out about after the fact, and model routing across the full frontier, including every model in this dataset, so a workflow isn't locked to whichever lab happens to be leading AutomationBench-AA this month.

Sign up at getmegabrain.com to route across 500+ models at zero markup and run agents that keep working, and keep a record of what they did, while you sleep.