The Best AI Agent Refuses a Harmful Order Only 53.9% of the Time

The benchmark that stopped grading homework

Every agent benchmark up to this point has effectively been a closed-book exam: solve a coding puzzle, answer a question, complete a single tool call in a sandbox that resets when you're done. EnterpriseOps-Gym, built by ServiceNow Research and published as an arXiv paper in March 2026, is different. It runs agents against a containerized, stateful copy of a real enterprise stack: 164 database tables, 512 functional tools, live MCP servers, spanning 8 business verticals (HR, IT service management, customer service, email, calendar, drive, teams, and cross-system hybrid work). Every action an agent takes permanently changes the database. There's no reset button mid-task, and grading happens on the final database state, not the conversation transcript: a hidden, expert-authored SQL script checks whether the world actually ended up the way it was supposed to.

On July 8, 2026, Artificial Analysis launched EnterpriseOps-Gym-AA, an independent leaderboard running the current frontier field against this same environment, in oracle tool mode with 3 repeats per task. That is the dataset behind every number in this post.

The good news: completion rate is climbing fast

When ServiceNow published the original paper in March 2026, the best model, Claude Opus 4.5, completed 37.4% of tasks. Four months later, the July leaderboard shows real movement.

That's a real, fast 13.7-point climb on a genuinely hard benchmark, in 4 months. Claude Fable 5 leads, with Gemini 3.5 Flash close behind. If completion rate were the only thing this benchmark measured, the story would end here: frontier labs are converging on production-grade reliability inside live enterprise systems, and it's just a matter of time before the field clears 60%, then 70%.

The number that didn't move

Completion rate isn't the only thing EnterpriseOps-Gym measures. Buried in the same 1,150-task set is a subset of requests agents are supposed to refuse: instructions that break a privacy rule, skip an approval workflow, or ask for something the system shouldn't allow. The agent's job on those tasks isn't to complete them. It's to recognize the request is out of bounds and decline.

53.9% is not a stale March number that just hasn't caught up yet. It's the current best, measured on the same July 2026 evaluation as the 51.1% completion score. While task completion climbed nearly 14 points in 4 months, the ability to recognize and refuse a harmful order sat still, close enough to a coin flip that it's hard to call it a solved capability at all.

Where it breaks: the highest-stakes domains score worst

The gap isn't evenly spread across the 8 business domains either. Artificial Analysis published per-domain scores for 4 of them, and the pattern is uncomfortable: the domains with the most compliance exposure score the worst, and the domains where a mistake is cheap to undo score the best.

HR tasks touch approval chains and access controls: who's allowed to see whose compensation history, who can approve a termination, who signs off on a policy exception. IT service management tasks touch privileged access and change-management workflows. Both score under 30%. Email and Drive, domains where an imprecise action is usually reversible and low-stakes, score roughly twice as high. The domains where you most need an agent to get it right, and to know when to stop, are exactly the domains where it's worst at both.

The hidden math: why 79% per-check accuracy becomes 51% task success

Part of the completion-rate story is a grading-mechanics detail worth understanding if you're evaluating any agent on a multi-step, stateful task. Each task in EnterpriseOps-Gym carries an average of 5.3 separate verification checks against the final database state. The leading model passes 79% of individual checks. On its own, that sounds solid. But there's no partial credit: every check on a task has to pass for the task to count as a success.

Pass most of a task's checks and still fail the task outright, and a model that looks 79% reliable at the step level turns into a model that's barely a coin flip at the outcome level. That gap matters beyond this one benchmark: any system that grades agents on step-level accuracy instead of end-state correctness is going to systematically overstate how production-ready an agent actually is.

Money doesn't fix it

You'd expect spending more per task to buy a meaningfully better score. Look at the 4 models with published cost-per-task figures on this leaderboard.

DeepSeek V4 Pro clears a task for roughly a cent and lands at 40%. Claude Fable 5, 93x the price, lands at 51.1%: an 11-point gap for a 93x cost gap. Claude Opus 4.8, at $0.79, scores lower than DeepSeek's cheaper sibling GLM-5.2 relative to what its price tag would predict. Whatever is capping scores around the low-50s on this benchmark, it isn't solvable by throwing more inference budget at the problem.

There's a second, related efficiency data point in the same leaderboard: turns spent per task. Claude Fable 5 averages 6.5 turns per task. Gemini 3.5 Flash averages 13.5. Grok 4.5 (high) averages 17.4. Fable 5 isn't just scoring highest, it's doing it in roughly a third of the turns some competitors need, which suggests the gap between models here is less about raw compute and more about how efficiently a model plans its path through a multi-step task in the first place.

The real bottleneck: planning, not tool access

ServiceNow ran one more ablation in the original paper that reframes everything above. What happens if you hand the agent a perfect, human-written plan instead of making it figure out the plan itself?

Across the models ServiceNow tested, oracle planning lifted scores by 14 to 35 percentage points, a bigger swing than the entire gap between the best and worst model on the July leaderboard. The exact per-model breakdown isn't published, but the direction is unambiguous: tool access and execution aren't the limiting factor here. Planning is. An agent that knows the right sequence of steps, and, by extension, which steps to refuse, executes at a dramatically higher success rate than the same agent working out that sequence on its own.

Building your own unattended-write-access gate

The practical takeaway for anyone deciding whether an agent gets standing write access to a real system: a single completion percentage hides the number that actually determines blast radius, whether the agent knows when to say no. A simple pre-deployment filter looks like this:

# refusal_gate.py — filter EnterpriseOps-Gym-AA candidates on refusal rate, not just completion rank
CANDIDATES = [
    {"model": "claude-fable-5",       "completion": 51.1, "safe_refusal": 53.9, "cost_per_task": 0.93},
    {"model": "gemini-3.5-flash-high","completion": 50.1, "safe_refusal": None, "cost_per_task": None},
    {"model": "gpt-5.5-xhigh",        "completion": 46.6, "safe_refusal": None, "cost_per_task": None},
    {"model": "claude-opus-4.8-max",  "completion": 44.0, "safe_refusal": None, "cost_per_task": 0.79},
    {"model": "glm-5.2-max",          "completion": 43.0, "safe_refusal": None, "cost_per_task": 0.10},
    {"model": "deepseek-v4-pro-max",  "completion": 40.0, "safe_refusal": None, "cost_per_task": 0.01},
]

def cleared_for_unattended_writes(candidates, min_refusal=80.0):
    """Only a published, verified safe-refusal score above the bar clears
    an agent for standing write access to irreversible production systems.
    Missing data is a fail, not a pass — an unverified refusal rate is not
    a safe refusal rate."""
    return [
        c for c in candidates
        if c["safe_refusal"] is not None and c["safe_refusal"] >= min_refusal
    ]

cleared = cleared_for_unattended_writes(CANDIDATES)
print(f"{len(CANDIDATES)} candidates -> {len(cleared)} cleared for unattended writes")
for c in CANDIDATES:
    mark = "✓ OK" if c in cleared else "⚠ FIX"
    refusal = f"{c['safe_refusal']}%" if c["safe_refusal"] is not None else "unverified"
    print(f"  {mark} {c['model']}: completion={c['completion']}% safe_refusal={refusal}")

$ python3 refusal_gate.py
6 candidates -> 0 cleared for unattended writes
  ⚠ FIX claude-fable-5: completion=51.1% safe_refusal=53.9%
  ⚠ FIX gemini-3.5-flash-high: completion=50.1% safe_refusal=unverified
  ⚠ FIX gpt-5.5-xhigh: completion=46.6% safe_refusal=unverified
  ⚠ FIX claude-opus-4.8-max: completion=44.0% safe_refusal=unverified
  ⚠ FIX glm-5.2-max: completion=43.0% safe_refusal=unverified
  ⚠ FIX deepseek-v4-pro-max: completion=40.0% safe_refusal=unverified

Zero candidates clear an 80% refusal bar today, because even the one model with a published number, Claude Fable 5, sits at 53.9%. That's the actual state of the field in July 2026: not “agents are close to ready for unattended production access,” but “agents are getting measurably better at the job while the safety-relevant number stays roughly where it started, and most vendors aren't even publishing it yet.”

Keeping a gate like this current, re-pulling the latest EnterpriseOps-Gym-AA numbers, re-running the filter against whatever your production routing config currently allows, flagging the moment a model you already ship drifts below your threshold, is exactly the kind of unglamorous, always-on job that shouldn't depend on someone remembering to check a leaderboard. A mhermes agent, running on its own isolated VM with shell and network access, can do that pull and re-check on a schedule and flag drift before it becomes an incident.

And once you've settled on which models actually clear your bar for which task type, MegaBrain gives you one API across 500+ models with transparent, at-cost pricing and automatic routing, so shifting HR or ITSM workflows to a higher-refusal-rate model the moment one clears your threshold is a routing-config change, not a re-integration of your agent stack.

What the data actually says

Sources for every figure above: ServiceNow Research's EnterpriseOps-Gym paper (arXiv:2603.13594, March 2026) and Artificial Analysis's independent EnterpriseOps-Gym-AA leaderboard and writeup (artificialanalysis.ai/articles/announcing-enterpriseops-gym-aa, July 8, 2026).