Computer Use Looked Solved at 85%. The Realistic Version of the Same Test Scores 20.6%.

The test that quietly became easy

OSWorld has been the reference benchmark for computer-use agents since 2024: can a model look at a real desktop, plan a sequence of clicks, drags, and keystrokes across real applications, and get the job done. OSWorld-Verified, the repaired 369-task release from July 2025, has been treated as the yardstick ever since. By July 2026, that yardstick stopped being useful, because the field had simply caught up to it.

Nine models above 78%. Three above 83%. Two tied at 85%. On paper, computer use reads like a solved category the way image classification did a decade ago, benchmark saturation, not genuine capability exhaustion. Saturated benchmarks stop discriminating between good and great systems, which is exactly why a team spanning Hong Kong University, Snorkel AI, and Google Research built a harder one instead of declaring victory.

What OSWorld 2.0 actually tests

OSWorld 2.0 (arXiv:2606.29537, submitted June 28, 2026, revised July 13, 2026) keeps the same core idea, an agent operating a real computer, and throws out the assumption that a task should resolve in a few minutes. Its 108 tasks are pulled from actual professional workflows across 7 domains and 21 sub-categories: research, creative production, engineering, personal services, business and finance, administration and compliance, and healthcare.

That last pair of numbers is the whole design philosophy in two figures. A median human needs 1.6 hours to finish one of these tasks, and more than two-thirds of them run longer than an hour. Nothing in that description fits inside a product demo. The benchmark is explicitly built to reward the part of computer use that a 10-minute screen recording can't show: staying coherent, tracking state, and recovering from surprises over a genuinely long session.

The collapse

Run the same frontier models, several of them scoring 78%+ on OSWorld-Verified, against OSWorld 2.0's 500-step budget, and the leaderboard looks like a different category of problem entirely.

The best system on the planet for this task, by a comfortable margin, still fails 4 out of every 5 realistic jobs outright. That is not a small regression from an 83.4% baseline; it is the difference between “mostly works” and “mostly doesn't.” And the gap is not explained by effort: GPT-5.5 at its highest reasoning setting still lands at 13%, and every open-weights model tested clusters under 5%.

The hidden cost: 10.6x the tool calls, for a fifth of the result

The completion gap alone understates how much harder OSWorld 2.0 actually is, because it doesn't capture how much more work the agent has to do just to get partial credit.

318 tool calls is 10.6x the old benchmark's average, for a model completing the task 4x less often. Running the full 108-task suite end to end isn't cheap either: Claude Opus 4.7 at standard settings costs roughly $3,870 to clear the whole benchmark once ($35.83 per task), GPT-5.5 at its highest effort costs about $2,750 ($25.46 per task), and Claude Sonnet 4.6 at maximum reasoning runs $2,410 ($22.31 per task). The cheaper open-weights models finish the suite for a few hundred dollars total, Kimi 2.6 at $708 ($6.56/task), Qwen3-7-Plus at $411.56 ($3.81/task), MiniMax M3 at $258.78 ($2.40/task), but complete a fraction as much work per run.

Cost and completion don't trade off cleanly here. Spending 15x more per task (Opus 4.7 standard vs. MiniMax M3) buys roughly 3x the completion rate (13.9% vs. 4.6%), not 15x. Neither end of that curve is a good deal if the job genuinely needs to get done unattended.

What actually breaks the agent

OSWorld 2.0's authors tag every task with which real-world phenomena it exercises, which turns “the agent failed” from a single number into a diagnosis.

Reconciling facts across multiple documents shows up in nearly half the tasks, and visual precision (exact placement, alignment, timing) in another 41.7%, so those aren't the rare edge cases. The two phenomena at the bottom, dynamic environments (the task changes mid-way through) and proactive interaction (the agent should stop and ask a question instead of guessing), are the rarest in the suite and, per the paper's breakdown, the ones agents fail hardest. That combination, rare in training and evaluation data, catastrophic in production, is exactly the failure mode a short, scripted demo never has a chance to surface.

The number that matters more than the headline score

Binary completion is a harsh, all-or-nothing metric. OSWorld 2.0 also tracks partial score, the share of a task's internal checkpoints an agent hits, whether or not it finishes.

That reframes the story. These agents are not collapsing at step one, unable to even begin a real task. On average they complete more than half of a task's checkpoints, roughly 27 of them, before losing the thread. The failure mode is a stall, not a crash, and it tends to land right around the point where a task stops being scriptable (open the file, click the button) and starts requiring judgment (which of these three conflicting numbers is the current one, should I ask before I send this).

Building your own long-horizon reality check

The practical lesson for anyone deciding which model to route a real, long-running agent workload to: a short-task leaderboard score and a long-horizon completion rate can diverge by 60+ points on the exact same model, and the leaderboard score alone won't tell you which side of that gap you're on. A simple pre-routing check looks like this:

# longhorizon_gate.py — flag models whose short-task score overstates long-horizon reliability
CANDIDATES = [
    {"model": "claude-opus-4-8-max-batched", "short_task_score": 83.4, "long_horizon_score": 20.6, "cost_per_task": 35.83},
    {"model": "gpt-5-5-xhigh",               "short_task_score": 78.7, "long_horizon_score": 13.0, "cost_per_task": 25.46},
    {"model": "claude-sonnet-4-6-max",       "short_task_score": 81.2, "long_horizon_score": 8.3,  "cost_per_task": 22.31},
    {"model": "kimi-2-6",                    "short_task_score": None, "long_horizon_score": 4.6,  "cost_per_task": 6.56},
    {"model": "qwen-3-7-plus-thinking",      "short_task_score": None, "long_horizon_score": 2.8,  "cost_per_task": 3.81},
    {"model": "minimax-m3",                  "short_task_score": None, "long_horizon_score": 4.6,  "cost_per_task": 2.40},
]

def gap(c):
    """Points of short-task score not backed up by long-horizon completion."""
    if c["short_task_score"] is None:
        return None
    return round(c["short_task_score"] - c["long_horizon_score"], 1)

def status(c):
    g = gap(c)
    if g is None:
        return "? N/A "
    return "⚠ FIX " if g > 40 else "✓ OK  "

for c in CANDIDATES:
    g = gap(c)
    print(f"  {status(c)}{c['model']}: short={c['short_task_score']} "
          f"long={c['long_horizon_score']}% gap={g} cost/task=${c['cost_per_task']}")

$ python3 longhorizon_gate.py
  ⚠ FIX claude-opus-4-8-max-batched: short=83.4 long=20.6% gap=62.8 cost/task=$35.83
  ⚠ FIX gpt-5-5-xhigh: short=78.7 long=13.0% gap=65.7 cost/task=$25.46
  ⚠ FIX claude-sonnet-4-6-max: short=81.2 long=8.3% gap=72.9 cost/task=$22.31
  ? N/A kimi-2-6: short=None long=4.6% gap=None cost/task=$6.56
  ? N/A qwen-3-7-plus-thinking: short=None long=2.8% gap=None cost/task=$3.81
  ? N/A minimax-m3: short=None long=4.6% gap=None cost/task=$2.40

Every frontier model with a public short-task score in this shortlist carries a gap over 40 points once you check it against the realistic number, which is the whole point: the headline benchmark that made these models look interchangeable at the top of the field is the wrong number to route on for anything that resembles a real 1.6-hour job.

Keeping a check like that current, re-pulling the latest OSWorld 2.0 and OSWorld-Verified numbers, re-running the gate against whatever models your agent stack is actually allowed to use, is exactly the kind of always-on, unglamorous job a mhermesagent is built for: it runs on its own isolated VM with shell and network access, and can flag the moment a model you already route to drifts on a benchmark you haven't re-checked in a month. And once you know which models clear a long-horizon bar for your workload, MegaBrain gives you one API across 500+ models at transparent, at-cost pricing, so testing Opus 4.8 against Kimi 2.6 against Qwen3-7-Plus on your own long-running tasks is a routing-config change, not a re-integration.

What the data actually says

Sources for every figure above: the OSWorld 2.0 paper (arXiv:2606.29537, submitted June 28, 2026, revised July 13, 2026) and project page (osworld-v2.xlang.ai), the Snorkel AI OSWorld 2.0 leaderboard (snorkel.ai/leaderboard/os-world-2-0), the OSWorld-Verified leaderboard snapshot via BenchLM.ai (benchlm.ai/benchmarks/osWorldVerified, July 17, 2026), and the xlang-ai/OSWorld-V2 GitHub repository.