Opus 4.8 Aces a Web-Editing Benchmark at 99.4%. On 273 Real User Requests, the Best AI Agent Fully Finishes 26.4%.

99.4% on the old test, 64.9% on real requests

Co-Design Bench (arXiv 2610.35138, submitted October 26) comes from a team at Alibaba Group and 5 universities. It starts from a simple complaint: web-editing benchmarks are saturated. To show it, the authors reran an existing one, WebCompass-edit, on 150 tasks.

Run single-shot, Qwen3.7-Plus completed 73.6% of the requested features. Run as an agent, with tools and a loop, the same model scored 95.1%. Claude Opus 4.8, as an agent, scored 99.4%. A benchmark with 0.6 points of headroom at the top can’t separate the best models.

So the team built a harder test from real work. They started with 39,325 production queries from 26,314 coding sessions, deduplicated them to 37,463, and favored the requests where users had to rephrase or roll back. Each of the 273 tasks pairs a working app with one change request. Experts wrote 2,331 hidden acceptance criteria to grade them.

Grading is strict. An LLM judge builds and opens the finished app in a browser and checks each criterion. A second verifier then challenges every pass, looking for stubs, hardcoded results and features nobody can reach. Visual claims need a screenshot, and interaction claims need recorded browser actions.

The leaderboard

All 10 models ran in the same harness, Claude Code, with the same prompts, containers and tools, capped at 3 hours and 120 turns per task. Strict completion is the average share of criteria fully met. Full pass means every criterion on a task was met.

From Tables 2 and 5 of arXiv 2610.35138 and the project’s data file. Strict completion converted from the paper’s 0 to 1 scale. Ranked by strict completion, as in the paper.

The gap between “mostly done” and “done” is the story. Averages near 60% to 70% sound fine. But the person who filed the request wanted their whole change, and on most tasks no model delivered it.

Repository edits are where agents fall apart

The multimodal track asks for changes to a page from an attached video, audio clip, image or document. The repository track asks for changes inside a real codebase, and 166 of its 173 tasks need 2 or more kinds of edits at once.

Every model did worse there. GPT-6 Astra’s full-pass rate falls from 41.0% on multimodal tasks to 17.9% on repository tasks. Claude Opus 5 falls from 30.0% to 11.6%.

Breadth hurts too. Pooled across all 10 models, repository tasks with 1 or 2 edit types score 65.4% strict completion, and tasks with 3 or 4 types score 59.1%. The share of tasks where a model met zero criteria rises from 0.9% to 5.8%. All 10 models scored lower on the broader tasks.

More tokens did not buy better work

The paper logs every run. On the multimodal track, GPT-6 Astra used a median of 32 turns and 1.24M tokens per task to reach 75.4% strict completion. GLM-5.2 used 69 turns and 7.29M tokens, nearly 6x as many, and reached 57.2%.

From Table 2 of arXiv 2610.35138. The paper treats tokens as a measure of execution effort, not a monetary cost, so we don’t convert them to dollars.

Doing too little hurts as much as flailing. On repository tasks, Grok 4.6 stopped after a median of 9 turns and 0.45M tokens, and only 1.8% of its tool calls ran a shell command. It finished last on that track.

Swap the harness, and the ranking flips

The most useful table in the paper is in Appendix H. The team took 2 models, Kimi K3 and Qwen3.8-27B, and ran all 273 tasks again in 3 other harnesses: OpenAI’s Codex CLI, OpenCode, and Alibaba’s own Qwen Code. Prompts, criteria and judges stayed the same.

Average completion, Table 12 of arXiv 2610.35138. This run is separate from the main leaderboard, so don’t compare these numbers with the table above.

Same model, same tasks: Qwen3.8-27B went from 0.749 in OpenCode to 0.560 in Qwen Code on repository edits, an 18.9-point swing. Qwen Code, built by the same company as the model, was its worst harness.

On multimodal tasks, the harness order reversed completely. Kimi K3 did best in Qwen Code and worst in OpenCode, and Qwen3.8-27B did the opposite. The authors’ conclusion: always report the model and the harness together.

Read it with these caveats

Alibaba grades Alibaba. Most authors work at Alibaba Group, which makes the Qwen models that place 3rd and 4th. The authors flag that Qwen3.8-27B’s second place on repository edits, 0.645 to Claude Opus 5’s 0.642, does not establish a general advantage.

The judges are models. Two front-end experts reviewed the judge’s verdicts on 200 task-model outputs. Rescoring with 3 other judge models moved scores by at most 0.053 and didn’t change the order of the 2 models checked.

One request, no follow-ups. Each task is a single fixed request. In real life you’d see the half-done change and send a correction, which the benchmark doesn’t test.

The task counts don’t quite match. The project page itself notes that Hugging Face lists 96 multimodal tasks and 521 criteria, against the paper’s 100 and 553.

What this means if you code with AI

Ignore benchmarks at 95% and above. A score near the ceiling tells you the test is solved, not that your request will be. Look for benchmarks built from real requests, with hidden criteria, that report a full-pass rate.

Expect partial delivery, and check for it. Even GPT-6 Astra left something undone on 73.6% of tasks. When you vibe code a change, click through every part of the request before you accept it. The verifier in this paper caught stubs and hardcoded results, and your review should too.

Split broad changes. Repository tasks that mix 3 or 4 kinds of edits scored lower for every model. 2 narrow requests beat 1 sprawling one.

Test the agent, not only the model. An 18.9-point swing from the harness alone is bigger than the 8.8-point gap between 1st and 6th on the main leaderboard. If a model disappoints you in one tool, try it in another before you drop it.

Full disclosure: we sell an AI coding agent. On MegaBrain you can bring your own API key or a local model on every plan, Free included, with no limits from us. Qwen, Kimi and GLM are included on the paid plans, and Claude, GPT and Gemini come from Pro. See the plans. MegaBrain is not affiliated with Alibaba, OpenAI, Anthropic or the paper’s authors.

Fun facts

To read audio attachments, agents downloaded models from Hugging Face on their own. GPT-6 Astra’s runs pulled in Whisper, CLAP and Qwen2-Audio-7B-Instruct.

Tasks with no attachment had the highest full-pass rate on the multimodal track, 37.0% pooled across all 10 models. Audio tasks scored lowest on strict completion, 58.4%.

Sources

Pages opened November 9, 2026. Every number above comes from the paper or the project’s own data file.

• arXiv 2610.35138 — Co-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models (Wei et al., submitted October 26, 2026; abstract)
• arXiv 2610.35138 — full paper, HTML (Tables 2, 4, 5, 6, 7 and 12; Appendices A, C, D and H)
• Co-Design Bench project page (headline results, dataset statistics, example runs)
• Co-Design Bench project data file (per-model, per-track scores, token counts and tool shares; generated October 19, 2026)
• The Decoder — AI coding agents still need human cleanup on nearly three out of four tasks in a real-world test