Swap the Model: 15-Point Swing. Swap Just the Search Engine: 10. Almost Nobody Tests the Second One.

The benchmark that tests the setup, not just the model

Every model leaderboard, Intelligence Index, Arena, LMSYS, answers one question: which model is smartest. OpenRouter's Live Web Search Benchmarks asks a different one: once you've wired a model into a search agent, which of the decisions around it actually move the score? The team ran 35 configurations, 4 model providers (Claude Opus 5, GPT-5.6 Sol, GPT-5.6 Luna, and DeepSeek V4 Flash) crossed with 3 search engines (Perplexity, Exa, and Parallel) and 5 task suites (BrowseComp, DeepSearchQA, WideSearch, HLE, and a turn-budget sweep), and logged accuracy, cost, and latency for each combination.

The framing matters. Nobody ships a model in isolation, they ship a model plus a search engine plus a turn budget, and a leaderboard only ever tells you about the first variable. This dataset is the first public attempt to price out the other two.

Turn budget: the knob that usually pays for itself

Start with the most obvious lever: how many searches the agent is allowed to take. OpenRouter ran Claude Opus 5 on BrowseComp through Perplexity at three budgets. At 1 turn, it scored 35.8% for $0.14 a question. At 5 turns, 66.5% for $0.51. At 25 turns, 89.0% for $0.99.

Going from 1 turn to 25 roughly doubles the score for 2.5 to 7 times the cost per question, a good trade by almost any unit-economics standard, and one OpenRouter's own writeup calls out as the highest-leverage single change most teams haven't made. If your search agent defaults to a low turn cap for latency reasons, that default is probably costing you more in wrong answers than it saves in API spend.

It doesn't hold everywhere, though. On the HLE task suite, GPT-5.6 Sol running through Perplexity scored about the same at 1 turn as it did at 25, for triple the cost. OpenRouter's data shows over a third of the 35 tested configurations were actually slower with fewer turns, not faster, an artifact of retry and backoff behavior under a tight budget. Turn budget is a real lever, but it's task-dependent, and testing it against your own workload matters more than assuming the BrowseComp curve transfers.

Model choice, held everything else constant

Fix the setup at 25 turns through Perplexity and vary only the model, and the spread is real but smaller than you might expect. Claude Opus 5 leads at 89.0%. GPT-5.6 Sol comes in at 82.4%. DeepSeek V4 Flash scores 77.0%, at 8 cents a question, roughly a twelfth of Opus 5's cost for 12 fewer points. GPT-5.6 Luna trails at 74.0%.

A 15-point spread between the best model and the cheapest-efficient one is a genuine finding, and it's the one every team already optimizes for. What the dataset shows next is that a second, mostly-ignored lever moves the score by nearly as much.

The knob nobody tunes: the search engine itself

Freeze the model at Claude Opus 5 and the turn budget at 25, and vary only which search engine answers the agent's queries. Perplexity scores 89.0% for 99 cents. Parallel scores 88.8%, functionally identical accuracy, for $2.42, nearly 2.5 times the price. Exa scores 82.2% for $1.29, worse than Perplexity and still pricier.

Across the entire 35-configuration dataset, OpenRouter reports that varying the engine while holding the model constant changed scores by an average of 10 points, versus a 15-point average gap between frontier and cost-efficient models. Engine choice is not a rounding error next to model choice, it's in the same weight class, and it moves your bill harder: the priciest engine in the comparison cost 2.5x the cheapest, versus 1.5x for the cheapest-efficient models.

The practical read: most teams run a bake-off between 2 or 3 frontier models before shipping a search agent, and then hard-code whatever search plugin their framework defaults to, without ever running the same bake-off on the engine. This data says that second bake-off is worth almost as much, and the engine you didn't test might be quietly costing you both accuracy and money at the same time.

Wrong answers always cost more searches, every suite, no exceptions

One more pattern holds across every task suite in the dataset with paired correct/wrong data. Agents that landed on the right answer consistently used fewer searches, on average, than agents that got it wrong.

The most extreme case in the dataset: the deepest single attempt OpenRouter recorded was 81 searches on one WideSearch table, and it was still graded incorrect. Extra search budget doesn't reliably rescue a wrong answer, it mostly just makes the wrong answer more expensive to reach. That's a useful early-stopping signal in its own right: an agent that's burned 2x its typical “correct” search count on a task is statistically more likely heading toward a wrong answer than converging on a right one, and cutting it off early might beat letting it run to the cap.

What this means: you're tuning 1 knob out of 3

Put the three findings together and a web-search agent's real accuracy is a function of at least three variables that each move the score by double digits: which model, which search engine, and how many turns you allow it. Every public leaderboard only shows you the first one, so it's the only one most teams ever formally benchmark before shipping. The other two are sitting untested in whatever config file your framework generated by default.

How to run your own 3-variable test

You don't need OpenRouter's full 35-configuration matrix to get a useful signal, a small grid over the variables you actually control is enough to catch a bad default. If you're calling models through OpenRouter, web search is available on any model by appending :online to the model slug, or explicitly via a plugins array, and search depth is controllable per request:

# sweep search depth against a fixed model, log accuracy on a labeled eval set
for max_results in 3 8 15 25; do
  curl -s https://openrouter.ai/api/v1/chat/completions \
    -H "Authorization: Bearer $OPENROUTER_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "model": "anthropic/claude-opus-5:online",
      "plugins": [{ "id": "web", "max_results": '"$max_results"' }],
      "messages": [{ "role": "user", "content": "'"$QUERY"'" }]
    }' | jq -r '.choices[0].message.content' \
    >> "results/depth_${max_results}.txt"
done
# then score each results/ file against your labeled answer key,
# the same way OpenRouter scored BrowseComp and WideSearch

Search-engine selection is provider-specific rather than a single universal request field, check your framework or gateway's docs for how it exposes engine choice, but the shape of the test is identical: hold the model and turn budget fixed, swap only the engine, and re-score against the same labeled set. If the accuracy delta between your 2 cheapest engine options is anywhere close to the 7-10 points OpenRouter measured, you have a live decision to make, not a rounding error to ignore.

The other cheap check is a stopping rule. Log the search count on every task your agent gets marked wrong on in eval, and compare it to the median search count on tasks it gets right. If wrong answers are consistently burning more searches than right ones, the way they did in every suite OpenRouter tested, an early-stop threshold around your “correct” median is worth testing before you widen the turn budget further.

What this means if you're running a search agent in production

The instinct to spend engineering time on model selection is reasonable, it's the variable every benchmark makes visible. But this dataset says that instinct is now covering roughly half the real picture. Engine choice moves accuracy almost as much as model choice and moves cost more. Turn budget can double your accuracy for a fraction of the cost, except on the tasks where it does nothing at all. And the cheapest diagnostic signal you already have, how many searches a task burned, tells you more about whether an answer is trustworthy than the model name on the invoice does. Benchmark the model if you want, that part is table stakes now. But the 3-variable test is the one that's actually still uncovered.

If you want a single gateway that makes it easy to run that comparison, without juggling separate API keys and separate plugin syntax per provider, MegaBrain routes to 500+ models through one API with transparent, at-cost pricing, so swapping the model side of the grid is a one-line change instead of a new integration. And for an agent that needs to keep re-running that comparison on a schedule, not just once before launch, a mhermes instance stays on 24/7 on its own isolated VM to re-test your production configuration and flag it the moment a cheaper engine starts beating your current default, or a provider you depend on quietly gets worse.