One Provider Sells GLM-5.2 at 51.9% Accuracy. Another Sells It at 100.3%, for the Same Price.

The benchmark nobody had built until 4 days ago

Every model leaderboard answers the same question: which model is smartest. Artificial Analysis's Endpoint Accuracy Index answers a different one: once you've picked a model, which server actually delivers it. Open-weight models like GLM-5.2, gpt-oss-120b, and DeepSeek V4 Pro run on dozens of competing serverless endpoints, each free to quantize the weights, tune sampling defaults, cap output tokens, or write custom serving kernels, all while advertising the identical model name.

The methodology is deliberately narrow and reproducible. Artificial Analysis self-hosts each model's official weights at the lab's own recommended precision, following the lab's published serving recipe, and calls that run the 100% reference. It then re-runs three evaluations against every commercial endpoint and against the reference: BFCL v4-500 (500 tool-calling questions, 3 repeats), HLE-250 (250 scientific-reasoning questions, 10 repeats), and AA-LCR-25 (25 long-context recall questions, 10 repeats). Each endpoint's score is the average of the three, expressed as a percentage of the reference, with a 95% confidence interval. An endpoint counts as “at parity” only if its result falls inside that interval.

Coverage launched with three models: GLM-5.2, gpt-oss-120b, and DeepSeek V4 Pro, with Kimi K3 promised soon. The full per-provider tables are public on Artificial Analysis's site, and everything below is pulled directly from them, dated August 4, 2026.

GLM-5.2: the same weights, 13 different grades

Sort GLM-5.2's 13 tested providers by score and the bottom of the list is not a rounding error. Blackbox AI comes in at 51.9%, well over 20 points below the next-lowest endpoint and roughly half of what FriendliAI, Fireworks, and Nebius deliver at 100.2%, 100.3%, and 100.3% respectively.

Look at the two ends of that table together and the “you get what you pay for” story falls apart. Blackbox AI is not the cheap, corner-cutting option; at $0.778 per million tokens it sits in the middle of the price range, more expensive than CoreWeave's 90.5%-accuracy endpoint at $0.492. Fireworks, at 100.3% accuracy, costs $0.818, just 5 cents more than Blackbox's 51.9% version. A buyer comparing only price and the model name on the label would see two nearly identical line items and have no way to know one of them delivers roughly half the tested accuracy of the other.

gpt-oss-120b: the priciest endpoint is also the worst deal

gpt-oss-120b has the deepest bench, 17 tested providers, and the pattern repeats with an extra twist. Cloudflare's endpoint scores 69.8%, the lowest of all 17, while charging $0.39 per million tokens, tied with Cerebras for the single most expensive option on the entire list. CoreWeave scores 97.6%, almost 28 points higher, for $0.044, nearly 9 times cheaper and more accurate at the same time.

Cloudflare's own engineering blog explains why: gpt-oss-120b runs on Workers AI “natively at FP4 quantization”, a lower-precision serving choice made for memory footprint and speed, not disclosed as an accuracy tradeoff on the pricing page. The Endpoint Accuracy Index is the first independent measurement of what that tradeoff actually costs in tested accuracy, and it costs more than 30 points.

Where the accuracy actually goes missing

The obvious assumption is that a lower-scoring endpoint is just running a “dumber” model across the board. The data says otherwise. Test the same 17 gpt-oss-120b endpoints on GPQA, a pure scientific-reasoning benchmark, and the spread shrinks to almost nothing: 70.7% to 79%, an 8-point band. Nearly every provider preserves the model's raw reasoning ability. That 8-point band cannot explain the 31-point gap the composite Endpoint Accuracy Index measures between Cloudflare and Amazon Bedrock.

The real damage is concentrated in the two sub-scores that matter most for an autonomous agent: tool calling and long-context recall. Artificial Analysis's own reference deployment of gpt-oss-120b scores 37% on BFCL-500, a test of whether the model can correctly format and issue a tool call. The worst commercial endpoint scores 22%, a 40% relative collapse in the exact skill that separates an agent that takes an action from one that just talks. Artificial Analysis attributes it directly to serving choices: providers parse and format tool calls differently, and several endpoints truncate reasoning tokens or long context before the model finishes the job, at identical requested settings.

The control group: this isn't inevitable

DeepSeek V4 Pro is the cleanest evidence that endpoint-level accuracy loss is a business decision, not an unavoidable cost of decentralized serving. Its 9 tested endpoints cluster from 97.2% to 107% of reference, with several landing inside the confidence interval and a couple edging slightly above it, effectively all at parity.

Same three evaluations, same methodology, a completely different outcome from GLM-5.2 and gpt-oss-120b. Some model families ship in a way that's harder to quantize down without the loss showing up immediately; some providers simply choose to protect accuracy over shaving latency or cost. Either way, the spread on GLM-5.2 and gpt-oss-120b is not the price of running an open-weight model at scale. It's the price of picking the wrong endpoint.

A second lab found the same shape, from a different angle

This isn't one benchmark's opinion. Days before Artificial Analysis published the Endpoint Accuracy Index, OpenRouter published its own provider-performance methodology, measuring the identical underlying problem from a latency angle instead of an accuracy one. Their worked example: Claude Sonnet 4.5's typical response (p50) lands in 6 to 8 seconds. Its worst-case, 99th-percentile response, on a different provider serving the same model, takes 77 to 92 seconds. OpenRouter's writeup names the same root causes Artificial Analysis found: quantization level, serving infrastructure, and load handling, all varying silently under one model name.

This is a different failure mode than the price spread you've already heard about

Multi-provider price and uptime spread on open-weight models is not new. OpenRouter has published routing tables showing DeepSeek V4 Pro's input price varying 4x and its throughput varying 14x across 16 providers on identical weights, and that spread is easy to reason about: a slower or pricier endpoint is a worse deal, but the answers it gives you are still correct. The Endpoint Accuracy Index measures something a price or throughput column can't show you at all: whether the answers themselves are still the answers the lab actually shipped. A DeepSeek V4 Pro endpoint that costs 4x more but stays at 100% accuracy is a bad deal you can see on the invoice. A GLM-5.2 endpoint that costs about the same as its best alternative but silently returns 51.9% of the reference model's tested accuracy is a bad deal you can only see by running an independent benchmark against it, which is precisely why nobody had caught it until 4 days ago.

That distinction matters most for anything agentic. A slow, expensive endpoint degrades your unit economics. A silently-quantized endpoint degrades your agent's judgment, specifically its ability to call the right tool with the right arguments, exactly the failure mode BFCL-500 is built to catch and exactly where this data shows the damage concentrating. An agent that fails a tool call doesn't throw a pricing alert. It just does the wrong thing, and the postmortem rarely thinks to ask which server answered that particular request.

How to check where your own traffic actually lands

The Endpoint Accuracy Index tables are public and worth bookmarking directly for GLM-5.2, gpt-oss-120b, and DeepSeek V4 Pro. But most teams aren't reading them, they're routing traffic through a gateway that picks a provider automatically. The fastest proxy check, without waiting on Artificial Analysis to add coverage for your model, is inspecting the quantization and context-window fields OpenRouter already publishes per endpoint:

curl -s "https://openrouter.ai/api/v1/models/openai/gpt-oss-120b/endpoints" \
  | jq '.data.endpoints[] | {
      provider: .provider_name,
      quantization: .quantization,
      context_length: .context_length,
      max_completion_tokens: .max_completion_tokens
    }'

A provider reporting fp4 or int4 quantization, a truncated context_length, or a capped max_completion_tokens well below the model's published limit is showing you the exact mechanism this article measured: the same knobs Artificial Analysis traced the GLM-5.2 and gpt-oss-120b gaps back to. It won't give you a precise accuracy percentage, but it will tell you which endpoints are worth cross-checking against Artificial Analysis's tables before you commit production traffic to them.

What this means if you're running agents on open-weight models

The Endpoint Accuracy Index is 2 days old as of this article and covers 3 models. Most of the open-weight inference running in production today has never been checked this way at all. If you picked a provider on price or brand recognition sometime in the last year, there's a real chance you've been running a materially worse model this whole time and calling it by its full name, especially if your workload leans on tool calling or long context, the two skills this data shows breaking first. The model card tells you what the lab built. It never told you what showed up on the other end of your API call, and until 4 days ago, nobody was checking.

If you want to stop guessing which endpoint you're actually running on, MegaBrain routes every request through a single gateway with transparent, at-cost pricing and provider-level visibility, so you can see and pin the endpoint behind a model instead of leaving it to whichever backend a black-box router picked this time. And for agents that need to keep checking, a mhermesinstance, running 24/7 on its own isolated VM, can poll a model's Endpoint Accuracy Index or OpenRouter's endpoint metadata on a schedule and alert you the moment a provider you depend on quietly drops precision.