The #1 AI Model for Phones Scored 63. Give It One More Minute to Think, and It Scores 18.

What actually got tested

Every mobile AI benchmark you've seen before this one measured intelligence in a simulator or measured speed in a simulator, rarely both, and almost never on an actual phone. Artificial Analysis built this one in partnership with Liquid AI, whose Pipette inference tool (open-sourced on GitHub) handled the on-device side: 41 quantized model builds, from 230 million to 9 billion parameters, each collapsed to 4-bit precision or smaller and served through llama.cpp on a real iPhone 17 Pro, with an 8GB memory ceiling that includes the KV cache at an 8K context. Intelligence is a simple average across 5 evaluations chosen to look like real phone usage rather than academic trivia: BFCL (tool-calling, a 640-task subset), IFBench (instruction-following under precise output constraints), AA-Omniscience (knowledge accuracy and hallucination resistance), GPQA Diamond (graduate-level science reasoning), and MATH-500 (competition math with symbolic answer checking).

That combination, real intelligence evaluations and real on-device inference on the same leaderboard, is what makes the next part possible: you can watch a model's score change not because it got smarter or dumber, but because you changed how long you let it run.

The leaderboard everyone will screenshot

At the study's default setting, a 16,000-token context limit, here's the top of the table, unedited:

Read that table as a developer, not a spectator, and the interesting line isn't the tie at the top. It's that LFM2.5-2.6B matches the field's best score at 2.6 billion parameters, running 2.7x faster and using 1.7x less memory than the 9B models it beat. Nanbeige4.2-3B matches it too, at a third of the parameter count of Ornith-1.0-9B, for one point more. Bigger didn't buy those two 9B models anything on this test, and it cost them real seconds and real gigabytes.

Six models, zero of them dominated

Plot every model's score against its generation time and a small set survives as a genuine frontier, meaning no other model beats it on both axes at once. On this test, 6 models make that cut:

Everything else tested, all 35 remaining builds, is dominated: some model on this list beats it on both intelligence and speed at once. That's the honest shortlist for anyone actually shipping a model to a phone, and it's a very different list from “whichever model has the highest single score,” which is the number most coverage of a benchmark like this actually leads with.

Change one dial, and the podium changes

The study didn't stop at 16K. It reran the same models at a 64K-token context limit, and the ranking that looked settled a paragraph ago moves:

Ling 3.0 Tiny wasn't in the 16K top four at all, and it's #1 at 64K. Nothing about any of these models changed between the two rows. What changed is how much room they were given to keep writing before the study cut them off, and models that reason in longer chains benefit from that room more than models that answer tersely and stop.

The real turn: what happens at a budget a human would actually accept

Context length is an abstraction. Nobody waits on tokens; they wait on a clock. So the study also capped every model at a flat 1-minute time budget on-device, the kind of patience an actual person has for their phone to answer a question, and this is where the leaderboard stops rearranging politely and starts falling apart:

LFM2.5-8B-A1B takes the crown here specifically because it loses the least, 11 points, not because it gains the most. Every model on this leaderboard gets worse under a time cap; the model that wins is the one that degrades slowest, which is a completely different property than the one the 16K leaderboard was measuring.

The mechanism: a verbosity tax, paid in tokens you never see

The study has a name for why this happens: Context Budget Overruns, the share of a model's answers that get cut off at the 16K ceiling before the model was actually done reasoning. A cut-off answer is graded as-is, wrong parts and all, so a model that overruns its budget gets penalized twice: once for the interruption, and once because an incomplete chain of reasoning is usually a wrong one.

Qwen3.5 9B (Reasoning) is the worst offender in the field, hitting the 16K ceiling on 29% of its generations. Across one full pass of the benchmark set, that single model consumed 74.5 million tokens, against 5.2 million for Gemma 4 E4B answering the same questions, 14x more tokens for a lower final score. That gap is the entire story of why the leaderboard flips under a time budget: a model that spends 14x the tokens per question also spends 14x the seconds, and a phone battery and a phone user both run out of patience at the same fixed rate regardless of how the model got there.

The math your users actually feel

The study states the concrete version of this directly: at roughly 55 output tokens per second on an iPhone 17 Pro, generating a full 64K-token answer takes on the order of 20 minutes, plus a real battery cost, which is exactly why 16K is the study's default ceiling rather than 64K. You can reproduce the shape of that math for any model, any device, and any budget with a five-line script:

# audit_context_budget.py — does this model finish inside your latency budget?
import argparse

p = argparse.ArgumentParser()
p.add_argument("--tokens-per-sec", type=float, required=True)
p.add_argument("--context", type=int, required=True)
p.add_argument("--budget-s", type=float, required=True)
args = p.parse_args()

seconds_needed = args.context / args.tokens_per_sec
tokens_that_fit = int(args.tokens_per_sec * args.budget_s)

print(f"target latency:      {args.budget_s:.0f}s")
print(f"tokens/sec (device): {args.tokens_per_sec:.0f}")
print(f"tokens that fit:     {tokens_that_fit:,}")
print(f"requested context:   {args.context:,}")
if seconds_needed <= args.budget_s:
    print(f"verdict: OK — finishes in {seconds_needed:.1f}s")
else:
    over = seconds_needed / args.budget_s
    print(f"verdict: OVERRUN — needs {seconds_needed / 60:.1f} min, {over:.1f}x over budget")

$ python3 audit_context_budget.py --tokens-per-sec 55 --context 65536 --budget-s 60
target latency:      60s
tokens/sec (device): 55
tokens that fit:     3,300
requested context:   65,536
verdict: OVERRUN — needs 19.9 min, 19.9x over budget

That's the pure decode-time estimate; real wall-clock is worse once you add prompt-processing latency, which is why the study's own “more than twenty minutes” framing is the conservative read, not an exaggeration. Run the same script against your own device's measured tokens/sec and your actual product's acceptable wait time, and you get a hard number for whether a model that looks great on a 16K leaderboard will actually finish inside your app.

What this means if you're picking a model by its badge, not its constraint

Every one of these three leaderboards, 16K context, 64K context, 1-minute wall clock, is a legitimate way to score the same 41 models, and every one of them crowns a different winner using data collected in the same study, on the same phone, the same week. None of them is lying. Each one is answering a question the reader didn't know had been asked: how much room did you give it to think before you decided who won? A model badge that says “63” or “#1 on the leaderboard” is silently reporting the answer to that hidden question along with the score, and most coverage of a benchmark like this one drops the constraint and keeps only the number.

The fix isn't picking a different single number. It's asking for the curve: how does this model's score move as you tighten the time budget toward the one your actual users will sit through? A model that holds its score under a 1-minute cap is telling you something a model that only holds its score at 64K tokens of unlimited patience never had to prove.

Ship models against the budget you actually have

The same principle holds off the phone, in every agent that runs against a real cost or latency ceiling instead of an unlimited research benchmark: the model with the best score on a leaderboard and the model that finishes your actual job inside your actual budget are frequently not the same model, and the only way to tell them apart is to measure against your constraint, not the leaderboard's.

If you're building with MegaBrain, every model call routes through one API at zero markup with full cost and latency visibility per task, so you can see which model actually finishes inside the budget you have, not just which one tops a chart built around someone else's. And if you're running agents continuously rather than one request at a time, mhermes, MegaBrain's always-on agent runtime, gives each one its own isolated VM with persistent storage and scheduling, so a model that's slow to finish costs you compute minutes, not a blown UX budget nobody was watching.

Sign up at getmegabrain.com to pick models against the constraint you actually ship under, not the one a leaderboard picked for you.