Your AI Agent's Cache Hit Rate Is Probably 7%. Here's How to Check.

The bug that cost nothing to fix and nobody found for months

ProjectDiscovery runs an AI security agent called Neo: 26 steps, 40 tool calls, a system prompt over 20,000 tokens. By every reasonable assumption, that's exactly the kind of workload prompt caching was built for: a large, mostly-static prefix, reused across many steps of the same task. According to ProjectDiscovery's own writeup, Neo ran at a 7% cache hit rate in production while its dashboard reported caching as enabled. The bug wasn't exotic: mutable working memory sat inside the prefix that was supposed to be static, so every step invalidated the entire cached block behind it.

The fix was to move that mutable state out of the cached prefix. One deploy took the hit rate from under 8% to 74%; further refinement pushed it to 84% within weeks. The result, measured across the fleet: a 59% overall cost reduction, climbing to 66-70% in the most recent window at time of writing. One single task cached 91.8% of 67.5 million input tokens across 1,225 steps. Across all of ProjectDiscovery's traffic, 9.8 billion tokens have been served from cache.

Nobody caught this by watching the bill. They caught it by checking the number that actually tells you whether caching is working: cache_read_input_tokens against total input tokens. Most teams never check that number at all, because the dashboard for “is caching on” and the metric for “is caching working” are two different things, and only one of them is a toggle.

What the vendors actually promise, in the fine print

Anthropic, OpenAI, and Google all sell the same pitch: mark a prefix as cacheable, reuse it across calls, and the repeated portion gets billed at a steep discount on the reread. On Anthropic's pricing, a cache read costs 0.1x the base input rate, genuinely close to the “90% off” headline. But that's only the read. Writing to the cache costs 1.25x base rate for a 5-minute TTL, or 2x for a 1-hour TTL. Cache a prompt you only end up reusing once, and you paid 25-100% more than if you had never marked it cacheable at all.

That asymmetry means caching is a bet, not a free discount: it only pays off once a prefix is reused enough times for the cheap reads to outweigh the expensive write. For a one-shot prompt, or a prefix that changes every call, caching is a strictly worse deal than doing nothing.

Where it breaks: multi-provider routing

Most production agents don't call a single model directly, they go through a router, whether that's a custom load balancer or a gateway spreading load across providers for uptime and price. Caching depends on hitting the same backend twice in a row so the cached prefix is actually there to read. OpenRouter's own worked example makes the cost of getting this wrong concrete: an agent that repeats the same 10,000 tokens over 6 turns costs 6.0x a single uncached turn with no caching at all. Add caching plus sticky routing (pinning the agent to the same provider across the session), and total cost drops to 1.75x. Add caching but route randomly across providers on every turn, the default behavior in most multi-provider setups, and you only get to 2.25x, nearly a third more than the sticky-routing number, for the exact same caching code.

If your agent framework calls a gateway or a load balancer, check whether it supports provider affinity or sticky sessions per conversation. If it doesn't, you're probably paying the 2.25x number and calling it 1.75x in your head.

Where it breaks: silently, below the minimum

Anthropic's documentation states this plainly, but it's easy to miss: prompts shorter than the model's minimum cacheable length, 512 tokens for the smallest models up to 4,096 tokens for others, cannot be cached, even if you mark them with cache_control. There is no error. The request is processed normally, as if caching had never been requested, and the only trace is that cache_creation_input_tokens and cache_read_input_tokens stay at zero in the response.

This is exactly the kind of failure that OpenRouter's own GitHub issue tracker shows happening in the wild, repeatedly, across completely unrelated projects. Zed editor, agent-zero, hermes-agent, obsidian-copilot, the AI SDK provider, and oh-my-pi have all filed separate issues reporting native_tokens_cached stuck at zero for Anthropic models routed through OpenRouter. Six independent teams, hitting the same silent failure, none of them aware the other five existed.

The only way to know for certain is to check the usage fields on every response, not just when something looks slow:

# cache_audit.py — verify a Claude call actually hit the cache
import anthropic

client = anthropic.Anthropic()

def call_with_cache(system_prompt: str, user_message: str) -> dict:
    response = client.messages.create(
        model="claude-sonnet-5",
        max_tokens=1024,
        system=[
            {
                "type": "text",
                "text": system_prompt,
                "cache_control": {"type": "ephemeral"},
            }
        ],
        messages=[{"role": "user", "content": user_message}],
    )
    u = response.usage
    return {
        "cache_creation_input_tokens": u.cache_creation_input_tokens,
        "cache_read_input_tokens": u.cache_read_input_tokens,
        "input_tokens": u.input_tokens,
    }

result = call_with_cache(SYSTEM_PROMPT, "What changed in the last deploy?")
hit = result["cache_read_input_tokens"] > 0
wrote = result["cache_creation_input_tokens"] > 0
mark = "OK" if (hit or wrote) else "FIX"
print(f"cache_read={result['cache_read_input_tokens']} "
      f"cache_write={result['cache_creation_input_tokens']} -> [{mark}]")

$ python3 cache_audit.py
cache_read=0 cache_write=0 -> [FIX]
⚠ FIX: prompt likely below the model's minimum cacheable length, caching silently skipped

# after moving the mutable message-history block out of the cached system prompt:
$ python3 cache_audit.py
cache_read=18432 cache_write=0 -> [OK]

Run that check on every call for a day, not once, since the first call in any session always shows a cache write and zero reads by design. It's the second and later calls that tell you whether the cache is actually paying off.

Where it breaks: the agent just pauses

Agents don't call the model in a tight loop, they wait on tool calls, human review, rate limits, or scheduled steps. A July 2026 paper, “Keeping the Cache Warm Pays: Keepalive Economics for Agentic Workloads” by Maxim Khailo, measured what those pauses actually do to a cache. At 600 seconds (10 minutes) idle with no keepalive ping, Anthropic's cache came back cold in 0 of 48 trials across three runs. Add one lightweight keepalive ping during the pause, and it held warm in 40 of 40. But keepalive pings aren't free: they're themselves cache reads. Ping too aggressively and the pings cost more than the cache saves.

The paper found an optimal ping interval near 4 minutes for Anthropic's 5-minute TTL: holding a 100,000-token prefix warm costs about $3.60/hour at 30-second pings, versus about $0.45/hour at that optimal interval, an 8x difference for the identical technique, purely from a tuning parameter almost nobody sets deliberately. And the optimal interval isn't the same across providers, because each prices cache reads differently. The paper reports break-even horizons of roughly 46 minutes on Anthropic, 36 minutes on OpenAI and DeepSeek, and just 12 minutes on Google, whose higher cached-read pricing compresses the window where keeping a cache warm is worth the ping cost at all.

What real production data says, at scale

Individual bugs and pricing quirks are one thing. The bigger question is what caching actually delivers once you average across a large number of real agent sessions, not a vendor's cherry-picked demo. “Don't Break the Cache,” a January 2026 study by Lumer et al., ran prompt caching across 500+ real agent sessions and roughly 10,000 API calls on the DeepResearch Bench, spanning OpenAI, Anthropic, and Google. The measured result: 41-80% cost reduction and 13-31% faster time-to-first-token, a real, substantial win, just not the “up to 90%” figure every vendor leads with. The paper's sharper warning is the one that doesn't make it into marketing copy at all: “naive full-context caching can paradoxically increase latency.” The lever that's supposed to be a strict improvement isn't always one.

The framework tax

Even a perfectly-priced, perfectly-routed cache does nothing if the calling code never gives it a stable prefix to reuse. TradingAgents, an open-source multi-agent trading framework on GitHub, makes 16-22 LLM calls per run. According to an issue filed against the project in May 2026, every one of those prompts is rebuilt from scratch with f-strings and ad hoc message-history construction on each call, which yields “approximately 0% LLM API prompt cache hit rate” and an estimated 30-40% unnecessary cost overhead, on a framework whose users likely have no idea caching isn't firing at all.

This is the pattern to check for in any framework or template you didn't write yourself: does the system prompt get assembled identically, byte for byte, across calls in the same conversation, or does it get rebuilt with fresh timestamps, reordered tool lists, or interpolated state on every turn? Any of those breaks the prefix match caching depends on, silently, with the same “no error” behavior as the token-minimum case above.

The goalposts moved again in July

Even the economics of trying keep shifting. Per OpenAI's current API docs, cache writes had no additional fee on every model before the GPT-5.6 family. On GPT-5.6 and every model since, a cache write now costs 1.25x the uncached input token rate. Before this change, caching a prompt you might only reuse once cost nothing extra to try. On the current flagship model, trying it and only getting one read means you paid 25% more than if you had never tried at all.

What the data actually says

If you want to stop guessing whether your cache is working

The audit script above works for a single call. The harder problem is running it continuously, across every session, catching the moment a cache hit rate quietly drops from 84% back toward 7% because someone shipped a change that put mutable state back inside the prefix. A mhermesagent, running on its own isolated VM with shell and scheduling access, can run that check against your production traffic on a fixed interval and page you the moment cache_read_input_tokens starts trending toward zero, instead of discovering it in next month's bill the way ProjectDiscovery almost did.

And once you're actually watching the number, MegaBrain gives you one API across 500+ models with transparent, at-cost pricing and automatic routing, so you can route agent traffic with real provider affinity instead of the random, cache-breaking routing that turns a 1.75x cost multiplier into a 2.25x one by default.