AI Hacking Just Hit 86.9%. In April, That Score Was “Too Dangerous to Release.”

The test that started at 1 in 5

CyberGym is a benchmark built at UC Berkeley and published as arXiv:2506.02548 in June 2025. The task is deliberately concrete: give an AI agent a vulnerability description and the actual pre-patch codebase, and see if it can produce a proof-of-concept that crashes the vulnerable build but not the patched one. No hints, no synthetic bugs. 1,507 real vulnerabilities pulled from 188 real open-source projects. At launch, the top-performing agent combinations reached roughly 20%. A useful, hard, narrow benchmark, in other words, not a headline number.

It also produced a side effect nobody was optimizing for. While the Berkeley team was building and validating the eval, the agents they ran against it surfaced 34 real zero-day vulnerabilities and 18 historically incomplete patches in the software under test, bugs nobody had previously found or fully fixed, in projects people actually run in production. The benchmark wasn't just measuring a capability. It was a live demonstration of it.

Watch the climb

Thirteen months after that 20% baseline, four separate model releases moved the CyberGym record almost in a straight line.

The April jump, Opus 4.6's 66.6% to Mythos Preview's 83.1%, is the one worth slowing down on. Anthropic's own Project Glasswing writeup describes Mythos chaining Linux kernel privilege escalations with zero human steering, building full working exploits from nothing but a CVE identifier and a git commit hash within hours, and autonomously turning up thousands of zero-day vulnerabilities across every major operating system and browser. A 16.5-point jump on a benchmark isn't usually a qualitative shift. This one was.

What Anthropic did with a model that good: not ship it

Anthropic's answer to that jump was to call it too dangerous for public release. Claude Mythos Preview went out to roughly 50 named organizations, AWS, Apple, Broadcom, Cisco, CrowdStrike, Google, JPMorgan Chase, the Linux Foundation, Microsoft, NVIDIA, and Palo Alto Networks among them, plus around 40 more, for defensive security work only. Anthropic backed the program with $100 million in usage credits and $4 million in donations to open-source security organizations.

That's a real gate: named partners, a public safety rationale, and capital specifically earmarked for the defensive side of the ledger. It set the bar for what “too dangerous to release” was supposed to mean in 2026.

By June, a second lab matched it, and gated it too

Two months later, OpenAI shipped a model that beat it. GPT-5.5-Cyber scored 85.6% on CyberGym, higher than the model Anthropic had just called too dangerous to release, and OpenAI gated its own version behind Daybreak, an invite-only cybersecurity access tier the company built after closed-door briefings with federal agencies. OpenAI paired the release with “Patch the Planet,” the same underlying capability tier pointed the other direction: auto-patching open-source vulnerabilities before anyone can weaponize them.

That pairing is the tell hiding in plain sight. If the offensive and defensive products run on the same underlying capability, the thing actually being gated was never the skill itself. It's who gets to point it, which makes the strength of the gate the only variable that actually matters.

3 weeks later: the same score, a much thinner lock

On July 21, 2026, a third lab, Sakana AI, posted a new CyberGym high score: 86.9%, alongside 72.1% on CTI-REALM, Microsoft's benchmark for AI-written threat-detection rules. The capability claim sits at the same tier as Anthropic's and OpenAI's. The access model does not.

Sakana's own release page confirms the access model: apply with an intended use case and verified contact information, wait for manual review, pay by usage. No open weights, no disclosed pool of underlying models. And the benchmark claim itself ships without the basics that let anyone outside Sakana check it:

It isn't even one model

The detail most coverage of the launch skipped: Fugu-Cyber isn't a single trained model. It's an orchestration system built on Sakana's ICLR 2026 research, TRINITY and Conductor, where a coordinator with roughly 600 million parameters decides in real time which model from an undisclosed pool handles each piece of an attack, then presents the whole thing as one API endpoint. The current record holder for autonomous vulnerability discovery is a router, not a monolith, and the pool it routes to is not public.

The one number we can actually check

Every figure in the CyberGym trajectory above is self-reported by the lab that produced it. There's exactly one place in this story where an independent evaluator measured something comparable: Microsoft's own CTI-REALM evaluation, which grades whether an agent can turn a raw threat-intelligence report into a working detection rule. Microsoft's own testing put the top three Claude configurations between 0.624 and 0.685. Sakana's self-reported Fugu-Cyber score for a comparable setup: 0.721, above the top of the band an independent evaluator actually measured.

That 0.036 gap above the top of Microsoft's own measured band isn't proof of anything dishonest on its own. Different setups, different runs, some genuine variance is expected. What it proves is narrower and more useful: on the single case where an outside party actually ran a comparable evaluation, the vendor-reported number came in hotter than the independently measured one. Extend that pattern to CyberGym, where no independent number exists at all, and the honest read is that nobody outside 4 companies can currently verify a single figure in the trajectory chart above.

The same crack, in a more dangerous category

This isn't the first time a self-reported number has outrun what anyone outside the lab that produced it could check. We covered OpenAI's retraction of SWE-Bench Pro after 27.4–34.1% of its 731 tasks turned out broken, and the money that had already moved to the routing layer as a result, in our July 21 piece on the coding-benchmark collapse. 2 days later we covered GPT-5.6 Sol topping the ITBench-AA infrastructure-agent leaderboard the same week METR measured the highest cheating rate it had ever detected on a publicly tested model, in our July 23 coverage. Both of those were coding and infrastructure benchmarks, where a wrong answer costs you a bad deploy. CyberGym and CTI-REALM measure autonomous exploit discovery, where an unverifiable capability claim isn't just embarrassing if it's wrong, it's the exact number 3 labs used to decide how carefully to gate a weapon. The pattern is identical each time: the number that gets screenshotted keeps climbing, and the number that would let you check it stays inside 1 company.

Scoring a benchmark claim before you trust it

The practical question this leaves isn't “is Sakana lying.” It's that a capability claim and a verifiable claim are two different things, and a launch post that leads with the score rarely volunteers which one you're getting. A minimal disclosure-completeness check, scored against the same 4 dimensions CyberGym coverage kept flagging as missing:

# benchmark_disclosure_audit.py — score a vendor's benchmark claim
# on how much of it you can actually verify, not just the headline number
DISCLOSURE_DIMENSIONS = [
    "trial_count",
    "run_dates",
    "agent_scaffold",
    "uncertainty_bounds",
    "third_party_replication",
]

# Pulled by hand from each lab's own launch materials, cited in this post.
CLAIMS = {
    "anthropic-mythos": {
        "score": 83.1,
        "trial_count": False,
        "run_dates": True,       # Apr 7, 2026 announcement date
        "agent_scaffold": True,  # described in Project Glasswing writeup
        "uncertainty_bounds": False,
        "third_party_replication": False,  # gated to ~50 partners, not independently public
    },
    "openai-gpt55cyber": {
        "score": 85.6,
        "trial_count": False,
        "run_dates": True,       # Jun 22, 2026 Daybreak launch
        "agent_scaffold": True,
        "uncertainty_bounds": False,
        "third_party_replication": False,
    },
    "sakana-fugu-cyber": {
        "score": 86.9,
        "trial_count": False,
        "run_dates": False,
        "agent_scaffold": False,  # orchestration pool undisclosed
        "uncertainty_bounds": False,
        "third_party_replication": False,
    },
}


def audit(name: str, claim: dict) -> dict:
    disclosed = [d for d in DISCLOSURE_DIMENSIONS if claim.get(d)]
    opacity_score = round(10 * (1 - len(disclosed) / len(DISCLOSURE_DIMENSIONS)), 1)
    return {
        "model": name,
        "headline_score": claim["score"],
        "disclosed": disclosed,
        "opacity_score": opacity_score,  # 0 = fully verifiable, 10 = fully opaque
    }


for name, claim in CLAIMS.items():
    result = audit(name, claim)
    print(
        f"{result['model']}: {result['headline_score']}% headline, "
        f"opacity {result['opacity_score']}/10, disclosed={result['disclosed'] or 'none'}"
    )

$ python3 benchmark_disclosure_audit.py
anthropic-mythos: 83.1% headline, opacity 6.0/10, disclosed=['run_dates', 'agent_scaffold']
openai-gpt55cyber: 85.6% headline, opacity 6.0/10, disclosed=['run_dates', 'agent_scaffold']
sakana-fugu-cyber: 86.9% headline, opacity 10.0/10, disclosed=['none']

Run against the 3 claims in this post, the highest headline score also carries the highest opacity score, fully undisclosed on every dimension checked. None of the 3 clear a “fully verifiable” bar, which is the point: even the best-documented claim here, Anthropic's, is still self-reported and gated to partners who signed something before they got to see it. A leaderboard number and a checkable number are not the same axis, and right now, in autonomous cyber-offense benchmarks, almost nothing sits on the second one.

If you want to build your own trust-aware monitor

The deeper problem isn't any single vendor's number. It's that this category moves fast enough that a claim like Fugu-Cyber's can go from launch post to widely cited “state of the art” within days, while the disclosure gaps that should discount it sit in a paragraph nobody screenshots. Checking a script like the one above against every new cyber-benchmark launch, by hand, on top of everything else a security or infra team already tracks, is exactly the kind of always-on job that shouldn't depend on someone remembering to re-read a launch page. A mhermes agent, running 24/7 on its own isolated VM, can watch the sources cited in this post on a schedule, re-run the disclosure audit against every new claim, and page you the moment a headline number ships with a 10/10 opacity score attached.

And once you've decided which of these gated models are actually worth routing to, MegaBrain gives you one API across 500+ models with transparent, at-cost pricing, so the moment access opens up on any of them, wiring it into your own pipeline is a one-line change, not a new integration.