$401 Billion in AI Infrastructure Spend. The Average GPU Runs at 5%.

The number nobody wants attached to their name

Cast AI's 2026 State of Kubernetes Optimization Report didn't survey anyone. It measured actual production telemetry from roughly 23,000 Kubernetes clusters running on AWS, Azure, and Google Cloud, the same three clouds that host most of the industry's AI workloads. The finding: average GPU utilization across those clusters is 5%. CPU utilization is 8%, down from about 10% a year earlier. Memory utilization is 20%, down from 23%. Every number is moving in the wrong direction, in the same year AI infrastructure spend is accelerating.

Cast AI co-founder and president Laurent Gil put the GPU number in blunt terms: a GPU sitting idle costs dollars an hour, a CPU sitting idle costs cents. Utilization dropping on the cheapest resource is a rounding error. Utilization dropping on the most expensive one, while spend on that same resource accelerates, is a structural problem.

It's also worth noticing what didn't happen. Kubernetes has spent a decade selling itself on exactly this kind of efficiency gain, bin-packing workloads onto shared infrastructure so utilization climbs as fleets scale. The report's CPU and memory trend lines say the opposite is happening for AI workloads specifically: both metrics fell year over year, at the same time overall cluster count and spend both grew. Scale alone isn't fixing this. If anything, it's masking it, because a bigger denominator makes a bad utilization percentage easier to ignore in a board deck.

Why a GPU-hour is not a CPU-hour

The reason this specific number matters more than the CPU and memory figures next to it is straightforward: cost concentration. A single idle H100 or B200 can run several dollars an hour in lease or amortized capex terms. An idle CPU core costs a fraction of a cent. When 95% of your GPU fleet is doing nothing at any given moment, you're not wasting a rounding error in the infrastructure budget, you're wasting the single largest line item in it, most of the time.

Put it in annual terms and the number stops being an abstraction. An 8-GPU node leased at roughly $2.49 per GPU-hour costs about $174,500 a year fully loaded. At 5% utilization, you're paying that full amount to receive about $8,725 worth of actual completed work, and writing off the other $165,775 as the cost of having capacity on hand. Multiply that by however many nodes sit in your fleet, and the 5% figure moves from a research-report statistic to a specific number your own finance team could put on a slide.

Scarcity that isn't scarce

If GPUs are this underused, lease prices should be falling. They're not. Reported H100 one-year lease pricing climbed roughly 40% over the 5 months entering 2026, during the exact window utilization kept dropping. Gil's explanation is that enterprises aren't renting the compute they need, they're hoarding compute they're afraid to be without, essentially manufacturing scarcity the way speculative buyers do in a housing market. He calls it “neo-real estate” rather than cloud computing.

FOMO-driven demand for capacity nobody is actually using is still bidding up the price of capacity everyone wants. That's the opposite of how a rational market with 95% idle supply should behave, and it's a strong signal that the current AI infrastructure market is pricing on fear of missing capacity, not on measured need.

The sellers don't feel any of this

None of the waste above shows up as a problem for the hyperscalers renting out that compute, because they get paid on committed or consumed capacity regardless of whether the buyer's workload actually saturates it. Microsoft's Azure AI business is now at a $37 billion annualized revenue run rate, with Azure overall revenue up 40% year over year. AWS reported $37.6 billion in quarterly revenue, up 28% YoY. Google Cloud reported $20 billion, up 63% YoY, the fastest of the three.

The waste is invisible to the people selling the shovels. It only becomes visible on the buyer's own invoice, and only if someone is actually looking at utilization instead of just at whether the workload finished on time.

Where the 5% actually comes from

The Cast AI number isn't a mystery once you look at enterprise AI adoption maturity data from the same period. 62% of organizations are still experimenting with or piloting AI. 23% are scaling it in a single function. Fewer than 10% have scaled it enterprise-wide. Only 14% consider their own data architecture ready for AI at scale. Most organizations aren't running steady, continuous AI workloads yet, they're running one-off pilots on infrastructure sized as if it were already production. That mismatch, pilot-scale usage on production-scale hardware, is the entire 5%, explained in four numbers.

The market is already repricing it

The clearest sign this is a recognized structural issue, not a rounding error someone will quietly fix, showed up in financing, not in a cloud provider's earnings call. On July 17, 2026, General Compute and Upper90 closed what's reported as the first loan of its kind: $400 million, collateralized not by training GPUs but by inference chips, the hardware that runs models that are already built and serving real, continuous traffic. Lenders are quietly pricing steady, always-on serving as safer collateral than bursty, on-demand training or pilot capacity. That's a financial market saying, in the most literal way possible, that idle-prone compute is now measurably riskier to lend against than compute that's actually kept busy.

Checking where your own cluster actually stands

You don't need Cast AI's dataset to find out if you're part of the 95%. `nvidia-smi` already exposes per-GPU utilization, and a few lines turn that into a real idle-cost estimate against whatever you're actually paying per GPU-hour:

# gpu_utilization_audit.py — real-time idle-cost check against nvidia-smi
import subprocess

HOURLY_RATE_USD = 2.49   # what you're paying per GPU-hour (on-demand H100, mid-2026)
UTILIZATION_FLOOR = 30   # below this, treat the GPU as idle capacity

def sample_utilization() -> list[int]:
    out = subprocess.run(
        ["nvidia-smi", "--query-gpu=utilization.gpu", "--format=csv,noheader,nounits"],
        capture_output=True, text=True, check=True,
    )
    return [int(line) for line in out.stdout.strip().splitlines()]

def audit(samples: list[int]) -> dict:
    idle = [u for u in samples if u < UTILIZATION_FLOOR]
    return {
        "gpus": len(samples),
        "idle_gpus": len(idle),
        "avg_utilization": round(sum(samples) / len(samples), 1),
        "wasted_usd_per_hour": round(len(idle) * HOURLY_RATE_USD, 2),
    }

samples = sample_utilization()
result = audit(samples)
mark = "OK" if result["idle_gpus"] == 0 else "FIX"
print(f"{result['gpus']} GPUs sampled, avg utilization {result['avg_utilization']}%, "
      f"{result['idle_gpus']} idle -> [{mark}]")
print(f"Wasted spend: ${result['wasted_usd_per_hour']}/hour if idle GPUs stay idle")

$ python3 gpu_utilization_audit.py
8 GPUs sampled, avg utilization 4.8%, 7 idle -> [FIX]
⚠ FIX: 7 of 8 GPUs below 30% utilization floor
⚠ FIX: wasted spend: $17.43/hour if idle GPUs stay idle

Run that against a typical pilot-stage cluster and you land close to Cast AI's industry average, not because the number is rigged, but because most GPU fleets today are provisioned for a production workload that hasn't arrived yet. The fix isn't a bigger GPU. It's someone, or something, actually watching this number continuously and scaling capacity to match real usage instead of anticipated usage.

What the data actually says

Sources for every figure above: Cast AI's 2026 State of Kubernetes Optimization Report; Gartner's worldwide AI spending forecast; VentureBeat's coverage of Cast AI's utilization data and Laurent Gil's commentary; Data Center Knowledge's reporting on hyperscaler earnings and AI adoption maturity; and TechCrunch's July 17, 2026 coverage of the General Compute / Upper90 financing.

If you want to stop being part of the 5%

The practical lesson isn't “buy fewer GPUs.” It's that nobody is watching utilization continuously, because it's a boring, always-on job that a human checks once a quarter if at all. A mhermesagent, running on its own isolated VM with shell access and scheduling, can run the audit script above against your cluster on a fixed interval, track the trend instead of a single snapshot, and page you the moment idle GPU-hours cross a dollar threshold you actually care about, instead of discovering it in next quarter's cloud bill.

And once you know which workloads are actually running continuously versus which ones are bursty pilots that don't need dedicated frontier-scale hardware, MegaBrain gives you one API across 500+ models with transparent, at-cost pricing and automatic routing, so a pilot workload can run on a right-sized model instead of quietly reserving GPU capacity it was never going to saturate.