DEV Community: Ingero Team

When GPUs Are Scarce, Each Stall Costs N Times More

Ingero Team — Mon, 22 Jun 2026 13:00:00 +0000

The per-percent dollar cost of a GPU stall multiplies with capex scale. The eBPF instrumentation cost does not.

TL;DR

Big Tech Q1 2026 AI capex came in at roughly $630B aggregate. Azure is supply-constrained. Stargate, Roze AI, Verda, Nscale, and OpenLight added tens of billions to AI-data-center construction in the same quarter. When GPU supply is constrained, the dollar cost of every percent of throughput lost to kernel-level inefficiency rises in proportion. The GPU stall cost at fleet scale crosses single-digit-million-per-year below 1% loss on a billion-dollar fleet. Kernel-level observability is the cheapest piece of the stack and the one that prevents the loss.

The capex line items

April closed with the largest concentration of AI-infra capex announcements in industry history:

Anthropic talking $50B raise at ~$900B valuation (decision in May).
SoftBank creating Roze AI as a $100B IPO target, robotics + AI-data-center construction.
Stargate Michigan $16B financing (Apr 27), Nscale $2B Series C at $14.6B (Apr 27), OpenLight $50M Series A-1 (Apr 29), Verda €100M for Nordic clean-power AI cloud (Apr 24), Parallel Web Systems $100M at $2B (Apr 30).
Big Tech Q1 earnings preview pegs aggregate capex at ~$630B, with Microsoft explicitly citing supply-constrained Azure capacity.

The cumulative move is not just that more GPU compute is being built. It is that the existing GPU compute is being used closer to its ceiling. When capacity is the gate, every percent of throughput lost to host-side stalls, co-scheduling contention, or NCCL imbalance is throughput that has to be rebuilt with new physical hardware.

The per-percent dollar

Take a representative number: a single H100 instance on a hyperscaler lists at roughly $4-$6 per hour. At ~$5/hour and 24/7 utilization a single H100 costs about $44K per year. A fleet of 100 H100s is $4.4M/year of GPU spend. A 1,000-host fleet is $44M/year. A 20,000-host training cluster (the rough scale of the largest 2026 frontier-model runs) is $880M/year.

At those scales, percent-loss math is straightforward:

Scenario	1% throughput loss costs	Visibility
1% throughput loss on a 100-host fleet ($4.4M/yr)	$44K/yr	(below SRE noise)
1% loss on a 1,000-host fleet ($44M/yr)	$440K/yr	(detectable on the budget line)
1% loss on a 10,000-host fleet ($440M/yr)	$4.4M/yr	(material to the team budget)
1% loss on a 20,000-host training cluster ($880M/yr)	$8.8M/yr	(material to the platform budget)

These are conservative numbers. The dispatcher off-CPU bug we wrote up in an earlier post produced a 3.7% loss on a single host. The MoE all-to-all imbalance in the hybrid-architecture post produced a 30%+ loss on the affected workload. Real losses hit higher percentages, and they often hit at exactly the workloads where the cost-per-percent is highest (large training runs).

What kernel-level observability costs

The Ingero agent runs at under 2% CPU overhead on the workloads we have measured, with memory in the tens of MB. There is no SaaS backend, no per-host SaaS bill, and the binary is open-source. A reasonable first-order budget: the host operator’s time to install, configure, and wire the kernel-trace database into an MCP-callable surface for investigations. Call that <$5K per host amortized over the life of the host.

On the largest scenario above ($880M/yr GPU spend, 20,000 hosts), the instrumentation budget at <$5K/host is $100M one-time, or roughly 11% of the annual GPU spend – and recovers itself the moment a single 1% stall is found and fixed. In practice the recovery is 10-100x in the first kernel-stall investigation.

Why the math is asymmetric

The asymmetry is structural. Building more GPU capacity is bounded by physical supply (silicon, energy, real estate) and takes years. Recovering throughput from existing GPU capacity is bounded by software and takes weeks. At every capex milestone, the existing fleet becomes more valuable per unit, which means each unit recovered through kernel-level efficiency is worth more in dollar terms.

This is not a hypothetical. The Wolfe Research note disclosing OpenAI’s use of Datadog for Codex tracing is one signal. The 10 consecutive days of Datadog GPU Monitoring press through April is another. Operators are looking for ways to make the existing fleet faster because the supply side is gated.

Public sources on the AI-infra capex cycle

Public sources for the capex numbers and the supply-side argument above: the CNBC report on Anthropic’s $50B round; the TechCrunch report on SoftBank’s Roze AI; the AWS EC2 P5 (H100) instance pricing page for the per-GPU-per-hour anchor; and the Crusoe Command Center launch as a representative neocloud-side response to the same capex pressure.

Per-percent dollars at hyperscaler scale

When GPU supply is the gate, every percent of throughput that returns to the workload is a percent that did not have to be bought as new silicon. The math at fleet scale makes kernel-level observability the cheapest line item in the AI-infra capex stack and the one most likely to recover its own cost in a single investigation. The numbers stop being abstract above ~1,000 hosts. They cross the seven-figure-recovery line above ~10,000.

Ingero – open-source eBPF agent for GPU debugging. One binary, zero deps, <2% overhead. Apache 2.0 + GPL-2.0. *GitHub ⭐** · Open an issue if you are operating GPU clusters at fleet scale and want kernel-side visibility into where throughput is going.*

Agent + MCP + eBPF: 10,869 CUDA Kernel Events, Now Queryable

Ingero Team — Tue, 21 Apr 2026 13:30:00 +0000

A vLLM inference server handles hundreds of requests per second. Then one request with n_completions=8 and logprobs=20 arrives, and every other request blocks for 9-11 seconds. GPU utilization monitors stay green. Kubernetes reports healthy pods. Latency dashboards show a spike but no why. An eBPF trace of every CUDA call is the only view that catches this.

Originally published at ingero.io.

This is a real issue (vLLM #37343). We reproduced it on an RTX 4090 running vLLM 0.18.0 with Qwen3.5 with 27B params. Ingero's eBPF trace captured everything: 10,869 events, 550 cudaLaunchKernel calls, 7,757 context switches, 6 causal chains. All stored in a 1.2 MB SQLite database.

Then we pointed Claude at it via MCP.

The session

The entire investigation below is real. These are actual MCP tool calls, actual data, actual conclusions. Nothing was edited except formatting for readability.

Claude's first move - it called get_trace_stats:

10,869 events traced:
  7,757 sched_switch (context switches)
    550 cudaLaunchKernel   p50: 20us  p99: 5,079us
  1,672 cuLaunchKernel     p50: 15us  p99: 892us
    734 cudaMemcpyAsync
     10 cudaDeviceSync     p50: 19us  p99: 4,720us (239x median)

Two numbers jump out: cudaLaunchKernel p99 is 254x the p50, and there are 7,757 context switches. That ratio is not normal.

Claude's second move - it called get_causal_chains:

The most severe chain:

cudaLaunchKernel p99 = 13.1ms (784x the p50 of 16.7us), correlated with 428 context switches totaling 8.9 seconds off-CPU.

This is the signal. The kernel launch itself takes microseconds. But the CPU thread that feeds the GPU is getting preempted 428 times, accumulating 8.9 seconds of idle time. The GPU sits waiting.

Claude's third move - without being asked, it called get_stacks to look at the actual code paths:

Kernel	Calls	Avg Latency	Max Latency
fused_add_rms_norm	118	54 us	3.0 ms
reshape_and_cache_flash	75	42 us	161 us
silu_and_mul	69	44 us	156 us
rotary_embedding	62	38 us	222 us

All running under VLLM::EngineCore, the single-threaded engine coroutine.

Claude's diagnosis (47 seconds from first tool call):

"The vLLM engine coroutine is running on a CPU shared with other processes. When those processes get scheduled, the coroutine is preempted, the GPU gets no new work, and throughput collapses. The CPU thread driving the GPU was context-switched out 5,347 times, causing 18.4 seconds of CPU unavailability. The root cause is CPU contention, not GPU contention."

Claude's fix recommendation:

Pin the engine thread to dedicated CPU cores (taskset)
Reduce priority of background processes (nice -n 19)
Do NOT use CUDA_LAUNCH_BLOCKING (it will make things worse)
Do NOT trust nvidia-smi (it shows utilization, not causation)

What happened here

No SSH. No log files. No dashboard hopping. No "let me check nvidia-smi on each node."

An AI agent made 4 MCP tool calls against a 1.2 MB SQLite database containing kernel-level eBPF traces. It identified the root cause (CPU scheduling contention), the specific code path (EngineCore coroutine), and the fix (CPU pinning) - all in under a minute.

The key insight: nvidia-smi would have shown 100% GPU utilization during this entire incident. The GPU was "utilized" - it was executing the work it was given. The problem was that it wasn't being given work fast enough because the CPU thread feeding it was being preempted. That distinction - between "GPU is busy" and "GPU is being fed work efficiently" - is invisible to every standard GPU monitoring tool.

What made this possible

This is not a wrapper around nvidia-smi. The eBPF trace attaches uprobes directly to libcudart.so (CUDA Runtime) and libcuda.so (CUDA Driver), plus tracepoints on the Linux kernel scheduler (sched_switch, sched_wakeup), memory allocator (mm_page_alloc), and I/O subsystem. Every CUDA API call is captured with nanosecond precision. Every context switch that preempted a GPU-feeding thread is recorded. The causal chain engine connects them automatically.

The MCP server exposes this data through 10 tools. The AI agent decides what to query. There is no pre-aggregation layer, no dashboard, no human selecting which metrics to look at. The agent gets the raw events and builds the diagnosis.

Try the eBPF trace yourself

The trace database is in the Ingero repo. The investigation works with any MCP-compatible AI:

# 1. Clone and build
git clone https://github.com/ingero-io/ingero.git
cd ingero && make build

# 2. With Claude Code
claude --mcp-config <(echo '{"mcpServers":{"ingero":{"command":"./bin/ingero","args":["mcp","--db","investigations/vllm-37343-logprobs-amplification.db"]}}}')

# 3. With Ollama (any open model)
pip install mcp-client-for-ollama
ollmcp -m qwen3.5:27b -j /tmp/ingero-mcp.json

Type /investigate to start the guided workflow. The AI will walk through the same investigation you just read.

The pattern repeats

This is not a one-off. We have traced dozens of GPU performance issues. The pattern is consistent:

124x slower PyTorch DataLoader - kernel tracing revealed 191,000 context switches and 299,000 page allocations in 40 seconds. The GPU was starved because DataLoader workers were fighting for CPU cores.
13x PyTorch slowdown from hidden NumPy sync - a tensor.cpu().numpy() call in a masking function triggered B x 2 implicit cudaStreamSynchronize calls per forward pass. On faster GPUs, the bottleneck got worse, not better.
GPU 97% utilized but training 3x slower - nvidia-smi reported healthy utilization while Prometheus node exporter and Fluent Bit were consuming 51.7% of available CPU time through 14,504 context switches.

Every one of these follows the same pattern: the GPU is fast, the host is the bottleneck, and standard GPU metrics cannot see it. The causal chain from host event to CUDA API call is the missing link.

What this means for GPU debugging

The traditional approach: alert fires, SSH into the machine, check nvidia-smi, check dmesg, check logs, open profiler, wait for reproduction, analyze flame graphs, correlate across tools. Hours.

The MCP-native approach: point an AI agent at the kernel traces, let it query what it needs, read the diagnosis. Minutes.

We are not saying the AI is smarter than a senior SRE. We are saying it has access to data the SRE cannot see (kernel scheduling decisions, per-CUDA-call latency distributions, automated causal chains) and it can query that data faster than a human can navigate dashboards.

The investigation databases are open source. The agent is open source. Try it locally.

Ingero - open-source eBPF agent for GPU debugging. One binary, zero deps, <2% overhead. Apache 2.0 + GPL-2.0. *GitHub** star - Open an issue if you are seeing vLLM or CUDA runtime issues. Investigation DB: investigations/vllm-cuda-kernel-events.db - Original issue: vllm-project/vllm#37343.*

11-Second Time to First Token on a Healthy vLLM Server

Ingero Team — Tue, 21 Apr 2026 13:30:00 +0000

TL;DR

A vLLM health endpoint says "ok." nvidia-smi says 95% utilization. But a user just waited 11 seconds for their first token. We reproduced a real vLLM issue on an RTX 4090 and traced every CUDA API call and Linux kernel event to find the root causes: head-of-line blocking during prefix caching. This is invisible to standard monitoring. The trace databases are available in the Ingero repo for independent investigation. We traced a production case of vLLM latency spikes down to kernel-level scheduling contention.

The Problem Nobody Can See

vLLM's continuous batching is one of the best things to happen to LLM serving. It lets the engine process multiple requests simultaneously, filling GPU capacity that would otherwise sit idle between sequential requests.

But continuous batching has a dark side: when requests compete for GPU resources inside the same batch, one expensive request can silently starve all others. No error. No health check failure. No metric spike. Just users waiting 10x-250x longer than expected for their first token.

We investigated a real vLLM issue reported in the last week (#37308) to understand what happens at the kernel level during these silent latency spikes.

Setup

The investigation used the same server configuration:

python -m vllm.entrypoints.openai.api_server \
 --model Qwen/Qwen2.5-0.5B-Instruct \
 --port 8000 \
 --gpu-memory-utilization 0.95 \
 --max-model-len 32768 \
 --enable-prefix-caching

Hardware: RTX 4090 (24GB), 4 vCPUs, Ubuntu 22.04, vLLM 0.17.1.

We ran Ingero alongside each test to trace CUDA Runtime/Driver API calls and host kernel events (scheduler context switches, memory allocations) simultaneously.

Prefix Caching Head-of-Line Blocking

Issue: vllm-project/vllm#37308

What happens

6 concurrent requests arrive within 40ms. 4 are heavy (2048-token prompts, 128-512 output tokens) and 2 are light (128-token prompts, 32-64 output tokens). All share a 32-token prefix so the prefix cache groups them together.

The light requests should complete in under 100ms. Instead:

Run	r08 (128 tok)	r05 (128 tok)	r07 (2048 tok)	r02 (2048 tok)
1	1,131ms	1,406ms	1,654ms	1,851ms
2	54ms	129ms	258ms	234ms
3	66ms	177ms	175ms	156ms

Run 1 is catastrophic: the light requests are 14x over threshold. Subsequent runs settle to 2-4x because the prefix cache warms up. But that first cold-cache batch is brutal.

What the tracer shows

3 causal chains detected. The most revealing one:

[MEDIUM] cudaLaunchKernel p99=444us (6.4x p50) - 371 sched_switch events
 Timeline:
 [HOST ] 371 context switches (5.9s off-CPU)
 [CUDA ] p99=444us (6.4x p50=70us)

The per-process breakdown tells the full story:

VLLM::EngineCore (the GPU scheduling loop):

24,347 context switches, max stall 2.5 seconds
40,632 cuLaunchKernel calls, avg 29us but max 34ms
34,087 cudaLaunchKernel calls, avg 96us but max 356ms

The engine core process – the single-threaded loop that decides which requests get GPU time – was descheduled for 2.5 seconds in the worst case. During that stall, the GPU kernel queue drained and the light requests had nothing submitted on their behalf.

The 356ms cudaLaunchKernel spike (3,700x the average) is the smoking gun. That's not the GPU being slow. That's the CPU failing to submit work to the GPU because the scheduling loop was preempted.

Why nvidia-smi misses this

nvidia-smi shows high utilization because the GPU IS working – on the heavy requests' prefills. The light requests are starving, but from the GPU's perspective there's always a kernel to run. The starvation is in the CPU-side scheduling loop, not on the GPU.

What Standard Tools Show vs What Kernel Tracing Shows

Signal	nvidia-smi	vLLM /health	vLLM metrics	Kernel tracing
GPU utilization	95%+	–	–	95%+ (but wrong work)
Server health	–	"ok"	requests_running=5	–
TTFT regression	–	–	Visible in histograms	Visible + root cause
Engine stall (2.5s)	Not visible	Not visible	Not visible	24,347 sched_switch events
Kernel launch drop (80%)	Not visible	Not visible	Not visible	1,051 -> 208 ops/s
Memory pressure	Not visible	Not visible	Not visible	43,606 mm_page_alloc
Which process is blocked	Not visible	Not visible	Not visible	VLLM::EngineCore PID 2438

The key insight: GPU utilization was high because the GPU was doing work. It was just doing the wrong work – processing heavy prefills or computation while light requests starved. No GPU-side metric can distinguish "GPU is busy computing my request" from "GPU is busy computing someone else's request while mine waits."

Implications for Production vLLM

If you're running vLLM in production with mixed workloads (different prompt sizes, some requests with or ), you're likely experiencing these silent regressions:

Monitor TTFT per-request, not just aggregate throughput. Aggregate metrics hide the tail – your p99 might be 100x worse than p50 during batch contention.
Be careful with . A single request with n=8 and =20 can block your entire server for 11+ seconds on a cold cache. Consider routing these to dedicated instances.
First-request-after-idle is the worst case. This issue showed the most extreme regression on Run 1 (cold prefix cache). If your traffic is bursty, the first batch after a quiet period will hit hardest.
GPU utilization is not a proxy for request health. Your dashboards might show 95% utilization while individual users experience 256x TTFT regression.

Investigate It Yourself

The trace database from this investigations are in the Ingero repo:

git clone https://github.com/ingero-io/ingero.git
cd ingero && make build

# View the causal chains
./bin/ingero explain --db investigations/vllm---amplification.db --since 5m

# Per-process breakdown
./bin/ingero explain --db investigations/vllm---amplification.db --per-process --since 5m

# Connect your AI assistant for interactive investigation
./bin/ingero mcp --db investigations/vllm---amplification.db

Investigate with AI (recommended)

You can point any MCP-compatible AI client at the trace database and ask questions directly. No code required.

First, create the MCP config file at /tmp/ingero-mcp-vllm.json:

{
  "mcpServers": {
    "ingero": {
      "command": "./bin/ingero",
      "args": ["mcp", "--db", "investigations/vllm-37308-hol-blocking.db"]
    }
  }
}

With Ollama (local & free: no data sent outside):

# Install ollmcp (MCP client for Ollama)
pip install ollmcp

# Investigate with a local model (no data leaves your machine)
ollmcp -m qwen3.5:27b -j /tmp/ingero-mcp-vllm.json

With Claude Code (with data sent to remote models / Anthropic):

claude --mcp-config /tmp/ingero-mcp-vllm.json

Then type /investigate and let the model explore. Follow up with questions like "what was the root cause?" or "which kernel calls had the highest latency spikes?"

Ask your AI assistant: "What caused the 80% throughput drop?" or "Which process had the most context switches?" The trace data has the full story.

The investigation database from this post is available for download. Investigations performed on TensorDock RTX 4090 (24GB), Ubuntu 22.04, vLLM 0.17.1, Qwen/Qwen2.5-0.5B-Instruct with prefix caching enabled.

GitHub (give us a star!): github.com/ingero-io/ingero. No NVIDIA SDK, no code changes, production-safe by design.

If you are seeing vLLM issues in your own workloads, we'd love to take a look. Drop an issue on GitHub and we will gladly dive into it together.

Ingero is free & open source software licensed under Apache 2.0 (user-space) + GPL-2.0/BSD-3 (eBPF kernel-space). One binary, zero dependencies, <2% overhead.

DEV Community: Ingero Team

When GPUs Are Scarce, Each Stall Costs N Times More

TL;DR

The capex line items

The per-percent dollar

What kernel-level observability costs

Why the math is asymmetric

Public sources on the AI-infra capex cycle

Per-percent dollars at hyperscaler scale

Related reading

Agent + MCP + eBPF: 10,869 CUDA Kernel Events, Now Queryable

The session

What happened here

What made this possible

Try the eBPF trace yourself

The pattern repeats

What this means for GPU debugging

11-Second Time to First Token on a Healthy vLLM Server

TL;DR

The Problem Nobody Can See

Setup

Prefix Caching Head-of-Line Blocking

What happens

What the tracer shows

Why nvidia-smi misses this

What Standard Tools Show vs What Kernel Tracing Shows

Implications for Production vLLM

Investigate It Yourself

Investigate with AI (recommended)

Related reading