A slow AI agent run is a chain of network calls: the model, tools, a database, outside APIs. An application-level trace tells you which call was slow. It can't tell you why, because a DNS stall, a slow connection and a slow server all happen inside one HTTP span. System-level eBPF tools see DNS time, connect time, retransmits and round-trip time, but they know nothing about the agent.
So I tested a simple hypothesis: joining agent traces with kernel-level evidence diagnoses faults better than agent traces alone. I wrote my predictions down and committed them to git before every run, and many were wrong. This post is the short version.
Code, benchmark, predictions and the full report: https://github.com/mac146/crosstrace (tag report-v1)
What I built
- A seeded fault-injection benchmark (slow DNS, network latency, packet loss, a slow API server, a slow database and more)
- An eBPF collector, first in bpftrace, then a compiled C/libbpf version that uses about 2.9x less CPU
- A correlator that aligns kernel events with OpenTelemetry spans (clock alignment, indexed matching)
- Rule-based diagnosers and an evaluation where every stage's predictions, with "I would be wrong if..." conditions, were committed before the runs
What held up
- Kernel evidence works where the cause is in the network path. Top-1 accuracy on DNS delay, latency and packet loss rose from about 30% (spans only) to about 97% on a fresh set of 90 runs, with no false accusations in 240 runs. The caveat is in the report: these are mock services on loopback, and the spans-only baseline can't separate those causes by construction.
- The join is robust. 160/160 runs passed integrity checks against ground-truth ports, and 5,120/5,120 spans got exactly their own connection with up to 512 simultaneous agent threads.
- On a real model over the internet, the rules found injected DNS delays in 10/10 runs and latency in 9/10 (10 runs per group, one network path, so observations rather than estimates).
What I had to narrow
- Per-request kernel timing carries the result, not the join itself. Host-wide counters plus the run's maximum server wait matched the join on single faults.
- I ran the real OpenTelemetry eBPF agent (OBI v1.0.0). Its per-request server duration names a slow server as well as my join (100%) but is at chance on network causes. With its per-connection RTT matched exactly to each call by client port, it equals my join (99.5-100% with 9 and 24 concurrent agents in different conditions). Matched by connection order, it falls to 63-69%.
- So the join is not better evidence. What my correlator adds is matching a call to its connection without the application declaring its socket. That works for thread-per-call agents. Async agents on one thread need the port declared, and a quick text search of the standard Python HTTP client instrumentations suggests they don't set it.
- Host-wide counters get worse as you add agents. One stray retransmit anywhere in a run gets pooled into every agent's evidence (rescoring without it fixed it: 64/64 instead of 24/64). Per-call evidence doesn't have this problem.
What didn't work (all kept in the report)
- On a real, NAT'd network path, packet loss can't be told from the background retransmits by count: 70% detection at best, and an alternative feature that looked better on the design set failed on fresh data.
- My CPU target was missed (2.9x less, predicted 3.4x), a clock-sampling prediction was wrong, a probe silently missed 0.1% of connections until I found a better hook, and a harness bug briefly made a latency fault look like packet loss.
- The report lists 13 numbered negative or wrong-prediction results.
Limits you should know about
Mock services on loopback, one host, scenarios I designed to discriminate, and one OBI version and configuration that I tuned to make it attach. No packet-loss, HTTP/3 or streaming tests. The collector needs root-equivalent access, so it is for dev machines only. I don't know how often slow agent runs in the real world are network problems. The one dataset I found suggests provider-side causes like rate limits dominate.
What's next
Two experiments are pre-registered but not run: whether async agents can be matched without a declared port, and whether the results hold with 8 concurrent real agents on a real model and network.
A question for you: if you run agents in production, how often are slow or failing runs network or infrastructure problems rather than model or tool problems? Your last two or three slow-run incidents, anonymized, would help me more than any benchmark row. Comments or issues on the repo are welcome, and so are corrections if I've misused OBI anywhere.
Report, code and predictions: https://github.com/mac146/crosstrace
Top comments (0)