Zero LLM Calls From a Working Agent. My Capture Hook Was Lying.

From the agentdrill v0.1.0 release, part of agentic-suite: open-source, harness-agnostic security tools for AI-agent defenses.

The agent produced a full trajectory. Tool calls, results, a final answer. The capture said it had made zero LLM calls.

I almost shipped the wrong diagnosis. I spent a debugging cycle inside the agent's adapter, convinced the llamaindex path was broken, because the evidence in front of me said the model never spoke. The evidence was the bug.

This is from the agentdrill v0.1.0 field test: 73 test units, 19 real agents across 8 frameworks, every LLM case required to prove its traffic with captured request and response pairs. The full story is in the field test report.

The symptom pointed at the wrong suspect

The setup was simple in principle. Patch the HTTP client inside the test harness so every LLM call gets written to disk as a JSON pair. The judge calls appeared immediately. Every one of them, 2xx, clean.

The agent calls did not appear at all.

Zero. Not errors, not 429s, not empty bodies. Nothing. When a capture records failures you debug the server. When it records silence you debug the client. So I debugged the client, which in this case meant the agent adapters, and burned an afternoon on a suspect that was innocent.

Here is what was actually happening. The judge client made its calls through a code path that went through httpx.Client.request(). My hook patched request(). The agent frameworks used the OpenAI Python SDK, and the OpenAI SDK speaks to httpx through send(). It never touches the polite public method I had hooked.

httpx.Client.send() is the single funnel. Every request goes through it, including the ones that skip request(). My hook was sitting one level too high, watching the road where only half the traffic drove.

The fix was two lines of understanding, not two hundred lines of code

The capture now hooks send() on both Client and AsyncClient, because agents love async. That is the whole fix. You can see it in llm_capture.py and the test harness.

What came back after the fix changed the shape of the rest of the run:

• Every one of the 19 Track B cases now has request plus response pairs, all 2xx
• The model pin is visible in each request body, which is how we proved the agent ran on Qwen3-4B-Instruct-2507-4bit and the judge on Qwen3.5-4B-4bit
• Zero non-2xx pairs across the suite

The raw pairs are committed under results/track-b/cases, one directory per case, so the claim is checkable without trusting me.

Why a silent gap is worse than no capture

A missing capture is an absence. A silently partial capture is a false witness. It did not just fail to record the agent traffic, it actively supported a wrong conclusion, because the shape of the missing data looked exactly like "the agent never called the model". I had a green case verdict and a zero on the chat counter, and those two facts together pointed at the agent.

There was a second, smaller version of the same disease earlier in the run. The capture assertion originally passed on an error stub: a response file existed, but its status code was 0. A file was present, so the check was happy. We hardened it to require a real 2xx with a non-empty body before it counts as a captured pair.

Both fixes have the same shape: the check must fail when the claim is false. Present is not the same as true.

How to know your hook sees everything

You cannot prove a capture is complete from inside the capture. What worked for us:

1. Cross-check against a counter you do not own. The server saw the requests. The artifacts showed trajectories. When the capture disagreed with the trajectories, the capture lost.
2. Hook the lowest funnel point that still sees everything. For httpx that is send(). Find the equivalent in your stack and patch that, not the convenience method.
3. Make each capture self-describing. Ours carry meta.model, so every file says which pin produced it. That detail later let us audit which runs were stale after a model switch, which is its own story.
4. Assert on the pair, not the file. A capture check that counts files will pass on garbage. Ours requires N requests and N responses, a real 2xx, and a non-empty body.

What I keep from this

• A capture that under-records does not just miss evidence. It manufactures diagnosis. The zero was louder than the truth.
• The polite public API of a library is not the same as the choke point. Find the choke point.
• Test infrastructure needs its own suspicion. We only caught this because a second signal (the trajectory) contradicted the first (the counter). One source of truth with no second opinion is a single point of failure for the truth itself.

The deeper pattern is uncomfortable: every observation tool is also a claim about how the observed system works. My hook claimed "requests flow through request()". The system disagreed, quietly, and the system was right.

Tip: before you trust a patched client, send one request and confirm the server and the capture both saw it.

If you patched a client library to record traffic today, how would you know if you missed half of it? That question is why every capture assertion now counts real pairs, checks for a 2xx, and gets re-derived from the files on disk instead of from the capture's own tally. I would rather explain a discrepancy than trust a counter.

Code and receipts: agentdrill · field test report · capture hook · raw LLM pairs

---

About agentdrill

This story is one of seven from the agentdrill v0.1.0 field test. agentdrill is the adversarial-testing engine in agentic-suite: open-source, harness-agnostic security tools for AI-agent defenses. It runs versioned attack packs against your deployed stack in CI and publishes honest results, misses included.

pip install agentsec-agentdrill   # v0.1.0 is live on PyPI
agentdrill quickstart             # first honest result in ~15 minutes, bundled demo

Dig deeper: the v0.1.0 field test report (73/73 units, every pass grounded), the release notes, and the rest of the suite at github.com/agentic-suite: agentdrill drills the defenses, agentwatch records the evidence.