I built agentrace to catch the subagent runs I should not trust

Directing agents is the easy half. The hard half is knowing which of their answers to trust.

I spent about two weeks fanning out research subagents in Claude Code, ten at a time, and the bottleneck was never getting them to produce output. A model is good at producing candidates and bad at knowing what counts as proof. So when ten background agents each return a confident wall of text, generation is not the problem. Verification is. And you cannot verify what you cannot see. By the time the last agent reports, the interesting details are buried in a transcript nobody is going to read.

agentrace reads the transcript for you.

The core idea: no instrumentation

The thing I like most about it is that there is nothing to add ahead of time. No wrapper, no SDK, no decorator around your agent calls. Claude Code already writes every session to disk at ~/.claude/projects/<slug>/<session-id>.jsonl, and that file already contains every Agent delegation and the result it returned. The data is on disk whether or not you planned to look at it, which means you can analyse the run you wish you had traced, after the fact.

So agentrace is a reader, not a runtime. It parses those JSONL transcripts, pairs each subagent invocation with its result, and then runs a set of text heuristics over the pair to point at the results worth a second look.

What it shows you

Three commands do most of the work. First, the aggregate:

$ agentrace stats
 subagent runs          152
 errored                7
 total agent time       2.3 h
 slowest run            9.5 min
 prompt chars written   350,134
 result chars returned  257,721

Then the part that matters, the flagging:

$ agentrace check
 36/152 runs flagged, 39 findings

Those are real numbers from the session that motivated the tool: a 34MB transcript with 152 subagent runs. Of the 36 flagged, 7 were agents that died on session limits mid-sweep, 17 were hedged claims, and 12 were prompts where I forgot to specify an output shape.

That last number is the one I keep pointing at. Most agent tooling assumes the model is the problem. A third of the flags here were mine.

You can narrow the view and read a single run in full:

agentrace list                  # every run: description, duration, sizes
agentrace check --severity high # only the ones that definitely matter
agentrace check --strict        # exit 1 on any high finding (CI-friendly)
agentrace show 6e7fAJ8T         # one run: prompt, result, findings
agentrace stats --json          # machine-readable aggregate

The checks come from real failures

Every check exists because it actually happened, not because it sounded plausible. A few of them:

• error: agents dying on session limits mid-sweep. Work silently lost, and nobody noticed until the report came back short.
• absence_as_evidence: an agent concluded a company was not hiring because an API returned an empty list. That API returns empty with HTTP 200 for accounts that do not exist. Absence of data is not evidence of absence.
• gave_up: "I was unable to find..." reads like an answer if you skim. It is not one.
• hedged_claim: an agent said a company "appears to be" hiring. That "appears to be" became a fact by the time it reached a decision. The hedge was honest; the bug was flattening it downstream.
• unverified_urls: twenty URLs cited, none opened. That is autocomplete, not research.
• no_output_contract and thin_prompt: the failure that is yours, not the model's. A task with no definition of done cannot be verified, because you never really asked the question.
• slow_run: a subagent running 25 minutes is usually looping or retrying.

Mechanically these are regex-and-length heuristics over the result and prompt text, plus the run duration. unverified_urls, for example, counts links in the result and only fires when there are at least five and nothing in the text mentions verifying, confirming, fetching, or an HTTP 200. Cheap, but it catches the shape of the failure.

The hard part was not crying wolf

Every check is a heuristic over text. It tells you what to go read; it does not tell you what is true. That distinction is load-bearing, because a checker that cries wolf gets switched off, which is worse than no checker at all. There is even a test, test_clean_run_produces_nothing, whose only job is to keep a clean run from producing findings.

The clearest example of tuning for that: thin_prompt used to fire on any prompt under 200 characters. But "Run the suite and report every failing test as node ids with its assertion message" is 113 characters and completely verifiable. Flagging it taught nobody anything while spending the reader's attention. Length was never the defect. Being short and never saying what done looks like is. So now both signals have to fire. On the bundled fixture that took findings from 16 down to 9 without losing a single true one.

The honest limitation

These are hints, not verdicts. Every finding is a heuristic over text, so it can be wrong in both directions. A hedged_claim flag does not mean the claim is false, only that a hedge is present and might get flattened downstream. An unflagged run is not certified correct; it just did not trip any of the seven patterns. agentrace narrows where a human should look. It does not replace that human, and it cannot judge whether the agent's answer is actually true. If you want a verdict, you still have to read the run it points you to.

A couple of parsing choices follow from wanting to look at live runs. It scans the transcript twice, because results can appear before every corresponding use in unusual orderings, and a 34MB file is cheap to scan twice compared to getting the pairing subtly wrong. A torn final line from a session still being appended to is skipped rather than treated as fatal, and a run with no result yet still shows up rather than vanishing, because a dead or still-running agent is exactly the one you want to see.

Closing

It is working today, has 17 tests, zero dependencies beyond rich, no API keys, and no network. It reads local files and nothing else. If you run subagents and have ever shipped one of their confident answers without reading it, this is the tool I wanted for exactly that moment.

Repo: https://github.com/royalpinto007/agentrace