You ship an AI agent. It calls tools, reads results, calls more tools, answers. Most of the time it works. Then a user reports something wrong, you open the trace, and you find it: the charge_card tool returned a 402, and the agent just... kept going and told the customer their order shipped.
That's not a hallucination in the "made up a fact" sense. It's a structural defect in the run — an ignored tool error. And here's the thing about structural defects: you don't need another LLM to find them. They're decidable by looking at the trace.
That's the whole premise of tracelint: a linter for agent runs. It reads the execution trace — what the agent actually did — and flags structural bugs deterministically, with the exact trace lines as evidence and a CI exit code. It runs after the run, on the trace, not on your code. No second model ever judges it.
Why not just use an LLM judge?
Because for this class of bug, a judge is the wrong tool. Published trace-error benchmarks show LLM judges have low localization accuracy — they'll tell you "something seems off" without reliably pointing at which step. They're also non-deterministic, cost money per trace, and can't gate CI (would you fail a build on a coin-flip?).
Meanwhile, a whole category of agent bugs is structurally decidable:
- A tool call whose arguments violate the tool's JSON Schema. That's not an opinion — you run the schema validator.
- A tool that returned an error, followed by the agent proceeding as if it hadn't.
- The same tool called 5 times with identical arguments and identical results (a stuck loop).
- Arguments that don't appear anywhere in what the agent observed (a candidate hallucinated value).
None of these needs a model. They need the trace and a validator. That's what tracelint does.
The 60-second version
pip install tracelint
tracelint demo --html demo.html
demo runs a keyless validation suite — one planted instance of every defect, plus clean controls — and writes an HTML report. No API key, no model download.
To gate CI on a real trace:
tracelint check ./trace.json --tools ./tools.json # exit 2 on a structural defect
Exit codes: 0 clean, 2 a structurally-provable defect, 3 an input error. Heuristic findings never fail CI on their own.
The part that matters: it runs on traces you already collect
Here's the distribution insight. You're probably already instrumenting your agent — with OpenInference (the OpenTelemetry semantic convention for AI), feeding Arize Phoenix, Langfuse, or an OTel collector. tracelint reads that telemetry directly. You don't learn a new trace format; you point it at the spans you've got.
tracelint check spans.json --format openinference # Phoenix, OTLP, TRAIL
tracelint check trace.json --format langfuse
tracelint check messages.json --format openai
Or straight from a running Phoenix instance, in Python:
import phoenix as px
from tracelint import lint_otel_trace
spans = px.Client().get_spans_dataframe().to_dict("records")
report = lint_otel_trace(spans)
print(report.exit_code) # 0 or 2
for f in report.active_findings:
print(f.rule, f.tier.value, f.summary)
I validated this against real OpenInference exports, not just hand-built fixtures — a real Phoenix trace, an OTel-SDK span export, the Phoenix dataframe shape. On one real Phoenix trace, tracelint deterministically localized a genuine tool failure:
[hard_event] R2a tool_error_event (step 9)
'add_spans_to_dataset' returned an error (GraphQL query 'exampleMutation' ... 'an unexpected error occurred')
No model in the loop. Just: this TOOL span has an ERROR status, at exactly this step, here's the message.
What it catches
| Rule | Finding |
|---|---|
| R1 | schema violation — args fail the tool's JSON Schema |
| R2 | tool returned an error / an errored value reused by a later side-effecting call |
| R3 | hallucinated argument — value not derivable from anything observed |
| R4 | loop — N identical no-progress calls |
| R5 | redundant call — identical call + identical result, no mutation between |
| R6 | malformed arguments — the tool-call arguments aren't valid JSON |
| R7 | unknown tool — a call to a tool absent from the declared toolset |
Here's a stuck-loop run through the OpenInference adapter:
[candidate] R4 loop (step 2,4,6)
'search' called 3 times in a row with identical arguments and no change in result state (ok)
[candidate] R5 redundant_call (step 2,6)
'search' repeats an earlier identical call with no mutating call in between
Two design decisions I'd defend
1. Candidate, not verdict. Only structurally-provable things (a schema violation, malformed JSON) are hard defects that fail CI. Heuristic signals — loops, redundant calls, suspicious arguments — are shown as candidates with their evidence, for a human to review, never asserted as truth and never failing your build on their own. A retry loop and a stuck loop look similar structurally; tracelint shows you the evidence and lets you decide, instead of pretending it knows.
2. It tells you what it couldn't check. This is the one I care about most. If a trace is missing a field a rule needs — no tool schemas, no result payloads — that rule doesn't silently pass. It suppresses with a stated reason, printed in the report:
suppressed (2) — not checked, not a clean pass:
R1 schema_violation: no tool schema available for any called tool
R7 unknown_tool: no tool registry supplied — cannot know which tools were declared
A clean report with hidden gaps is worse than no report — it's false confidence. tracelint refuses to give you that.
Honest limitations
- It catches structural defects, not whether the final answer was correct. It won't tell you the agent gave bad advice; it'll tell you the agent ignored a failed tool call on the way there.
- Hallucinated-argument, loop, and redundant-call findings are candidates unless structurally proven. Legitimate value transforms and intentional retries can trip them — that's why they're shown with evidence, not asserted.
- A trace is only as good as its instrumentation. Missing fields mean suppressed rules, not fabricated ones.
Try it
pip install tracelint
tracelint demo --html demo.html
It's open source (MIT), dependency-light (jsonschema + stdlib), Python 3.10–3.12, and the whole test suite is offline and deterministic.
If you're collecting agent traces and want deterministic checks on them, I'd genuinely like to know what breaks on your real exports — that's how the last three real-world shape fixes happened. Issues and traces welcome.
Top comments (2)
This is the right kind of boring gate. I’d still keep the heuristic checks out of the blocking path until they have a false-positive budget, but the deterministic cases belong in CI. Ignored tool errors and schema misses are trace defects, not vibes.
This is a sensible boundary for CI. If a failure is decidable from the trace, adding a judge mostly adds variance and cost. The harder bit is deciding which warnings are allowed to block a release.