You ship an AI agent. It calls tools, reads results, calls more tools, answers. Most of the time it works. Then a user reports something wrong, you open the trace, and you find it: the charge_card tool returned a 402, and the agent just... kept going and told the customer their order shipped.
That's not a hallucination in the "made up a fact" sense. It's a structural defect in the run — an ignored tool error. And here's the thing about structural defects: you don't need another LLM to find them. They're decidable by looking at the trace.
That's the whole premise of tracelint: a linter for agent runs. It reads the execution trace — what the agent actually did — and flags structural bugs deterministically, with the exact trace lines as evidence and a CI exit code. It runs after the run, on the trace, not on your code. No second model ever judges it.
Why not just use an LLM judge?
Because for this class of bug, a judge is the wrong tool. Published trace-error benchmarks show LLM judges have low localization accuracy — they'll tell you "something seems off" without reliably pointing at which step. They're also non-deterministic, cost money per trace, and can't gate CI (would you fail a build on a coin-flip?).
Meanwhile, a whole category of agent bugs is structurally decidable:
- A tool call whose arguments violate the tool's JSON Schema. That's not an opinion — you run the schema validator.
- A tool that returned an error, followed by the agent proceeding as if it hadn't.
- The same tool called 5 times with identical arguments and identical results (a stuck loop).
- Arguments that don't appear anywhere in what the agent observed (a candidate hallucinated value).
None of these needs a model. They need the trace and a validator. That's what tracelint does.
The 60-second version
pip install tracelint
tracelint demo --html demo.html
demo runs a keyless validation suite — one planted instance of every defect, plus clean controls — and writes an HTML report. No API key, no model download.
To gate CI on a real trace:
tracelint check ./trace.json --tools ./tools.json # exit 2 on a structural defect
Exit codes: 0 clean, 2 a structurally-provable defect, 3 an input error. Heuristic findings never fail CI on their own.
The part that matters: it runs on traces you already collect
Here's the distribution insight. You're probably already instrumenting your agent — with OpenInference (the OpenTelemetry semantic convention for AI), feeding Arize Phoenix, Langfuse, or an OTel collector. tracelint reads that telemetry directly. You don't learn a new trace format; you point it at the spans you've got.
tracelint check spans.json --format openinference # Phoenix, OTLP, TRAIL
tracelint check trace.json --format langfuse
tracelint check messages.json --format openai
Or straight from a running Phoenix instance, in Python:
import phoenix as px
from tracelint import lint_otel_trace
spans = px.Client().get_spans_dataframe().to_dict("records")
report = lint_otel_trace(spans)
print(report.exit_code) # 0 or 2
for f in report.active_findings:
print(f.rule, f.tier.value, f.summary)
I validated this against real OpenInference exports, not just hand-built fixtures — a real Phoenix trace, an OTel-SDK span export, the Phoenix dataframe shape. On one real Phoenix trace, tracelint deterministically localized a genuine tool failure:
[hard_event] R2a tool_error_event (step 9)
'add_spans_to_dataset' returned an error (GraphQL query 'exampleMutation' ... 'an unexpected error occurred')
No model in the loop. Just: this TOOL span has an ERROR status, at exactly this step, here's the message.
What it catches
| Rule | Finding |
|---|---|
| R1 | schema violation — args fail the tool's JSON Schema |
| R2 | tool returned an error / an errored value reused by a later side-effecting call |
| R3 | hallucinated argument — value not derivable from anything observed |
| R4 | loop — N identical no-progress calls |
| R5 | redundant call — identical call + identical result, no mutation between |
| R6 | malformed arguments — the tool-call arguments aren't valid JSON |
| R7 | unknown tool — a call to a tool absent from the declared toolset |
Here's a stuck-loop run through the OpenInference adapter:
[candidate] R4 loop (step 2,4,6)
'search' called 3 times in a row with identical arguments and no change in result state (ok)
[candidate] R5 redundant_call (step 2,6)
'search' repeats an earlier identical call with no mutating call in between
Two design decisions I'd defend
1. Candidate, not verdict. Only structurally-provable things (a schema violation, malformed JSON) are hard defects that fail CI. Heuristic signals — loops, redundant calls, suspicious arguments — are shown as candidates with their evidence, for a human to review, never asserted as truth and never failing your build on their own. A retry loop and a stuck loop look similar structurally; tracelint shows you the evidence and lets you decide, instead of pretending it knows.
2. It tells you what it couldn't check. This is the one I care about most. If a trace is missing a field a rule needs — no tool schemas, no result payloads — that rule doesn't silently pass. It suppresses with a stated reason, printed in the report:
suppressed (2) — not checked, not a clean pass:
R1 schema_violation: no tool schema available for any called tool
R7 unknown_tool: no tool registry supplied — cannot know which tools were declared
A clean report with hidden gaps is worse than no report — it's false confidence. tracelint refuses to give you that.
Honest limitations
- It catches structural defects, not whether the final answer was correct. It won't tell you the agent gave bad advice; it'll tell you the agent ignored a failed tool call on the way there.
- Hallucinated-argument, loop, and redundant-call findings are candidates unless structurally proven. Legitimate value transforms and intentional retries can trip them — that's why they're shown with evidence, not asserted.
-
Error detection is only as structured as the trace. A tool failure is caught when it arrives as a structured signal — an explicit error status, an HTTP status ≥ 400, or an
errorfield. A domain failure hidden in an otherwise-successful payload (HTTP 200 carrying{"status": "declined"}) isn't recognized unless the tool declares what failure looks like — and today that's a silent miss for that one result, not a suppression. Teachingtools.jsona per-tool failure predicate is the fix I'm working toward; until then, treat error detection as covering transport/structured errors, not arbitrary domain semantics. - A trace is only as good as its instrumentation. Missing fields mean suppressed rules, not fabricated ones.
Try it
pip install tracelint
tracelint demo --html demo.html
It's open source (MIT), dependency-light (jsonschema + stdlib), Python 3.10–3.12, and the whole test suite is offline and deterministic.
If you're collecting agent traces and want deterministic checks on them, I'd genuinely like to know what breaks on your real exports — that's how the last three real-world shape fixes happened. Issues and traces welcome.
Top comments (41)
How does R5 decide that nothing mutating happened between two identical calls? That looks like it needs the toolset to declare which tools have side effects, and if it's inferred from the name or the shape of the call, custom tools seem easy to miss.
Good eye — and yes, it's declared, not inferred. mutating_between walks the calls sitting between the two identical ones and asks the registry metadata_for(name).side_effecting for each. There's deliberately no name or shape heuristic: tracelint won't guess that delete* mutates or that get_* is safe, because that guess is exactly the kind of vibes-inference the tool exists to avoid — a custom refresh_index that writes, or a get_or_create that mutates despite the name, would fool any such rule. So a tool's side-effect status comes from tools.json or it isn't known. This is the same "declare per-tool semantics" model as x-value-origin (and the failure_when predicate from the other thread).
But the failure mode for an undeclared custom tool is the opposite of a missed bug. An unknown tool defaults to non-side-effecting, so _mutating_between returns false and R5 will fire — a false-positive candidate ("redundant"), never a silent miss. And there's a backstop before that even matters: R5 only triggers when the second call's result is byte-identical to the first (a full result fingerprint, not just matching args). If a real mutation happened in between and actually changed the data, the re-fetch returns different bytes, the fingerprints diverge, and R5 never fires. So the side_effecting metadata is a secondary guard for the narrow case where a mutation occurred but didn't change this particular read; the primary guard is "same call, same result." And it's a candidate — shown with evidence, never gating CI.
Where I think you've found a real sharp edge: if the in-between tool is entirely unknown
This is the right kind of boring gate. I’d still keep the heuristic checks out of the blocking path until they have a false-positive budget, but the deterministic cases belong in CI. Ignored tool errors and schema misses are trace defects, not vibes.
Agreed, and that's the default. Only structurally-provable defects — a schema violation, malformed args — carry the tier that returns a non-zero exit. Everything heuristic (loops, redundant calls, suspicious args, even "a tool errored") is a candidate that never gates CI on its own. Your false-positive-budget framing is the right lens: today the budget is implicitly zero for the blocking path and unbounded/advisory for candidates. Promoting any heuristic into the gate should be an explicit opt-in with a measured FP rate, never a default.
is_structured_erroris doing more load-bearing work here than the post lets on. It fires on an explicit ERROR status, a top-levelhttp_statusorstatus_codeof 400+, or a top-levelerrorkey, and the OTel adapter's_is_error_spanadds span status plus anexceptionevent while still reading only those same top-level fields out ofoutput.value. So error-ness arrives from whatever wrote the span. The trace never decides it.That makes the opening example the shakiest case in the post. A declined charge usually lands as a transport-level success carrying
{"status":"declined","decline_code":"insufficient_funds"}, or with the error nested a level down under a result wrapper, and neither shape trips the structured tier. R2a's regex fallback won't save it, since that only runs when the result content is a string, so a dict payload yields no candidate and no suppression. Clean pass. Worse, R2b is gated on the same check, so the reuse chain, the "told the customer their order shipped" half of the story, is never examined at all.The fix looks like it's already half-built.
tools.jsoncarriesmetadata.side_effecting, and R3 accepts declared per-field semantics throughx-value-origin. A siblingmetadata.failure_when, say{"pointer":"/status","in":["declined","failed"]}, evaluated insideis_structured_error, would declare domain failure once per tool and keep it structural. Where a side-effecting tool has none declared, R1's per-call suppression is the honest fallback ("no failure predicate for tool X") rather than counting that call as checked.You're right. is_structured_error reads exactly three things — an explicit error status, http_status >= 400, or a non-null error field — all populated by whatever wrote the span. The result content is never inspected for domain failure, so {"status":"declined","decline_code":"insufficient_funds"} over HTTP 200 sails through: no hard_event, and since R2a's regex fallback is string-only it can't touch a dict payload, so no candidate and no suppression either. R2b gates on the same predicate, so the reuse chain is never examined. And you've named the worst part: it's a silent miss — the fail-closed guarantee lives at the rule level (rule can't run → suppress), not the per-result level, so a side-effecting tool that fails in a shape the vocabulary doesn't know looks clean.
Your failure_when proposal is exactly where I'd take it, and it fits the existing model — side_effecting and x-value-origin already push per-tool/per-field semantics into tools.json. A sibling like {"failure_when": {"pointer": "/status", "in": ["declined","failed"]}} evaluated inside is_structured_error keeps the decision structural and declared once per tool. The honest fallback you describe is the important half: a side-effecting tool with no predicate should emit a per-call suppression ("no failure predicate for tool X"), not a silent pass — which closes the fail-closed gap precisely where it matters. I'm going to prototype this. Best critique the post has gotten — thank you.
One caution on the predicate before you build it: it moves the trust rather than removing it. Whoever writes tools.json now owns the definition of failure for that tool, which is the same self-authorship I flagged in the span writer, one floor up. It is still clearly better, and for a reason worth being precise about. The declaration happens once, ahead of any run, in a file that gets reviewed and diffed. A wrong failure_when is a bug someone can see and argue with; a wrong per-call judgement is invisible.
There is a class the pointer can't reach either way. It only catches failures the payload names. {"status":"declined"} announces itself. What stays dark is a 200 with a well-formed, entirely plausible success body that is wrong about the world: the transfer landed on the wrong account, or the write only partially applied. No JSON pointer into that response finds it, because the failure isn't sitting in the response. It's in the distance between the response and the actual state, and the response is still the caller's own account of what it did.
A static linter can't check whether that account is true. What it can check is whether anything ever went and looked. For a side-effecting call, is there a later read that confirms the effect through a path other than the one that wrote it? The write API echoing its own result back shouldn't count, since that is the same channel grading itself. Fetching the resource by id afterwards, or reading a balance, at least brings something else into it.
Score the presence of that read-back and not its verdict, because the verdict is precisely what you cannot evaluate statically. Then "side effects with no independent observation" becomes a countable property of a trace, and one you can watch over releases: if it climbs as tools get added, the trace vocabulary is falling behind the tool surface. It also composes with what you already have. Per-call suppression names the tools with no predicate. This names the effects nobody checked, and it is expressible in the same per-tool vocabulary, as a declaration of what a confirming read of that effect looks like.
Yes — "moves the trust, doesn't remove it" is the honest way to put it. failure_when relocates authorship to tools.json, so whoever writes that file now owns the definition of failure. What I'd defend is exactly your reason: the declaration happens once, ahead of the run, in a file that's reviewed and diffed. A wrong failure_when is a visible, arguable bug; a wrong per-call judgment is invisible. The trust becomes legible, not absent — and it's worth being precise that that's a smaller claim than "removed."
And you've named the boundary I don't think a static linter crosses: a 200 with a well-formed, plausible success body that's wrong about the world — the transfer on the wrong account, the partial write. The failure isn't in the response, so no pointer into the response reaches it. The response is the caller's own account of what it did, and a check over the response can only grade the account, never the world.
The read-back idea is the part I keep turning over, though, because it's the first thing that stays static and still gets traction here. The linter can't verify the account is true — but it can check whether anything independent ever went and looked. For a side-effecting call, is there a later read of the same resource through a different path than the one that wrote it? The write echoing its own result is the same channel grading itself and shouldn't count; a later get_transfer(id) or a balance read brings a second source in. And scoring the presence of that read and never its verdict is exactly right — the verdict is what's unavailable statically.
What makes it real for me is that it's countable and it composes. "Side-effecting calls with no independent observation" is a property you can put a number on and watch across releases: if it climbs as tools get added, the trace vocabulary is falling behind the tool surface — useful on its own. And it drops into the vocabulary I already have: per-call suppression names the tools with no failure predicate; this names the effects nobody confirmed, declared the same per-tool way — the write tool states what a confirming read looks like (a later call to tool X keyed on the id it returned), and tracelint counts the writes that never got one.
The one honesty tax I'd state up front: it measures confirmation within the trace. A rising count could mean the vocabulary is behind, or that confirmations happen off-trace where the linter can't see them — the same instrumentation ceiling as everything else here. But as a countable, declarable, fail-closed signal, this is the strongest idea anyone's put in these comments, and it's the one I most want to build next.
Different path is doing a lot of work here. In this design it is another declared property, like failure_when, since two spans can name different tools while sharing the same session, same credential, same connection, and same client-side cache. A read served from the cache populated by the write is just the write echo with a fresh label, and it still passes the count. The property I actually want is counterfactual: could this read have returned a different result if the write had never actually happened? I do not think a static linter can compute that in general. The useful move is still declarative: make the confirming read declare what it does not share with the writer, endpoint and credential and connection and cache boundary. Then the diff contains an independence claim that can be argued with, instead of smuggling that claim through a tool name.
I also think the metric needs pressure from the cheapest way to move it. A number is only worth what it costs to improve without improving the underlying behavior. If presence is scored and verdict is ignored, the cheapest improvement is to add a read whose output dies immediately. That is the original defect shifted one hop over. The observation exists in the trace, then gets dropped at the consumption point.
That part is visible statically. Agreement is unavailable, yes. Consumption is often available: does the read output appear in later inputs, or does later control flow depend on it? I would split the count into effects with no independent observation and observations with no downstream consumption. The second bucket is the easy one to fake, so mixing it into the first makes the main number look healthier than the run really is.
The honesty tax should follow the same pattern. Off-trace confirmation should be a declared exemption in the same diffable vocabulary, with undeclared cases still counted as unconfirmed. That does not prove the exemption is true. It does make the escape hatch visible, and it decomposes the headline number into unconfirmed effects and declared-exempt effects instead of hiding both inside one count.
This is a great framing: treating silent tool-call failures as a testable, deterministic signal instead of relying on the LLM to self-report. The CI gating angle makes it practical for teams that already have regression pipelines.
Thanks — "testable, deterministic signal instead of self-report" is exactly the framing I was going for. The regression-pipeline angle is the part I care about most too: if you're already running the agent in CI, you're already producing the trace, so linting it is one more assertion step, not new infrastructure.
The one gap I keep hitting is that a lot of those pipelines run the agent and then throw the trace away — capturing it is the actual work. Once it's persisted, the check is basically free.
The candidate-versus-verdict split is exactly right. I would add one temporal check to the trace contract: a fresh heartbeat must not count as progress. A worker can stay responsive while repeating the same call or waiting on a child process, so I would persist a monotonic progress sequence only when a meaningful boundary is recorded, then let CI or the watchdog flag fresh liveness with a stale sequence as STALLED. That keeps a stuck loop visible without asking a second model to guess.
This is the boundary I'd underline hardest. A structurally clean trace is evidence the run didn't self-contradict — not that it accomplished the task. The tool says so out loud: structural ≠ correct, and the recovery scorecard only claims correctness when you hand it a success oracle. Your "deterministic evidence producer, then a separate completion contract" split is the right architecture. Checks like "did the artifact land at the recorded commit" or "did an authorized reviewer accept it" are contracts over external state; folding them into trace lint would just make it lie more confidently. Keep them separate.
The HTTP-200-carrying-a-failure case is the one that keeps biting. Half the MCP tools I've wired up report errors as plain text in the content field: transport fine, no error status, just a string starting with "Error:". A per-tool failure predicate only works if the tool author declares it, and in practice they don't. We ended up with a crude heuristic that regexes the first chunk of the result for error/failed/exception and marks the span suspect. False positives, but it beats the agent cheerfully summarizing a stack trace.
Also glad suppression is loud rather than a silent pass. A clean report with unrun rules is worse than no report.
On R3: how strict is "derivable from anything observed"? Models normalize values constantly. User says "next Friday", the call carries 2026-08-28. String-level provenance would flag that as hallucinated. Do you compare semantically at all, or is that exactly the class that stays candidate-only?
The HTTP-200-with-a-text-error case is the one, yeah. A couple of things there:
tracelint already does roughly what your heuristic does, just tiered. R2a runs a regex over string result content (traceback / *Error / http 4xx–5xx / errno) and emits a candidate — flagged possible-false-positive, shown with the matched text, and it never fails CI on its own. Same "mark it suspect, accept some FPs, beats a silent pass" instinct. One gap you'd hit immediately: my pattern matches error/exception but not "failed"/"failure" — trivial add, I'll make it.
On the predicate not getting declared — worth clarifying that failure_when lives in your tools.json, not the tool author's. You declare how a tool you've wired up reports failure, so third-party MCP tools aren't a blocker in principle. The real blocker is that the predicate today only does structured matches (a JSON pointer like /status ∈ {…}), which is useless against a bare "Error: …" string. It needs a contains/matches mode to cover free text — which is exactly your MCP case. That would turn your regex-the-first-chunk heuristic into a per-tool declared signal instead of a global guess. Adding it to the list.
And glad the loud-suppression call landed — that one's non-negotiable for me.
On R3: it's strictly value-level, and the transform set is deliberately bounded — exact match (after case/whitespace normalization), digit-reformat (so 1,234.56 == 1234.56), substring extraction, and concatenation of two observed values. No semantics. So "next Friday" → 2026-08-28 is exactly the class that stays candidate-only: the date isn't in anything observed and no bounded transform reaches it, so it's flagged possible-false-positive, never asserted. That's on purpose — resolving "next Friday" needs a reference date and a calendar, i.e. assumptions that aren't in the trace and aren't deterministic, which is the point where I'd be doing the model's job with worse tools. The one thing I'd warn against: don't annotate a date/unit field x-value-origin: provided. "provided" promotes underivable → hard defect, and normalized values are precisely where that fires falsely. Reserve "provided" for values that must arrive verbatim from context — an order id, an account number — not ones the model legitimately reformats.
The “suppressed, not clean” distinction is the strongest design choice here. In our coding runs, we found one more boundary: a structurally clean trace still did not prove that the expected artifact existed at the recorded commit or that an authorized reviewer had accepted it. Treating trace lint as a deterministic evidence producer, followed by a separate completion contract, worked well for that split.
Good distinction — liveness isn't progress. R4 leans on that in a narrow way already: "no progress" keys on the coarse result class, so a poll advancing pending → completed changes class and isn't flagged, while three identical oks with identical args is. But that's per-call, not a persisted monotonic sequence across the run. Your version — bump a progress counter only at a meaningful boundary, then flag fresh liveness against a stale counter as STALLED — catches the "responsive but not advancing" case that per-call identity misses (waiting on a child process, re-emitting the same call). That's a cleaner contract; I'm noting it for the trace model.
The ignored tool error is the failure mode that makes chat summaries look fine while the run is already wrong. I have watched agents treat a failed write or a 4xx as a soft signal and keep going, then report success because the last sentence sounded confident. Linting the trace for "error then proceed as if ok" is the right layer for CI, because that class of bug is decidable without a second model. A judge that shares the writer's blind spots will not catch it reliably. The receipt has to come from the tool result itself.
"The receipt has to come from the tool result itself" is the whole thesis in seven words. A judge built from the same model that wrote the confident final sentence shares its blind spot — it rates the summary, not the run. One caveat another commenter surfaced: the receipt only works if the failure is actually encoded in the result structurally. A decline buried in a 200 body needs the tool to declare what failure looks like, or the linter can't read the receipt either. But when the signal is there, reading it deterministically beats asking a second model to guess.
This is a sensible boundary for CI. If a failure is decidable from the trace, adding a judge mostly adds variance and cost. The harder bit is deciding which warnings are allowed to block a release.
Right, and that's deliberately policy, not baked in. The default is conservative — only structurally-provable defects block; everything else is advisory. "Which warnings graduate into the gate" should follow a team's own false-positive tolerance and its own declared tools, not a default the linter imposes. The tool's job is well-tiered evidence with the receipts attached; the release policy on top is yours.
Silent tool-call failures are one of the nastiest agent bugs because the run still returns something that looks like success. I started asserting on the tool result and not just the final answer, which caught a whole class of cases where the agent quietly worked around a broken call. Putting that check in CI the way you describe is the part most people skip until it burns them.
"Assert on the tool result, not just the final answer" is the whole shift. The final answer is written by the same model that made the mistake — it'll narrate success right over a failed call, so it's the least reliable place to check. The tool result is the ground truth.
That's exactly why tracelint reads the trace instead of grading the output: "tool returned an error, then the agent proceeded (or reused that value in a later action)" is decidable from the trace with no model in the loop. And you're right that moving it into CI is the part people skip — usually because asserting per-result by hand is tedious, which is what I'm trying to make declarative.
One sharp edge from your "quietly worked around a broken call": the check only fires if the failure is actually encoded in the result. A 500 or an error field, sure — but a decline that comes back as HTTP 200 with {"status":"declined"} reads as success unless the tool declares what failure means. So "assert on the result" quietly assumes the result is honest about failing — teaching the checker that (a per-tool failure predicate) is the one place a human still has to weigh in, and it's what I just added.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.