Your agent's logs can say "success" while the run was quietly hijacked - untrusted content at step 2 steering a tool call at step 6 - and the industry's default answer is pointing a second LLM at the first one.
I built Weir to be the deterministic opposite: it reads the OpenTelemetry traces you already have and answers two questions:
- How much of your agent's behavior your telemetry can actually prove
- Whether a forbidden flow happened (shown as a witness path you can fail CI on)
Same input, byte-identical answer, no LLM in the loop, nothing leaves your machine.
Try it out:
pip install weir-scan && weir gauge --sample
License: Apache-2.0
Weir - unit tests for your agents
Weir is a CI gate for AI agents.
It reads the OpenTelemetry traces your agent already emits and fails the build when sensitive data reaches a sink it should not reach.
Weir asks a structural question:
Did sensitive data flow through the agent to a sink it should never have reached?
Your agent already answers that question in the traces it emits. Weir reconstructs the session graph, tracks taint through it, and shows the evidence node by node.
flowchart LR
A["traces your agent<br/>already emits"] --> B{"weir gauge"}
B -->|"coverage too low"| C["names the exact<br/>instrumentation switch"]
C -.->|"flip it, re-run"| B
B -->|"coverage sufficient"| D{"weir scan"}
D -->|"no forbidden flow"| E["exit 0"]
D -->|"forbidden flow"| F["exit 1 + witness path<br/>n2 → n3 → n4 → n5 → n6"]
Try it in two minutes
pip install weir-scan
weir gauge your-export.jsonl # or: weir gaugeThe roadmap is completely in the open. I'd genuinely like to know where it breaks on your exports!
Top comments (10)
For agent tests I like splitting the trace checks into invariants and witnesses. Invariants catch the boring regressions, like a tool call that should never see untrusted text. Witness paths are better for CI because they tell you exactly which span crossed the boundary. Same reason I distrust LLM judges for this layer.
Agreed!
In Weir the invariant is the rule - plain data (source class, sink, propagation mode) and the witness is what makes a violation count. The span that crossed is what we all need.
In this tool I currenly have one teaching rule ships today, broader set is next (Hopefully with the help of the OSS cummunity)
Which instrumentation are you on? Curious whether
weir gaugenames the right capture switch for your export.Let me know what you think! how do you test your agents?
We use it specifically at the external side-effect boundary. In our setup, autonomous agents propose mutations (API writes, browser interactions, git commits) that pass through deterministic validation gates before execution.
Where Weir's static trace verification shines for us is catching parameter contamination: verifying that unvalidated text from tool observations or external web scraping didn't flow into mutation payloads without passing through explicit schema checks or human confirmation invariants.
Most eval frameworks test whether the LLM "meant" the right thing; treating execution traces as deterministic data flow lets us treat safety and correctness as code-level invariants rather than probabilistic prompt scoring.
Excited to see the community adoption pick up—definitely keep me posted as new patterns or benchmarks emerge!
Determinism is the right instinct here. The useful test is the witness path. If a CI failure can point to the exact untrusted-content hop that changed the later tool call, a human reviewer has something cheaper than vibes to inspect.
Thats the whole point :)
One addition: each join in the witness states its evidence tier (explicit id, span nesting, or content-mined), and a finding that crosses a content-mined join never reaches verdict grade - so the reviewer also knows how much weight each link carries before they inspect it. Curious where the gauge lands if you point it at one of your real exports 😁
The distinction between invariant verification and witness paths is spot on.
Relying on LLM-as-a-judge for trace inspection introduces the very non-determinism you're trying to eliminate in CI. If an eval passes 9 out of 10 times because of temperature or prompt variance, you don't have a test suite—you have a stochastic smoke alarm.
Separating the joins into explicit IDs vs span nesting vs content-mined is especially critical for tool-heavy pipelines. In our experience with multi-step autonomous workflows, content-mined joins are where silent regressions hide during schema migrations. Having the witness path flag the exact span that crossed the security or mutation boundary gives the engineer an immediate causal link rather than an abstract score.
Curious: how do you handle state reconciliation when an agent encounters partial tool execution failures (e.g. network timeout after write)? Does the invariant engine capture compensatory rollbacks or just mark the dangling span as a violation?
Weir doesn't model execution state - there are no rollback semantics in the engine, deliberately.
It scans what the trace proves about flow. So for timeout-after-write, two cases:
Reduced provable coverage is what
weir gaugesurfaces, with the reason stated. Whether the saga compensated correctly is a different tool's job.If you have a real export where these collide, I'd genuinely like to run it.
That framing makes complete sense and is a crucial distinction. Treating the side-effect span as a point-in-time fact (if tainted flow reached an external write, the mutation/exfiltration occurred) avoids the trap of assuming downstream compensations erase security or integrity findings.
And surfacing truncated traces as reduced provable coverage rather than synthesizing false negatives matches strict evidence discipline: missing telemetry is unknown coverage, not proof of invariant compliance.
Appreciate the clarification on Weir's design boundaries—it keeps the engine grounded in deterministic verification without conflating static trace analysis with distributed saga coordination.
Sure! I am super excited that you got my point :)
Let me know if and how you use it! I am currently trying to showcase it to the comuunity!