Your agent's logs can say "success" while the run was quietly hijacked - untrusted content at step 2 steering a tool call at step 6 - and the industry's default answer is pointing a second LLM at the first one.
I built Weir to be the deterministic opposite: it reads the OpenTelemetry traces you already have and answers two questions:
- How much of your agent's behavior your telemetry can actually prove
- Whether a forbidden flow happened (shown as a witness path you can fail CI on)
Same input, byte-identical answer, no LLM in the loop, nothing leaves your machine.
Try it out:
pip install weir-scan && weir gauge --sample
License: Apache-2.0
Weir - unit tests for your agents
Weir is a CI gate for AI agents.
It reads the OpenTelemetry traces your agent already emits and fails the build when sensitive data reaches a sink it should not reach.
Weir asks a structural question:
Did sensitive data flow through the agent to a sink it should never have reached?
Your agent already answers that question in the traces it emits. Weir reconstructs the session graph, tracks taint through it, and shows the evidence node by node.
flowchart LR
A["traces your agent<br/>already emits"] --> B{"weir gauge"}
B -->|"coverage too low"| C["names the exact<br/>instrumentation switch"]
C -.->|"flip it, re-run"| B
B -->|"coverage sufficient"| D{"weir scan"}
D -->|"no forbidden flow"| E["exit 0"]
D -->|"forbidden flow"| F["exit 1 + witness path<br/>n2 → n3 → n4 → n5 → n6"]
Try it in two minutes
pip install weir-scan
weir gauge your-export.jsonl # or: weir gaugeThe roadmap is completely in the open. I'd genuinely like to know where it breaks on your exports!
Top comments (5)
For agent tests I like splitting the trace checks into invariants and witnesses. Invariants catch the boring regressions, like a tool call that should never see untrusted text. Witness paths are better for CI because they tell you exactly which span crossed the boundary. Same reason I distrust LLM judges for this layer.
Agreed!
In Weir the invariant is the rule - plain data (source class, sink, propagation mode) and the witness is what makes a violation count. The span that crossed is what we all need.
In this tool I currenly have one teaching rule ships today, broader set is next (Hopefully with the help of the OSS cummunity)
Which instrumentation are you on? Curious whether
weir gaugenames the right capture switch for your export.Let me know what you think! how do you test your agents?
Determinism is the right instinct here. The useful test is the witness path. If a CI failure can point to the exact untrusted-content hop that changed the later tool call, a human reviewer has something cheaper than vibes to inspect.
Thats the whole point :)
One addition: each join in the witness states its evidence tier (explicit id, span nesting, or content-mined), and a finding that crosses a content-mined join never reaches verdict grade - so the reviewer also knows how much weight each link carries before they inspect it. Curious where the gauge lands if you point it at one of your real exports 😁