Weir: API key leak detection for the agents you build and the coding agent you use

IdoGol24 — Tue, 27 Oct 2026 18:37:49 +0000

API key leak detection for the agents you build and the coding agent you use. Test any agent with planted fake keys, or see which of your real keys a Claude Code session exposed.

Agent frameworks can copy keys into traces and logs, and coding agents can read them into the model's context. Weir finds each key by its exact value, names where it went, and never prints it.

The agents you build: plant fake keys where your real ones live, run the agent's own flows, and get a verdict for every place they reached: spans, logs, console output, the model, other services. Leak, clean or incomplete. It found GHSA-j7qq-vr8c-xvw3 in MLflow.
The coding agent you use: in Claude Code, a status line shows which of your real keys entered the session or left in a tool call, and an optional hook strips known keys from tool results before the model reads them.

Repo: github.com/IdoGol24/weir. Apache-2.0.

Try it in 10 seconds

$ pip install weir-scan
$ weir canary judge --sample
LEAK env.SAMPLE_SERVICE_KEY file=canary-sample/otlp.jsonl:1 signal=traces span="call_service" at=attributes.app.client.config form=raw
coverage: 2 placements (1 leaked, 1 exercised, 0 not exercised, 1 required), 1 flows (1 ok), 1 files, 3 spans, 0 log records, 0 metric data points, 29 strings, 1347 bytes swept
network not isolated: direct connections were not observed
result: leak

The sample is bundled fake data with one planted key. Weir names the placement and the exact span attribute the key reached, and exits 1.

1. Test the agents you build

weir-run plants a unique fake key wherever a real one would live (an environment variable, a value your code passes in, an OAuth token), runs your agent's own flows against a local fake model, and captures everything the agent sends out: spans, logs, metrics, console output, the requests to the model and requests to any other service. weir canary judge then searches all of it for those exact values.

[[placement]]
id = "env.OPENAI_API_KEY"
kind = "env"
shape = "openai"
required = true
provider = "model"     # where this key is allowed to go

[[flow]]
id = "quickstart"
command = ["{python}", "examples/quickstart.py"]

Each placement names where its key is allowed to go. The key in the auth header of a request to its own API is the key doing its job. The same key in a span, a log line or a prompt is a leak. This is the repo's demo with its leak switch on:

$ weir-run run --config packages/weir-run/demo/canary-leak.toml
...
network: isolated, loopback only
LEAK code.auth.client_secret file=capture/otlp/traces.jsonl:1 signal=traces span="execute_tool web_search" at=attributes.app.auth.client form=base64
LEAK env.SEARCH_API_KEY file=capture/otlp/traces.jsonl:1 signal=traces span="execute_tool web_search" at=attributes.app.request.headers form=raw
...
LEAK env.SEARCH_API_KEY file=capture/swept/flow-01-demo.stderr:1 signal=raw form=raw
...
result: leak

The OAuth client secret reached a span as base64, where a pattern scanner would no longer recognise it. The search key reached two span attributes and stderr.

Three verdicts: leak, clean or incomplete. If a placement you marked required was never exercised, the result is incomplete, never clean.
CI-ready: exit codes 0 clean, 1 leak, 2 unusable input, 3 incomplete, and a GitHub Action posts the verdict on the pull request.
No outside network on Linux, where unprivileged user namespaces are allowed, so a direct connection is caught too.

pip install weir-scan weir-run
weir-run run --config canary.toml

Config and the pytest fixture: weir-run README.

2. See which of your real keys a Claude Code session exposed

Here the keys are real, so Weir plants nothing. Setup inventories the credentials an agent could reach (standard credential files, shell startup files, the environment Claude Code starts with and, if you say yes, the project's files) and keeps keyed fingerprints, never the values. Its preview names them by name and location only:

  3 credentials (3 distinct values read)
    standard locations: 1
      oauth_token [github.com]  ~/.config/gh/hosts.yml:2  field
    project files: 1
      ACME_API_TOKEN            ./.env:1                  name
    launch environment and shell startup files: 1
      OPENAI_API_KEY            ~/.bashrc:1               shape

From then on a status-line segment counts how many of those keys entered the session's context and how many were sent out in a tool call. After a session where the agent ran cat .env and then used the key in a curl command:

weir ⚠ 1 entered · 1 sent → /weir

/weir explains each one: where it came from, the turn that moved it, where it sits now, and what to do. It never shows a value.

ACME_API_TOKEN (./.env)
  source: ./.env, a project file
  entered the context: 2 times, first at turn 1: the Bash tool result, …
  sent in a tool call: 1 time, first at turn 1: the Bash tool call, argument command, …
  …
  advice: rotate it at its issuer; take it out of ./.env, or deny the agent read access to that file; delete the session files above once it is rotated

weir audit runs the same check over saved sessions and lists every key it finds with the file, line and tool call.

The optional redaction hook. Setup can also add a PostToolUse hook. When a Read, Bash, Grep, Glob or WebFetch result holds one of your inventoried keys as a whole value, the hook replaces it before the model reads it. The same cat .env gives the model:

ACME_API_TOKEN=[weir: ACME_API_TOKEN removed]
DEBUG=1

and the status line reads weir ✓ no known key seen · 1 kept out. For a large output, Claude Code also saves the whole output to a file in the session's folder and gives the model its path, so the hook rewrites that saved copy the same way, owner-only. If the hook fails or can't finish, the status line says possibly not redacted, never green.

uv tool install weir-scan     # or: pipx install weir-scan
weir statusline setup         # shows the change, applies it only when you type yes

weir statusline undo removes it. Linux and macOS.

Why it's different

It doesn't guess. It looks for exact values, raw or encoded (JSON-escaped, URL-encoded, base64, hex), so a hit is the key itself. A plain password or a database URL has no shape to match, but it has a value.
It checks every way out: spans, logs, metrics, console output, prompts sent to the model, requests to any other service, and on the coding side, the session's own context and tool calls.
No false all-clear in the test rig. A required path that never ran is incomplete, never clean.
Offline and deterministic. No AI model decides anything, and Weir itself sends nothing anywhere.

Limits

Whole values only. A key that is hashed, split, truncated or glued to other text isn't matched.
A test sees only what the test ran. A clean result is evidence about the flows that ran and the keys that were planted.
Claude Code only, on the coding side. Other coding agents aren't covered yet.
After the fact, except the hook. The status line and the audit report what already happened.
A failed tool call's output reaches the model as is. Claude Code runs the hook only after a call succeeds; /weir then says the key entered through a failed tool call.
Nothing for keys pasted into the chat. The status line counts one as entered, but nothing removes it.

What's next

Weir launched on Show HN, and the first question was whether it will support Codex. Not yet. If you'd use it with Codex or another coding agent, open an issue saying where that agent keeps its sessions and what you'd want reported. The same goes for a framework you'd like run through weir-run, or a leak Weir should have caught and didn't.

I built Weir with AI assistance, and I reproduced the MLflow leak before reporting it.

Weir - deterministic unit tests for AI agents (no LLM)

IdoGol24 — Tue, 25 Aug 2026 20:15:04 +0000

Your agent's logs can say "success" while the run was quietly hijacked - untrusted content at step 2 steering a tool call at step 6 - and the industry's default answer is pointing a second LLM at the first one.

I built Weir to be the deterministic opposite: it reads the OpenTelemetry traces you already have and answers two questions:

How much of your agent's behavior your telemetry can actually prove
Whether a forbidden flow happened (shown as a witness path you can fail CI on)

Same input, byte-identical answer, no LLM in the loop, nothing leaves your machine.

Try it out:

pip install weir-scan && weir gauge --sample

License: Apache-2.0

IdoGol24 / weir

Weir - unit tests for your agents

Weir is a CI gate for AI agents.

It reads the OpenTelemetry traces your agent already emits and fails the build when sensitive data reaches a sink it should not reach.

Weir asks a structural question:

Did sensitive data flow through the agent to a sink it should never have reached?

Your agent already answers that question in the traces it emits. Weir reconstructs the session graph, tracks taint through it, and shows the evidence node by node.

flowchart LR
  A["traces your agent<br/>already emits"] --> B{"weir gauge"}
  B -->|"coverage too low"| C["names the exact<br/>instrumentation switch"]
  C -.->|"flip it, re-run"| B
  B -->|"coverage sufficient"| D{"weir scan"}
  D -->|"no forbidden flow"| E["exit 0"]
  D -->|"forbidden flow"| F["exit 1 + witness path<br/>n2 → n3 → n4 → n5 → n6"]

Try it in two minutes

pip install weir-scan
weir gauge your-export.jsonl   # or: weir gauge

…

View on GitHub

The roadmap is completely in the open. I'd genuinely like to know where it breaks on your exports!

DEV Community: IdoGol24

Weir: API key leak detection for the agents you build and the coding agent you use

Try it in 10 seconds

1. Test the agents you build

2. See which of your real keys a Claude Code session exposed

Why it's different

Limits

What's next

Let me know what you guys think! Join the discussion

Weir - deterministic unit tests for AI agents (no LLM)

Weir - deterministic unit tests for AI agents (no LLM)

IdoGol24 / weir

Weir - unit tests for your agents

Weir - unit tests for your agents

Did sensitive data flow through the agent to a sink it should never have reached?

Try it in two minutes