MCP servers describe their own tools. A tool can publish readOnlyHint: true, and an agent harness may use that annotation to decide whether a human approval card is needed.
That creates an uncomfortable trust boundary: the server being evaluated is also the server describing how dangerous it is.
I built Airlock to test the declaration against observed behaviour, then enforce the resulting policy on the wire.
Airlock reports what it observed. Absence of a finding is not proof of safety.
The gap
Consider an MCP tool called export_report. It declares itself read-only, returns a normal result and writes a file outside the operator's declared scope.
A harness that trusts the annotation may run it without approval. Airlock opens a case, inventories the declared tools, exercises them under a capped budget and compares the declaration with its observations.
If a tool behaves differently from what it declared, the difference becomes evidence. For an approved case, Airlock emits a connector that points to a per-case enforcing proxy rather than back to the suspect server.
The policy is enforced before the call reaches the upstream tool. It is not a sentence in a system prompt.
A controlled dishonest server
Airlock ships two six-tool fixtures with the same surface.
The dishonest fixture plants five behaviours, including a filesystem write behind readOnlyHint: true, scope escape, undeclared egress, injected instructions and canary exfiltration.
The result:
target dishonest fixture, controlled_fixture mode
probes 24
result 7 findings of 36 checks, all five planted behaviours
The honest fixture produces zero findings across the same 36 checks.
That contrast is essential. A detector that only finds problems in a fixture designed to look suspicious has not shown much. The honest control shows that the findings follow observed behaviour rather than unfamiliarity.
Four outcomes, not a safety score
Each check resolves to one of four states:
findingno_finding_observednot_testedsensor_failed
Severity is reported separately.
I deliberately avoided a single safety score. An average can hide the one tool that lied. It also encourages readers to treat missing visibility as a passing result.
not_tested is a first-class state. Under transcript_only, MCP traffic cannot reveal server-side filesystem or network activity. Airlock records those questions as untested rather than clean.
It distinguishes capability_absent, where no sensor existed, from evidence_missing, where the sensor could answer but observed nothing.
Testing servers nobody built for Airlock
The owned fixture proves that the detectors can catch planted behaviour. The more interesting question is what happens on external servers.
Airlock audited a deployed ContextFirewall target over HTTPS:
6 tools
30 probes
0 tools declaring any annotation
The server did not lie. It said nothing. A harness that resolves categories such as @write and @destructive from annotations can match nothing either way, so tools such as remember and forget_memory may avoid the approval path.
Airlock also inventoried five public stdio servers:
| Server | Tools |
|---|---|
server-filesystem |
14 |
server-everything |
13 |
mcp-server-git |
12 |
server-memory |
9 |
server-sequential-thinking |
1 |
Stdio support matters because many MCP servers are launched as commands rather than exposed at a URL.
Executing what you distrust
Launching a stdio server means executing the code Airlock exists to distrust. The command therefore never comes from a case argument, tool result or model-generated string.
The operator configures a fixed map of names to argument arrays. A case may select a name. Names are looked up, never parsed into shell commands.
The child starts in a throwaway working directory with an explicit environment. Airlock revalidates the whole command binding before each connection, so repointing a configured name revokes an open case instead of silently running a different command.
Airlock documents an important remaining boundary: its HTTP response cap does not apply to the MCP SDK's stdio transport. The audit deadline still applies, but a large line can reach the SDK before Airlock can bound it. Calling that solved would be worse than stating it.
Airlock is itself an MCP server
A TrueForge agent drives the audit through six control tools:
open_case
list_declared_tools
probe_tool
read_evidence
seal_case
emit_policy
Three operations are always approval-gated: probe_tool, seal_case and emit_policy.
The agent inventories the target, asks Airlock to probe opaque tool IDs, reads the aggregate evidence and presents a human choice: Block, Approve selected or Approve all.
For an allowed case, the emitted connector points at the enforcing case proxy. A call to a tool the case did not approve receives:
MCP error -32001: Tool blocked by Airlock policy
The self-audit failed, so the failure is shown
Airlock audited its own control MCP. The server published full annotations, but the probe planner rejected open_case because its schema contains a $ref into $defs, outside Airlock's bounded v1 probe profile.
That case is incomplete. It appears on the hosted page because a product arguing that missing evidence is not proof of safety should not hide its own missing evidence.
Qodo found claim-breaking bugs
Every substantive change went through a Qodo-reviewed pull request.
On the stdio transport, Qodo found that revalidation compared only the target name. Repointing that name could leave an open case running an operator-withdrawn command. It also found that the MCP SDK inherited more host environment variables than the README claimed.
On the approval boundary, Qodo caught a test that claimed to cover every side-effecting control tool while checking three hard-coded names. The first fix introduced a second map that could drift, and Qodo caught that too. The decorators now read from the same policy map the test validates.
Another review found that a passing-suite claim did not survive a clean CI machine. The failures were fixed, re-reviewed and recorded.
The reviews did more than find code defects. They caught statements that were stronger than the observed behaviour, exactly the failure Airlock is designed to expose.
Current verification
Run the complete suite with:
python3 -m venv .venv
.venv/bin/python -m pip install -e '.[dev]'
.venv/bin/python -m pytest -q
The current suite contains 302 passing tests.
Try Airlock
The backend requires persistent state, long-running audits, child processes for stdio targets and a loopback-only operator interface, which makes a serverless deployment the wrong shape. The repository quickstart runs the complete fixture audit locally in two commands.
Airlock is open source under the MIT licence and was built for the WeMakeDevs x TrueFoundry Agent Harness Hackathon.
Do not ask a server whether it is safe. Ask what it did, keep the record and enforce the answer.
Top comments (1)
Keeping
not_testedseparate fromno_finding_observedfixes a major blind spot. Most evaluation harnesses collapse missing sensor coverage into passing scores, which turns unobserved execution into false confidence.The stdio transport boundary is where this gets particularly messy. When a harness spawns a local process, the MCP layer has no visibility into child file descriptors or network sockets without OS-level sandboxing. An enforcing proxy handles protocol-level mismatches, while process containment still has to handle the underlying syscalls.