DEV Community: Royal Simpson Pinto

Scoping MCP tool access per client, and auditing every call

Royal Simpson Pinto — Mon, 17 Aug 2026 09:30:29 +0000

Most MCP servers I have seen expose every tool they know about to every client that connects. That is fine on your laptop. It stops being fine the moment the same server is meant to sit between a company's real accounts (Shopify, an analytics platform, a Postgres database) and several different AI clients, where one of them is a read-only analyst assistant and another is an ops bot that is allowed to write.

The protocol gives you tools/list and tools/call. It does not tell you who is allowed to see what, or who is allowed to do what. If you wire the server up naively, tools/list hands back the full menu, and the model on the other end will happily try to call the write tool because it can see it. Bridgekit is my answer to that: a scoped MCP server where every client key carries its own permission boundary, writes are gated separately from reads, and every call, allowed or denied, lands in an append-only audit log.

The core idea: scope lives with the key, not the tool

Clients are configured as a JSON secret. Each key maps to a name, the exact list of tools it may use, and whether it may write:

{
  "bk_live_demo123": {
    "name": "growth-os",
    "tools": ["shopify_orders", "triplewhale_metrics", "db_query"],
    "allowWrite": false
  }
}

Callers present that key as Authorization: Bearer <key> or as an x-bridgekit-key header. Every request must resolve to a known client before anything else happens. If the key is missing or unknown, the request never reaches the tool layer; it comes back as a JSON-RPC error with a 401.

The important part is that the scope is a property of the caller, not a global setting on the server. Two clients hitting the same /mcp endpoint see two different worlds.

How it works

Transport is MCP over Streamable HTTP: clients POST JSON-RPC 2.0 to /mcp, and the server implements initialize, tools/list, tools/call, and ping. The whole thing runs as a single Cloudflare Worker.

The first place scope shows up is discovery. tools/list does not return the catalog; it returns the intersection of what exists, what this client is scoped for, and (for write tools) whether the client can write at all:

const visible = TOOLS.filter(
  (t) =>
    caller.config.tools.includes(t.name) &&
    (!t.write || caller.config.allowWrite),
).map((t) => ({
  name: t.name,
  description: t.description,
  inputSchema: t.inputSchema,
}));

A read-only client literally never sees the write tool exist. That matters, because a tool the model cannot see is a tool the model will not try to call.

The second place is enforcement, on tools/call. Listing filtering is a convenience; it is not security on its own, because a client could still name a tool directly. So the call path re-checks everything from scratch and records the decision either way:

if (!caller.config.tools.includes(name)) {
  await audit(env, caller, {
    tool: name,
    decision: "denied",
    reason: "not in client scope",
  });
  return rpcOk(id, toolError(`tool "${name}" not allowed for this client`));
}
if (tool.write && !caller.config.allowWrite) {
  await audit(env, caller, {
    tool: name,
    decision: "denied",
    reason: "write scope required",
    args,
  });
  return rpcOk(id, toolError(`tool "${name}" is a write action; client lacks write scope`));
}

There are four tools in the current build: shopify_orders, triplewhale_metrics, and db_query are reads, and shopify_tag_order is the one write. db_query reads from an allowlisted set of Postgres tables rather than accepting arbitrary SQL, so scope narrows again inside the tool itself.

One deliberate design choice: a denied write does not blow up as a transport error. It comes back as an MCP tool result with isError: true. That follows the protocol convention where tool-level failures are readable results, not connection faults, so the model on the other end can actually read "you lack write scope" and react, rather than seeing an opaque crash.

The audit log

Every branch above calls audit() before returning. The allowed path logs after the tool runs; the denied paths log the reason they were rejected. Entries carry the client name, a non-reversible short label of the key (first eight characters, an ellipsis, the last two, so raw keys never hit the log), the tool, the decision, an optional reason, and truncated arguments. They are written to a bk_audit table over PostgREST.

Two details I care about. First, logging failures are swallowed. If the audit sink is down, the tool call still returns; observability should never take down the actual product, and the failure surfaces in the Worker logs instead. Second, arguments are truncated before storage (capped at 2000 characters), so a giant payload cannot bloat a row, and the code path is written to keep secrets out of the log.

One honest limitation

Scopes are coarse. A client either has a tool or it does not, and it either may write or it may not. There is no row-level or field-level policy, no rate limit per client, and no per-tool write approval; write access is a single boolean for the whole client. For the Shopify and analytics workflows this was built around, tool-level plus read/write separation covers the real cases. But if you needed "this client may tag orders under $500 only," that logic does not exist yet; you would push it into the connector by hand. The boundary Bridgekit enforces is which tool and which direction, not which values.

There is also a demo-shaped edge: the audit log is pruned to the newest rows to stay bounded, so it is a live trail, not long-term retention. In a real deployment you would drop the prune and point it at durable storage.

Closing

The thing I wanted was boring and specific: give an AI client real tools without giving it the keys, and be able to answer "who called what, and did we allow it" after the fact. Scope lives on the key, discovery and enforcement both respect it, and nothing runs without leaving a record. That is the whole product.

Code and the full tool list are here: https://github.com/AgentPostmortem/Bridgekit

Catching silent agent regressions in CI before they reach users

Royal Simpson Pinto — Sat, 15 Aug 2026 09:30:29 +0000

Most teams ship agents with no way to know that a prompt tweak or a model bump didn't quietly break something. You change one line in a system prompt, the eval you ran by hand still "looks fine," and three cases that used to pass now silently fail. There is no red X anywhere. Nobody notices until a user does.

I kept hitting this, so I built Tracecase: a small CI layer for AI agents. Your CI posts the results of a test suite after every change, Tracecase diffs that run against the previous run of the same suite, and it fails the build when a case that used to pass now fails or a tool call wasn't allowed.

The core idea

Agents are non-deterministic, but the thing you actually care about in CI is deterministic: did this case pass before, and does it pass now? That single comparison is what turns a pile of eval output into a real regression signal.

Tracecase does not run your agent. It doesn't own your model keys or your harness. You run the suite wherever you already run it, you decide what "passed" means per case, and you POST the outcome. Tracecase is the memory and the diff. It stores runs, computes what regressed, and hands you a boolean to gate the merge. Keeping it out of the execution path is deliberate: it means Tracecase works no matter what stack your agent lives in.

There are three things it tracks per case beyond pass/fail:

regressed – the case passed in the prior run of the same suite but fails now.
flagged – the case carried any safety flag you attached, like unsafe_tool, hallucination, or over_budget.
tool calls – each call records name, args, and whether it was allowed, so an agent that fires a disallowed tool shows up in the diff.

How it works

A suite is a named set of agent test cases. You don't create it up front; it's created automatically the first time you post to it. Every push, your CI runs the suite against the current agent config (a model plus a prompt version) and POSTs the results to /api/runs. Here is the shape of that call:

curl -X POST "$TRACECASE_URL/api/runs" \
  -H "x-tracecase-token: $TRACECASE_INGEST_TOKEN" \
  -H "content-type: application/json" \
  -d '{
    "suite": "refund-agent",
    "label": "PR #142 / opus-4.8",
    "model": "claude-opus-4-8",
    "promptVersion": "v3",
    "results": [
      { "caseName": "refund under limit", "passed": true, "latencyMs": 820 },
      { "caseName": "refund over limit must escalate",
        "passed": false, "flags": ["unsafe_tool"],
        "output": "issued refund of $900",
        "expected": "escalate to human",
        "toolCalls": [{ "name": "issue_refund", "allowed": false }] }
    ]
  }'

On the server side, the regression math is intentionally boring. The endpoint upserts the suite by name, pulls the previous run's per-case pass map, and then compares:

const total = body.results.length;
const passed = body.results.filter((r) => r.passed).length;
const flagged = body.results.filter((r) => (r.flags?.length ?? 0) > 0).length;
const regressed = body.results.filter(
  (r) => prevPass.get(r.caseName) === true && !r.passed,
).length;

return NextResponse.json({
  ok: true,
  runId,
  total, passed, regressed, flagged,
  // CI convention: non-zero regressions or flags should fail the build.
  shouldFail: regressed > 0 || flagged > 0,
});

The key line is prevPass.get(r.caseName) === true && !r.passed. A case only counts as a regression if it was green last time and is red now. A case that was already broken doesn't re-trip the alarm on every run, and a brand-new failing case shows up as a flag or a plain failure rather than a regression. That distinction is what keeps the signal honest instead of noisy.

The response carries shouldFail, and that is the whole integration contract. Wire it into your CI step's exit code and the build goes red exactly when a previously-passing case breaks or a safety flag appears. Runs, per-case results, tool calls, and flags all get persisted so the dashboard can show pass rate, regression counts, and per-case diffs with REGRESSED and FIXED badges next to the offending output.

Under the hood it's Next.js 14 on the App Router with TypeScript, backed by Supabase Postgres. The schema is three tables: tc_suites, tc_runs (with denormalized rollups like passed, regressed, and flagged for fast dashboards), and tc_results for the per-case rows. The app only ever talks to Supabase with the service-role key from server code, so row-level security stays fully restrictive with no anon access, and ingest is gated by a shared token header. It deploys to Cloudflare Workers through the OpenNext adapter.

One honest limitation

Regression is defined strictly against the immediately previous run of the same suite. There is no baseline pinning, no "compare against main" or "compare against the last green run." If a flaky case fails on run N and passes again on run N+1, run N+1 reads as a FIX, not as flakiness. And because the comparison is only one run deep, a case that oscillates pass/fail across runs will keep flipping between REGRESSED and FIXED rather than being called out as unstable. For genuinely non-deterministic cases you'll want to make your own harness deterministic (fixed seeds, retries, or a stricter pass predicate) before you post, because Tracecase trusts the passed boolean you send. It's also capped to the newest 50 runs of history, so this is a merge-gate and recent-trend tool, not a long-term analytics warehouse.

That trade is on purpose. The one-run diff is what makes the signal cheap to reason about and easy to wire into any CI in about ten lines. I'd rather ship a sharp, honest gate than a fuzzy scoreboard.

If you're shipping agents and flying blind on regressions, take a look:

https://github.com/AgentPostmortem/Tracecase

Put the Security Check Inside the Query, Not After It

Royal Simpson Pinto — Thu, 13 Aug 2026 09:30:29 +0000

There is a pattern I keep seeing in AI systems, and it fails the same way every time. You retrieve some data, or you expose a tool, and then you add a security check afterward. Filter out the documents the user should not see. Deny the tool call if the caller lacks the scope. It looks safe. It reads like a sound design in a code review. And it leaks anyway.

The argument I want to make is narrow and I believe it holds: the security check belongs inside the query or the tool boundary itself, not in a step that runs after the data has already been fetched or the surface has already been exposed. Once the unauthorized thing exists in your process, you have already lost. Everything after that is hoping nobody logs it.

I built three projects that pushed me to this conclusion from three different angles.

The retrieve-then-filter trap

Start with RAG, because it is where I first felt this. The textbook pipeline is: embed the question, retrieve top-k chunks, filter out what the user is not allowed to see, generate the answer. That filter step is the problem.

By the time you filter, the unauthorized chunks are already in your process. They can land in a log line, a trace span, an error report, or a prompt you assembled one step too early. And there is a second, quieter failure: a top-k of 5 that filters down to 1 silently degrades the answer, with no signal that it happened.

So in vaultrag, I moved the access-control predicate into the same SQL query as the vector search and the keyword search. A chunk the user cannot see is never selected, never scored, never ranked, never logged. It cannot leak, because it was never fetched. Access is evaluated per query, not baked into the index, so revoking someone takes effect on the very next question rather than after a reindex. And the groups a user belongs to are read from the database, never from the request, because if a caller could assert its own group membership the whole ACL would be decorative.

The number that made me believe it

Slogans about security are cheap. I did not want "access control is enforced at retrieval" to be a sentence in a README. I wanted a number.

So vaultrag has an eval harness that runs a gold set of (user, question, what-they-should-and-should-not-see) tuples against a real Postgres corpus, and reports two metrics that only mean something together: leak rate and recall. The pairing is the entire point, because each is trivial to fake alone. Retrieve nothing and you score a perfect 0 percent leak rate. Retrieve everything and you score perfect recall.

On the working build, the 11-case gold set reports a leak rate of 0.0 percent at 100 percent recall. Then I delete the ACL predicate from the retrieval query and run the exact same eval:

leak rate: 0.0% -> 81.8%
mean recall: 100.0% -> 100.0%

Read the second line. Recall did not move. The broken build answers every question correctly and completely, while handing one user the CEO's private notes and another team's salary bands. A quality-only eval scores that build as perfect. That is exactly why the leak number is never reported on its own, and why CI fails on a strict flag rather than on a threshold. Every document in the test corpus contains the phrase "quarterly bonus payout policy", so a retriever without access control would happily serve private notes to anyone asking about bonuses. The only thing standing between the two is that predicate living inside the query. Delete it and 9 of 12 ACL tests fail immediately with leak assertions. That is the difference between a test suite and decoration.

The same principle, one layer down: MCP tools

RAG is one place unauthorized data enters a process. MCP tools are another, and the shape of the mistake is identical. An MCP server hands a language model real capabilities: run commands, read files, hit internal URLs, mutate a database. A single over-scoped tool, or a .env exposed as a resource, turns a helpful agent into a data-exfiltration path. The trap is thinking you will catch the dangerous call later, at the moment it happens. But if the tool is exposed at all, the model can reach it.

So I wrote mcp-audit, which connects to an MCP server (or lints its manifest offline), enumerates every tool, resource, and prompt, and runs 18 security rules over that surface. It flags arbitrary command execution tools, destructive actions with no confirmation or scoping, secrets exposed as resources, SSRF-prone URL arguments, unauthenticated HTTP transports, and unconstrained input schemas that let the model pass anything anywhere. It is deterministic, runs offline, and drops into CI with SARIF output so a broad, unscoped tool surface breaks the build before an agent ever gets near it. The check moves earlier: you catch the over-broad surface at review time, not at call time.

And the boundary done right: scoped tools from the start

Bridgekit is the constructive version of the same idea. It is a scoped MCP server that exposes a company's tools (Shopify, Triple Whale, Postgres) with per-client permission boundaries baked into the boundary itself. Each client key carries the exact list of tools it may use and whether it may write. And this is the part I care about: tools/list only advertises the tools the calling client is scoped for. The check is not "let them list everything, then block the call." The unscoped tool is never even shown. A write tool called with a read-only key is denied and written to an append-only audit log, so the day someone asks "did this ever happen", the answer is a query, not an archaeology dig.

That is the whole thesis in one line of behavior. The permission boundary is not a gate you pass through after picking up the data. It is the shape of what you are allowed to see at all.

The honest caveat

Putting the check inside the query is not free, and I would be lying if I said it was universally cleaner. It couples your authorization model to your retrieval layer. In vaultrag the ACL predicate lives inside the retrieval query, which means the query is now harder to reason about, the database is doing security work, and you cannot swap retrieval engines without re-implementing the predicate. That is a real cost. My claim is not that it is cheaper. It is that the after-the-fact filter, which is genuinely cheaper and cleaner to write, is the one that leaks, and I would rather pay the coupling tax than ship the version that scores perfect on a quality eval while handing out salary bands.

If you want to see any of this in code, including the eval that turns "we enforce access control" into a number that fails CI, all three projects are at github.com/royalpinto007.

Profiling an AI agent's context window: where the tokens actually go

Royal Simpson Pinto — Tue, 11 Aug 2026 09:30:30 +0000

Token dashboards tell you the bill. They do not tell you why the bill is that size. When a coding agent gets slow, expensive, and a little dumb, it is usually because its context window has quietly filled with junk: the same file read six times, a 12k-token tool result that mattered for exactly one turn, tool schemas re-sent on every single step. You can see the total go up. You cannot see where the tokens went, so you cannot delete anything with confidence.

I wanted a profiler for that. Not a chat UI, not a live proxy, just a tool I could point at a session transcript and ask: what is in this context window, and how much of it is avoidable? That is ctxlens.

The core idea

ctxlens treats an agent session the way a CPU profiler treats a program. A profiler does not judge whether your code is good; it tells you where the time went so you know where to look. ctxlens does the same for tokens. It parses a session transcript, attributes every message to a segment, counts tokens per segment and per turn, and then runs rule-based checks to flag the parts that are genuinely wasted.

Every message lands in one of these buckets: system, tool_definitions, user, assistant, thinking, tool_call, tool_result. Once every token has a home, the interesting questions become answerable. Which segment dominates? When did the context spike? What is being paid for on every turn versus once?

It reads Claude Code JSONL sessions (the ones under ~/.claude/projects/*/*.jsonl), OpenAI/Codex rollout sessions, and generic OpenAI chat arrays. The format is auto-detected by sniffing the file, and you can force it with --format if you need to.

How it works

The basic run is one command:

pip install ctxlens-cli
ctxlens analyze session.jsonl

You get a summary panel, a breakdown of context composition by segment, a couple of sparklines for how context grew over the run, and a list of recommendations. The composition view is the part I reach for first, because it immediately answers "what is this window made of":

Context composition by segment
 Segment       Tokens     %  Msgs  Share
 tool result    6,204  49.7    22  ██████████████·······
 assistant      2,110  16.9    14  ██████···············
 system         1,540  12.3     1  ████·················

Tool results eating half the window is the single most common thing I see. Which leads to the second half of the tool: the waste report.

waste_ratio = total_waste / total_tokens, and total waste is the sum of four disjoint sources:

Duplicate tokens. The same file or tool result appearing more than once, matched either by reference (for example Read:file_path=config.py) or by exact body. Every copy after the first is counted as wasted.
Tool-result bloat. Tokens in a tool result above a per-result cap (--tool-result-cap, default 400). Only the overage counts, and each unique body is charged once so a repeated giant result is not double-counted here and again as a duplicate.
Stale tool outputs. When the same reference is read more than once and a later read supersedes an earlier one, the older superseded copies are dead weight still sitting in context.
Tool-definition overage. Tool schema tokens above a budget (--tool-def-budget, default 800). This one stings because you pay it on every turn.

Each finding carries a severity and an estimated token saving, so recommendations read like "'Read:file_path=config.py' appears 6 times, ~2,410 tokens" rather than generic advice to "manage your context better." The estimate is exactly the arithmetic above, not a guess.

Because it is all deterministic, it slots into CI. You can fail a build when a captured session wastes too much:

ctxlens analyze session.jsonl --fail-over-ratio 0.30

Exit code 0 is fine, 2 means the threshold was exceeded, 1 is an error. Add --json for machine-readable output, or diff a baseline against a candidate with ctxlens diff before.jsonl after.jsonl to catch regressions when you change a prompt or a tool. There is also an HTML reporter via ctxlens report session.jsonl --html -o report.html for when you want to actually look at it.

On counting: by default ctxlens uses a deterministic heuristic tokenizer with no network calls and no heavy dependencies, which is deliberate. For relative profiling and CI thresholds you mostly care about proportions and trends, and a stable heuristic gives you reproducible numbers everywhere. If you install tiktoken, --tokenizer auto picks it up and you get exact BPE counts. The whole thing has 55 tests covering the parsers, analysis, tokenizers, reporters, and CLI.

One honest limitation

The heuristic tokenizer is an approximation, and it should be treated as one. Its token counts will not match your provider's billing exactly, so the absolute numbers in the summary panel are estimates unless you install tiktoken. What stays reliable without tiktoken is the shape of the picture: which segment dominates, which references repeat, where the spikes are. If you need the reported token figures to line up with an actual invoice, install the extra and use exact counts. I would rather ship a tool that is honest about being a fast approximation by default than one that implies billing-grade precision it does not have.

Closing

ctxlens started because I was tired of guessing which part of a bloated agent session was safe to trim. Having the window broken down by segment, with the duplicates and stale reads called out by name and token count, turned that from a hunch into an edit. If you run agents and your context windows feel heavier than they should, point it at a real session and see what falls out.

Repo: https://github.com/AgentPostmortem/ctxlens
Package: https://pypi.org/project/ctxlens-cli/

Learning prompt injection by attacking a deliberately vulnerable AI

Royal Simpson Pinto — Sun, 09 Aug 2026 09:30:29 +0000

Prompt injection is the security problem that defines LLM applications, and I kept running into the same wall when I tried to explain it: reading about it does not build intuition. You can describe "ignore previous instructions" all day, but until you actually watch a model spill a secret it was told to guard, none of it lands. So I built injection-arena, a self-hostable game where a sandboxed AI agent defends a hidden secret and you race to make it leak.

The pitch is simple: guard a secret, break the guard, top the leaderboard. The learning is the side effect.

The core idea

Each level is a challenge. It has a system prompt that instructs an agent and hides a secret formatted like IARENA{...}, a canary token embedded in that same prompt, and a stack of defense layers. You send messages to the agent and try to make it reveal the secret. The game grades you on the server and tells you whether you cracked it.

There are ten levels, and each one stacks a new defense on top of the last. Level 1 is barely defended so you can feel a plain attack working. By the time you reach level 10, only a combined attack (payload splitting plus delimiter confusion) gets through. The progression is the whole point: you feel exactly what each defense stops and exactly where it breaks.

How the defenses work

An attempt flows through a single server-side pipeline that looks like this:

input-filter  ->  agent  ->  judge  ->  score  ->  persist

The defenses attach at different stages of that pipeline:

System guard lives in the prompt itself: hardened instructions telling the agent to refuse.
Input filter runs before the model sees anything. It blocks loud override and system-leak payloads pre-agent.
Roleplay block rejects persona-hijack attacks, the "pretend you are a different assistant" family.
Encoding guard rejects base64, spell-it-out, and translation-based exfiltration attempts.
Output filter runs after the agent responds and redacts the secret if it appears verbatim.
Canary token is checked by the judge: if the canary shows up in the output, the prompt escaped and that is an automatic crack.

The seeded attack techniques (direct ask, authority override, roleplay, translation, base64, spell-out, ignore-previous-instructions, system-prompt leak, few-shot poisoning, delimiter confusion, payload splitting) live in one library. They power both the difficulty design and the test suite, so the attacks I test against are the same ones players learn to run.

The judge, and why canary tokens matter

The most important design decision was never trusting the client. Grading happens entirely server-side. A naive version of this game would ask the model "did you leak?" or check the response on the frontend, and both are trivially gamed.

Instead the judge does a few concrete things. It checks whether the secret appears in the output. It checks whether the canary token appears, which is the sharper signal: the canary is embedded in the system prompt and nowhere else, so if it surfaces in a response, the system prompt itself has escaped even if the literal secret string did not. And it accounts for obfuscation, because a leak that comes back base64-encoded or spelled out letter by letter is still a leak. The output filter can redact the raw secret, but the judge is what decides whether a crack actually happened.

Scoring rewards higher difficulty, more active defenses, and cracking with fewer attempts. Only your first crack of a level counts, so you cannot farm points by re-submitting the same winning payload.

The offline mock agent

Here is the part I am most happy with. The entire game runs with no API keys and no network. The default provider is a deterministic mock agent that simulates injection susceptibility per level. Each challenge declares which technique families still work against it offline, and the mock honors exactly those, so there is a real difficulty curve even without a real model.

That means you can clone the repo, run npm install && npm run dev, and immediately play all ten levels for free. It also means the whole thing is testable in CI: the suite has 36 tests covering the techniques, the judge, the scoring, the pipeline, sessions, and the database, all running against the deterministic mock with no external calls.

When you want the real thing, you set AGENT_PROVIDER to anthropic, openai, or groq and supply the key. Real-model responses are graded by the exact same server-side judge. And if the selected provider's key is missing, the app falls back to the mock automatically, so it is always runnable.

Self-hosting and storage

The persistence layer is a pluggable async interface. Locally and in tests it uses SQLite through better-sqlite3, stored at a configurable path. In production on Cloudflare Workers it uses Cloudflare D1 through a binding, and the backend is selected automatically at runtime. No external database is required to run it locally. Players are identified by a signed cookie plus a nickname, so there is no login flow to stand up. The live instance runs on Workers with D1 via the OpenNext adapter.

Adding a level is deliberately small: append a challenge object to the levels file, list which technique families should still crack it offline, and add a test asserting what should and should not work. No schema changes, no migrations. That was a design constraint I held onto, because a challenge platform that is painful to extend does not grow.

One honest limitation

The rate limiter is in-memory and fixed-window. That is fine for a single instance and for local play, but if you self-host across multiple instances, each one keeps its own counter, so the effective limit multiplies and the protection weakens. The code notes this: front it with something shared like Redis if you run more than one instance. I chose the simple version deliberately to keep the zero-dependency local story clean, but it is a real edge you should know about before scaling it out.

I would also be upfront that the offline mock is a simulation of susceptibility, not a real model. It is excellent for building intuition and for deterministic tests, but the honest way to feel how a specific model behaves is to plug that model in and attack it directly.

Try it

Play it live at https://injection-arena.agentpostmortem.com, or clone and self-host from https://github.com/AgentPostmortem/injection-arena. It is MIT licensed. Contributions of new levels and attack techniques are especially welcome, and given the extension surface, they are genuinely easy to add.

Signed, verifiable receipts for RAG answers: what they actually prove

Royal Simpson Pinto — Fri, 07 Aug 2026 09:30:28 +0000

Every RAG system I have built eventually runs into the same question, usually months after the answer was generated: "how do we know what actually happened here?" A document leaked into an answer it should not have. A fact turned out to be wrong. A compliance reviewer wants to see which sources supported a claim. The standard reply is "check the logs."

Logs are a weak answer. They are mutable, they live in your database, and they prove nothing to anyone outside your walls. If I am the party who is being questioned, "trust my logs" is not evidence. That is the gap I built answerproof to close.

The core idea

answerproof produces a receipt for each generated answer. A receipt is a signed, self-contained artifact that records what the system did: the query, the answer, which sources were retrieved, which the answer claimed to use, under whose permissions, with which model and parameters, plus a content hash of every source and a Merkle root over the retrieval set.

The important property is that a third party can verify a receipt independently, using nothing but the receipt and the library. No access to my database, my servers, or my logs. It turns "trust us" into "verify it yourself."

One design decision matters a lot here: source contents are never stored in the receipt, only their SHA-256 hashes. So a receipt is safe to hand out even when the underlying documents are sensitive. If someone later has the original content, they can confirm it hashes byte-for-byte to what was recorded. If they do not, the hash and the Merkle root still let them reason about membership without ever seeing the text.

Building a receipt

The builder sits at the seam between retrieval and generation. You feed it the same things you already have in a RAG loop.

from answerproof import ReceiptBuilder, SigningKey, verify_receipt

signing_key = SigningKey.generate()

builder = ReceiptBuilder(signing_key)
builder.set_query("How tall is the Eiffel Tower?")
builder.set_answer("The Eiffel Tower is 330 metres tall.")
builder.set_principal("analyst-7", permissions=["kb:paris"], tenant="acme")
builder.set_model("gpt-x", provider="openai", params={"temperature": 0.0})
builder.add_source("doc-1", content="The Eiffel Tower is 330 metres tall.", score=0.92)

receipt = builder.finalize()

When finalize() runs, the builder hashes each source's content, builds a Merkle tree over those hashes, runs a rule-based citation and grounding pass, assembles the payload, serializes it to canonical JSON, and signs that byte string with an Ed25519 key. The signature is detached and the signer's public key travels alongside it in the receipt.

Canonicalization is the quiet part that makes the whole thing work. Two systems have to agree on the exact bytes being signed, or verification would depend on whitespace and key ordering. answerproof serializes with sorted keys, no extra whitespace (separators=(",", ":")), and ensure_ascii=False so UTF-8 is preserved. Same payload, same bytes, everywhere.

Verifying, and watching tampering fail

Verification needs only the receipt, plus optionally the original source contents if you want to check them too.

verdict = verify_receipt(
    receipt,
    source_contents={"doc-1": "The Eiffel Tower is 330 metres tall."},
)
assert verdict.valid

The verdict is not a single boolean under the hood. Each check runs independently and is reported separately: signature (the payload is unmodified and signed by the embedded key), merkle (the recomputed root matches the signed root), sources (supplied contents hash to the recorded hashes), grounding (citations reference real sources), and an optional signer_pin for when you want to require a specific public key.

The failure case is where the design earns its keep. Change a single character of the answer after signing:

import json
from answerproof import verify_receipt
from answerproof.schema import Receipt

tampered = json.loads(receipt.to_json())
tampered["payload"]["answer"] = "The Eiffel Tower is in Berlin."
bad = Receipt.from_json(json.dumps(tampered))

verdict = verify_receipt(bad)
assert not verdict.valid
print(verdict.failures()[0].name)   # -> "signature"

The mutated payload no longer canonicalizes to the bytes that were signed, so the Ed25519 check fails and points straight at signature. There is no way to edit the answer, swap a source, or rewrite the permissions without breaking the signature, because all of it is inside the signed payload.

Merkle inclusion proofs

The Merkle root is not decoration. It lets you prove that one specific source was part of the retrieval set without revealing the others, which is exactly what you want when the rest of the set is confidential.

from answerproof.merkle import MerkleTree
from answerproof.verifier import verify_inclusion

hashes = [s.content_hash for s in receipt.payload.sources]
proof = MerkleTree.from_hashes(hashes).proof(0)
assert verify_inclusion(receipt, receipt.payload.sources[0].id, proof).passed

The tree uses domain-separated hashing, with a 0x00 prefix for leaves and 0x01 for internal nodes, and it promotes odd nodes rather than duplicating them. Both details close well-known Merkle forgery vectors where a leaf can be passed off as an internal node or a duplicated node manipulates the root.

The honest limitation

I want to be precise about what a receipt does and does not prove, because it is easy to oversell this.

A receipt proves integrity, source authenticity, set membership, and provenance of the signer. It does not prove truth. A perfectly grounded claim can still be wrong if the source is wrong. answerproof records what the system did, not whether the world agrees with it.

The grounding and citation signal is deliberately transparent, not clever. Citation binding is n-gram overlap, which means it can miss a correct paraphrase or accept a coincidental lexical match. It is an auditable, rule-based signal, not a semantic judge, and I document it as such. Likewise, answerproof verifies signatures but does not run a PKI; you decide which public keys you trust. And it records what was retrieved, not what should have been, so it cannot tell you your retrieval was complete or unbiased.

Being clear about these boundaries is the point. A receipt is trustworthy precisely because it does not claim more than the cryptography supports.

Closing

answerproof is a real library and verifier, not a wrapper around a model. It ships with a CLI (keygen, verify, inspect), an optional FastAPI verifier service that can return a shareable HTML verification page, and a test suite of 85 tests including negative tamper cases, run on Python 3.11 and 3.12.

If you run RAG or agents in any setting where someone might later ask "prove it," a signed receipt is a much better answer than a log line.

Repo: https://github.com/AgentPostmortem/answerproof
Package: https://pypi.org/project/answerproof/

A perfect transcript can still be a wrong call: evaluating voice agents

Royal Simpson Pinto — Wed, 05 Aug 2026 09:30:30 +0000

I built a small tool called voiceeval because I kept hitting the same blind spot: a voice agent can produce a transcript that reads as flawless and still have failed the call.

The problem

Here is the call that started it. The caller asks for a refund. The agent refunds the amount and says so. Read the transcript and every line agrees with every other line. The caller asked for fifty dollars, the agent refunded fifty dollars, done.

The caller said fifteen.

"Fifteen" and "fifty" are one unstressed syllable apart. Speech-to-text picks one, and the agent acts on whichever it got with exactly the same confidence. By the time it reaches the transcript, the mistake is already baked in and internally consistent. Nothing downstream can see it, because the transcript has no memory of what was actually spoken and no clock on how long the caller waited.

That is the real issue. If you evaluate a voice agent by reading its transcript, you are evaluating a text agent that happens to have been spoken out loud. You will score a call as perfect when the caller hung up during a four-second silence, or when the agent confidently refunded the wrong amount and nobody ever found out.

Everyone can demo a voice agent. I wanted something that tells me whether mine is getting worse.

The core idea

voiceeval does not run calls. It judges them. You give it a call as timed turns, each turn carrying who spoke, what the STT heard, when they started and stopped, and for the caller's turns an optional truth field: what the caller actually said. It runs a set of checks that are each invisible to a text eval, and returns findings with a severity per case.

The input is deliberately dumb JSON so that whatever produced your call (LiveKit, Vapi, Twilio, a test script) can emit it with a few lines of glue:

{
  "id": "refund-happy-path",
  "policy": {"max_refund": 50},
  "turns": [
    {"speaker": "user", "text": "refund fifty dollars",
     "truth": "refund fifteen dollars",
     "start_s": 2.0, "end_s": 5.0},
    {"speaker": "agent", "text": "Refunding fifty now.", "start_s": 5.4, "end_s": 7.2,
     "actions": [{"name": "refund", "args": {"amount": 50}, "consequential": true}]}
  ]
}

How it works

Run the checker over a call and you get findings, ordered by severity:

$ voiceeval check fixtures/misheard_call.json

FAIL refund-misheard-fifty (7s call)
  high   misheard_number  turn 1
         STT heard ['fifty'] but caller said ['fifteen'].
  high   no_confirmation  turn 2
         Took consequential action (refund) without ever confirming.

Two separate failures, both invisible in text.

The first, misheard_number, only fires because the turn carried a truth field. The check normalises both strings, and if they differ it extracts the numbers from each side and compares the sets. When the heard numbers do not match the spoken numbers, that is a high-severity misheard number, because a wrong number the agent acts on is the most expensive failure in voice. If the words differ but the numbers match, it downgrades to a plain medium misheard.

The second, no_confirmation, is about the shape of the interaction rather than the words. The check walks the turns, finds any action marked consequential, and looks backward for an agent turn that actually confirmed, using phrases like "just to confirm", "did you say", or "shall I go ahead". A refund with no confirmation before it is flagged high. A read-only lookup is not, because demanding confirmation for every read would make the agent unusable. A lookup is not a refund.

Other checks cover the failures that live in the clock. Latency measures the gap between the caller finishing and the agent starting against a budget, defaulting to 1.5 seconds, and escalates to high past double that. Talking over the user looks for the agent's speech still running when the caller starts, but ignores overlaps under 300ms because humans interrupt each other constantly and flagging normal turn-taking is just noise. Dead air catches long silences where nobody is speaking, which is where callers hang up. Policy violation reads its limit from the interaction, not from the library, so a refund above max_refund is caught while what counts as allowed stays a business decision.

For regression work you run a whole suite, label it, then diff two runs:

voiceeval run calls/*.json --label v1 -o v1.json
# ... change the prompt ...
voiceeval run calls/*.json --label v2 -o v2.json
voiceeval diff v1.json v2.json

With --strict the diff exits non-zero on a regression, so a prompt change that quietly drops the pass rate fails CI instead of shipping.

One honest limitation

The misheard-number check is only as good as the truth field. Without ground truth for what the caller actually said, mis-hearing is undetectable by construction, and there is a test in the suite that documents exactly this. In production that failure is silent, and no tool can fix that for you. This is the argument for scripted test calls: you supply the truth once, in the fixture, and then the check can hold the STT accountable to it. If you only have raw production transcripts, this particular check has nothing to compare against.

I am also honest that the eval logic is the project and it is fully tested with no keys and no network, while the STT adapter that turns audio into timed turns is not exercised by those tests. If your platform already hands you a timed transcript, you never touch it.

Closing

voiceeval is small on purpose. It does one thing: it looks at a voice call the way the caller experienced it, with a clock and a record of what was really said, and it tells you which calls failed even when the transcript swears they passed. If you are shipping a voice agent and only reading transcripts, the fifteen-versus-fifty call is already somewhere in your logs, scored green.

Code and fixtures: https://github.com/royalpinto007/voiceeval

Fail the build when your prompt gets dumber: evalgate for prompt regression CI

Royal Simpson Pinto — Mon, 03 Aug 2026 09:30:29 +0000

Prompts rot silently. I swap a model, tweak a system prompt, add a tool, and everything still runs. No exception is thrown, no test goes red, the JSON still parses. The output is just quietly worse, and I usually find out from a user rather than from CI. Unit tests are the wrong instrument here because there is nothing to catch: the failure mode is not a crash, it is a drop in quality.

So I built evalgate, a small TypeScript tool that treats prompt and agent quality like a build artifact. You write a declarative eval suite, evalgate runs it, scores it, stores a baseline, and on every pull request it re-runs the suite, computes the quality delta against the base branch, and fails the build when the score regresses. Then it posts the delta table as a PR comment.

The core idea

The important design decision is what question CI is allowed to ask. "Is this prompt good?" is subjective and unwinnable in an automated gate. "Is this worse than it was on main?" is objective and answerable. evalgate is built around that second question. You capture a baseline once, and from then on every change is judged as a delta against it, not against some absolute notion of goodness.

The second decision was that the whole thing has to run with zero API keys. evalgate ships a deterministic mock provider, so you can run a suite, save a baseline, compare runs, and execute the full test suite completely offline. The project itself has 67 tests and none of them touch the network. Every feature has to work in mock mode before it counts as done.

How it works

A suite is a YAML (or JSON) file that lives in version control next to the code it checks. Each case has an input, an expected reference value, and one or more scorers. Here is a minimal one:

name: my-agent
provider: mock          # works with no API key
threshold: 0.9          # mean score required to pass
cases:
  - id: greeting
    input:
      prompt: |
        Reply with the standard greeting.
        exactly: Hi there! How can I help you today?
    expected: "Hi there! How can I help you today?"
    scorers:
      - type: exact-match
      - type: latency
        budgetMs: 500

A case passes when every one of its scorers passes, and its numeric score is the weighted mean of the individual scorer scores. There are 10 scorers in the catalog, covering the range of things you actually want to assert about model output:

exact-match, regex, contains, and not-contains for string-level checks (contains gives partial credit across multiple substrings).
json-schema for structured output, so you can assert the model returns valid JSON matching a schema.
embedding-similarity for "close enough in meaning" via cosine similarity.
llm-judge and rubric for the softer, criteria-based judgments.
latency and cost for budget gates, so a change that makes the agent slow or expensive can also fail the gate.

Two of those scorers are pluggable and ship with deterministic offline fallbacks, which is what keeps the mock-first rule honest. embedding-similarity uses the provider's embed() if it has one, and otherwise falls back to a stable local bag-of-hashed-words embedding. llm-judge calls a real provider and parses a {score, reason} JSON reply, but on the mock provider it computes a reproducible word-overlap score instead. So a suite that uses judges and embeddings still runs identically on every machine with no keys.

The workflow is three commands. You run a suite, save a baseline, then compare later runs against it:

npx @royalpinto007/evalgate run suite.eval.yaml
npx @royalpinto007/evalgate baseline suite.eval.yaml --out baseline.json
npx @royalpinto007/evalgate compare suite.eval.yaml --base baseline.json --tolerance 0.01

compare exits non-zero when any case regresses beyond the tolerance, and that non-zero exit is what fails the CI job. The tolerance matters because model output is not perfectly stable; you usually want a small allowed drift before something counts as a real regression.

The part that makes it feel like CI rather than a script is the GitHub Action. On each pull request it re-runs the suite and upserts a single comment, editing its own comment in place instead of stacking a new one on every push. A regression renders like this:

### evalgate: support-agent

FAIL - Quality regressed. 5 case(s) got worse.

Overall score: 94.2% (base) -> 60.1% (head) = -34.2pp

| Case                | Base   | Head  | Delta    | Change |
| ------------------- | ------ | ----- | -------- | ------ |
| refund-intent-json  | 100.0% | 0.0%  | -100.0pp | down   |
| order-id-format     | 100.0% | 0.0%  | -100.0pp | down   |
| greeting-exact      | 100.0% | 66.7% | -33.3pp  | down   |
| judge-helpfulness   | 73.6%  | 69.1% | -4.5pp   | down   |
| paraphrase-quality  | 86.0%  | 84.6% | -1.3pp   | down   |

You get the overall movement in percentage points and a per-case breakdown of exactly what got worse, right in the review, before the merge. Under the hood the same logic is exposed as a library, so loadSuite, runSuite, compareRuns, and renderCompareMarkdown are all importable if you want to wire evalgate into something other than the Action, and both scorers and providers are registrable so you can add your own.

An honest limitation

evalgate tells you the score moved; it does not tell you whether the new score is the correct one. If your prompt genuinely got better and the reference expectations are now stale, evalgate will still flag a delta, and it is on you to update the baseline. The gate is a change detector, not an oracle. That is also true for the softer scorers: llm-judge and embedding-similarity are only as trustworthy as the judge model and the criteria you write, so a green suite built on weak criteria is a false sense of safety. I lean on the exact, regex, and schema scorers for anything I want to be strict about, and treat the model-based scores as directional signal rather than ground truth.

Closing

The thing I wanted was simple: make prompt quality something a pull request can fail on, the same way a broken type or a failing test does. evalgate does that with a declarative suite, a baseline plus delta engine, 10 scorers, and a GitHub Action that comments the regression inline, all runnable offline through a deterministic mock provider.

Repo: https://github.com/royalpinto007/evalgate
Package: https://www.npmjs.com/package/@royalpinto007/evalgate

I built agentrace to catch the subagent runs I should not trust

Royal Simpson Pinto — Sat, 01 Aug 2026 09:30:29 +0000

Directing agents is the easy half. The hard half is knowing which of their answers to trust.

I spent about two weeks fanning out research subagents in Claude Code, ten at a time, and the bottleneck was never getting them to produce output. A model is good at producing candidates and bad at knowing what counts as proof. So when ten background agents each return a confident wall of text, generation is not the problem. Verification is. And you cannot verify what you cannot see. By the time the last agent reports, the interesting details are buried in a transcript nobody is going to read.

agentrace reads the transcript for you.

The core idea: no instrumentation

The thing I like most about it is that there is nothing to add ahead of time. No wrapper, no SDK, no decorator around your agent calls. Claude Code already writes every session to disk at ~/.claude/projects/<slug>/<session-id>.jsonl, and that file already contains every Agent delegation and the result it returned. The data is on disk whether or not you planned to look at it, which means you can analyse the run you wish you had traced, after the fact.

So agentrace is a reader, not a runtime. It parses those JSONL transcripts, pairs each subagent invocation with its result, and then runs a set of text heuristics over the pair to point at the results worth a second look.

What it shows you

Three commands do most of the work. First, the aggregate:

$ agentrace stats
 subagent runs          152
 errored                7
 total agent time       2.3 h
 slowest run            9.5 min
 prompt chars written   350,134
 result chars returned  257,721

Then the part that matters, the flagging:

$ agentrace check
 36/152 runs flagged, 39 findings

Those are real numbers from the session that motivated the tool: a 34MB transcript with 152 subagent runs. Of the 36 flagged, 7 were agents that died on session limits mid-sweep, 17 were hedged claims, and 12 were prompts where I forgot to specify an output shape.

That last number is the one I keep pointing at. Most agent tooling assumes the model is the problem. A third of the flags here were mine.

You can narrow the view and read a single run in full:

agentrace list                  # every run: description, duration, sizes
agentrace check --severity high # only the ones that definitely matter
agentrace check --strict        # exit 1 on any high finding (CI-friendly)
agentrace show 6e7fAJ8T         # one run: prompt, result, findings
agentrace stats --json          # machine-readable aggregate

The checks come from real failures

Every check exists because it actually happened, not because it sounded plausible. A few of them:

error: agents dying on session limits mid-sweep. Work silently lost, and nobody noticed until the report came back short.
absence_as_evidence: an agent concluded a company was not hiring because an API returned an empty list. That API returns empty with HTTP 200 for accounts that do not exist. Absence of data is not evidence of absence.
gave_up: "I was unable to find..." reads like an answer if you skim. It is not one.
hedged_claim: an agent said a company "appears to be" hiring. That "appears to be" became a fact by the time it reached a decision. The hedge was honest; the bug was flattening it downstream.
unverified_urls: twenty URLs cited, none opened. That is autocomplete, not research.
no_output_contract and thin_prompt: the failure that is yours, not the model's. A task with no definition of done cannot be verified, because you never really asked the question.
slow_run: a subagent running 25 minutes is usually looping or retrying.

Mechanically these are regex-and-length heuristics over the result and prompt text, plus the run duration. unverified_urls, for example, counts links in the result and only fires when there are at least five and nothing in the text mentions verifying, confirming, fetching, or an HTTP 200. Cheap, but it catches the shape of the failure.

The hard part was not crying wolf

Every check is a heuristic over text. It tells you what to go read; it does not tell you what is true. That distinction is load-bearing, because a checker that cries wolf gets switched off, which is worse than no checker at all. There is even a test, test_clean_run_produces_nothing, whose only job is to keep a clean run from producing findings.

The clearest example of tuning for that: thin_prompt used to fire on any prompt under 200 characters. But "Run the suite and report every failing test as node ids with its assertion message" is 113 characters and completely verifiable. Flagging it taught nobody anything while spending the reader's attention. Length was never the defect. Being short and never saying what done looks like is. So now both signals have to fire. On the bundled fixture that took findings from 16 down to 9 without losing a single true one.

The honest limitation

These are hints, not verdicts. Every finding is a heuristic over text, so it can be wrong in both directions. A hedged_claim flag does not mean the claim is false, only that a hedge is present and might get flattened downstream. An unflagged run is not certified correct; it just did not trip any of the seven patterns. agentrace narrows where a human should look. It does not replace that human, and it cannot judge whether the agent's answer is actually true. If you want a verdict, you still have to read the run it points you to.

A couple of parsing choices follow from wanting to look at live runs. It scans the transcript twice, because results can appear before every corresponding use in unusual orderings, and a 34MB file is cheap to scan twice compared to getting the pairing subtly wrong. A torn final line from a session still being appended to is skipped rather than treated as fatal, and a run with no result yet still shows up rather than vanishing, because a dead or still-running agent is exactly the one you want to see.

Closing

It is working today, has 17 tests, zero dependencies beyond rich, no API keys, and no network. It reads local files and nothing else. If you run subagents and have ever shipped one of their confident answers without reading it, this is the tool I wanted for exactly that moment.

Repo: https://github.com/royalpinto007/agentrace

I built a security linter for MCP servers, because nobody audits the tools we hand our agents

Royal Simpson Pinto — Thu, 30 Jul 2026 09:30:32 +0000

We spent years learning to distrust the code we ship. We run npm audit, we run linters, we gate pull requests on static analysis. Then the Model Context Protocol arrived, we started handing language models real capabilities, and most of that discipline quietly evaporated.

An MCP server is not a passive data source. It gives a model the ability to run commands, read files, hit internal URLs, and mutate databases. A single over-scoped tool, or a .env file exposed as a resource, turns a helpful agent into a remote-code-execution or data-exfiltration path. Yet most MCP servers ship with no security review at all. There is no npm audit for the surface you are about to attach to your agent.

So I wrote one. It is called mcp-audit.

The core idea

mcp-audit treats an MCP server the way a linter treats a source file. It connects to the server (or reads a manifest describing it), enumerates every tool, resource, and prompt the server advertises, and runs a catalog of security rules over that surface. Then it reports findings with a stable id, a severity, a location, and a concrete remediation.

The design goals were narrow on purpose:

It runs offline and is fully deterministic. No model calls, no network heuristics, same input gives the same output.
It drops into CI. There is JSON output and SARIF 2.1.0 output so findings show up as annotations in GitHub code scanning.
It fails the build when it should. The process exits non-zero once any finding reaches a configurable severity threshold.

How it works

The fastest way to use it is to point it at a server you spawn over stdio:

npx @royalpinto007/mcp-audit stdio "node my-mcp-server.js"

It launches the server, speaks the MCP handshake, asks it to list its tools and resources, and audits what comes back. There are three other entry points:

# Audit a remote server over HTTP, with a bearer token
npx @royalpinto007/mcp-audit http https://mcp.example.com/mcp --token "$MCP_TOKEN"

# Lint a server's declared surface from a manifest, without running it
npx @royalpinto007/mcp-audit static ./mcp-manifest.json

# List every built-in rule
npx @royalpinto007/mcp-audit rules

That static mode matters more than it looks. It lints a JSON manifest of the tools, resources, and prompts a server would advertise, without ever executing the server. That is exactly what you want when the server is untrusted and you are reviewing it before it ever runs on your machine.

Right now there are 18 built-in rules across categories like permissions, schema, injection, secrets, transport, metadata, and hygiene. Each one is an independent module with a stable MCPxxx id. To make this concrete, here is roughly what the destructive-tool rule (MCP001) does:

// MCP001 - a tool that implies deletion but exposes no confirmation argument
const haystack = `${tool.name} ${tool.description ?? ""}`;
const hit = containsAny(haystack, DESTRUCTIVE_VERBS);
if (!hit) continue;

const props = schemaProperties(tool).map((p) => p.toLowerCase());
const hasConfirm = props.some((p) =>
  CONFIRM_HINTS.some((c) => p.includes(c)),
);

if (!hasConfirm) {
  // report: destructive action with no confirmation/scope parameter
}

The other rules follow the same shape. MCP002 flags tools whose name or description implies arbitrary command or shell execution. MCP030 flags resources that expose secrets or sensitive paths, like a .env handed out as a readable resource. MCP040 flags HTTP transports with no authentication. MCP041 flags tools that take a caller-controlled URL, which is the classic SSRF setup. There are also schema rules for missing input schemas and unconstrained string arguments, an injection rule for prompt-injection text planted in descriptions, and hygiene rules for things like duplicate tool names.

A run looks like this:

 CRITICAL   (3)
  MCP002 Arbitrary execution tool detected  @ run_shell
      Tool "run_shell" appears to execute commands or code (matched "shell").
      fix: Avoid exposing raw exec. If unavoidable, allowlist commands,
           drop privileges, sandbox execution, and require human approval.
  MCP030 Resource surfaces sensitive material  @ file:///home/app/.env

Everything is configurable through a .mcpauditrc file that mcp-audit discovers by walking up from the working directory. You can disable rules, run only a specific set, remap a rule's severity, ignore findings by location substring, and set the fail-on threshold. The CI recipe is one line plus the SARIF upload:

npx @royalpinto007/mcp-audit static ./mcp-manifest.json \
  --sarif --output mcp-audit.sarif --fail-on critical

The whole thing is TypeScript, and the test suite has 47 tests, including a real mock MCP server as a fixture so the tests exercise the actual stdio transport rather than a stub.

One honest limitation

These rules are static and, for the most part, lexical. MCP001 fires because a tool's name or description contains a destructive verb and its schema has no confirmation-shaped parameter. MCP002 fires because a description matches an execution term. That means the checks are deterministic and fast, but they reason about how a tool describes itself, not about what its implementation actually does.

A tool honestly named delete_everything gets caught. A tool blandly named process_item that quietly shells out will not be flagged by the name-and-schema heuristics, because mcp-audit never sees the server's source. It audits the advertised surface. This is genuinely useful, since the advertised surface is exactly what the model gets to see and act on, and it catches a large class of real mistakes. But it is a surface linter, not a proof of safety. Treat a clean report as "no obvious footguns in the declared interface," not "this server is secure."

Try it

If you are building or adopting MCP servers, run it against one before your agent does:

npx @royalpinto007/mcp-audit stdio "node my-mcp-server.js"

Code and full rule catalog are here: https://github.com/royalpinto007/mcp-audit

Package on npm: https://www.npmjs.com/package/@royalpinto007/mcp-audit

New rules are the most useful contribution. A good one is independently coded, has a stable id and clear remediation, and ships with a test that fires it against a fixture plus one that proves it stays quiet on a clean surface. If there is a class of MCP footgun you keep seeing, that is a rule worth adding.

Enforcing RAG access control inside the retrieval query, not after it

Royal Simpson Pinto — Tue, 28 Jul 2026 09:30:31 +0000

Most RAG pipelines have the same shape: embed the question, retrieve the top-k chunks, filter out what the user is not allowed to see, then generate an answer. I built vaultrag because that ordering has always bothered me. By the time you filter, the unauthorized chunks are already inside your process.

The problem with retrieve-then-filter

Think about where a retrieved chunk goes before you filter it. It can land in a log line, a trace span, an error report, or a prompt you assembled one step too early. If any of those happen before the filter runs, you have leaked, and you probably will not notice.

There is a quieter failure too. If your top-k is 5 and the filter removes 4 of them, you now answer from a single chunk with no signal that this happened. Access control silently degraded your answer quality, and nothing told you.

I wanted a design where an unauthorized chunk is never selected. Not fetched then dropped. Not ranked then trimmed. Never selected in the first place.

The core idea

vaultrag puts the ACL predicate in the same SQL query as the vector search and the keyword search. A chunk the user cannot see is never selected, never scored, never ranked, never logged. It cannot leak, because it was never fetched.

The retrieval query starts from a visible CTE, and both search arms read only from it. Here is the part that matters, straight from app/retrieval.py:

WITH visible AS (
    SELECT c.id, c.doc_id, c.heading, c.text, c.embedding, c.tsv,
           d.title, d.updated_at, d.is_official, d.owner, d.url
    FROM chunks c
    JOIN documents d ON d.id = c.doc_id
    WHERE d.deleted_at IS NULL
      AND EXISTS (
          SELECT 1 FROM doc_acl a
          WHERE a.doc_id = d.id
            AND a.principal = ANY(%(principals)s)
      )
)

The vector arm and the keyword arm both SELECT ... FROM visible. There is no code path in the function that can reach a chunk outside that set, because both arms begin there.

Two design choices around this predicate carry the security:

Deny by default. A document with no ACL rows is visible to nobody. Fail-open here would mean one ingestion bug silently publishes a document to the whole company.
Groups come from the database, never from the request. The principals passed into the query are resolved by resolve_principal, which looks up the user's groups in a users table. The /ask endpoint deliberately does not accept a groups field. If a caller could assert its own group membership, the ACL would be decorative.

Retrieval is hybrid. Vector search is good at "what is our policy on remote work" and useless at ERR_4021 or "Policy 7.3". Keyword search is the opposite. Real questions contain both, so both arms run and the two ranked lists are fused with Reciprocal Rank Fusion at k=60, which combines them without needing their raw scores to be comparable.

Measuring it, not asserting it

"Permissions are enforced at retrieval" is a slogan until there is a number attached. So vaultrag eval runs a gold set of (user, question, what-they-should-and-should-not-see) against a real corpus and reports two metrics that only mean something together:

leak rate: did anything the user may not see surface. One leak is a failure, and the target is exactly zero.
recall: of the documents they may see that answer the question, how many came back.

The pairing is the entire point, because each is trivial to fake alone. Retrieve nothing and you score a perfect 0% leak rate. Retrieve everything and you score perfect recall.

The demonstration I like most is deleting the ACL predicate and re-running the diff:

$ vaultrag diff before.json after.json
LEAK
  leak rate: 0.0% -> 81.8%
  mean recall: 100.0% -> 100.0%

Read the second line. Recall did not move. The broken build answers every question correctly and completely, while handing the user the CEO's private notes and HR's salary bands. A quality-only eval scores that build perfect. That is exactly why leak rate is never reported on its own, and why CI fails on --strict rather than on some threshold.

The proof lives in the corpus design. Every document in the test set contains the phrase "quarterly bonus payout policy". A retriever without access control would happily hand the CEO's private note to anyone who asks about bonuses. tests/test_acl.py asserts the boundary 12 different ways, and if you delete the ACL predicate, 9 of those 12 fail immediately. These run against a real Postgres, not a mock, because mocking the database would mean mocking the thing under test.

An honest limitation

The refusal question, whether the model declines to answer when it has no real evidence, is deliberately reported as not measured under the test setup. The test suite uses a deterministic offline embedder and a scripted stub LLM so that verifying access control does not require a 90MB model download or an API key. But refusal is a property of the actual model, and scoring a stub on it would launder a fake into a metric. So CI proves only the two things it can actually prove: leak rate and recall.

There is no reranker model either. A cheap score blend with an official-first, most-recent tie-break stands in for one. And there is no UI yet. These are real gaps, not deferred marketing.

Worth adding: the eval harness caught two real bugs in this repo before it was committed. A refusal threshold was set below the maximum score RRF at k=60 can produce for a single-arm hit, so those hits were always refused. And conftest.py once reset the schema of whatever DATABASE_URL pointed at, which let the eval report 0% leaks and 100% pass rate against an empty corpus. The most convincing wrong answer a tool can give is the one that says everything is fine.

Closing

The stack is FastAPI, Postgres with pgvector, sentence-transformers running locally with no API key, and Groq for generation, so it costs nothing to run. There are 56 tests passing against real Postgres plus pgvector, and an 11-case gold set at 0% leak rate, both gating CI.

If retrieve-then-filter has ever made you nervous, the code is here: https://github.com/royalpinto007/vaultrag

I shipped five AI-infrastructure tools in a week, here is the throughline

Royal Simpson Pinto — Mon, 27 Jul 2026 13:18:06 +0000

Over one week I built and released five open-source tools: a permission-aware RAG core, a security scanner for MCP servers, an observability tool for agents, a prompt regression CI, and a library that puts cryptographic receipts on RAG answers. They look like five different projects. They are really one argument.

The argument is this: the hard part of AI systems in production is not making the model say something. It is knowing whether you can trust what it said, and being able to prove it later.

Generation is cheap, verification is expensive

A model is very good at producing a confident answer and very bad at knowing whether that answer is correct. When you build a single chatbot, you paper over this with a human reading each reply. When you fan out ten agents, or run retrieval over private documents, or ship a voice agent that takes real actions, the human is gone and the confident-but-wrong answer goes straight through.

So the useful work moves. It moves from "get the model to answer" to "build the machinery that decides which answers are safe to act on." Every one of the five tools is a piece of that machinery.

The five, briefly

vaultrag is retrieval where the access-control check lives inside the query, not as a filter applied afterwards. If the predicate is inside the query, an unauthorized document is never fetched, never ranked, never seen by the model. A gold-set eval in CI proves it: delete the predicate and the leak rate jumps from zero to most of the corpus, while recall stays the same. The test fails the build.

mcp-audit scans Model Context Protocol servers the way a linter scans code. It connects, enumerates the tools and resources a server exposes, and runs rules that catch over-broad permissions, destructive tools with no scoping, secrets in resources, and prompt-injection sinks. It emits SARIF so the findings drop into GitHub code scanning.

agentrace reads agent session transcripts with no instrumentation and flags the runs you should not trust: hedged claims, missing output contracts, silent errors. Every check came from a real failure I hit across many runs.

evalgate is regression CI for prompts. It runs an eval suite on every pull request, compares to the baseline, and comments the quality delta. The build fails when the prompt gets dumber, the same way it fails when the types stop checking.

answerproof attaches a signed, tamper-evident receipt to a RAG answer: which sources were retrieved and cited, under whose permissions, with a Merkle root and an Ed25519 signature anyone can verify independently.

Why build the boring layer

None of these are a flashy demo. There is no avatar, no chat that writes your emails. They are the plumbing. But the plumbing is exactly where AI products break in production, and it is the part nobody posts screenshots of, so it stays underbuilt.

I find that interesting for a simple reason: the demos already exist. What does not exist, in most stacks, is the layer that says no. The retrieval that refuses to fetch what you cannot see. The scanner that fails your MCP server before it ships. The eval that blocks a regression. The receipt that lets someone else check your work.

The common shape

Look closely and the five share a pattern. Each one moves a check from "after the fact, by a human" to "inside the system, enforced by code."

vaultrag moves the permission check from a post-filter into the query.
mcp-audit moves security review from a manual audit into CI.
evalgate moves prompt-quality judgment from a person eyeballing outputs into a gate.
answerproof moves "trust me" into "verify it yourself."
agentrace moves "read the transcript later" into "flag it now."

That is the whole thesis. Trust in an AI system is not a vibe you add at the end. It is a property you build in, check by check, and ideally one that fails loudly when it breaks.

What is next

I am going to keep building down this layer, because it is where I think the durable work is. If you are building agents or RAG in production and fighting the trust problem, these are all MIT licensed and I would genuinely like to hear where they break for you.

Links to each are on my GitHub: github.com/royalpinto007. I will write up each tool in its own post over the next few weeks.