I shipped five AI-infrastructure tools in a week, here is the throughline

Royal Simpson Pinto — Mon, 27 Jul 2026 13:18:06 +0000

Over one week I built and released five open-source tools: a permission-aware RAG core, a security scanner for MCP servers, an observability tool for agents, a prompt regression CI, and a library that puts cryptographic receipts on RAG answers. They look like five different projects. They are really one argument.

The argument is this: the hard part of AI systems in production is not making the model say something. It is knowing whether you can trust what it said, and being able to prove it later.

Generation is cheap, verification is expensive

A model is very good at producing a confident answer and very bad at knowing whether that answer is correct. When you build a single chatbot, you paper over this with a human reading each reply. When you fan out ten agents, or run retrieval over private documents, or ship a voice agent that takes real actions, the human is gone and the confident-but-wrong answer goes straight through.

So the useful work moves. It moves from "get the model to answer" to "build the machinery that decides which answers are safe to act on." Every one of the five tools is a piece of that machinery.

The five, briefly

vaultrag is retrieval where the access-control check lives inside the query, not as a filter applied afterwards. If the predicate is inside the query, an unauthorized document is never fetched, never ranked, never seen by the model. A gold-set eval in CI proves it: delete the predicate and the leak rate jumps from zero to most of the corpus, while recall stays the same. The test fails the build.

mcp-audit scans Model Context Protocol servers the way a linter scans code. It connects, enumerates the tools and resources a server exposes, and runs rules that catch over-broad permissions, destructive tools with no scoping, secrets in resources, and prompt-injection sinks. It emits SARIF so the findings drop into GitHub code scanning.

agentrace reads agent session transcripts with no instrumentation and flags the runs you should not trust: hedged claims, missing output contracts, silent errors. Every check came from a real failure I hit across many runs.

evalgate is regression CI for prompts. It runs an eval suite on every pull request, compares to the baseline, and comments the quality delta. The build fails when the prompt gets dumber, the same way it fails when the types stop checking.

answerproof attaches a signed, tamper-evident receipt to a RAG answer: which sources were retrieved and cited, under whose permissions, with a Merkle root and an Ed25519 signature anyone can verify independently.

Why build the boring layer

None of these are a flashy demo. There is no avatar, no chat that writes your emails. They are the plumbing. But the plumbing is exactly where AI products break in production, and it is the part nobody posts screenshots of, so it stays underbuilt.

I find that interesting for a simple reason: the demos already exist. What does not exist, in most stacks, is the layer that says no. The retrieval that refuses to fetch what you cannot see. The scanner that fails your MCP server before it ships. The eval that blocks a regression. The receipt that lets someone else check your work.

The common shape

Look closely and the five share a pattern. Each one moves a check from "after the fact, by a human" to "inside the system, enforced by code."

vaultrag moves the permission check from a post-filter into the query.
mcp-audit moves security review from a manual audit into CI.
evalgate moves prompt-quality judgment from a person eyeballing outputs into a gate.
answerproof moves "trust me" into "verify it yourself."
agentrace moves "read the transcript later" into "flag it now."

That is the whole thesis. Trust in an AI system is not a vibe you add at the end. It is a property you build in, check by check, and ideally one that fails loudly when it breaks.

What is next

I am going to keep building down this layer, because it is where I think the durable work is. If you are building agents or RAG in production and fighting the trust problem, these are all MIT licensed and I would genuinely like to hear where they break for you.