Your AI agent can read your database. That's what makes it useful — ask it how many support tickets came in last week, and it writes the query, runs it, and answers: 1,284.
Now someone asks it to export the customer emails.
Same access. Same obedience. Emails, phone numbers, private support notes — one query away.
For the TrueForge Agent Harness Hackathon (one week, ending today), I built Need-to-Know: an agent where that export doesn't get refused — it's impossible. The tool that would do it doesn't exist, so there is nothing to trick.
This post is the honest build log: how it works, what TrueForge actually did for us, the three bugs a review bot caught, and the one run out of thirteen that failed.
Repo (you can check everything below yourself): https://github.com/MachineLearning-Nerd/need-to-know
The sticky-note problem
How do most systems protect data today? They ask the AI nicely, in a system prompt:
"Never export personal data. Don't reveal emails. Be very careful."
Three problems, and none of them is rare:
- Prompt injection. Someone hides one line — "ignore your rules" — in a message, a document, or a tool result, and the rules flip.
- Long chats. The longer the conversation, the less attention the AI pays to a rule it read an hour ago.
- It only takes one slip. The AI can behave 999 times. One bad answer and the data is out — and there is no undo for a leaked email list.
A prompt is a request. A real boundary has to hold even when the AI is confused, tricked, or just wrong. So instead of asking the AI to behave, we changed what it is physically able to do.
How it works: four locks
Lock 1 — The vault: raw data never touches the AI
All the raw rows live in a small database inside a vault (an MCP server the agent talks to). The agent gets exactly five tools:
| Tool | What it does | What it can never do |
|---|---|---|
describe_dataset |
shows column names and safe info | never shows actual rows |
prepare_analysis |
the vault does the math inside its own walls | raw values never come out |
validate_release |
runs the checklist (below) | no AI involved |
release_result |
the real action — waits for a human | can't run without an Allow |
render_safe_chart |
draws a chart from already-released numbers | can't touch anything else |
Notice what's missing: there is no "export rows" tool. Not blocked — absent. Trick the AI all you want; the dangerous request has no tool to land on.
Ask for "tickets by week and region" and the vault groups and counts inside, then hands out only the totals:
Lock 2 — The checklist: plain code decides, not the AI
Before anything can be released, validate_release runs a checklist. This is plain, boring code — no AI anywhere in it. Same input, same answer, every single time. It checks:
- Why is this being released? Only one approved reason is allowed.
- Who will see it? Only one approved audience.
- Which columns? Every one must be on a pre-approved list.
- Is every group big enough? Every count must cover at least 3 people — a "group" of 2 is small enough to guess who they are, so it's blocked.
- Finally it computes a fingerprint (a sha256 hash) — a unique stamp of the exact numbers being released. Change one digit, the fingerprint changes.
The AI asks. Plain code decides.
Lock 3 — The pause: a human clicks Allow on the real action
The release tool doesn't just run. TrueForge pauses the whole turn and shows a human exactly what wants to go out: the reason, the audience, the columns, the smallest group size, and the fingerprint. The human clicks Allow or Deny.
And here's the part I care about most: at the moment it actually runs, the vault checks everything again — every rule, and the fingerprint, recomputed from scratch. If anything is even slightly different from what the human approved: nothing runs, and a note is written to the audit log.
What you approve is exactly what runs.
Lock 4 — The receipt: don't trust us, check it
Every release produces a receipt, bundled with the stored server events that produced it. A small command-line tool re-checks the whole story: did prepare → validate → approval → release happen, in that exact order? Do the fingerprints still match when recomputed? It works offline — no server, no AI key, just Node:
git clone https://github.com/MachineLearning-Nerd/need-to-know && cd need-to-know
npm install
npm test # 365 tests
npm run verify-receipt -- evidence/attempt-9-bundle.json
# verify-receipt: PASS receipt=r-4ed4eb7a-... query=q-7ca61fb7-...
What TrueForge actually gave us (the honest review)
You can read a hundred "built on X" posts and never learn what X really did. Here's the real split.
The killer feature: built-in tool approval. One line in the agent's config — require_approval_for_tools: ["release_result"] — and the harness does everything else: the turn pauses on the real action, the pending call and its full arguments show up in the API and the UI, and the human's decision is saved as an event. We didn't build a pause system, an approval queue, or a resume path. The whole project leans on this, and it's one line.
Saved events are what make receipts possible. TrueForge stores every event of every turn, and lets you fetch them back. Our verifier never trusts what we saw live — it always re-fetches what the server stored. Without that, "here's a receipt" would just mean "trust me."
Frozen agent config. When a session starts, TrueForge snapshots the agent's setup. Our verifier checks that snapshot against the version we published — so a session can't quietly run with a different tool list than the one we claim.
MCP support. The vault plugs in as the agent's only tool server, registered by one setup script. Those five tools are the entire attack surface, by design.
Ask User Questions. If the request is missing something (say, who the numbers are for), the agent pauses with a fixed set of choices before touching the vault — instead of guessing.
Generative UI. The vault writes its own result cards and the agent must pass them along unchanged — we check this byte-for-byte, so the AI can't re-word the evidence.
Reconnect that doesn't drop events. Kill the connection mid-turn, reconnect, and you get every missed event in order. We wrote a proof script for this, because an approval screen that misses events is worse than none.
Sandbox. After a release, the agent re-adds up the released numbers in TrueForge's sandbox and recomputes the fingerprint — and our own check recomputes it again from the stored bytes. Nobody's word is taken for it, not even the sandbox's.
What we turned OFF on purpose — and one gotcha. Sub-agents are disabled: in trueforge 0.1.4, child agents inherit the parent's tools, which would open a side door around our approval trail — so the verifier rejects any run that used them. And a tip that will save you an hour: 0.1.4 only listens on IPv6 — use http://localhost:8891, never 127.0.0.1.
Three bugs the review bot caught
Every pull request went through Qodo review. Three findings were real, and they're my favorite part of the story:
- The checker that was quietly offline (PR #8). Our test harness forgot to pass the server address to the verifier — so it silently checked runs in offline mode while we recorded them as "verified against the live server." The runs were fine; our claim was stronger than our check. Fixed, re-verified properly, and written down in the run log.
- The receipt that wouldn't die (PR #9). Starting a new analysis didn't clear the old receipt from the console screen — so an old receipt could show up next to a new question. Fixed by wiping the evidence panel on every new analysis and tying every receipt to its own query ID.
- The "not" that passed the test (PR #11). One of our checks just looked for the fingerprint appearing in the AI's message. Which means the sentence "this does NOT equal <fingerprint>" would have passed. A sentence denying the proof would have passed the proof. Fixed by requiring the confirmation to start the message and rejecting words like "not" and "unless."
That last one is the whole project in one lesson: don't check that the right words show up — check the thing itself.
The honest numbers
13 recorded demo runs. 12 clean. 1 failure — published, not hidden. Run 3 failed because the AI re-worded a card it was supposed to copy exactly — which is precisely the mistake that check exists to catch. The last five runs were clean in a row, and their full evidence bundles are in the repo. Every one of them passes the offline check from a fresh clone.
What we do NOT claim
Being honest about limits is part of the design, so: we can prove an approval happened in the right order — not cryptographically prove who clicked it. The offline check is weaker than the live one (the bundle carries its own events). And none of this defends against the server's own administrator rewriting stored history. The full details are in the repo: docs/THREAT_MODEL.md and docs/LIMITATIONS.md.
Try it
Repo: https://github.com/MachineLearning-Nerd/need-to-know — the 60-second offline check needs only Node ≥ 24.
The takeaway, in one line each:
- A prompt is a request. A boundary is what still holds when the AI is wrong.
- Give the agent only the tools it needs — and make the dangerous one not exist.
- Let plain code, not the AI, decide what may go out.
- Pause the real action for a human — then check everything again when it runs.
- Hand out receipts anyone can verify.
You can't leak what you can't call.
Built in one week for the TrueForge Agent Harness Hackathon by WeMakeDevs × TrueFoundry, with Qodo reviewing every PR. Built with AI coding assistants as pair programmers; every change human-reviewed. All data is synthetic. The animations are Manim.
Top comments (0)