My homelab agent can only recommend what the dataset can prove

This is a submission for the Sanity Challenge, Path One: Ship an Agent That Queries Real Content

What I Built

Toolsmith is a weekly agent that asks one question about my self-hosted homelab: is there a better tool than the one we run, and is that actually true? It measures what's running, researches replacements, tests up to two in a sandbox with no network, and writes recommendations. A validator then withholds any recommendation whose numbers aren't in the facts it cites.

After two weekly runs it had written 8 recommendations and published 3. The problem was that the validator worked on prose. It pulled digits out of sentences, so Apache-2.0 became the number 2.0, and it held back correct claims along with wrong ones.

For this challenge I moved Toolsmith's output into Sanity and built an agent that answers questions about it over Sanity Context MCP:

• Structured content is the evidence. Every fact is a document with typed measurements[]{value, unit}. Every number a recommendation relies on is a numbersUsed[] entry holding a reference to the fact that must contain it. Licences are an SPDX field, not a phrase.
• A Knowledge Base holds the recall. Each run's full report and the validator's "withheld" file are report documents, indexed as a Context Knowledge Base, so the agent can explain why something was held back in the run's own words.
• Only the structure can make a claim "shown". The model can explore both freely, but it can present a recommendation only through one tool. That tool fetches the recommendation with its facts joined in, through Context MCP, and checks every number against the referenced fact. A number that turns up only in the prose backs nothing.

The agent only works because the content is structured. Ask it "should we replace our reverse proxy?" and it has to say that all three recommendations were withheld, not paraphrase them into something that sounds confident.

Demo

"What did the observability runs find?" Five recommendations exist for that area. Four are withheld and one (Grafana OSS) is shown. The answer guard also caught six numbers that no shown recommendation holds, and made the model take them out:

The shown recommendation claims the target container has 4440 MB free. Here is the join the validator runs, in Vision. The claim's number points at fact F037, and F037's measurements contain 4440:

"Why were the reverse-proxy recommendations withheld?" This one uses both endpoints. The agent reads the outline of the Knowledge Base, pulls the reverse_proxy and fact_verification entries through knowledge_base_read, and then presents each recommendation through the validator. The Knowledge Base prose is full of figures. The guard strips every one of them, because recall isn't evidence. What's left is an answer built entirely from the validator's structured result:

The Studio, for anyone who wants to see the shape of the data:

Code

https://github.com/casareanderson/toolsmith (MIT). The Sanity work is in sanity/:

• convert/convert.py and convert/reports.py: Toolsmith run artefacts → NDJSON. Deterministic (stable _ids), stdlib only, and sanitised, because the dataset is public.
• studio/: the schema. It's deployed because Context only serves datasets that have a deployed Studio (more on that below).
• agent/mcp_client.py: a ~130-line Streamable HTTP MCP client, stdlib only.
• agent/validator.py: validate(), which works on the joined document instead of prose.
• agent/agent.py: the loop, the present_recommendation tool and the answer guard. It works with any OpenAI-compatible model. I used qwen/qwen3.7-flash on OpenRouter.

python -m agent --trace "What did the observability runs find?"   # via Context MCP
python -m agent --offline "..."                                   # public GROQ API, no token needed

How I Used Sanity

1. The schema is the validator's contract. The original validator asked "does this number appear in the text of fact F037?" The new one asks "does the fact this numbersUsed entry references have this value in its measurements?" That's one GROQ projection:

*[_id == $id][0]{
  title, verdict, claim,
  numbersUsed[]{ value, unit, "fact": fact->{ factId, measurements } },
  citedFacts[]->{ factId, licenceSpdx }
}

A licence is licenceSpdx: "Apache-2.0", so it can't be mistaken for the number 2.0. That was the false positive that started all this.

2. Two Context MCP endpoints, two jobs. An endpoint serves either GROQ tools or Knowledge Base tools. If you attach both source types, the dataset wins and the Knowledge Base is silently ignored. So there are two endpoints. toolsmith runs in GROQ mode over the whole dataset, and the validator's joins go through it. toolsmith-kb runs in Knowledge Base mode over the report documents. The agent connects to both and namespaces the KB tools as kb_*.

3. The Knowledge Base is deliberately not evidence. This is the design decision I'd defend hardest. Prose is exactly what a Knowledge Base is good at: the reports are long, and the reasons for withholding are narrative. But if a number in a report could back a claim, the validator would be back to trusting prose. So the answer guard only accepts numbers from a recommendation that present_recommendation has returned as shown. Anything else gets sent back to the model once, and after that it's replaced with [not verified].

4. What caught me out:

• -32004 Only datasets with deployed Studio applications are supported. My first MCP handshake failed with this. Context needs a deployed Studio for the dataset, even if nobody ever opens it. One sanity deploy fixed it.
• The endpoint name is permanent, and a GROQ filter that matches nothing looks exactly like a broken connection. The toolsmith endpoint uses an empty filter, so it serves the whole dataset.
• Context wants an organisation token, not a project token. The client says so when it gets a 401 or 403.
• The step limit. Each present_recommendation call is one model round. An area with five recommendations used up most of a 10-round budget before the model reached the one that passes. The limit is now 16.
• My own redaction broke run IDs. The answer guard replaced an unbacked 2305 inside run-20260914-2305, so the demo printed run-20260914-[not verified]. Redaction now skips the same ID, date and URL spans the detector skips, and there's a test for it.
• The Knowledge Base isn't a copy of your documents. From 4 report documents, the build produced 5 entries organised by topic (observability, reverse_proxy, fact_verification, candidates, recommendations), each with an outline of what it covers and what it leaves out. The agent navigates that outline and doesn't search raw documents.
• The free plan caps a Knowledge Base source at 150 documents, and editing a filter doesn't re-index. My first source pointed at the whole dataset (275 documents), so Build entries stayed greyed out. Saving a _type == "report" filter on the existing source changed nothing. I had to remove the source and add it again with the filter (4 documents) before it would build.
• Withheld reasons contained the very numbers the guard forbids. "The claim states 119.1 but no cited fact backs it" came out as "the claim states [not verified]…". Now present_recommendation gives reasons in words ("six figures in the claim have no structured source"), and counts are spelled out because a digit would itself trip the guard.
• When a Knowledge Base is connected, prose goes to it. The GROQ endpoint serves the whole dataset, reports included, so at first the model just read report bodies over GROQ and never touched the Knowledge Base. Now, when a KB is connected, a GROQ query for report documents gets a pointer to kb_knowledge_base_read instead.
• The guard polices numbers, not words. The model can read any document over GROQ, including a withheld claim, and paraphrase it: "heavier than the incumbent" gets through, "4 GB" doesn't. I chose that on purpose, because hiding withheld claims from the model would also hide the context it needs to explain them. But it's a real limit, and the demo's answer shows it.
• The measurement labels are ugly. The converter derives labels from the statement text (# MB free of 60167 MB). They're correct but hard to read. I left them alone because the validator matches on value, not labels.

5. Honest numbers, including where it's stricter than I'd like. 245 facts, 8 recommendations, 7 candidate tools, 3 runs, 12 tool areas and 4 reports. The structured validator shows 1 of the 8. The old text validator published 3. The two it now withholds are SigNoz and HyperDX, both REJECTs, and I checked why:

• It's mostly the right call. The old validator only checked a number when a fact ID sat next to it. The new rule is that every number in a claim needs a structured source. SigNoz's "4 GB minimum RAM" and HyperDX's "4 GB + 2 cores" come from the vendors' docs, not from my estate. Web evidence has no document type yet, so those numbers have nothing to point at. The next step is a sourceClaim type (URL, fetched-at, quoted value) that numbersUsed can reference, the same way it references facts.
• Some of it was my bug, and I fixed it. The first comparison also flagged 2026 and 14 from "Sep 14, 2026", 208 from a CISA bulletin ID (sb26-208) and a sandbox run ID read as a number. The identifier patterns now cover all of these, each with a test (27 tests in total).

Every withhold now names the exact number and says what's missing, instead of a regex failing to match somewhere in a sentence.

Sanity Project Details

• Project ID: en9phc0q
• Public dataset: toolsmith. Try it: https://en9phc0q.apicdn.sanity.io/v2025-02-19/data/query/toolsmith?query=*[_type=="recommendation"]{title,verdict}
• Studio: https://toolsmith-kb.sanity.studio (members only; it exists so that Context will serve the dataset)
• Knowledge Base: kbReXSyq7xIs ("Toolsmith"), built from *[_type == "report"] in the toolsmith dataset: 4 documents, 5 entries.
• Context MCP endpoints: toolsmith (GROQ mode) and toolsmith-kb (Knowledge Base mode) in organisation omdh8fw7q. They need an org token with Context Viewer, so to try the agent without one, run --offline. That mode runs the same GROQ against the public dataset.

No Agent Session is attached: the build transcripts are full of my network's internal details. The code and the dataset are both public instead.