The two-year itch
Every six months or so I stare at my homelab and wonder if the thing doing DNS should be doing SSO, and the thing doing SSO should be doing MFA, and the thing doing backups should be doing... you get the picture. Tool fatigue is real when you're running fifteen services on a low-power box and each one made sense when you installed it but the landscape has shifted twice since then.
I wanted an agent that would do the research for me. Not a chatbot that vomits a comparison table, but something that comes back every week with a measured brief: here are the three alternatives to your current tool, here's why you'd switch, and here's why you wouldn't.
I built it. I called it toolsmith. And the most important piece of it is the component that refuses to answer the question.
What toolsmith actually is
Toolsmith is a weekly research agent that lives in a Docker sandbox and has one job: given a homelab tool you're running, find credible replacements and write up what you'd gain and lose by switching. The repository lives on GitHub if you want the full picture.
The architecture hits three constraints that shaped everything else.
Measured fact packs. Every recommendation must cite where the information comes from. Not a link dropped at the bottom — the claim and the source are paired in the output. If the source says one thing and the marketing page says another, the agent is told to surface the conflict, not pick a side.
Capped Docker sandbox with no egress by default. The agent runs in a container that can't reach the internet unless a specific research phase explicitly opens an outbound channel. That means no "let me just check the latest blog post mid-run" drift. The research window is defined upfront, the sandbox opens, the data comes in, the sandbox closes. It keeps the agent honest about working with what it gathered rather than live-navigating and getting distracted.
A validator that withholds unsupported recommendations. This is the one I want to talk about.
The validator
Most agent projects I've built or studied optimise for giving an answer. The agent is embarrassed if it says "I don't know" — so it finds something, anything, to recommend. Maybe it conflates two tools, maybe it extrapolates from a forum post with three upvotes, maybe it just makes something plausible.
Toolsmith's validator does the opposite. It looks at the proposed recommendation and checks: is there a direct source for every factual claim? If the answer is no for any claim that the recommendation depends on, the validator strips that recommendation from the output. Not marks it as low confidence — withholds it entirely.
The first time I saw this in action I thought I'd broken something. The agent ran, did its research, produced a perfectly reasonable-looking analysis, and the validator gutted half of it. Two of three tool recommendations vanished. What remained was thin. I nearly patched the validator to be more lenient.
I'm glad I didn't.
What remained was correct. The surviving recommendation cited actual documented feature matrices, a real changelog entry about the breaking change I was trying to avoid, and a comparison that accounted for the specific hardware constraints of my homelab box. The two that got cut were built on forum speculation, a six-month-old Reddit thread that the agent had treated as authoritative, and one that was just plain hallucinated version numbers.
The commit history shows a few rounds of "validator identifier fixes" — the nitpicky work of getting the validator to recognise source material reliably. It is not a solved problem. But the direction is the right one.
What broke
A few things didn't survive contact with reality.
The knowledge base was only as good as its last sync. I integrated a Sanity Context MCP knowledge base to give the agent a curated starting point. Great in theory. In practice, the knowledge base needed prose updates, identifier corrections, and new demo images constantly. If I wasn't tending it weekly, the agent was working from stale ground truth. The knowledge base became a second project to maintain, which defeats the purpose of an agent that's supposed to save you time.
The sandbox isolation added friction to iteration. Every time the research scope changed — "oh, also check if this tool supports LDAP" — I had to redeclare the sandbox permissions. The no-egress-by-default design is principled, but when you're iterating on prompts it feels like the system is working against you. I don't have a clean solution for this. I suspect the answer is better tooling around ephemeral sandbox configuration rather than softening the isolation.
The Buy Me a Coffee button went up before the thing was useful. I added it early, before the validator was tuned, before the knowledge base was populated. Nobody bought me a coffee, which is exactly what should happen when you ask for support on a project that still eats its own homework. The commit that added that link sits there as a reminder to ship value before asking for gratitude.
The lesson that transfers
The takeaway isn't about toolsmith specifically. It's about where to put the defensive boundary in an AI-assisted workflow.
Every agent project I see optimises for throughput — more completions, faster answers, lower latency, broader knowledge. The implicit assumption is that more information is better and the human will filter out the bad parts. But that's exactly wrong for homelab work. A bad recommendation in production infrastructure doesn't waste an hour, it wastes a weekend of rebuilding something that was working fine.
The right defensive boundary is the one that says "I have an answer but I cannot vouch for it, so I will not give it to you." Not a confidence score, not a disclaimer paragraph, not a footnote. Withhold.
That design choice — a validator with teeth — changes the optimisation target of the whole system. The agent stops trying to sound smart and starts trying to be verifiable. The research phase becomes about collecting sources that can survive inspection rather than assembling a plausible narrative. The prompt engineering shifts from "be thorough and helpful" to "be correct or be silent."
I don't think this applies only to agents. Any automation that makes decisions about infrastructure should have the same bias. A backup script that half-works is worse than one that fails loudly. A migration tool that mostly migrates data is worse than one that refuses to proceed on schema mismatch. Sometimes the most helpful thing a tool can do is tell you it can't help.
Where it stands now
Toolsmith runs weekly in my homelab. Some weeks it produces useful research. Some weeks the validator cuts everything and I get an empty report. Those weeks I curse it, then I go manually look up what I needed, and usually I find that the validator was right — there wasn't a clear enough answer to automate.
That's fine. An empty report from an honest system is worth more than a confident recommendation from one that doesn't want to disappoint you.
The repo is at casareanderson/toolsmith if you want to see the current state. It's early, it's rough, and the validator eats its own output regularly. But the direction — measured facts, isolated research, and a hard no when the evidence isn't there — is one I think more homelab tools should take.
Built for the shelf, not for the shop window.
🤖 Drafted with AI assistance from my own homelab notes, logs and repos, then reviewed and edited before publishing.
Top comments (0)