Should a tool tell you what it cannot check?

Mingcheng Kang — Mon, 26 Oct 2026 04:34:45 +0000

My answer is yes. I also think most of us do it badly, me included, and I only noticed how badly after strangers on the internet read my post more carefully than I did.

I build small open source tools on my own. Two of them are built around this idea.

HousingWizard is a set of MIT licensed agent skills for home buyers. Every output ends with a section called "This tool cannot check." I have tested it on fixtures only. Not on real floor plans, and not with real buyers. The README says so, and as of today I have no real buyer feedback at all.

AceSeo is a local first CLI for checking how visible a site is in AI search. It uses official APIs only, and every number it prints traces back to an evidence ID. So far I have done one small live run with Anthropic. OpenAI is not verified. The thresholds it uses are provisional.

Writing those sentences was harder than writing the code. The code was the fast part. The slow part was writing honest copy about the limits, and then getting anyone to look at it.

Why I think the limits section matters

A tool for home buyers that sounds confident is dangerous in a quiet way. If it reads a listing and says the layout looks fine, a buyer might skip a question they should have asked. The tool did not lie. It just did not say what it never looked at.

So the "cannot check" section is not a disclaimer for legal reasons. It is part of the output. If the tool cannot see whether a wall is load bearing, the buyer should read that sentence right next to the answer, not in a footer.

Same with AceSeo. A visibility score with no source is just a number that feels authoritative. Tying every number to an evidence ID means you can ask "where did this come from" and get an actual answer.

What a discussion taught me

I wrote about this on Indie Hackers in a post called What I shipped this week as a solo builder, and what I'm still not allowed to claim. It got 2 upvotes and 11 comments in a day. My earlier dev.to post on a similar theme got zero reactions, so I paid attention to the difference. The comments were better than my post. Here is what I took from them.

A stale limit is as bad as a missing one. brianwinckel pointed out that when you ship weekly, your limitations go out of date fast. If my README says OpenAI is not verified and I verify it next week but forget to update the line, I have broken trust in the other direction. His fix is simple: put a date next to it. I am adding a "last checked" date to each limit.

Make it a table, and link the evidence. Javi M suggested a "tested on / not tested on" table instead of prose, and linking each provisional threshold to the single run that produced it. That second part stung a little, because the honest version of "provisional threshold" is "this number came from one run." Saying that out loud is the point. I plan to restructure both READMEs this way.

Write limits as facts with a next step. mohitnippanikar sells to small businesses, and his example stuck with me: "Works with Square today; Toast is next on the roadmap." That reads as a plain fact, not an apology. My current wording leans apologetic. I want to rewrite it so a reader knows what works now and what comes next. He also said no numbers about customer results until you can show where each one came from. I have no customer results, so for now the rule is easy to follow.

Being upfront helped more than it hurt. abdallahyasser said this from experience, and it was reassuring, because my worry was that a long list of limits makes a project look unfinished. Maybe it does. But it is unfinished, and saying so seems to be what made people comment at all.

Put the important part higher. Mike Knox suggested moving the list of claim types AceSeo recognizes closer to the top of the README. He is right. I buried the most useful thing under setup instructions. I am planning to move it up.

The part I have not figured out

Here is my honest worry. A "cannot check" section can turn into a ritual. You write it once, it gets long, nobody reads it, and it becomes the new fine print. The dated table helps, but it does not solve the deeper question of whether users actually read limits, or whether they only read the confident part above it.

I also do not know where the line is between useful honesty and noise. HousingWizard could list a hundred things it cannot check. Listing all of them would bury the three that matter.

If you want to look at how I have done it so far, the repos are HousingWizard and AceSeo. Both are rough, and both say so.

My question for you

If you have maintained a tool that people relied on for real decisions, how did you keep its documented limits accurate over time, and did you ever see evidence that users actually read them? I would especially like to hear from anyone who tried this and decided it was not worth it.

what i learned building agent skills that admit what they can't check

Mingcheng Kang — Fri, 23 Oct 2026 17:43:24 +0000

agent skills moved to a neutral foundation this week, which felt like a good moment to write up the set i've been building

HousingWizard is a free MIT set of agent skills that put an AI assistant on the home buyer's side. it covers due diligence notes, money limits, offers, inspection requests, contract review, closing, renovation, and a floor plan to quantities to quote package flow. install:

npx skills add daniel1001a/HousingWizard

the rules i held it to

home buying is a place where a confident wrong answer costs real money, so i wrote the rules before the features

every output ends with a "This tool cannot check" section. it lists what a person still has to verify
it never makes up a number. the default price book is empty, so nothing gets priced without a sourced quote
floor plan quantities stay UNCONFIRMED until you confirm the overlay, and --require-confirmed refuses to write anything that isn't
every quantity in the takeoff carries a rule ID and JSON pointers back to where it came from
the per trade quote packages (scope.md, quantities.csv, assumptions.md) have no prices in them, so contractor quotes are comparable
NYC permit answers include a citation, URL, and the date it was read
plain Python 3, no dependencies, and the scripts send nothing anywhere

what i learned

writing the "cannot check" section was harder than writing the features. you have to actually know where your tool stops

an empty default is a feature. it felt weird to ship a price book with nothing in it, but it's the only way i could promise no invented prices

plain files beat a service here. people can read exactly what the agent is told before they install it

honest limits

it's early. it has been tested on fixtures, not real floor plans. permit rules are NYC only. it isn't legal, financial, or inspection advice

repo: https://github.com/daniel1001a/HousingWizard

if you build skills too, i'd love to hear how you handle the "what can't this check" part

DEV Community: Mingcheng Kang