My answer is yes. I also think most of us do it badly, me included, and I only noticed how badly after strangers on the internet read my post more carefully than I did.
I build small open source tools on my own. Two of them are built around this idea.
HousingWizard is a set of MIT licensed agent skills for home buyers. Every output ends with a section called "This tool cannot check." I have tested it on fixtures only. Not on real floor plans, and not with real buyers. The README says so, and as of today I have no real buyer feedback at all.
AceSeo is a local first CLI for checking how visible a site is in AI search. It uses official APIs only, and every number it prints traces back to an evidence ID. So far I have done one small live run with Anthropic. OpenAI is not verified. The thresholds it uses are provisional.
Writing those sentences was harder than writing the code. The code was the fast part. The slow part was writing honest copy about the limits, and then getting anyone to look at it.
Why I think the limits section matters
A tool for home buyers that sounds confident is dangerous in a quiet way. If it reads a listing and says the layout looks fine, a buyer might skip a question they should have asked. The tool did not lie. It just did not say what it never looked at.
So the "cannot check" section is not a disclaimer for legal reasons. It is part of the output. If the tool cannot see whether a wall is load bearing, the buyer should read that sentence right next to the answer, not in a footer.
Same with AceSeo. A visibility score with no source is just a number that feels authoritative. Tying every number to an evidence ID means you can ask "where did this come from" and get an actual answer.
What a discussion taught me
I wrote about this on Indie Hackers in a post called What I shipped this week as a solo builder, and what I'm still not allowed to claim. It got 2 upvotes and 11 comments in a day. My earlier dev.to post on a similar theme got zero reactions, so I paid attention to the difference. The comments were better than my post. Here is what I took from them.
A stale limit is as bad as a missing one. brianwinckel pointed out that when you ship weekly, your limitations go out of date fast. If my README says OpenAI is not verified and I verify it next week but forget to update the line, I have broken trust in the other direction. His fix is simple: put a date next to it. I am adding a "last checked" date to each limit.
Make it a table, and link the evidence. Javi M suggested a "tested on / not tested on" table instead of prose, and linking each provisional threshold to the single run that produced it. That second part stung a little, because the honest version of "provisional threshold" is "this number came from one run." Saying that out loud is the point. I plan to restructure both READMEs this way.
Write limits as facts with a next step. mohitnippanikar sells to small businesses, and his example stuck with me: "Works with Square today; Toast is next on the roadmap." That reads as a plain fact, not an apology. My current wording leans apologetic. I want to rewrite it so a reader knows what works now and what comes next. He also said no numbers about customer results until you can show where each one came from. I have no customer results, so for now the rule is easy to follow.
Being upfront helped more than it hurt. abdallahyasser said this from experience, and it was reassuring, because my worry was that a long list of limits makes a project look unfinished. Maybe it does. But it is unfinished, and saying so seems to be what made people comment at all.
Put the important part higher. Mike Knox suggested moving the list of claim types AceSeo recognizes closer to the top of the README. He is right. I buried the most useful thing under setup instructions. I am planning to move it up.
The part I have not figured out
Here is my honest worry. A "cannot check" section can turn into a ritual. You write it once, it gets long, nobody reads it, and it becomes the new fine print. The dated table helps, but it does not solve the deeper question of whether users actually read limits, or whether they only read the confident part above it.
I also do not know where the line is between useful honesty and noise. HousingWizard could list a hundred things it cannot check. Listing all of them would bury the three that matter.
If you want to look at how I have done it so far, the repos are HousingWizard and AceSeo. Both are rough, and both say so.
My question for you
If you have maintained a tool that people relied on for real decisions, how did you keep its documented limits accurate over time, and did you ever see evidence that users actually read them? I would especially like to hear from anyone who tried this and decided it was not worth it.
Top comments (0)