I built a website audit tool. It crawls a site in a real browser and writes one HTML report: accessibility, performance, security headers, SEO, broken links, JavaScript errors, forms that accept anything. It runs on your machine. No account, no upload, no server.
Before telling anyone about it, I did the obvious thing: I pointed it at its own website.
It came back with three findings. Two of them were false.
[HIGH] Page returns HTTP 403
[LOW] No sitemap.xml found
[INFO] No robots.txt
The site publishes both files. Bing had crawled the sitemap successfully the day before. I had opened robots.txt in a browser an hour earlier.
Here is what happened. The site sits behind Vercel, and Vercel's managed mitigations answered my automated browser with 403 on everything — the homepage, robots.txt, sitemap.xml, all of it. My crawler saw the 403, could not read the files, and reported them as missing.
That is a worse failure than crashing. A crash tells you something went wrong. A confident "No sitemap.xml found" sends you off to fix a file that is already there, and quietly teaches you that the tool cannot be trusted.
Absence is only ever proven by a 404
That is the rule I settled on. A refusal, a rate limit, a timeout, a server error — all of those mean the check could not be made, which is a different sentence entirely.
Now a blocked robots.txt reads "Could not check robots.txt". And when the whole site turns the crawler away, the report says so once, at the top, and stops making claims it cannot support:
[HIGH] The site refused TesterBot (HTTP 403) - this report is incomplete
One curl tells you which situation you are in. If curl gets a 200 from the same machine that the crawler got a 403 from, nothing is wrong with the site — the block is aimed at the automated browser, and it is invisible to you because your own Chrome passes and curl is waved through as a harmless script.
Then I pointed it at a hundred real sites
Three more, in the first hour:
www is not a subdomain boundary. A crawl seeded at example.com that redirected to www.example.com treated every internal link as out of scope and stopped after one page. Eight of ten sites came back with a single page each and I assumed they were blocking me. They were not. Sites move between the two forms freely and a crawler has to treat them as one place.
A domain the crawler could not reach was recorded as crawled. It vanished from the queue, would never be retried, and nothing in the output said so. When the machine's network dropped mid-run, nine domains disappeared from the index that way — silently. Failures are now counted instead, the domain keeps its place in the queue, and three consecutive DNS failures stop the run with "this machine looks offline" rather than burning the whole queue on a dead network.
One slow government site spent half of a 45-minute crawl on its own while fifteen others waited. Breadth beats depth for this kind of work: twenty-five pages from a hundred domains says far more than twenty-five pages from twenty. There is a per-domain time cap now.
The part worth keeping
None of these showed up in the test suite. All of them showed up in the first hour of pointing the thing at the real web. Fixtures test the paths you thought of; the open web tests the ones you did not.
But the first bug is the one I would take away from this. If you build anything that reports on the state of the world — a monitor, a linter, a health check, a scraper — decide early what your program is allowed to say when it could not look. Mine said "not there". It should have said "I could not see".
The tool is MIT licensed, runs locally, and needs Python 3.9+.
- GitHub: https://github.com/thisisismayill/testerbot
- Site and a real sample report: https://testerbot-web.vercel.app
Top comments (0)