Four traps for a naive SEO checker (and a single-file plain-Python one that avoids them)

I wanted a small script that answers the questions a small-business site owner actually has:
is my page indexable, does it work on phones, are there broken links, is the title cut off in Google?
No API key, no headless browser, standard library only.

The checks are easy. Not reporting false problems is the hard part.

1. robots.txt groups. python.org's robots.txt contains Disallow: /, but only inside the groups for a few old bots (HTTrack, puf, MSIECrawler, Nutch). The User-agent: * group blocks just two directories. A grep for Disallow: / would call the site blocked. The checker uses urllib.robotparser and asks the real question, can_fetch("Googlebot", url), with a test for that file shape.
2. Soft 404s. Some static hosts answer every unknown path with the home page and HTTP 200, so /robots.txt and /sitemap.xml "exist" but are HTML. The checker looks at content type and first bytes and reports "soft 404" instead of a green tick.
3. <h1>A<br>B</h1> must read A B, not AB. Line-break and block tags add a space.
4. Certificate errors on Windows Python look like "site unreachable". The checker uses certifi when installed and says clearly when the cause is the local certificate store.

Every flagged item in a client report should be checked next to curl output before sending.

What it does not do: JavaScript-rendered pages, Core Web Vitals, backlinks. Say that up front.

It is one Python file with 61 tests against a local test server.

Built with help from Claude.