My website audit crawler named Google as the company that built the site

I build a tool that reads a business's website before I talk to the owner. It crawls up to 60 pages, works out what the site runs on (booking system, chat widget, analytics, form handler, cookie banner) and writes one internal markdown brief for whoever makes the call. Nobody outside sees it.

One field in that brief is "who built this site". When the first build went through review, a page carrying a standard reCAPTCHA badge got this answer: policies.google.com.

Google does not build small-business websites. This post is about how that happened, the second bug the same review found, and the rule I now apply to anything that reads a website and makes a claim about it.

How a reCAPTCHA badge became a web agency

Plenty of small-business sites carry a credit in the footer from the agency that built them. The giveaway is often not the "Website by X" text. It's the legal links. An agency that hosts its clients' privacy and terms pages on its own domain leaves footer links like agency.example/privacy-policy/ and agency.example/website-terms/ on every page.

So the first version of the detector did this: for every link on every page, if it points off-site and the path contains privacy, terms or cookies, count that host as a candidate builder. Most frequent host wins.

Now look at what a standard reCAPTCHA badge puts on a page: policies.google.com/privacy and policies.google.com/terms. Off-site, on every page with a form, both paths matching. My heuristic was built to find exactly that shape. The saved site I use as the main test fixture, a real dental practice, loads assets from www.google.com too, so this was not a contrived case.

The fix had two parts. First, a short list of known infrastructure hosts that can never be the builder:

_AGENCY_INFRASTRUCTURE_HOSTS = frozenset({
    "policies.google.com", "www.google.com", "google.com",
    "support.google.com", "myaccount.google.com", "accounts.google.com",
})

Second, any host that is already a named vendor in one of the other categories (a cookie-consent provider's own policy page, say) is skipped too. When nothing is left, the field says not detected. A reCAPTCHA-only site now returns not detected, and a real agency footer still resolves.

What bothered me was where the wrong answer would end up. The brief is read by a person about to ring a real business, and "your site was built by Google" is a false statement about that business.

The bug that every test agreed with

The same review found a second problem, worse because the test suite was green.

For each detected vendor the brief reports how many pages it appeared on: len(pages) against total_pages. The reviewer tried replacing len(pages) with total_pages, which turns "1 of 8" into "8 of 8" for every vendor. The suite stayed at 115 passed.

The only test checking the page count used a review widget that appeared on 8 of 8 pages. In that case the right number and the wrong number are the same number, so the test could never tell them apart.

I now think of this as the main way a test suite lies: an oracle where the correct value and a plausible wrong value coincide. The fix was a small corpus where a vendor appears on exactly 1 of 3 pages, and a test asserting pages=1, total_pages=3. Put the mutation back and it fails.

The rule: trust where an asset came from, never what the page says

Both bugs led back to a rule that was already in the spec for vendor detection, and that I now apply to everything the tool claims.

A vendor counts as present only when a host served something to the page. The evidence is the set of external hosts in src and href attributes (scripts, iframes, stylesheets, images) and the hosts that forms post to. The detector never reads the rendered body text.

def _match_vendor(host: str, table: dict) -> Optional[str]:
    host = host.lower()
    if host in table:
        return table[host]
    for known_host, vendor in table.items():
        if host.endswith("." + known_host):  # booking.vendor.example matches vendor.example
            return vendor
    return None

Two cases in the test fixtures show why the body text is off limits.

Across the dental practice's pages, the word "dentally" appears six times, every one of them in the phrase "dentally phobic patients". Dentally is also the name of a dental booking system. A substring match over the page text reports that the practice books through Dentally. It doesn't.

On the same site, the word "crisp" appears, but only inside a WordPress CSS custom property, --wp--preset--shadow--crisp. Crisp is also a live-chat product. A text matcher reports a chat widget that isn't there.

Both are kept as negative tests. Plant a body-text matcher and they go red.

The other half of the rule is the vocabulary. When no host evidence turns up, the brief says not detected, never absent. A booking system can sit behind a button the crawler never clicked, or be wired up server-side. "We didn't see one" is a fact. "They don't have one" is a guess, and it becomes a question for the call instead.

Keeping the model away from the page text

The brief also has a judgement layer. Three model-based reviewers each look at one area: whether the site gets found and turns visits into enquiries, how enquiries are followed up, and what software the business runs on. They write prose; code checks it.

The obvious design gives each reviewer the crawl, which would undo the rule above. A model handed the dental site's HTML sees "dentally" six times and is likely to conclude the practice uses Dentally. Origin-based detection only helps if nothing downstream re-reads the text.

So the reviewers never see the crawl. The engine dumps its evidence once, then cuts a projection per reviewer: worked-out findings (vendors with their hosts, structured-data findings, which sector workflows had evidence) plus page metadata such as titles, headings and schema types. No page body, not even truncated. In the sample dump the raw pages were 127KB of a 174KB file, all of it material the rule forbids.

Page metadata is still text the business wrote, and "dentally phobic" can turn up in a meta description. So each projection also carries a contract block saying vendor claims come from the stack findings only. The projection removes most of the risk. The contract covers the rest.

Two more limits are computed in code and travel inside the projection, so a reviewer can't widen them:

• The follow-up reviewer is forbidden a verdict. A website can show there are three ways to get in touch and no booking widget. It cannot show who answers the phone at 6pm.
• The website reviewer loses its verdict when the crawl fetched less than 60% of the URLs it discovered. Judging a whole site from a fifth of it is the same mistake as judging something invisible.

The gate writes the file, not the person running it

The last guard is a gate script that runs over the composed brief before it is saved. One check is a deliberately blunt regular expression:

G7_PATTERN = re.compile(r"/10\b|/100\b|[0-9]/5\b|losing £|lost revenue")

It exists because this tool replaced an older audit that produced scores. That version scored eleven areas, but six couldn't be measured from free public data, so its headline number was an average over only the areas it could score. The new brief has no score, no grade and no money estimate, and this pattern keeps them out. It over-catches on purpose (a URL path like /100-club/ trips it), because clearing a false positive takes a minute and a scored brief that ships costs the call.

The other change is who writes the file. Originally the gate printed pass or fail and the person running it copied the draft across, so a failing draft could land because nobody checked an exit code. Now the gate takes --write-to and does the write itself. On failure it writes nothing and leaves the previous brief untouched.

A smaller one: the crawler refuses to audit itself

The older audit once reported that a live business had no HTTPS. It had been pointed at a local copy of the site.

The new runner refuses loopback addresses and exits with code 2. The first version compared hostnames as strings, which misses most of the ways to write "this machine". The current check normalises first, then parses with both ipaddress and socket.inet_aton, which accepts the short, decimal and hex IPv4 forms a browser accepts:

try:
    ip = ipaddress.ip_address(h)
except ValueError:
    try:
        # 127.1, 2130706433 and 0x7f000001 all parse here
        ip = ipaddress.ip_address(socket.inet_ntoa(socket.inet_aton(h)))
    except OSError:
        return False
if isinstance(ip, ipaddress.IPv6Address) and ip.ipv4_mapped is not None:
    ip = ip.ipv4_mapped
return bool(ip.is_loopback or ip.is_unspecified)

[::1], [::ffff:127.0.0.1], 0.0.0.0 and a trailing-dot 127.0.0.1. are refused; a real domain like localhost-dental.co.uk is not. There's no DNS lookup, on purpose: the offline modes must make no network calls, and a lookup during argument parsing would be one.

What I'd tell anyone writing a site analyser

• Decide what counts as evidence before you write a detector. "A host served an asset" is evidence. "The page mentions a word" is not.
• Write the negative fixtures early, from real pages that contain the vendor's name without using the vendor.
• Plant the wrong answer and watch for red. If the suite stays green, the test is checking a case where right and wrong coincide.

The brief is an internal document that prepares a first conversation with a business, the kind of discovery work described on our AI consultancy page, and its unanswered questions are asked there rather than guessed at. Every one of the fixes above came from asking what that brief might say about a real business that wasn't true.

This article was drafted with AI assistance, working from the project's own code, commit history and test fixtures.

Comments

[Sid Mishra]

The 1-of-3 fixture is the best part of this. Tests where the right and wrong answers coincide are so easy to write by accident. One more false-positive source on the "who built it" field, since evidence now comes from asset hosts: shared CDNs and tag managers. jsDelivr, cdnjs, unpkg and anything loaded through GTM will look like vendors unless you denylist them as infrastructure too. The two strongest agency signals I've found are a footer credit link whose anchor text says "website by" / "designed by", and the <meta name="generator"> tag plus theme paths (e.g. /wp-content/themes/<agency-name>/). Those beat link-path heuristics by a wide margin.

[STAE HIM]

Having the gate own --write-to removes a real failure path. Keeping the previous brief untouched needs a freshness check too, though: a downstream reader could mistake that old file for the new run's output after a failure. Would a success manifest containing the run ID and crawl revision make that boundary explicit?

[오피셜메일]

not detected와 absent를 구분한 부분이 특히 중요하네요. 관찰하지 못한 것을 부재로 단정하지 않고 다음 통화의 질문으로 넘기니까요. 1/3과 3/3이 갈리는 고정 데이터를 넣어 변이를 잡은 사례도, 테스트 개수보다 오답을 구별하는 입력이 중요하다는 점을 잘 보여줍니다.