I was comparing crypto trading terminals and did what I usually do first: curl the
top-ranking page and read the HTML instead of the rendering.
The site ranks first for its target keyword. Twelve pages, server-rendered, sub-second
response, clean markup. And when I grepped the raw HTML for a competitor's brand
name, it appeared thirty-two times across eight of the twelve pages.
Here is the whole audit, in commands you can run against any site in about five minutes.
1. Fetch the page, don't screenshot it
curl -sSL -A "Mozilla/5.0" https://example.com/ -o page.html
wc -c page.html
-L follows redirects, -A sets a normal user agent because plenty of sites serve
something different to a bare curl.
First surprise: some marketing sites return 80 KB of HTML where almost none of it is
content — it is a JavaScript bundle manifest. Others return 2 KB because the whole page
is client-rendered. Both tell you something before you have read a word.
2. Read the head
grep -aoE '<title>[^<]*</title>' page.html
grep -aoE '<meta name="description" content="[^"]*"' page.html
grep -aoE '<link rel="canonical" href="[^"]*"' page.html
The
-aflag matters.greptreats files with unusual bytes as binary and silently
prints nothing instead of your matches. I lost twenty minutes to this once.-a
forces it to treat everything as text.
3. Count the structured data
grep -ao 'application/ld+json' page.html | wc -l
This returned zero. No Organization, no WebSite, no Product — nothing. For a
site ranking first on a branded query that is a surprising gap, and a useful signal:
whoever built it was not optimising, they were shipping a template.
4. Check the sitemap actually points at itself
curl -sS https://example.com/sitemap.xml | grep -oE '<loc>[^<]+' | head
Every URL pointed at a different domain. The sitemap had been copied from a sibling
site and never updated, which means most of it is being ignored by every crawler that
reads it.
Note the counting trick here:
# WRONG — counts lines, and XML is often one single line
curl -sS .../sitemap.xml | grep -c "<loc>" # → 1
# RIGHT — counts occurrences
curl -sS .../sitemap.xml | grep -o "<loc>" | wc -l # → 19
I have watched people conclude a sitemap was broken because of exactly that.
5. The one that mattered
Pull every page, then look for a name that should not be there:
for p in product trading fees faq security rewards portfolio; do
curl -sSL -A "Mozilla/5.0" "https://example.com/$p" -o "$p.html"
sleep 1
done
for f in *.html; do
printf "%-16s %s\n" "$f" "$(grep -aoi 'competitorname' "$f" | wc -l)"
done
Eight files, thirty-two hits. The fees page had the competitor's name inside an <h2>.
The architecture page opened with "CompetitorName is built on industry-leading
infrastructure."
Why this happens
Affiliate templates get resold. Someone buys a site built for product A, swaps the logo
and the hero copy, and ships it for product B.
The visible headings get changed, because those are what you see in a screenshot. Body
copy on secondary pages does not, because nobody reads their own page five.
The errors are not cosmetic either. That template credited the wrong MEV protection
provider, a perpetuals integration the product does not have, and a fiat on-ramp that
does not exist on it. Anyone choosing between tools based on that page would be choosing
on invented specifications.
Three things that generalise
Grep the HTML, not the screenshot. Rendered pages hide their history. Raw markup
keeps it — in leftover brand names, in stale sitemaps, in og:image URLs still pointing
at a builder's preview domain. That last one told me exactly which no-code tool built
the site.
grep -aoE '<meta property="og:image" content="[^"]*"' page.html
Render only when you must. If a plain fetch returns a 2 KB shell, the content is
client-side and you need a real browser. Do not conclude a page is empty because curl
saw nothing — and do not skip the raw fetch either, because the browser will not show
you the leftovers.
Ranking first does not mean it is correct. This page outranks better-researched
pages because it has an older domain, an exact-match title, and a thin competitive
field. Search rank measures how well a page matches a query. It has never measured
whether the page is right.
If you are building a comparison or documentation site in any vertical, that gap is the
opportunity: correctness is a differentiator precisely because ranking does not require
it.
A small script to keep
#!/usr/bin/env bash
# audit.sh <url> [term-to-hunt]
set -euo pipefail
url="$1"; term="${2:-}"
tmp=$(mktemp)
curl -sSL -A "Mozilla/5.0" "$url" -o "$tmp"
printf "bytes %s\n" "$(wc -c < "$tmp")"
printf "title %s\n" "$(grep -aoE '<title>[^<]*' "$tmp" | head -1 | cut -c8-)"
printf "canonical %s\n" "$(grep -aoE 'rel="canonical" href="[^"]*"' "$tmp" | head -1)"
printf "json-ld %s block(s)\n" "$(grep -ao 'application/ld+json' "$tmp" | wc -l | tr -d ' ')"
printf "og:image %s\n" "$(grep -aoE 'og:image" content="[^"]*"' "$tmp" | head -1)"
[ -n "$term" ] && printf "'%s' %s hit(s)\n" "$term" "$(grep -aoi "$term" "$tmp" | wc -l | tr -d ' ')"
rm -f "$tmp"
chmod +x audit.sh && ./audit.sh https://example.com competitorname
Five checks, one file, no dependencies. It has caught a stale canonical, a sitemap
pointing at the wrong host, and a competitor's brand name in production — all in sites
that looked completely fine in a browser.
I write up the corrected data for the product in question at padrebot.com,
and keep the machine-readable version in a public repo.
That site is affiliate-funded and says so.
What is the strangest thing you have found in someone's raw HTML?
Top comments (0)