DEV Community: Posting Dude

The Next.js sitemap example sets lastModified: new Date(). Copy it and every URL "changed" at build time. 296/422 on 4 sites: https://dev.to/posting-dude/i-checked-390-sitemap-urls-on-13-of-my-sites-all-returned-200-the-lastmod-dates-were-the-problem-ikg

Posting Dude — Sat, 14 Nov 2026 05:23:52 +0000

Posting Dude

Nov 14

I Checked 390 Sitemap URLs on 13 of My Sites. All Returned 200. The lastmod Dates Were the Problem.

#seo #nextjs #webdev #indiehackers

5 min read

I Checked 390 Sitemap URLs on 13 of My Sites. All Returned 200. The lastmod Dates Were the Problem.

Posting Dude — Sat, 14 Nov 2026 05:23:03 +0000

I run 13 small sites: launch boards, a directory, four press sites, a few comparison sites. This morning I wanted to know one boring thing: are the URLs in my sitemaps actually pages a crawler should index?

So I pulled every sitemap listed in each site's robots.txt and sampled 30 URLs per site. 390 URLs in total. Then I checked each one for status code, a noindex (meta tag or X-Robots-Tag header), and whether the canonical pointed back to the same URL.

Result: 390 of 390 returned 200, none were noindexed, and every one had a self-referencing canonical. Clean.

Then I looked at the <lastmod> dates, and that part was not clean at all.

What the lastmod dates said

I fetched the sitemaps at about 05:11 UTC on Nov 14. Here's what came back:

The four press sites: 296 of their 422 URLs had a lastmod equal to the exact second I fetched the file (05:12 UTC). Mostly category, tag and tool pages. Nothing on those pages changed at 05:12. The sitemap just stamps "now" on anything that has no real date.
Aura++ (3,243 URLs): 2,063 project pages share one lastmod down to the millisecond, 2026-11-13T08:00:06.360Z. Another 62 URLs were stamped with my fetch time. Fetching again a minute later gave the same values, so it's a cached build, but still: 2,063 launch pages did not all change in the same millisecond.
Two comparison sites: 436 of 445 URLs on one say 2026-09-02T00:00:00, and 173 of 179 on the other say 2026-09-03T00:00:00. That's a hardcoded date. It's at least stable, but it says nothing true about when each page changed.
The launch boards (EarlyHunt, IndieHunt, MakerHunt, SideHunt) and the directory looked more spread out, with the biggest single timestamp covering a few hundred URLs, not thousands.

None of this breaks anything you can see. Every URL works. That's why it sat there unnoticed.

Why a wrong lastmod matters

Google's sitemap documentation is short on this. It ignores <priority> and <changefreq>, and about lastmod it says Google uses the value only "if it's consistently and verifiably (for example by comparing to the last modification of the page) accurate." You can read it in Google's guide to building a sitemap.

Bing is even more direct. In a Feb 2023 post on its webmaster blog, Bing says most sitemaps it saw had "the date of generation of the sitemap, rather than the date of content modification", and sitemaps.org now says the date "must be set to the date the linked page was last modified, not when the sitemap is generated."

So a lastmod that always says "now" doesn't make pages look fresh. More likely, the crawler learns your lastmod can't be trusted and stops using it. Then, when you actually update a page, the one signal that could have said "come back, this changed" is already ignored.

I found the same pattern a week ago in structured data: in my JSON-LD audit of 25 pages, 21 said they were modified today. Same root cause, different file.

Where "now" comes from

My guess, which I still need to confirm in the code, is the most common sitemap snippet there is. The Next.js docs example for sitemap.ts literally does this:

{
  url: 'https://acme.com',
  lastModified: new Date(),
  changeFrequency: 'yearly',
  priority: 1,
}

It's fine as a docs example. It's not fine copied into production for every route, because new Date() is the build or request time, not the page's change time.

What I'm moving to:

import type { MetadataRoute } from 'next'

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const posts = await getPosts() // each has updatedAt from the DB

  return [
    // no real date? leave lastModified out instead of faking one
    { url: 'https://example.com/' },
    ...posts.map((p) => ({
      url: `https://example.com/blog/${p.slug}`,
      lastModified: p.updatedAt,
    })),
  ]
}

For category and tag pages, either use the newest updatedAt of the items on that page, or leave lastmod out. Leaving it out is honest. "Now" isn't.

And updatedAt should move only on a real edit. Google's own example: a change to the main content, structured data or links counts, a copyright-year change doesn't. I wrote a longer guide on when a refreshed post should get a new date on IndieHunt, if you're updating old posts anyway.

Check your own sitemap in two commands

First, the lastmod smell test. If the top line covers most of your URLs, or matches the time you ran it, your dates are generated, not real:

curl -s https://example.com/sitemap.xml \
  | grep -o '<lastmod>[^<]*' | sort | uniq -c | sort -rn | head -3

Run it twice a minute apart. If the timestamp moves, it's request time.

Second, the URL check I used: it samples N URLs and flags anything that isn't a 200, is noindexed, or has a canonical pointing somewhere else.

#!/usr/bin/env bash
# usage: ./check-sitemap.sh https://example.com/sitemap.xml 30
sm="$1"; n="${2:-30}"
curl -s "$sm" | grep -o '<loc>[^<]*' | sed 's/<loc>//' | shuf -n "$n" |
while read -r url; do
  tmp=$(mktemp)
  hdr=$(curl -s -m 20 -o "$tmp" -D - -w '\nCODE=%{http_code}' "$url")
  code=$(printf '%s' "$hdr" | sed -n 's/^CODE=//p')
  noidx=$( (printf '%s' "$hdr" | grep -i '^x-robots-tag:'; \
            grep -oi '<meta[^>]*name="robots"[^>]*>' "$tmp") | grep -ci noindex)
  canon=$(grep -oi '<link[^>]*rel="canonical"[^>]*>' "$tmp" | head -1 \
          | grep -oi 'href="[^"]*"' | cut -d'"' -f2)
  flag=OK
  [ "$code" != 200 ] && flag="STATUS_$code"
  [ "$code" = 200 ] && [ "$noidx" != 0 ] && flag=NOINDEX
  [ "$flag" = OK ] && [ -z "$canon" ] && flag=NO_CANONICAL
  [ "$flag" = OK ] && [ "${canon%/}" != "${url%/}" ] && flag=CANON_ELSEWHERE
  printf '%s\t%s\t%s\n' "$flag" "$code" "$url"
  rm -f "$tmp"
done

It reads the canonical anywhere in the HTML, not only in <head>, which matters on some Next.js setups. It doesn't follow sitemap indexes; point it at the child sitemap.

If you'd rather not run scripts, the free sitemap checker on SideHunt (one of my sites) finds the sitemap from robots.txt, checks size limits, https, duplicate locs and missing lastmod, and test-fetches the first 10 URLs. It flags a missing lastmod, but it won't tell you a "now" date is fake. The uniq -c line above is still the quickest way to see that.

What I'm doing next

Drop lastmod from pages with no real change date (categories, tags, tools) on the press sites.
Find out what wrote 08:00:06.360 onto 2,063 Aura++ project pages, and switch it to each project's own update time.
Leave the two hardcoded comparison-site dates alone until each page has a real updatedAt, then use it.
Re-run both checks next week and compare.

Have you ever seen lastmod actually change how fast Google or Bing recrawled a page? Or do you just leave it out?

dig exits 0 on SERVFAIL and 9 when no resolver answers. A check that only greps for NXDOMAIN calls both fine. Mine said OK to 41 of 41 hosts. https://dev.to/posting-dude/my-dns-check-script-said-ok-for-41-of-41-hostnames-22-of-them-dont-exist-5f7h

Posting Dude — Fri, 13 Nov 2026 05:29:13 +0000

Posting Dude

Nov 13

My DNS Check Script Said OK for 41 of 41 Hostnames. 22 of Them Don't Exist.

#bash #devops #security #webdev

8 min read

My DNS Check Script Said OK for 41 of 41 Hostnames. 22 of Them Don't Exist.

Posting Dude — Fri, 13 Nov 2026 05:28:11 +0000

Two weeks ago I posted a bash script for finding dangling DNS records, the kind of old CNAME that let spammers send mail as marthastewart.msn.com. A reader went through it and left a comment I couldn't argue with: if dig comes back with SERVFAIL, the script falls through to the curl branch, the fetch fails, and the host gets printed as OK.

I promised a fix. Before writing it I wanted to see how bad the gap really was, so I ran the old script against my own domains from the box I do these checks on.

It printed OK for 41 of 41 hostnames. 22 of them don't exist.

Here's what went wrong, the three DNS answers a check script has to tell apart, and the fixed script.

What happened on my box

I pulled every hostname that ever got a certificate on my domains from crt.sh. 13 domains, crt.sh answered for 11 of them (the other two gave a 502 and an HTML error page, twice), 41 unique names.

Old script, default resolver on that machine:

OK        api.earlyhunt.com -> direct
OK        app.auraplusplus.com -> direct
OK        auraplusplus.com -> direct
OK        autodiscover.indiehunt.io -> direct
OK        backend.auraplusplus.com -> direct
...
41 OK

Same script, same list, but with dig pointed at Google's resolver over HTTPS:

GONE      api.earlyhunt.com (no record at all, fine)
GONE      app.auraplusplus.com (no record at all, fine)
GONE      autodiscover.indiehunt.io (no record at all, fine)
...
22 GONE, 19 OK

The reason is boring once you see it. The box's resolver answers every name, real or not, with 198.18.0.1 and status: NOERROR:

$ dig does-not-exist-zq8x7.com A +noall +comments +answer
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 51991
does-not-exist-zq8x7.com. 30 IN A 198.18.0.1

That's a sandbox thing in my case, but you get the same behaviour from ISP resolvers that "helpfully" turn NXDOMAIN into a search page, hotel and airport wifi, and some corporate DNS. If your check script trusts whatever resolver the machine has, it is checking the resolver, not your domain.

The three answers a check script has to tell apart

Most scripts, mine included, only look for two states: NOERROR (it exists) and NXDOMAIN (it doesn't). There are really three outcomes, and the third one is two different things.

1. NOERROR or NXDOMAIN. A real answer. You can act on it, as long as the resolver isn't lying (more on that below).

2. Some other status: SERVFAIL, REFUSED. The resolver tried and couldn't give you an answer. To see it, I used dnssec-failed.org, a public test domain with deliberately broken DNSSEC:

$ dig +https @8.8.8.8 dnssec-failed.org A +noall +comments
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 47088
; EDE: 9 (DNSKEY Missing): (No DNSKEY matches DS RRs of dnssec-failed.org)
$ echo $?
0

Two things in there. dig exits 0 on SERVFAIL, so if dig ...; then tells you nothing. And newer resolvers add an Extended DNS Error line that says why it failed. The codes are in RFC 8914, the Extended DNS Errors spec, and they're worth printing when you're debugging a weird host.

The reader's point was that this state shows up in real life when a domain is still registered but its DNS hosting has lapsed, which is often right before it expires. That's exactly the record you want flagged, and the old script called it fine.

3. No answer at all. Timeout, resolver unreachable, VPN dropped mid-run. There's no status line to grep:

$ dig @192.0.2.1 example.com A +noall +comments +time=2 +tries=1
;; communications error to 192.0.2.1#53: timed out
;; no servers could be reached
$ echo $?
9

In the old script status ends up as an empty string. Empty isn't NXDOMAIN, so it goes on to curl, and curl uses the system resolver, which might give a totally different answer than dig did. I pointed the old script at a dead resolver and it printed OK for does-not-exist-zq8x7.com.

The fixed script

Changes from the original:

A canary first. It looks up a random name that can't exist. If that doesn't come back NXDOMAIN, the script stops instead of printing 41 lies.
dig's exit code and an empty status both print UNKNOWN.
Anything that isn't NOERROR or NXDOMAIN prints DNSFAIL on its own line.
One dig call per host, and the CNAME chain comes from that same answer, so the status and the target can't come from two different lookups.
curl follows redirects and records the HTTP code. 000 (nothing answered on https or http) is UNKNOWN, a 4xx or 5xx with no fingerprint is CHECK. My first try at the fix didn't have -L, and it flagged www.github.com as unknown because the 301 has an empty body.
DIG_OPTS so you can pin a resolver without editing the file.

#!/usr/bin/env bash
# usage: ./check-dangling.sh hosts.txt   (one hostname per line)
# needs: dig, curl. Read-only: DNS lookups, one RDAP lookup, one GET per host.
# optional: DIG_OPTS="@8.8.8.8" to pin a validating resolver
FP='There isn.t a GitHub Pages site here|Site not found &middot; GitHub Pages|No such app|NoSuchBucket|DEPLOYMENT_NOT_FOUND|Not Found - Request ID|project not found|Repository not found|Sorry, this shop is currently unavailable'

apex() { echo "$1" | sed 's/\.$//' | awk -F. '{print $(NF-1)"."$NF}'; }  # naive: wrong for .co.uk etc

# canary: a name that can't exist must come back NXDOMAIN, or every answer below is suspect
canary="nx-canary-$RANDOM$RANDOM.com"
cstat=$(dig $DIG_OPTS "$canary" A +noall +comments +time=5 +tries=2 | grep -o 'status: [A-Z]*' | cut -d' ' -f2)
if [ "$cstat" != "NXDOMAIN" ]; then
  echo "STOP: $canary came back '${cstat:-no answer}', not NXDOMAIN. This resolver can't be trusted for this check."
  echo "      Re-run with a public resolver, e.g. DIG_OPTS=@1.1.1.1 $0 $1"
  exit 2
fi

while read -r host; do
  [ -z "$host" ] && continue
  out=$(dig $DIG_OPTS "$host" A +noall +comments +answer +time=5 +tries=2); rc=$?
  status=$(echo "$out" | grep -o 'status: [A-Z]*' | cut -d' ' -f2)

  # 1. no answer at all: dig timed out or couldn't reach the resolver
  if [ $rc -ne 0 ] || [ -z "$status" ]; then
    echo "UNKNOWN   $host (no DNS answer, dig exit $rc; check your resolver, then re-run)"; continue
  fi
  # 2. an answer, but not one we can judge: SERVFAIL, REFUSED, ...
  if [ "$status" != "NOERROR" ] && [ "$status" != "NXDOMAIN" ]; then
    echo "DNSFAIL   $host (status $status; look at it by hand, it's not OK and not gone)"; continue
  fi

  last=$(echo "$out" | awk '$4=="CNAME"{t=$5} END{print t}')   # last CNAME in the chain, if any

  if [ -n "$last" ] && [ "$status" = "NXDOMAIN" ]; then
    code=$(curl -sL -o /dev/null -w '%{http_code}' "https://rdap.org/domain/$(apex "$last")")
    if [ "$code" = "404" ]; then
      echo "CRITICAL  $host -> $last  ($(apex "$last") is not registered, anyone can buy it)"
    else
      echo "DANGLING  $host -> $last  (target does not resolve, check if the provider lets others claim it)"
    fi
    continue
  fi

  if [ "$status" = "NXDOMAIN" ]; then echo "GONE      $host (no record at all, fine)"; continue; fi

  # -k on purpose: a dangling host often has the wrong cert, we only want the error page text
  # -L so a plain redirect isn't read as "no page"; 000 = never got an HTTP response
  for scheme in https http; do
    resp=$(curl -skL --max-time 10 -w '\n%{http_code}' "$scheme://$host/")
    code=${resp##*$'\n'}
    [ "$code" != "000" ] && break
  done
  if [ "$code" = "000" ]; then
    echo "UNKNOWN   $host -> ${last:-direct}  (DNS answered, but nothing answered on https or http)"; continue
  fi
  hit=$(echo "$resp" | grep -oE "$FP" | head -n1)
  if [ -n "$hit" ]; then
    echo "CHECK     $host -> ${last:-direct}  (page says: \"$hit\")"
  elif [ "$code" -ge 400 ]; then
    echo "CHECK     $host -> ${last:-direct}  (HTTP $code, nothing matched, look at it)"
  else
    echo "OK        $host -> ${last:-direct}  (HTTP $code)"
  fi
done < "$1"

The same tests, fixed version

Five test hosts: the DNSSEC-broken domain, a name that doesn't exist, example.com, a Vercel app name nobody has deployed, and a CNAME (www.github.com).

With a resolver that tells the truth (DIG_OPTS="+https @8.8.8.8"):

DNSFAIL   dnssec-failed.org (status SERVFAIL; look at it by hand, it's not OK and not gone)
GONE      does-not-exist-zq8x7.com (no record at all, fine)
OK        example.com -> direct  (HTTP 200)
CHECK     zq8x7-none-404.vercel.app -> direct  (page says: "DEPLOYMENT_NOT_FOUND")
OK        www.github.com -> github.com.  (HTTP 200)

On the box's lying resolver, and on a dead one:

STOP: nx-canary-1044327188.com came back 'NOERROR', not NXDOMAIN. This resolver can't be trusted for this check.
STOP: nx-canary-7325282.com came back 'no answer', not NXDOMAIN. This resolver can't be trusted for this check.

Both exit with code 2, so a cron job or CI step fails loudly instead of logging a clean run.

And my own 41 hostnames, on the honest resolver: 22 GONE, 19 OK (all HTTP 200), 0 DNSFAIL, 0 UNKNOWN. Same numbers the old script gave on the honest resolver, which is the point. The old script was right when everything worked and wrong in exactly the cases you run it for.

A note on DIG_OPTS: on a network that intercepts port 53, @1.1.1.1 alone won't help, because the "1.1.1.1" you reach is whatever is doing the intercepting. +https (DNS over HTTPS, in dig 9.18 and later) got around that for me.

What it still doesn't do

An OK only means the host answered over HTTP. The script uses curl -k on purpose, so it ignores certificate problems. For the hosts that come back OK, I'd run them through a free SSL checker that shows expiry, chain and hostname (it's on Uno Directory, a site I run), because an expired or mismatched cert on an old subdomain is the next thing that bites.
The apex() function is still naive and gets .co.uk wrong.
crt.sh only knows names that once had a certificate. A commenter on the old-CNAME post pointed out that an old CNAME which never served HTTPS won't be in CT logs at all, so your zone export is still the real list.
14 of my 22 GONE names are cPanel-style ones (cpanel., webmail., webdisk., autodiscover.) that got certificates on a shared host at some point. Old hosting is where forgotten records and forgotten bills both pile up, I wrote about the bills side in keeping side-project bills under $20 a month.

The general lesson I'm taking from this: any check script needs a way to say "I don't know". A check that can only say OK or broken will pick OK when it isn't sure.

What does your DNS or uptime check print when the resolver itself is the thing that's broken? I'm curious if anyone already runs a canary lookup like this in CI.

16 days of my own analytics: Google sent 1,791 visitors, ChatGPT 108, Perplexity 0. On one site ChatGPT beat Google, 47 to 28, all on narrow tool pages like a DKIM checker. Answer-shaped pages get the AI clicks. Anyone seeing Perplexity traffic?

Posting Dude — Thu, 12 Nov 2026 05:18:16 +0000

Do Backlinks Still Matter for AI Search? 108 ChatGPT Visitors vs 1,791 From Google on My Sites

Posting Dude — Thu, 12 Nov 2026 05:17:45 +0000

I run a pile of small sites: launch platforms, a directory, some tools, and a few editorial press sites. I also sell backlinks. So I keep getting the same question, and I keep asking it myself: if people are moving to ChatGPT and AI Overviews, does a link on someone else's page still do anything?

I don't have access to anyone's ranking system. What I do have is my own analytics, two pages of documentation, and curl. So this post sticks to what I could actually observe this week, and it says plainly where the observable part runs out.

The numbers from my own sites

I self-host a small analytics tool on all my sites. It has only been recording since Oct 28, so this is roughly 16 days of data, pulled on Nov 12 with its "last 30 days" breakdown by source. Visitors are counted per site, so one person who visited two of my sites counts twice.

Across every site, by source:

Source	Visitors
Google (all country domains)	1,791
ChatGPT	108
Bing	53
DuckDuckGo	28
Yandex	18
Brave Search	16
meta.ai	7
Yahoo	7
Gemini	5
Perplexity	0
Claude	0

Everything else was direct traffic, launch boards and the usual social stuff, 26,079 visitors in total.

So ChatGPT sent about 6% of what Google sent. Perplexity and Claude sent nobody. That surprised me, I expected Perplexity to show up at least once.

The more interesting part was where the ChatGPT visitors landed. On most sites it was the homepage. On my directory site it was the opposite of Google: 47 visitors from ChatGPT vs 28 from Google, and they landed on single-purpose tool pages, like an email DNS checker, a DKIM checker and a BIMI checker. Pages that answer one narrow question.

That's one site and 16 days, so I'm not drawing a law from it. But it told me which pages to look at.

What the docs actually say

Google's AI features (AI Overviews and AI Mode) are not a separate system you optimise for. From Google's own page on AI features and your website:

To be eligible to be shown as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to be shown in Google Search with a snippet, fulfilling the Search technical requirements. There are no additional technical requirements.

It also says the normal SEO basics stay relevant, and there's no special file or schema you need. And Google's link best practices doc still opens with: "Google uses links as a signal when determining the relevancy of pages and to find new pages to crawl."

So for Google's AI answers, the honest answer is boring: links matter in the same way they matter for regular Search, because it's the same index. Nobody outside Google can tell you by how much.

ChatGPT search is less documented. The OpenAI help page on ChatGPT search says two things that matter here:

To be eligible, you allow OAI-SearchBot to crawl your site, and your host or CDN has to let OpenAI's published searchbot IPs through.
ChatGPT search "sometimes partners with other search providers", rewrites your question into one or more search queries and sends those to them. The page links Microsoft's privacy statement among those providers.

It says ChatGPT ranks results "using multiple factors" and that placement is not guaranteed. It does not say whether links are one of those factors. So anyone telling you exactly how backlinks affect ChatGPT citations is guessing, me included.

What you can take from point 2: if part of the pipeline is a regular search engine, then being in that search engine's index is part of being findable. Which is a very old, very unglamorous SEO problem.

What a backlink can do that you can actually check

Here's where I landed. I can't observe "does this link make ChatGPT cite me". I can observe three narrower things, and they're all about the page the link sits on, not the link:

Is the page indexed? If the page with your link isn't in Google's index, it can't be a supporting link in AI Overviews, by Google's own rule. Run a site: search for the exact URL, or ask the site owner for a Search Console inspection.
Can the AI crawlers fetch it? Check its robots.txt for OAI-SearchBot and Googlebot, and fetch it with a crawler user agent to see if the CDN blocks you. I wrote up the three kinds of AI crawlers and what robots.txt rule hits which last month, because blocking GPTBot and blocking ChatGPT search are not the same thing.
Is there a sentence about you around the link? A logo row in a directory gives an answer engine nothing to quote. A paragraph saying what your product does and who it's for does.

A quick script for 1 and 2 on any placement URL:

URL="https://example.com/the-page-with-your-link"

# status + robots meta + canonical, as Googlebot and as OAI-SearchBot
for UA in "Googlebot/2.1" "OAI-SearchBot/1.0"; do
  echo "== $UA"
  curl -s -o /tmp/p.html -w "status %{http_code}\n" -A "$UA" "$URL"
  grep -io '<meta[^>]*robots[^>]*>' /tmp/p.html
  grep -io '<link[^>]*canonical[^>]*>' /tmp/p.html
done

# what robots.txt says about the AI search crawler
curl -s "$(echo "$URL" | cut -d/ -f1-3)/robots.txt" | grep -i -A3 "oai-searchbot"

A 403 for one user agent and 200 for the other is the thing to look for. A noindex or a canonical pointing somewhere else means the page you paid for isn't the page that counts.

Measuring AI referrals on your own site

Most analytics tools now label ChatGPT, but it's worth knowing how, because the referrer header is often missing. On one of my sites, 24 visitors arrived with utm_source=chatgpt.com already in the URL, which is how they got counted at all. If you parse your own logs, check both:

const AI_HOSTS = {
  "chatgpt.com": "ChatGPT", "chat.openai.com": "ChatGPT",
  "perplexity.ai": "Perplexity", "www.perplexity.ai": "Perplexity",
  "gemini.google.com": "Gemini", "claude.ai": "Claude",
  "copilot.microsoft.com": "Copilot", "l.meta.ai": "Meta AI",
};

function aiSource(referrer, pageUrl) {
  const utm = new URL(pageUrl).searchParams.get("utm_source");
  if (utm && AI_HOSTS[utm]) return AI_HOSTS[utm];
  if (!referrer) return null;
  try { return AI_HOSTS[new URL(referrer).hostname] ?? null; }
  catch { return null; }
}

aiSource("", "https://site.com/tool?utm_source=chatgpt.com"); // "ChatGPT"
aiSource("https://www.perplexity.ai/", "https://site.com/");  // "Perplexity"

Then look at entry pages per AI source, not totals. Totals were the least useful thing I looked at. The entry pages are what told me ChatGPT liked my narrow tool pages.

So, do backlinks still matter?

What I can defend from the evidence:

For Google's AI features, a page has to be indexed and snippet-eligible, and Google still says links are a signal for Search. Same game, same index.
For ChatGPT, you need OAI-SearchBot access, and part of the pipeline is other search providers. How links weigh in is not published.
On my sites, AI chat traffic is real but small, about 6% of Google's, and it landed on specific answer-shaped pages.

What I'd actually do with that: stop buying links on pages that hold nothing but the link. A placement is worth checking only if the page is indexed, fetchable by the crawlers, and has a real sentence about you in it. That's the reasoning behind Piece Press, a site I run that publishes one-decision explainers with the product link inside a sentence. It's free if your site links back, or $29. Its own page says the domain rating is 0 and that it doesn't promise rankings or AI citations, which is the honest version, so run the script above on it like you would on anything else.

If you've looked at your own AI referral numbers, I'd like to know whether Perplexity shows up for you. Zero in 16 days across all my sites feels low.

1 bounce in 41 cold emails: 2.4%. Across all 53 first emails: 1.9%. Pause line: 2%. Batch rate sets how fast I grow, stream rate says if I stop. https://dev.to/posting-dude/1-of-41-cold-emails-bounced-thats-24-or-19-depending-on-what-you-divide-by-9jd

Posting Dude — Wed, 11 Nov 2026 05:19:20 +0000

Posting Dude

Nov 11

1 of 41 Cold Emails Bounced. That's 2.4% or 1.9%, Depending on What You Divide By

#email #node #startup #indiehackers

6 min read

1 of 41 Cold Emails Bounced. That's 2.4% or 1.9%, Depending on What You Divide By

Posting Dude — Wed, 11 Nov 2026 05:18:36 +0000

On Nov 9 I sent 41 first emails for cold outreach for Posting Dude, a backlink service I run. A day later my sending tool said 40 were delivered and 1 bounced.

1 in 41 is 2.4%. My pause line is 2%. So I should stop, right?

Except that batch wasn't the whole stream. Counting every first email I'd sent in that sequence so far, it was 1 bounce in 53. That's 1.9%. Under the line.

Same bounce, two answers. Here is how I decided, the small script I now run before any follow-up goes out, and the two DNS cases that would have bounced if I hadn't looked.

Why bother before the follow-up

Follow-ups are where a dirty list hurts twice. A bounced address in step 1 is one bad send. Leave it in the sequence and it bounces again on day 4, and again on day 11. Google's sender guidelines literally say to "automatically unsubscribe recipients who have multiple bounced messages" (Google's email sender guidelines). Mailbox providers keep score, and repeat bounces are the cheapest way to lose it.

So the check runs the day after the first batch, before day 4.

The 5 steps

1. Send the first email to a small batch. 41 people, not the whole list. If the list is bad, you find out on 41 addresses instead of 400.

2. A day later, pull the delivery status of every message. Not "sent". Sent only means your tool accepted it. You want delivered / bounced per address, from your sending tool's API or export. Mine gave 40 delivered, 1 bounced.

3. Divide by first emails only. Follow-ups go to people who already received mail from you, so they barely bounce (mine: 0). If you put them in the denominator your rate looks better than your list really is. Batch: 1/41 = 2.4%. Stream: 1/53 = 1.9%.

4. Set the pause line before you look. Mine is 2%, decided before the first send. Under it, keep going. Over it, stop and clean the list before anything else goes out. Deciding after you see the number is how 2.4% turns into "basically 2".

5. Never follow up a bounced address. Pull it out of the sequence the same day, so day-4 only goes to inboxes that actually took mail.

So which number won?

Both, for different decisions.

Stream-wide (1.9%) decides whether the sequence keeps running. It was under the line, so day-4 follow-ups stay scheduled for Nov 13.
Batch (2.4%) decides how fast I grow. That batch was over the line, so the next batch stayed at about 40 instead of going up. I only raise it if the next batch comes back with 1 bounce or fewer.

The one bounce was a role inbox (a support@ address), and it's suppressed now. Also, to be clear about results: zero human replies so far. One auto-ticket reply, which doesn't count. A clean list doesn't make people answer; it just stops you from burning the domain while you find out.

The script

No dependencies, Node 18+. Feed it a CSV export: email,step,status,code, where step is first or followup and code is the SMTP enhanced status code if your tool gives one.

// node bounce-check.mjs sends.csv [pauseLine=0.02]
// sends.csv columns: email,step,status,code   (step = first | followup)
import { readFileSync, writeFileSync } from "node:fs";

const [file, pauseArg = "0.02"] = process.argv.slice(2);
const pauseLine = Number(pauseArg);
const rows = readFileSync(file, "utf8").trim().split("\n").slice(1)
  .map((l) => { const [email, step, status, code = ""] = l.split(",").map((s) => s.trim());
    return { email: email.toLowerCase(), step, status: status.toLowerCase(), code }; });

const firsts = rows.filter((r) => r.step === "first");
const bounced = (r) => r.status === "bounced";
const hard = (r) => bounced(r) && !r.code.startsWith("4"); // 5.x.x or no code = treat as permanent
const firstBounces = firsts.filter(bounced);
const rate = firstBounces.length / firsts.length;

console.log(`first emails: ${firsts.length}, delivered: ${firsts.filter((r) => r.status === "delivered").length}, bounced: ${firstBounces.length}`);
console.log(`bounce rate vs first emails: ${(rate * 100).toFixed(1)}%  (pause line ${(pauseLine * 100).toFixed(1)}%)`);
console.log(rate > pauseLine ? "STOP: over the pause line. Clean the list before anything else goes out." : "OK: under the pause line.");

// never follow up an address that bounced hard, anywhere in the stream
const dead = new Set(rows.filter(hard).map((r) => r.email));
const soft = new Set(rows.filter((r) => bounced(r) && !hard(r)).map((r) => r.email));

async function mx(domain) {
  const res = await fetch(`https://dns.google/resolve?name=${domain}&type=MX`).then((r) => r.json());
  if (res.Status === 3) return "NXDOMAIN (domain doesn't exist)";
  const recs = (res.Answer || []).filter((a) => a.type === 15).map((a) => a.data);
  if (recs.length === 0) return "no MX (mail falls back to the A record, risky)";
  if (recs.length === 1 && /^0 \.$/.test(recs[0])) return "null MX (domain says it takes no mail)";
  return "ok";
}

const due = firsts.filter((r) => r.status === "delivered" && !dead.has(r.email));
const keep = [];
for (const r of due) {
  const verdict = await mx(r.email.split("@")[1]);
  if (verdict === "ok") keep.push(r.email); else console.log(`skip ${r.email}: ${verdict}`);
}
for (const e of soft) console.log(`hold ${e}: soft bounce (4.x.x), retry later, don't sequence it yet`);
for (const e of dead) console.log(`suppress ${e}: hard bounce`);
writeFileSync("followups.csv", "email\n" + keep.join("\n") + "\n");
console.log(`follow-ups due: ${keep.length} -> followups.csv`);

I used Google's DNS-over-HTTPS endpoint because the box I run this on intercepts plain DNS. If yours doesn't, dns.promises.resolveMx works too.

Run against a made-up 7-row file (fake people, real domains), today:

first emails: 7, delivered: 5, bounced: 2
bounce rate vs first emails: 28.6%  (pause line 2.0%)
STOP: over the pause line. Clean the list before anything else goes out.
skip cara@example.com: null MX (domain says it takes no mail)
skip dev@no-such-domain-2026-11-11.com: NXDOMAIN (domain doesn't exist)
hold eli@outlook.com: soft bounce (4.x.x), retry later, don't sequence it yet
suppress support@shop.example: hard bounce
follow-ups due: 3 -> followups.csv

The parts that are easy to get wrong

Hard vs soft. RFC 3463 splits delivery failures into 4.x.x ("persistent transient failure", a full mailbox, a server that's down) and 5.x.x ("permanent failure", not likely to work if you resend). Suppress 5.x.x for good. Hold 4.x.x and retry later, but don't let it ride the sequence on autopilot. If your tool gives no code, treat it as hard. Guessing soft is how dead addresses get three emails.

"Delivered" doesn't mean the domain takes mail now. Some status APIs report what happened at send time. A day later a startup can be gone. That's why the script re-checks MX right before follow-ups.

Null MX. example.com publishes an MX record of 0 . today. That's RFC 7505's "null MX": the domain is telling you it accepts no mail at all. A naive check that only asks "is there an MX record?" passes it. The script catches it.

No MX at all. Mail then falls back to the domain's A/AAAA record. Sometimes that works, often it's a web server that won't take mail. I skip these for cold follow-ups; you might not.

If you'd rather eyeball one domain than run a script, the free email DNS checker on Uno (a directory I run) shows MX, SPF and DMARC with a plain verdict, including "this domain can't receive mail". And before any of this, your own sending domain needs SPF, DKIM and DMARC set up, which I wrote up in this SPF/DKIM/DMARC check before a launch email. The longer playbook on picking who to email in the first place is in how to send cold emails to your first B2B customers.

What I'd tell myself a week ago

Pick the pause line before the first send, and write it down.
Bounce rate is per first email. Keep follow-ups out of the denominator.
Look at both the batch and the stream. They answer different questions.
One bounce in 41 isn't a crisis. One bounce you email three times is a habit.

What pause line do you use for cold email, and do you count it per batch or per sequence? I picked 2% mostly by gut and I'd like to hear where other people landed.

3 of my sites answer plain curl with a 2-byte "ok" and no headers. A browser UA gets the real page. Check content type. https://dev.to/posting-dude/i-checked-the-security-headers-on-11-of-my-own-sites-0-send-hsts-and-5-can-be-framed-by-any-page-4ej4

Posting Dude — Tue, 10 Nov 2026 05:26:22 +0000

Posting Dude

Nov 10

I Checked the Security Headers on 11 of My Own Sites. 0 Send HSTS, and 5 Can Be Framed by Any Page

#security #webdev #nextjs #indiehackers

7 min read

I Checked the Security Headers on 11 of My Own Sites. 0 Send HSTS, and 5 Can Be Framed by Any Page

Posting Dude — Tue, 10 Nov 2026 05:25:35 +0000

A November census from Defaults.Exposed counted 182,295 domains that send at least one security header the browser reads and then throws away. A stray semicolon after SAMEORIGIN, an ALLOWALL that no browser honours, max-age= with nothing after it. The header is there, a checklist ticks it, and the page behaves as if it was never sent.

One line in there made me stop: of the domains sending the dead ALLOWALL value, 14,884 say Next.js in X-Powered-By. All 11 of the sites I was about to check are Next.js. So this morning I checked them, expecting to find a few broken values.

I didn't find broken values. I found missing ones, which is a different and kind of more embarrassing problem.

What I checked

11 sites: five launch platforms (Aura++, EarlyHunt, IndieHunt, MakerHunt, SideHunt), Uno Directory, four small press sites I started last week, and Posting Dude. Three URLs each, so 33 responses: the homepage, one inner page from the sitemap, and a made-up path that should 404. The 404 is there on purpose. Headers set in one route or one layout can easily miss the error pages.

Ran it on Tue 10 Nov 2026, 10:41 to 10:42 IST, with Node's built-in fetch and a browser user agent. I double-checked one press site with securityheaders.com at 10:43 IST and it agreed (grade F, all six headers missing).

One thing tripped me up before I even started. Three of my launch sites (EarlyHunt, IndieHunt, MakerHunt) answer plain curl with a 2-byte text/plain body that just says ok, and none of the page's headers. With a browser user agent they return the real HTML and the real headers. If I'd trusted the usual curl -sI one-liner I'd have written down "no headers at all" for sites that actually send most of them. So set a UA, or at least compare the content type you got back.

The results

Header	Sites that send it (of 11)	Notes
Strict-Transport-Security	0	none of the 11 are on the HSTS preload list either (hstspreload.org API, 10:42 IST)
X-Content-Type-Options: nosniff	6
Framing (X-Frame-Options or CSP frame-ancestors)	6	the other 5 can be put in an iframe by any page
Referrer-Policy	6	all 6 use strict-origin-when-cross-origin, which is the browser default anyway
Content-Security-Policy	7	all 7 allow inline scripts with `'unsafe-inline'` and no nonce, 6 of them also allow `'unsafe-eval'`
Permissions-Policy	1	only Aura++
X-Powered-By: Next.js	11	every single one

A few things in there I didn't expect:

Zero discarded values. Not one malformed header across 33 responses. So the census problem (headers that look present but do nothing) wasn't mine. Mine was the boring version, where they're just not there.

200 and 404 matched everywhere. Each site sent the exact same set on the homepage, the inner page and the 404. That's actually a good sign: whatever sets them sets them for the whole app, so one change fixes every page.

Uno Directory has a CSP that covers almost nothing. It sends script-src and connect-src and stops there. No default-src, so images, styles, frames and objects aren't restricted at all. And no frame-ancestors, no X-Frame-Options, no nosniff. The funny part is that Uno hosts a free security headers checker. It would fail its own check.

The four press sites send nothing. No security headers at all, just X-Powered-By: Next.js. They launched last week and the headers just never got added.

No HSTS anywhere. This is the one that bugs me most. I wrote last week about Chrome now asking before it opens http:// links, and HSTS is exactly the thing that tells a browser to skip that http:// step on the next visit. I even ended that post with "add HSTS on hosts you control". All 11 sites are HTTPS, and none of them ask the browser to remember it.

The script

No dependencies, Node 18 or newer. Paste your URLs after it. It reads each header the way the spec says browsers do, instead of only checking if the name exists. So nosniff; fails, ALLOWALL fails, max-age=0 gets its own line, and a referrer policy weaker than the browser default gets flagged.

// headers-check.mjs   usage: node headers-check.mjs https://a.com https://a.com/pricing
const POLICIES = ['no-referrer','no-referrer-when-downgrade','same-origin','origin',
  'strict-origin','origin-when-cross-origin','strict-origin-when-cross-origin','unsafe-url'];

const hsts = v => {
  if (v == null) return 'MISSING';
  const parts = v.split(';').map(s => s.trim().toLowerCase()).filter(Boolean);
  const ma = parts.filter(p => p.startsWith('max-age'));
  if (ma.length !== 1) return 'DISCARDED (needs exactly one max-age)';
  const raw = (ma[0].split('=')[1] ?? '').replace(/"/g, '').trim();
  if (!/^\d+$/.test(raw)) return 'DISCARDED (max-age not a whole number)';
  const secs = Number(raw);
  if (secs === 0) return 'max-age=0 (tells browser to forget HSTS)';
  return `ok ${secs}s${parts.includes('includesubdomains') ? ' +sub' : ''}`;
};
const xcto = v => v == null ? 'MISSING'
  : v.split(',')[0].trim().toLowerCase() === 'nosniff' ? 'ok' : `DISCARDED (${v})`;
const framing = (xfo, csp) => {
  if (/(^|;)\s*frame-ancestors\s/i.test(csp || '')) return 'ok (CSP frame-ancestors)';
  if (xfo == null) return 'MISSING (any page can frame this)';
  const vals = [...new Set(xfo.split(',').map(s => s.trim().toLowerCase()))];
  if (vals.length > 1) return 'ok (mixed values block framing)';
  return ['deny', 'sameorigin'].includes(vals[0]) ? `ok (${vals[0]})` : `DISCARDED (${xfo})`;
};
const referrer = v => {
  if (v == null) return 'default (strict-origin-when-cross-origin)';
  const known = v.split(',').map(s => s.trim().toLowerCase()).filter(t => POLICIES.includes(t));
  if (!known.length) return `DISCARDED (${v})`;
  const p = known.at(-1);
  return ['unsafe-url', 'no-referrer-when-downgrade'].includes(p) ? `WEAKER than default (${p})` : `ok (${p})`;
};
const csp = v => {
  if (v == null) return 'MISSING';
  const d = Object.fromEntries(v.split(';').map(s => s.trim()).filter(Boolean)
    .map(s => { const [k, ...r] = s.split(/\s+/); return [k.toLowerCase(), r.join(' ')]; }));
  const n = [];
  if (!('default-src' in d)) n.push('no default-src');
  const script = d['script-src'] ?? d['default-src'] ?? '';
  if (/'unsafe-inline'/.test(script) && !/'nonce-|'sha(256|384|512)-|'strict-dynamic'/.test(script)) n.push("'unsafe-inline' scripts");
  if (/'unsafe-eval'/.test(script)) n.push("'unsafe-eval'");
  if (!('base-uri' in d)) n.push('no base-uri');
  return n.length ? `present, ${n.join(', ')}` : 'ok';
};

for (const url of process.argv.slice(2)) {
  let res;
  try { res = await fetch(url, { headers: { 'user-agent': 'Mozilla/5.0 headers-check' } }); }
  catch (e) { console.log(`\n${url}\n  fetch failed: ${e.cause?.code || e.message}`); continue; }
  const h = n => res.headers.get(n);
  console.log(`\n${url} -> ${res.status} ${h('content-type')}`);
  console.log('  HSTS        ', hsts(h('strict-transport-security')));
  console.log('  nosniff     ', xcto(h('x-content-type-options')));
  console.log('  framing     ', framing(h('x-frame-options'), h('content-security-policy')));
  console.log('  referrer    ', referrer(h('referrer-policy')));
  console.log('  CSP         ', csp(h('content-security-policy')));
  console.log('  permissions ', h('permissions-policy') ? 'present' : 'MISSING');
  if (h('x-powered-by')) console.log('  powered-by  ', h('x-powered-by'));
}

It prints the content type on purpose, so a 2-byte text/plain answer like the one I got from plain curl jumps out. Run it against a real page and a 404, not just your homepage.

The fix list, in order

This is a list for small sites, not a bank. Nothing here was an emergency, and I don't want to pretend it was.

nosniff and framing on the five sites that have neither. Zero risk of breaking anything. The press sites have sign-in and a submit flow, so being frameable by any page matters more there than on a static homepage.
HSTS on all 11, starting short. Something like max-age=300 first, then a day, then a year once nothing breaks. includeSubDomains only after I've checked every subdomain actually answers on HTTPS. The census found 682,019 domains sending max-age=0, which tells browsers to forget the rule, so check what you end up shipping.
poweredByHeader: false in next.config.js. Tiny, but it's one line and it's on all 11.
Fix Uno's half CSP, at least default-src, object-src 'none', base-uri 'self' and frame-ancestors.
The 'unsafe-inline' problem, later. The Next.js CSP guide is clear that nonces need dynamic rendering, because a static page is built before any request exists. 9 of my 11 homepages already come back with cache-control: private, no-store, so they render per request anyway and that cost mostly doesn't apply to me. It's still the biggest change on the list, so it goes last.

For the first four, Next.js lets you set them all in one place with headers() in next.config.js:

// next.config.js
module.exports = {
  poweredByHeader: false,
  async headers() {
    return [{
      source: '/:path*',
      headers: [
        { key: 'Strict-Transport-Security', value: 'max-age=300' }, // raise later
        { key: 'X-Content-Type-Options', value: 'nosniff' },
        { key: 'X-Frame-Options', value: 'SAMEORIGIN' },
        { key: 'Referrer-Policy', value: 'strict-origin-when-cross-origin' },
      ],
    }];
  },
};

Then run the script again, including the 404 URL.

What I'm not doing

I'm not chasing an A+ grade. Missing headers are also one of the most common findings in those "I found a critical vulnerability on your site, how much is the bounty" emails, and a missing X-Frame-Options on a page with no logged-in actions is low impact on its own. I wrote up how I'd triage beg bounty emails and real security reports if you get those too. The point of fixing headers is that it's cheap, not that it's urgent.

Also, new headers keep arriving. Chrome 152 shipped a Connection-Allowlist header in September for locking down where a page can connect to. I'm not touching that until the four boring ones are in.

If you don't want to run Node, the free security headers checker on Uno Directory (a site I run, and yes, currently one that fails it) shows the same six headers for any public URL plus the OWASP suggested value next to each.

What did your 404 page send when you checked? And did anyone else find their framework's starter template was the reason a whole batch of sites shipped with nothing?

Curled my own page as GPTBot: the canonical lands after the head closes. As Bingbot it sits inside the head. Same URL. https://dev.to/posting-dude/i-curl-checked-48-backlink-pages-on-my-own-sites-0-were-noindexed-27-had-the-canonical-in-the-body-233a

Posting Dude — Mon, 09 Nov 2026 05:55:23 +0000

Posting Dude

Nov 9

I Curl-Checked 48 Backlink Pages on My Own Sites. 0 Were Noindexed, 27 Had the Canonical in the Body

#seo #webdev #nextjs #indiehackers

12 min read

I Curl-Checked 48 Backlink Pages on My Own Sites. 0 Were Noindexed, 27 Had the Canonical in the Body

Posting Dude — Mon, 09 Nov 2026 05:53:32 +0000

Most "is this backlink any good?" advice stops at rel="nofollow". I went through what rel actually says on editorial and directory pages last week, so this time I wanted the other stuff: the checks that can quietly cancel a link even when the anchor tag looks perfect.

So this morning I pointed a small Node script at 48 pages on 10 sites I run:

22 product pages on my launch sites and directory (Aura++, EarlyHunt, IndieHunt, MakerHunt, SideHunt, Uno Directory)
22 pages on my four small press sites (14 published articles, 8 project pages)
4 of the Markdown copies those press sites publish next to each article

These are the kind of pages people submit to, or pay for, to get a link. They're mine, so I can poke at them all I want and tell you what's broken without pointing at anyone else's site.

Checked Monday 9 Nov 2026, between 10:54 and 11:14 IST. Four full runs, two with Googlebot's user agent string and two with a Chrome one. Raw HTML only, no JavaScript executed.

What the script checks

For each page, plus the URL it is supposed to link to:

Status and redirects, hop by hop, so a 301 to a 302 to a 200 shows up as exactly that.
noindex, in <meta name="robots">, <meta name="googlebot"> and the X-Robots-Tag header. The header is the one people forget, because View Source never shows it.
Page-level nofollow, in meta or header. That one cancels every link on the page, even an anchor with no rel at all.
robots.txt for Googlebot. Google's "Block Search indexing with noindex" doc (last updated 2025-12-10) says: "For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file." A blocked page can't even tell Google it's noindexed.
Canonical: missing, pointing at another URL, two different ones, sent as a Link header, or sitting outside <head>.
Sitemap: is the page in the site's own sitemap. It's a cheap orphan check, not proof that anything links to the page.
The link itself: is there a real <a href> to your domain in the raw HTML, or only inside script data. Is it inside <noscript> or a hidden element, does it go through a redirect URL, and does your URL still load.

Point 7 matters because of one line in Google's "SEO link best practices" doc (last updated 2025-12-10): "Generally, Google can only crawl your link if it's an <a> HTML element (also known as anchor element) with an href attribute."

The boring part, which is good news

Across all 48 pages:

0 noindex (meta or header)
0 page-level nofollow
0 robots.txt blocks (all 10 sites only disallow things like /admin, /dashboard and /api)
0 redirects, 0 non-200 pages
0 canonicals pointing at a different URL
0 pages missing from their sitemap
0 links that only existed in JavaScript data, 0 inside <noscript>

I half expected at least one stray noindex, so that was nice to see.

Finding 1: 27 pages send the canonical in the body

26 pages had the canonical <link> after </head> on every run: all 4 Aura++ project pages and all 22 press-site pages. One or two EarlyHunt and SideHunt pages flipped between runs. One SideHunt page, curled three times in a row with the same user agent, came back body, head, head. The total per run was 28, 26, 27 and 27.

The cause is Next.js streaming metadata. The Next.js generateMetadata docs say that when it resolves, "the resulting metadata tags are appended to the <body> tag", and that this is "interpreted correctly by bots that execute JavaScript and inspect the full DOM (e.g. Googlebot)". So for Google, this is probably fine once the page is rendered.

Not every crawler renders, though. Google's own canonical doc (last updated 2026-07-10) says the rel="canonical" link element "is only accepted if it appears in the <head> section of the HTML", and raw HTML is all a non-rendering crawler gets. So I curled one press page with 8 user agents. facebookexternalhit and bingbot got the canonical in <head>, which fits Next.js keeping a built-in list of "HTML-limited" bots it waits for. GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, AhrefsBot and plain curl all got it in the body.

I wrote up how this happens, and the htmlLimitedBots setting that fixes it, in a canonical in your Next.js root layout lands on every page. The lesson here is simpler: when you check a placement, check it with more than one user agent.

Finding 2: on 30 pages the product link sits inside a hidden div

This one looked scary. On 30 pages the link to the product was inside an element with the hidden attribute. My first thought was that a template was hiding links.

It isn't. When part of a React page streams in late, React sends that chunk inside <div hidden id="S:2"> near the end of the HTML, and a small inline script moves it into place. I went back over all 30 and the hidden wrapper was always one of those S:n boundaries (21 of them were S:2). None were display:none.

Google renders JavaScript, so this should be fine for Google. A parser that skips hidden elements and doesn't run JavaScript would skip the link. I don't know which crawlers actually do that, so I won't pretend it's a disaster. It does mean "the link is in the HTML" isn't the whole answer, and any checker that flags hidden links needs a second look at what is hidden.

Finding 3: the Markdown copies have no canonical

The press sites publish a .md version of each article for AI agents, and those URLs are in the sitemap. All 4 I checked returned 200, text/markdown, X-Robots-Tag: index, follow, and no canonical. The file has a "Canonical:" line in its text, but crawlers don't treat text as a canonical. Google accepts canonicals as an HTTP header, so the fix is one response header on each .md URL:

Link: <https://example.com/publications/some-article>; rel="canonical"

Finding 4: rel, briefly

7 pages had a followed, visible link to the product. 28 had nofollow on every link to it. 9 had a followable link, but only inside one of those streamed hidden divs. On the press sites the Markdown copy prints the plan, and 10 of 14 articles say Plan: nofollow, which matches their links. So it's not a bug, just a reminder to read the HTML instead of the sales page.

Finding 5: some of the linked sites are gone

Of the 44 product URLs these pages link to, 1 returned 404, 1 returned 402 and 2 didn't respond to my requests at all. The listing page is fine. The thing it points at isn't. That's the listing site's cleanup job, not yours, but it's a good reminder to re-check your own link targets now and then.

The script

Node 18+, no dependencies. Give it a text file with one <page-url> <your-url> pair per line (use - instead of a URL for page-only checks):

node placement-check.mjs pairs.txt
UA="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/130.0 Safari/537.36" node placement-check.mjs pairs.txt

#!/usr/bin/env node
// placement-check.mjs: checks the things beyond rel="nofollow" that quietly stop a backlink from counting.
// Usage: node placement-check.mjs pairs.txt      (UA="..." to change the user agent)
//   pairs.txt: one "<page-url> <your-url>" per line (# for comments, "-" = page checks only)
// Node 18+, no dependencies. Reads raw HTML only; it does not run JavaScript.
import { readFileSync } from "node:fs";

// Default: Googlebot's user agent string, because some sites (Next.js streaming metadata, for one)
// send crawlers different HTML than browsers. Run again with UA="Mozilla/5.0" to compare.
const UA = process.env.UA || "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)";
const lines = readFileSync(process.argv[2] || "pairs.txt", "utf8").split("\n")
  .map(s => s.trim()).filter(s => s && !s.startsWith("#"));
const attr = (tag, name) => (tag.match(new RegExp(`\\s${name}\\s*=\\s*("([^"]*)"|'([^']*)'|([^\\s>]+))`, "i")) || [])
  .slice(2).find(v => v !== undefined);
const host = u => { try { return new URL(u).hostname.replace(/^www\./, ""); } catch { return ""; } };
const norm = u => { try { const x = new URL(u); return (x.origin + x.pathname).replace(/\/$/, ""); } catch { return u; } };
const decode = s => s.replace(/&amp;/g, "&");

// 1. follow redirects by hand so we can see every hop
async function get(url) {
  const hops = [];
  for (let i = 0; i < 6; i++) {
    const r = await fetch(url, { redirect: "manual", headers: { "user-agent": UA } });
    hops.push(r.status);
    const loc = r.headers.get("location");
    if (r.status >= 300 && r.status < 400 && loc) { url = new URL(loc, url).href; continue; }
    return { res: r, url, hops, body: await r.text() };
  }
  throw new Error("too many redirects");
}

// 2. robots.txt: longest matching Allow/Disallow in the Googlebot group, else the * group
const robotsCache = {};
async function robotsBlocks(url) {
  const u = new URL(url);
  if (!(u.origin in robotsCache)) {
    let txt = "";
    try { const r = await fetch(u.origin + "/robots.txt", { headers: { "user-agent": UA } }); if (r.ok) txt = await r.text(); } catch {}
    const groups = []; let cur = null, lastWasUA = false;
    for (const raw of txt.split("\n")) {
      const [k, ...v] = raw.replace(/#.*/, "").split(":"); const key = k.trim().toLowerCase(), val = v.join(":").trim();
      if (key === "user-agent") { if (!lastWasUA) groups.push(cur = { agents: [], rules: [] }); cur.agents.push(val.toLowerCase()); lastWasUA = true; }
      else if ((key === "allow" || key === "disallow") && cur) { cur.rules.push([key, val]); lastWasUA = false; }
      else lastWasUA = false;
    }
    const g = groups.filter(x => x.agents.includes("googlebot"));
    robotsCache[u.origin] = (g.length ? g : groups.filter(x => x.agents.includes("*"))).flatMap(x => x.rules);
  }
  const path = u.pathname + u.search; let best = ["allow", ""];
  for (const [k, v] of robotsCache[u.origin]) {
    if (!v) continue;
    const re = new RegExp("^" + v.replace(/[.+?^${}()|[\]\\]/g, "\\$&").replace(/\*/g, ".*").replace(/\\\$$/, "$"));
    if (re.test(path) && (v.length > best[1].length || (v.length === best[1].length && k === "allow"))) best = [k, v];
  }
  return best[0] === "disallow" ? best[1] : null;
}

// 3. walk the tags so we know if a link sits inside <noscript> or a hidden parent
const VOID = new Set("area base br col embed hr img input link meta source track wbr".split(" "));
function anchors(html) {
  const out = [], stack = []; const re = /<(\/?)([a-zA-Z][\w-]*)([^>]*)>/g; let m;
  while ((m = re.exec(html))) {
    const [, close, name0, rest] = m, name = name0.toLowerCase();
    if (!close && (name === "script" || name === "style")) { const end = html.toLowerCase().indexOf(`</${name}`, re.lastIndex); re.lastIndex = end < 0 ? html.length : end; continue; }
    if (close) { const i = stack.map(s => s.name).lastIndexOf(name); if (i >= 0) stack.length = i; continue; }
    const style = (attr(m[0], "style") || "").replace(/\s/g, "").toLowerCase();
    const bare = rest.replace(/"[^"]*"|'[^']*'/g, ""); // attribute names only, so class="md:hidden" doesn't count
    const hidden = /(^|\s)hidden(\s|=|\/|$)/i.test(bare) || /display:none|visibility:hidden/.test(style);
    if (name === "a") out.push({ href: decode(attr(m[0], "href") || ""), rel: (attr(m[0], "rel") || "").toLowerCase(),
      noscript: stack.some(s => s.name === "noscript"), hidden: hidden || stack.some(s => s.hidden) });
    if (!VOID.has(name) && !rest.trim().endsWith("/")) stack.push({ name, hidden });
  }
  return out;
}

const sitemapCache = {};
async function inSitemap(url) {
  const o = new URL(url).origin;
  if (!(o in sitemapCache)) {
    const locs = new Set(); const queue = [o + "/sitemap.xml"]; let n = 0;
    while (queue.length && n++ < 10) {
      try { const x = await (await fetch(queue.shift(), { headers: { "user-agent": UA } })).text();
        for (const [, l] of x.matchAll(/<loc>\s*([^<\s]+)\s*<\/loc>/g)) (/<sitemapindex/i.test(x) ? queue.push(l) : locs.add(norm(decode(l))));
      } catch {}
    }
    sitemapCache[o] = locs;
  }
  return sitemapCache[o].has(norm(url));
}

const tally = {}; const bump = k => (tally[k] = (tally[k] || 0) + 1);
for (const line of lines) {
  const [page, target] = line.split(/\s+/); const flags = [];
  let r; try { r = await get(page); } catch (e) { console.log(`\n${page}\n  FETCH ERROR ${e.message}`); bump("fetch error"); continue; }
  const { res, url, hops, body } = r, ctype = res.headers.get("content-type") || "";
  if (hops.length > 1) flags.push(`redirects: ${hops.join(" -> ")} -> ${url}`);
  if (res.status !== 200) flags.push(`final status ${res.status}`);
  const xr = (res.headers.get("x-robots-tag") || "").toLowerCase();
  if (/noindex|none/.test(xr)) flags.push(`X-Robots-Tag: ${xr}`);
  if (/nofollow/.test(xr)) flags.push(`X-Robots-Tag nofollow (every link on the page): ${xr}`);
  const isHtml = ctype.includes("html");
  const head = body.split(/<\/head>/i)[0];
  for (const t of isHtml ? body.match(/<meta[^>]+>/gi) || [] : []) {
    const name = (attr(t, "name") || "").toLowerCase();
    if (name !== "robots" && name !== "googlebot") continue;
    const c = (attr(t, "content") || "").toLowerCase();
    if (/noindex|none/.test(c)) flags.push(`meta ${name}: ${c}`);
    if (/nofollow/.test(c)) flags.push(`meta ${name} nofollow (every link on the page): ${c}`);
  }
  const canon = isHtml ? (body.match(/<link[^>]+>/gi) || []).filter(t => /\brel\s*=\s*["']?canonical/i.test(t)).map(t => attr(t, "href")) : [];
  const hdrCanon = ((res.headers.get("link") || "").match(/<([^>]+)>\s*;\s*rel="?canonical/i) || [])[1];
  if (hdrCanon) canon.push(hdrCanon);
  if (canon.length > 1 && new Set(canon.map(c => norm(new URL(c, url).href))).size > 1) flags.push(`${canon.length} different canonicals: ${canon.join(" , ")}`);
  else if (canon[0] && norm(new URL(canon[0], url).href) !== norm(url)) flags.push(`canonical points elsewhere: ${canon[0]}`);
  if (isHtml && canon.length && !(head.match(/<link[^>]+>/gi) || []).some(t => /canonical/i.test(t)) && !hdrCanon) flags.push("canonical is outside <head> in the raw HTML");
  if (!canon.length) flags.push(isHtml ? "no canonical" : `no canonical (non-HTML: ${ctype.split(";")[0]}, no Link header)`);
  const blocked = await robotsBlocks(url); if (blocked) flags.push(`robots.txt Disallow: ${blocked}`);
  if (!(await inSitemap(url))) flags.push("not in the site's sitemap");

  let linkSummary = "page checks only";
  if (target && target !== "-") {
    const th = host(target);
    const all = isHtml ? anchors(body) : [];
    const mine = all.filter(a => host(new URL(a.href, url).href) === th);
    // e.g. /out?url=https://you.com : a param whose whole value is your URL
    const viaRedirect = all.filter(a => { try { const x = new URL(a.href, url); return host(x.href) !== th && [...x.searchParams.values()].some(v => /^https?:/.test(v) && host(v) === th); } catch { return false; } });
    const followed = mine.filter(a => !/nofollow|sponsored|ugc/.test(a.rel) && !a.noscript && !a.hidden);
    const inScripts = (body.match(/<script[\s\S]*?<\/script>/gi) || []).some(s => s.includes(th));
    linkSummary = `${mine.length} <a> to ${th} (${followed.length} followed & visible)` + (mine.length ? `, rel: ${[...new Set(mine.map(a => a.rel || "(none)"))].join(" | ")}` : "");
    if (!mine.length && inScripts) flags.push(`link to ${th} only inside <script> data (JS-rendered, not in raw HTML)`);
    else if (!mine.length) flags.push(`no link to ${th} at all`);
    if (mine.length && !followed.length) flags.push(`every link to ${th} is nofollow/sponsored/ugc, hidden or in <noscript>`);
    if (mine.some(a => a.noscript)) flags.push(`a link to ${th} sits inside <noscript>`);
    if (mine.some(a => a.hidden)) flags.push(`a link to ${th} sits in a hidden element`);
    if (viaRedirect.length) flags.push(`${viaRedirect.length} link(s) go through a redirect URL instead of straight to ${th}`);
    try { const t = await get(target); if (t.res.status >= 400) flags.push(`your URL returns ${t.res.status}`); } catch (e) { flags.push(`your URL failed to load: ${e.message}`); }
  }
  console.log(`\n${page}\n  HTTP ${hops.join(">")} | ${ctype.split(";")[0]} | ${linkSummary}`);
  for (const f of flags) { console.log("  FLAG " + f); bump(f.replace(/https?:\/\/\S+/g, "").replace(/(to|straight to) [\w.-]+\.[a-z]+/g, "$1 your URL").replace(/: .*|\(.*?\)/g, "").replace(/^\d+ /, "").trim()); }
  if (!flags.length) console.log("  ok");
  bump(flags.length ? "pages with any flag" : "clean pages");
}
console.log(`\n== summary (${lines.length} pages, UA: ${UA})`); for (const [k, v] of Object.entries(tally).sort((a, b) => b[1] - a[1])) console.log(`${String(v).padStart(3)}  ${k}`);

Before the real run I tested it on 15 local pages: one clean, and 14 each broken in one way (noindex meta, noindex header, page-level nofollow, canonical elsewhere, canonical in body, robots.txt block, JS-only link, noscript, display:none, redirect link, rel=nofollow, 301 to 302, 404, dead target). Every flag fired. It's regex over raw HTML, so treat it as a first pass and open DevTools on anything it flags.

If you only have two minutes, these cover most of it:

curl -sI "$PAGE" | grep -i -E "^(HTTP/|location:|x-robots-tag:|link:.*canonical)"
curl -s -A "GPTBot/1.2" "$PAGE" | grep -o -i -E '<meta[^>]+robots[^>]*>|<link[^>]+canonical[^>]*>|</head>'
curl -s "$PAGE" | grep -o -E '<a [^>]*href="https?://(www\.)?yourdomain\.com[^>]*>'

If </head> prints before the canonical (or the robots meta), that tag is in the body for that user agent. On my own press page with GPTBot's user agent, both were.

What I'd check before paying for a link

Open the live page. Check the status, X-Robots-Tag, and meta robots for noindex and nofollow.
Check robots.txt doesn't block it.
Check the canonical points at the page itself and sits in <head>, with more than one user agent.
Check your link is a real <a href> in the raw HTML, and what is wrapping it.
Check the page is in the site's sitemap and linked from somewhere a crawler will find.

Lede Press is a press site I run, and it's in this audit, warts and all. It's one of the sites that sends the canonical in the body to GPTBot, ClaudeBot and plain curl. It publishes editorial backlinks listed in public: one article about your product with one link in the text, for $29, or free if your site links back. The site says the link is dofollow on those two plans (there's also a free nofollow plan). Run the script on the live page and check it yourself. No promises about rankings.

Which of these do you actually check before you pay for a link? And has a hidden-div or body-canonical flag ever turned out to be a real problem for you?