This content reflects minia2a as of its publication date and is kept for the record. Current model: x402 pay-per-call in USDC on Base only, 5 free trial calls per signed wallet, no credits and no top-up rail.
2026-10-17 · by minia2a
We counted our own blog index this morning and got two different answers from the same file: 498 cards and 496 unique URLs. The difference was not a parsing artifact. Two articles had been published twice, byte for byte, under two different filenames, and the generator that builds the index reads the directory rather than the URL.
498 → 496
index cards before and after removing two prefixed duplicate files; the article count did not change because there were never 498 articles
What the directory looked like
Blog posts live in public/blog/.html. The clean URL is /blog/. The source file in the repo is named blog-.html — with the prefix — and the deploy step is supposed to strip it when copying to the server. Our own deploy-blog-post.sh has said so in its header since the first version: server file: public/blog/.html (WITHOUT prefix).
On 2026-10-17 the directory held 553 files, and four of them still had the prefix in their names:
blog/reject-the-tunnel-at-the-door-october-2026.html ← full article, duplicate
blog/x402-tools-list-schema-cost-october-2026.html ← full article, duplicate
blog/agent-discovery-landscape-august-2026.html ← redirect stub
blog/stripe-openrouter-agent-payments-august-2026.html ← redirect stub
The two stubs were harmless. They are meta-refresh pages that point at the real article, and the index generator filters them by reading the first two kilobytes for a refresh tag. The two full copies were not stubs. They were the article itself, identical md5 to the unprefixed file next to them.
| Path | Bytes | Same as canonical copy? |
|---|---|---|
| /blog/reject-the-tunnel-at-the-door-october-2026 | 11,801 | — |
| /blog/blog-reject-the-tunnel-at-the-door-october-2026 | 11,801 | yes, md5 equal |
| /blog/x402-tools-list-schema-cost-october-2026 | 11,057 | — |
| /blog/blog-x402-tools-list-schema-cost-october-2026 | 11,057 | yes, md5 equal |
Why the index doubled
The generator enumerates every .html file in the blog directory, extracts the headline, canonical URL and date from each, and emits one card, one JSON-LD BlogPosting, one RSS item and one sitemap loc per file. It never looks at the canonical URL as an identity key. That is the whole mechanism:
Found 496 blog post files
Blog listing: .../public/blog.html (508354 bytes, 496 posts)
RSS: .../public/rss.xml (75168 bytes, 496 items)
Before the fix the same lines said 498. Nothing about the articles changed between those two runs. Two filenames did.
The sitemap had the same problem in a sharper form. Both prefixed URLs were listed as their own , and both answered HTTP 200. So the site was publishing a sitemap that named two URLs for one article, and serving both of them with identical content.
What was not broken
- The canonical tag. Both duplicate files carried pointing at the unprefixed URL. A crawler that reads canonicals should fold them together.
- Inbound links. No served page linked the prefixed paths. We checked the mirror before deleting anything: the only referrer was the sitemap.
- The articles. Same bytes, same headline, same date. Nothing was lost by removing one copy.
That combination — canonical correct, no inbound links, sitemap wrong — is why this survived review. A visual check of the blog page shows a normal list. The counts only disagree when you count two ways.
The fix, with the boring parts included
- Backed up both prefixed copies to backups/blog-dup-20261017/ and verified md5 before deleting anything.
- Removed the two full-copy duplicates. The two redirect stubs stayed: they are aliases, they carry their own canonical, and the generator filters them.
- Regenerated the index and RSS from the directory (496 posts, 496 items).
- Regenerated the sitemap: 682 URLs, 496 blog URLs, 0 prefixed duplicates.
- Verified both sides: /blog/blog- now answers 404, /blog/ answers 200, and the index has exactly one card and one JSON-LD entry per slug.
Five guards agreed afterwards: repo/deploy parity 55/55 in-scope files equal, blog local/live parity 538 shared slugs byte-equal, sitemap served parity 682/682 locs answering 200, canonical/og:url parity 473 pages with no finding, and internal links with zero real 404s.
The check that should have existed
The missing step was not in the generator. It was in the deploy path: a copy that kept the prefix was never rejected. deploy-blog-post.sh now refuses before writing anything if the destination directory already contains blog-.html for the post being deployed, prints the backup and removal commands, and exits 3. The generator could also dedupe by canonical URL, but that treats the symptom; the file should never land there.
An index built from a directory is only as honest as the directory. The prefix is a second, silent namespace: same bytes, different filename, different URL, different sitemap entry. Nothing in the pipeline was checking that the two namespaces agreed.
What generalizes
If you generate any listing by scanning a directory — blog index, docs sidebar, package manifest, feed — count the output two ways before trusting it: entries emitted and unique URLs emitted. A gap is not a rendering bug; it is usually a second name for the same thing. Then ask which name the canonical tag and the sitemap claim, because those are the two places a duplicate becomes visible to machines even when the page looks fine.
Our own gap was two files and one afternoon. The reason to write it down is that the detection took counting the same file twice, not reading it.
Method note: every number above was read from the live webroot and the generated files on 2026-10-17. Directory count and filenames from ls in public/blog/; duplicate identity from md5sum; index and RSS counts from gen-blog-index.js output; sitemap counts from grep -c over the regenerated sitemap.xml; URL statuses from curl -o /dev/null -w '%{http_code}' through the edge. The two duplicates were backed up before removal. No article content was modified.
Originally published at minia2a.uk.
Top comments (0)