I pulled every external link out of the prose of every Turkish article on this
blog and checked them one by one this afternoon. One thousand three hundred and
thirty-three articles, one thousand seven hundred and fifty-eight unique
addresses.
Seven return 404. Honestly, I had braced for more.
The number that bothers me is seventy-five: these addresses return 200, but
they no longer sit where I wrote them. One was a security glossary entry; click
it today and you land on the product page of the firm that acquired the
company. The server answers successfully and my validation script is satisfied.
Only the reader arrives at a page that does not contain the sentence I pointed
to.
The measurement was wrong three times
The extraction rule is simple: http(s) addresses outside code blocks in
Turkish .mdx files that carry a publishDate. I did not count code blocks;
of the 168 unique addresses in them, 160 appear nowhere else — they are all
command examples. There are also 25 invented endpoints such as
api.example.com, vault.alanadim.com and 192.168.1.100:2283, which are
narrative material too.
I ran the first pass with fourteen parallel requests. Result: 132 addresses
returned something other than 200. I will admit I felt relieved — 92.5 percent
standing is a decent number.
Then I asked those same 132 again, serially, with 1.2 seconds between requests.
Side by side, the two rounds look like this:
| Code | First round (14 parallel) | Serial round |
|---|---|---|
| 200 | 0 | 60 |
| 503 | 40 | 1 |
| 403 | 25 | 25 |
| 429 | 24 | 7 |
| 000 (timeout) | 20 | 20 |
| 418 | 9 | 9 |
| 302 | 5 | 5 |
| 404 | 5 | 5 |
| 400 | 4 | 0 |
Sixty came back 200. In other words, 45 percent of the links I had called
"broken" were broken by my own scanner: thirty-nine of the forty 503s and
seventeen of the twenty-four 429s were the bill for my parallelism. A textbook
case of blaming the instrument's error on the thing being measured.
On the third pass, for the 72 addresses that still would not return 200, I
raised the connect timeout to 25 seconds and the total to 60. That is where the
true picture appeared: criu.org answers 200 in 24 seconds,
packages.ubuntu.com/noble/procps in 22, and
packages.ubuntu.com/resolute/valkey-server in 15.6. None of them is broken;
they are merely slow. Dead for an impatient script, alive for a patient reader.
The second mistake is more embarrassing. There was an eighth address on the
dead list: codefinity.com/blog/How-Python. The actual link in the article is
codefinity.com/blog/How-Python's-GIL-Works — an apostrophe. My extraction
pattern treats an apostrophe as the end of an address, so the truncation was
mine. The full address returns 200 today and
the page is right where it was. No other link in the archive fell into the same
trap; this single one was enough to manufacture one eighth of my "dead links".
I spotted the third while finishing this piece: the corpus holds 1,333
articles, but one of them was never scanned past its halfway point. The
Stirling PDF installation guide contains an unclosed code fence — the file holds
an odd number of triple-backtick markers — so the extractor treats everything
after that point as code. The eight addresses in that article's source section were never
asked. That is both a hole in the corpus and a real rendering bug on that page,
and the second one I have to fix separately.
Merging the four passes gives the clean result: 1,720 of 1,758 addresses
(97.8 percent) returned 200 on at least one attempt. Of the remaining 38, seven
are 404, eight time out, twenty-three are 403.
What a 200 proves
RFC 9110 defines 200 in a single sentence: "The 200 (OK) status code indicates
that the request has succeeded." It says the request succeeded. Not that my
claim is still supported.
For seventy-five addresses the final stop is not the address I wrote — discount
two of them, which differ only by www or a query parameter. In sixty-four the
host is the same and the path changed: MDN moved the animation-fill-mode page
under Reference/Properties, Ansible put all of its docs behind a
/projects/ansible/latest/ prefix, and GitHub split its merge documentation in
two — the two addresses I took from the same old folder
(incorporating-changes-from-a-pull-request/) now live in separate trees, one
under reference/, the other under how-tos/.
In eleven cases the host changed too:
entro.security -> www.sailpoint.com/products/agentic-ai-security
www.landh.tech -> depi.security/blog/...
www.openssh.com -> www.openssh.org (4 addresses)
owasp.org/blog/... -> owasp.github.io/blog/...
owasp.org/www-community -> community.owasp.org
go.dev/issue/46279 -> github.com/golang/go/issues/46279
guidelines.raterhub.com -> static.googleusercontent.com/...
raw.githubusercontent.com -> github.com
The first line teaches the most. The page I cited was a concept definition; the
company was acquired this summer, and today that address goes to the buyer's
product pitch. The link is healthy, the source is gone.
Three more are stranger: two bug reports on bugs.debian.org and a thread in
Apple's developer forum all return 200 — but the destination is
.internal/challenge.html and verify-human/verify.html respectively. Both
are bot walls: behind the response that reports success to a machine sits a
page asking the visitor who they are.
The spec also answers the question of who will fix the redirect. For 301 it
says a user agent with link-editing capability can permanently replace the
references; it then adds that the suggestion is usually ignored unless the
agent is actively editing content, the connection is secured, and the origin
server is a trusted authority. What it describes is exactly the tool in my
hands: a client that writes content. Only that client looks at new articles;
nobody opens the old ones.
A 404 is not an obituary
The same RFC is explicit about what 404 is not: "A 404 status code does not
indicate whether this lack of representation is temporary or permanent."
There is 410 for announcing permanence, but using it is not mandatory — the
text says it is "not necessary to mark all permanently unavailable resources
as 'gone'" and leaves the decision to the server owner.
In practice that means nobody has promised to notify me when a source
disappears. The distribution of the seven 404s shows it.
| Address | Article | Published |
|---|---|---|
aws.amazon.com/turkiye/cloud-database/ |
mobile database comparison | 23 May |
github.com/mitchcarroll/Plex-to-Jellyfin.py |
moving from Plex to Jellyfin | 21 Jun |
github.com/Actual-Budget |
Actual Budget installation | 29 Jul |
telnyx.com/resources/sms-pumping-fraud |
hardware key instead of SMS 2FA | 12 Aug |
manpages.ubuntu.com/.../questing/... (3 addresses) |
ZFS snapshot and send/receive | 25 Aug |
Five articles, three categories. Two of the stories are worth telling.
github.com/Actual-Budget is 404; github.com/actualbudget is 200. The
address lost a hyphen. GitHub ignores capitalisation — github.com/ACTUALBUDGET
returns 200 as well — but it does not ignore the hyphen, and
github.com/actual-budget is 404 too. GitHub's documentation says renaming a
repository sets up redirects, and warns that if a new repository is created
with the original name, the redirect stops working. That page is about
repository names; it makes no such promise for an account rename. The result is
that the link a reader of an installation guide is most likely to click — the
project's own address — returns 404.
The table in the ZFS article
is sharper. That piece points to seven
manpages.ubuntu.com/manpages/questing/... addresses; today three are 404 and
four time out. I tried a random page from the same tree — questing/man1/ls.1.html
is 404 as well. So the problem is not the pages I picked but that this release
tree is no longer served. The noble equivalent
(noble/man8/zfs-allow.8.html), which I checked by hand outside the corpus,
returns 200 in 15.7 seconds; the fix is one word. But the article passed the
source gate on 25 August and the gate never looked again.
Tim Berners-Lee wrote it in 1998: "URIs don't change: people change them."
Twenty-eight years later I am holding a seventy-five-item footnote to that
sentence.
The answer depends on who asks
Everything so far resembles any article you have read about link rot. The real
oddity starts here.
I ran the first pass "in good faith" with a Chrome identity — the reader will
see it that way too, I reasoned. Then I asked the same twelve addresses with
three different identities, twice each: curl's own default, my own gate's
identity (mustafaerbay-content-source-check/1.0), and Chrome. The table shows
eight of the twelve:
| Address | curl | gate | Chrome |
|---|---|---|---|
w3.org/TR/css-animations-1/ |
200 | 200 | 403 |
freedesktop.org/.../systemd.exec.html |
200 | 200 | 418 |
linux.die.net/man/1/sha256sum |
200 | 200 | 403 |
source.android.com/.../dm-verity |
200 | 200 | 302 → login |
ai.google.dev/.../rate-limits |
200 | 200 | 302 → login |
techcommunity.microsoft.com/... |
403 | 200 | 200 |
docs.redhat.com/.../10.0_release_notes |
200 | 403 | 403 |
akamai.com/glossary/what-is-sms-pumping |
403 | 403 | 403 |
In these seven rows the difference landed in exactly the same place in both
rounds. Two of the remaining five addresses moved between rounds — Debian's bug
report gave curl a 302 in the first round and not in the second; GitHub
returned 503 to two identities at once in the second. Identity is not the only
variable, but it is the most stable one.
The direction is not uniform either. W3C's own specification page shuts the
door when I impersonate a browser and opens it to the identity that honestly
says "I am a validation bot". Red Hat does the opposite: it rejects both named
identities and allows curl's default. Microsoft's community site dislikes curl
and likes the other two.
The cost of that distinction is measurable across the corpus: 19 addresses
returned 200 only to my gate's identity — nine freedesktop man pages, three
on GitHub, three Google AI docs, two on the Android source site, one W3C page
and one linux.die.net page. A check running as fake Chrome would have
reported all nineteen as broken.
Cloudflare's own documentation explains the mechanism: User Agent Blocking
rules look at the User-Agent header and offer blocking or various challenge
screens as the action, and Cloudflare recommends custom rules instead. So when
I see a 403 I am not reading the state of the page; I am reading the decision a
policy in front of that page made about me.
The distribution of the 23 stubbornly closed addresses says the same: six are
Red Hat docs, two are Akamai glossary entries, one an Akamai blog post, one
OpenAI's API reference, one a US health department report; the remaining twelve
are Medium/DZone-style writing sites and small vendor blogs. I am not sad about
that last group — my current source policy does not treat them as primary
anyway. What stings is the first nine, and above all the six Red Hat release
and documentation pages.
Across the whole corpus the two identities returned a different code for 58
addresses. I am not using that number as a claim: the two passes ran at
different concurrency and at different times, and GitHub's rate limiting did
not treat them alike. Quoting a figure from a measurement whose noise exceeds
its effect is the hole I fell into an hour earlier. What holds is the
controlled twelve-address experiment above.
What my own gate actually measures
My pre-publish source gate is scripts/validate-content-sources.mjs. I read
its code again.
The gate only looks at links under the ## Resmî Kaynaklar heading and its
English equivalents. 1,599 of the corpus addresses come from those sections;
159 appear only in the body and have never been machine-checked. Another 539 of
the section addresses belong to articles published before the gate existed, so
they were never asked either. In total 698 addresses — 40 percent of the corpus
— have never once been checked automatically. I had already seen that a rule
does not retroactively fix the archive in
an earlier measurement;
what is new here is the size of the gap.
The gate runs with --staged: it looks at each link exactly once, on the day it
is born. Its timeout is 12 seconds. Combine that with the measurement above —
packages.ubuntu.com/noble/procps answers in 22 seconds, criu.org in 24, and
my gate would reject both today as "could not open link". Treating 401, 403 and
429 as alive, on the other hand, is a sound decision; half of this article is
evidence for it. It also turns out to have a subtlety I had forgotten: the gate
checks the final stop of a redirect against the allowlist. Half the mechanism
for noticing that an address has moved is already sitting there — it just never
compares the result with the address I cited.
The links between my own articles are in good shape — of 304 internal links
only one has no target (transactional-outbox-pattern-ile-guvenilir-mesajlasma)
— which is only to be expected, since every one of those addresses is mine.
A decision framework
Here is what I am changing in the gate after this measurement:
- A citation gets three parts: the address, the access date, and the sentence I quoted. The sentence will live in a file next to the article; when a page moves, searching the target site for that sentence tracks it down far better than the address does. I will also send a copy to the Wayback Machine at publish time — it is the only cheap way to show a reader what a moved page used to say.
- 404 is not the alarm criterion. The real signal is "final stop ≠ cited address". Today's numbers are seven red against seventy-five amber; a check that only watches red merely looks like it is working.
- Timeout goes from 12 seconds to 30, with one concurrent request per host. Calling a slow server dead writes a measurement error into the content, and firing fourteen parallel requests at one host is rude.
- Ask with two identities: curl's default and the gate's own name. If both get 403, bring it to a human. I have dropped the browser impersonation — those nineteen addresses looked closed precisely because of it.
- The ambers get a queue. The monthly run will write every moved address into a list, and a line closes only when the new address lands in the text. A monthly job with no owner reproduces exactly the "gate installed, job done" feeling this article criticises.
- I am leaving the dead addresses listed. All seven are written out here; I will fix them in a separate pass, finding the right target for each one.
I also measured the bill for the cadence. Scanning the whole corpus with
fourteen parallel requests took 4 minutes; asking the suspicious ones again
serially took 11 minutes; the final generous-timeout pass over 72 addresses
took 12 minutes. Under half an hour. For a job that runs monthly that cost is
negligible; running it on every deploy would be both pointless and unfair to
the servers on the other side. I picked monthly.
Conclusion
I started measuring to answer "are my links healthy". What I found is that the
question is posed wrongly: a link's health is not a property of the link but a
property of the conversation between me and the server. It varies by who asks,
how long they wait, and what hour they ask; and a 200 says my request
succeeded, not that my claim did.
There is a part of this that is much bigger than a blog. The health checks we
keep in production work the same way: they produce a binary answer and bury
"who was asking" inside it. When the check is green we are not seeing that the
server is genuinely well, only that it was polite to us. For a long time those
two look like the same thing — until the day the page the reader sees and the
answer I see part ways.
Top comments (0)