<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: linkbruno.com</title>
    <description>The latest articles on DEV Community by linkbruno.com (@linkbruno).</description>
    <link>https://dev.to/linkbruno</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4210738%2F542819f7-f281-4f04-bf5d-bf348a14ce29.png</url>
      <title>DEV Community: linkbruno.com</title>
      <link>https://dev.to/linkbruno</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/linkbruno"/>
    <language>en</language>
    <item>
      <title>Case study (hypothetical): the 'View receipt' links in payment emails that expired after 7 days</title>
      <dc:creator>linkbruno.com</dc:creator>
      <pubDate>Mon, 16 Nov 2026 11:45:15 +0000</pubDate>
      <link>https://dev.to/linkbruno/case-study-hypothetical-the-view-receipt-links-in-payment-emails-that-expired-after-7-days-elm</link>
      <guid>https://dev.to/linkbruno/case-study-hypothetical-the-view-receipt-links-in-payment-emails-that-expired-after-7-days-elm</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Scheduled broken-link and redirect checks that tell you the moment a link breaks — &lt;a href="https://linkbruno.com" rel="noopener noreferrer"&gt;linkbruno.com&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;This is a hypothetical case study, built from a pattern that shows up in a lot of billing systems. The company and numbers are invented.&lt;/p&gt;

&lt;h2&gt;
  
  
  The setup
&lt;/h2&gt;

&lt;p&gt;A small B2B SaaS company billed around 2,000 customers monthly. After each successful payment, it sent a receipt email with a big "View receipt" button. The button linked to a page showing the invoice and a PDF download.&lt;/p&gt;

&lt;p&gt;The link looked like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;https://app.example.com/receipts/inv_8731?expires=1700000000&amp;amp;sig=4b9f...
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It was a signed URL: the server computed an HMAC over the invoice ID and the expiry time, and the receipt page accepted the request only if the signature matched and the expiry hadn't passed. No login was needed, which was the point. The person reading the email was often not the person with an account.&lt;/p&gt;

&lt;p&gt;The expiry was set to seven days. Someone had copied the value from the password reset email, where seven days is generous.&lt;/p&gt;

&lt;h2&gt;
  
  
  What went wrong
&lt;/h2&gt;

&lt;p&gt;Receipts are not password resets. Nobody uses a receipt link in the first seven days. People use them:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;At the end of the month, when the finance team collects receipts for expense reports.&lt;/li&gt;
&lt;li&gt;At the end of the quarter, for bookkeeping.&lt;/li&gt;
&lt;li&gt;At year end, when an accountant asks for every receipt from a supplier.&lt;/li&gt;
&lt;li&gt;During an audit, sometimes years later.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;In every one of those cases, the "View receipt" button went to a page saying "This link has expired." There was no way forward from that page: no login link, no "send me a fresh link" button, no mention of where else to find the receipt.&lt;/p&gt;

&lt;p&gt;Customers did what people do. They emailed support. "Your receipt links don't work." Support replied with the PDF attached, closed the ticket, and moved on. Each ticket looked like a one-off.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why it took a year to notice
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;No link checker would have found it.&lt;/strong&gt; A crawler or monitor that requested a receipt URL during the first week would get a 200. The links weren't broken when they were sent; they broke on a timer.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Support tagged the tickets as "billing question".&lt;/strong&gt; About 40 tickets a month, which nobody connected because they arrived spread out, with different wording, mostly from people who weren't the main account contact.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The expired page returned 403.&lt;/strong&gt; Nothing alerted on 403s from the receipt route, because 403 is also what you'd expect from someone tampering with a signature. The logs showed a steady stream of them with no one looking.&lt;/p&gt;

&lt;p&gt;It was found when a large customer's finance department sent a formal complaint: they'd needed 140 receipts for an audit and every link had failed.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Expiring links are easy to miss until a customer complains.&lt;/strong&gt; &lt;a href="https://linkbruno.com" rel="noopener noreferrer"&gt;linkbruno.com&lt;/a&gt; rechecks the same URLs on a schedule and alerts you when a receipt or account link starts failing.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  The fix
&lt;/h2&gt;

&lt;p&gt;The team made four changes.&lt;/p&gt;

&lt;h3&gt;
  
  
  1. Separate the link's lifetime from the receipt's lifetime
&lt;/h3&gt;

&lt;p&gt;The signed link now lasts 13 months instead of 7 days. A receipt is low-sensitivity compared with, say, account access: it contains the company name, the amount and the last four digits of the card. The team decided that someone who has the email can reasonably keep seeing the receipt for a year.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Make the expired page useful
&lt;/h3&gt;

&lt;p&gt;When a link does expire, the page no longer dead-ends. It says what happened and offers two ways forward: a button to email a fresh link to the address the receipt was sent to (never to an address typed on the page), and a link to the billing section of the app for anyone who can log in.&lt;/p&gt;

&lt;p&gt;That single change would have prevented almost all of the support tickets, even with the original seven-day expiry.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Attach the PDF to the email
&lt;/h3&gt;

&lt;p&gt;The receipt email now includes the PDF as an attachment. Finance teams forward emails into expense tools and archives; an attachment survives that. A link only works as long as the server agrees.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. Measure link failures on routes that matter
&lt;/h3&gt;

&lt;p&gt;The team added a dashboard panel for 403 and 410 responses on the receipt, invoice and download routes, with the reason (expired, bad signature, not found). Expired-link hits from real users are a product problem, not a security event, and they're now visible.&lt;/p&gt;

&lt;h2&gt;
  
  
  How to test expiring links
&lt;/h2&gt;

&lt;p&gt;The team added a test that would have caught the original bug in a minute. It generates a receipt link with the real signing code, then calls the receipt route with the server's clock set 35 days, 6 months and 14 months ahead (using a fake clock in the test, not by changing the system time). The first two must show the receipt. The third must show the friendly expired page with the "email me a fresh link" button, and must not reveal any invoice details. The same pattern works for any signed URL: download links, unsubscribe links, invitation links and magic login links all deserve a test at the age people actually click them.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I'd take from it
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Expiry is part of a link's design.&lt;/strong&gt; Pick it based on when people will actually use the link, not on the last expiry value you set somewhere else.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A link in an email is a promise that outlives the email.&lt;/strong&gt; Receipts, invoices, contracts and downloads get opened months or years later.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Never dead-end an expired link.&lt;/strong&gt; Offer a safe way to get a fresh one.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Some broken links are time bombs.&lt;/strong&gt; Link checkers verify a link at one point in time. For links that expire, test them at the age people actually use them, which in this case was a request made with a link signed 35 days earlier.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Tools like &lt;a href="https://linkbruno.com" rel="noopener noreferrer"&gt;linkbruno.com&lt;/a&gt; are good at telling you which links are broken right now. Expiring links need one extra test in your own suite: generate a link, move the clock forward, and check what the user sees.&lt;/p&gt;

&lt;h2&gt;
  
  
  Related reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/how-to-check-every-link-in-your-npm-package-before-you-publish-readme-homepage-repository-and-h28"&gt;How to check every link in your npm package before you publish (README, homepage, repository and bugs)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/every-url-in-your-error-messages-is-a-promise-a-10-point-checklist-for-links-baked-into-software-47p0"&gt;Every URL in your error messages is a promise: a 10-point checklist for links baked into software&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/you-fixed-the-link-but-its-still-broken-how-cached-404s-and-redirects-outlive-your-fix-27o7"&gt;You fixed the link, but it's still broken: how cached 404s and redirects outlive your fix&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/soft-404s-your-page-not-found-returns-200-and-your-link-checker-thinks-its-fine-3624"&gt;Soft 404s: your 'page not found' returns 200, and your link checker thinks it's fine&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;How long do the links in your transactional emails last? And what does someone see when they click one after it expires?&lt;/p&gt;

</description>
      <category>webdev</category>
      <category>backend</category>
      <category>security</category>
      <category>email</category>
    </item>
    <item>
      <title>How to check every link in your npm package before you publish (README, homepage, repository and bugs)</title>
      <dc:creator>linkbruno.com</dc:creator>
      <pubDate>Mon, 16 Nov 2026 11:44:50 +0000</pubDate>
      <link>https://dev.to/linkbruno/how-to-check-every-link-in-your-npm-package-before-you-publish-readme-homepage-repository-and-h28</link>
      <guid>https://dev.to/linkbruno/how-to-check-every-link-in-your-npm-package-before-you-publish-readme-homepage-repository-and-h28</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Scheduled broken-link and redirect checks that tell you the moment a link breaks — &lt;a href="https://linkbruno.com" rel="noopener noreferrer"&gt;linkbruno.com&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;There's a detail about npm that surprises a lot of package authors: the README shown on npmjs.com is the one from the published tarball. It doesn't update when you fix your GitHub README. If version 2.3.1 shipped with a broken link to the docs, every visitor to the package page sees that broken link until you publish 2.3.2.&lt;/p&gt;

&lt;p&gt;The same goes for the URLs in &lt;code&gt;package.json&lt;/code&gt;: &lt;code&gt;homepage&lt;/code&gt;, &lt;code&gt;repository&lt;/code&gt;, &lt;code&gt;bugs&lt;/code&gt; and &lt;code&gt;funding&lt;/code&gt; are shown in the package page sidebar, used by &lt;code&gt;npm docs&lt;/code&gt; and &lt;code&gt;npm bugs&lt;/code&gt;, and read by tools that build dependency reports. A typo there is a broken link in thousands of places at once.&lt;/p&gt;

&lt;p&gt;So the right moment to check those links is just before &lt;code&gt;npm publish&lt;/code&gt;. Here's a small script that does it.&lt;/p&gt;

&lt;h2&gt;
  
  
  What to check
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Every absolute URL in the README.&lt;/strong&gt; Documentation, badges, examples, license, changelog.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Relative links in the README.&lt;/strong&gt; This is the big one, covered below.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;package.json&lt;/code&gt; URL fields&lt;/strong&gt;: &lt;code&gt;homepage&lt;/code&gt;, &lt;code&gt;repository.url&lt;/code&gt;, &lt;code&gt;bugs.url&lt;/code&gt; (or &lt;code&gt;bugs&lt;/code&gt; as a string), and &lt;code&gt;funding&lt;/code&gt; (string, object or array).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Images.&lt;/strong&gt; Badge services change URLs; screenshots stored in a branch that got renamed stop loading.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  The relative-link trap
&lt;/h2&gt;

&lt;p&gt;On GitHub, a README link like &lt;code&gt;[Configuration](docs/config.md)&lt;/code&gt; or &lt;code&gt;![demo](assets/demo.gif)&lt;/code&gt; resolves relative to the repo. On npmjs.com, it's resolved against the package page, or against the repository URL if npm can work one out, depending on the link and the npm website's current rendering rules. It's often not where you intended, and files that aren't in the published tarball (&lt;code&gt;docs/&lt;/code&gt;, &lt;code&gt;assets/&lt;/code&gt;, anything in &lt;code&gt;.npmignore&lt;/code&gt; or not in &lt;code&gt;files&lt;/code&gt;) can't be served from the package at all.&lt;/p&gt;

&lt;p&gt;The safe approach is to use absolute URLs in the README for anything that must work on npm. Link to a docs site, or to a specific tag on GitHub (&lt;code&gt;https://github.com/OWNER/REPO/blob/v2.3.1/docs/config.md&lt;/code&gt;) so the link matches the published version. The script below flags any relative link so you can make that choice deliberately.&lt;/p&gt;

&lt;h2&gt;
  
  
  The script
&lt;/h2&gt;

&lt;p&gt;Save as &lt;code&gt;scripts/check-links.mjs&lt;/code&gt;. It needs Node 18 or later (for the built-in &lt;code&gt;fetch&lt;/code&gt;) and no dependencies.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;readFile&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="s2"&gt;"node:fs/promises"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;pkg&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;JSON&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;readFile&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"package.json"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"utf8"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;readme&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;readFile&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"README.md"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"utf8"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;urls&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nb"&gt;Map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="c1"&gt;// url -&amp;gt; where it came from&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;add&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;u&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;u&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="n"&gt;urls&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;u&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;/^&lt;/span&gt;&lt;span class="n"&gt;git&lt;/span&gt;\&lt;span class="o"&gt;+/&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;\&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;git&lt;/span&gt;$&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="nf"&gt;add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pkg&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;homepage&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"package.json homepage"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="nf"&gt;add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;typeof&lt;/span&gt; &lt;span class="n"&gt;pkg&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;repository&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="s2"&gt;"string"&lt;/span&gt; &lt;span class="o"&gt;?&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;pkg&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;repository&lt;/span&gt;&lt;span class="o"&gt;?&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"package.json repository"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="nf"&gt;add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;typeof&lt;/span&gt; &lt;span class="n"&gt;pkg&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;bugs&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="s2"&gt;"string"&lt;/span&gt; &lt;span class="o"&gt;?&lt;/span&gt; &lt;span class="n"&gt;pkg&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;bugs&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;pkg&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;bugs&lt;/span&gt;&lt;span class="o"&gt;?&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"package.json bugs"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt; &lt;span class="k"&gt;of&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;pkg&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;funding&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;flat&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;Boolean&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="nf"&gt;add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;typeof&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="s2"&gt;"string"&lt;/span&gt; &lt;span class="o"&gt;?&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"package.json funding"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;relative&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="c1"&gt;// Markdown links/images and HTML href/src attributes&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt;\&lt;span class="p"&gt;]&lt;/span&gt;\&lt;span class="p"&gt;(&lt;/span&gt;\&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="o"&gt;*&amp;lt;?&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;\&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;?&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;?:&lt;/span&gt;\&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="s2"&gt;"[^"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="s2"&gt;")?\s*\)|(?:href|src)=["&lt;/span&gt;&lt;span class="s1"&gt;']([^"'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"']/g;
for (const m of readme.matchAll(re)) {
  const u = m[1] || m[2];
  if (/^https?:\/\//.test(u)) add(u, "&lt;/span&gt;&lt;span class="no"&gt;README&lt;/span&gt;&lt;span class="s2"&gt;");
  else if (!u.startsWith("&lt;/span&gt;#&lt;span class="s2"&gt;") &amp;amp;&amp;amp; !u.startsWith("&lt;/span&gt;&lt;span class="n"&gt;mailto&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="s2"&gt;")) relative.push(u);
}

async function check(url) {
  for (const method of ["&lt;/span&gt;&lt;span class="no"&gt;HEAD&lt;/span&gt;&lt;span class="s2"&gt;", "&lt;/span&gt;&lt;span class="no"&gt;GET&lt;/span&gt;&lt;span class="s2"&gt;"]) {
    try {
      const res = await fetch(url, { method, redirect: "&lt;/span&gt;&lt;span class="n"&gt;follow&lt;/span&gt;&lt;span class="s2"&gt;", signal: AbortSignal.timeout(15000),
        headers: { "&lt;/span&gt;&lt;span class="n"&gt;user&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;agent&lt;/span&gt;&lt;span class="s2"&gt;": "&lt;/span&gt;&lt;span class="n"&gt;pkg&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;link&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;check&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mf"&gt;1.0&lt;/span&gt;&lt;span class="s2"&gt;" } });
      if (res.ok || res.status === 429) return [res.status, res.url];
      if (method === "&lt;/span&gt;&lt;span class="no"&gt;GET&lt;/span&gt;&lt;span class="s2"&gt;") return [res.status, res.url];
    } catch (e) {
      if (method === "&lt;/span&gt;&lt;span class="no"&gt;GET&lt;/span&gt;&lt;span class="s2"&gt;") return [0, String(e.cause?.code || e.name)];
    }
  }
}

let failed = 0;
const results = await Promise.all([...urls].map(async ([u, from]) =&amp;gt; [u, from, ...(await check(u))]));
for (const [u, from, status, final] of results) {
  const bad = status === 0 || status &amp;gt;= 400;
  if (bad) failed++;
  console.log(`${bad ? "&lt;/span&gt;&lt;span class="no"&gt;FAIL&lt;/span&gt;&lt;span class="s2"&gt;" : "&lt;/span&gt;&lt;span class="n"&gt;ok&lt;/span&gt;  &lt;span class="s2"&gt;"} ${status} ${u}  (${from})${final &amp;amp;&amp;amp; final !== u ? ` -&amp;gt; ${final}` : ""}`);
}
for (const r of relative) console.log(`WARN relative link in README: ${r}`);

if (failed) { console.error(`\n${failed} broken link(s)`); process.exit(1); }
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The script tries &lt;code&gt;HEAD&lt;/code&gt; first and falls back to &lt;code&gt;GET&lt;/code&gt;, because many servers answer &lt;code&gt;HEAD&lt;/code&gt; with 404 or 405 even when the page is fine. It treats 429 (rate limited) as success, since that means the server is up. Redirects are followed and the final URL printed, which catches the common case of a repository URL that redirects after the project was renamed.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;The README isn't the only place a package link can break.&lt;/strong&gt; &lt;a href="https://linkbruno.com" rel="noopener noreferrer"&gt;linkbruno.com&lt;/a&gt; keeps checking those URLs after publish, so a dead homepage or docs link doesn't sit unnoticed.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Run it before every publish
&lt;/h2&gt;

&lt;p&gt;Add it to &lt;code&gt;package.json&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"scripts"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nl"&gt;"check-links"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"node scripts/check-links.mjs"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"prepublishOnly"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"npm run check-links"&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;prepublishOnly&lt;/code&gt; runs on &lt;code&gt;npm publish&lt;/code&gt; and stops the publish if the script exits with an error. It doesn't run on &lt;code&gt;npm install&lt;/code&gt;, so it won't bother your users.&lt;/p&gt;

&lt;p&gt;If you publish from CI, run the same script in the release workflow, before the publish step. You might also want a weekly scheduled run on the main branch: links in the README can break between releases, and it's better to fix them in the next release than to find out from an issue.&lt;/p&gt;

&lt;h2&gt;
  
  
  Check what's actually in the tarball
&lt;/h2&gt;

&lt;p&gt;The README you check should be the one that gets published. Two quick commands:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;npm pack --dry-run        &lt;span class="c"&gt;# lists every file that would be published&lt;/span&gt;
npm pack &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; tar -xzf *.tgz &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; cat package/README.md
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If you generate the README at build time, or your &lt;code&gt;files&lt;/code&gt; field excludes it by accident, this is where you'll notice.&lt;/p&gt;

&lt;h2&gt;
  
  
  Other places your package's links live
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The &lt;code&gt;bin&lt;/code&gt; script's &lt;code&gt;--help&lt;/code&gt; text&lt;/strong&gt; and error messages often contain URLs. They ship in the tarball and live as long as the version stays installed.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The CHANGELOG&lt;/strong&gt;, if you publish it. Links to issues and pull requests are usually stable; links to blog posts and docs pages aren't.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;JSDoc &lt;code&gt;@see&lt;/code&gt; and &lt;code&gt;@link&lt;/code&gt; tags&lt;/strong&gt;, which editors show on hover.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For the docs site those links point to, a scheduled crawl from something like &lt;a href="https://linkbruno.com" rel="noopener noreferrer"&gt;linkbruno.com&lt;/a&gt; covers the other half: making sure the pages your package sends people to still exist.&lt;/p&gt;

&lt;h2&gt;
  
  
  Related reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/experiment-what-share-of-the-links-in-your-github-readmes-still-work-measure-every-repo-you-own-kl2"&gt;Experiment: what share of the links in your GitHub READMEs still work? Measure every repo you own in 15 minutes&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/every-url-in-your-error-messages-is-a-promise-a-10-point-checklist-for-links-baked-into-software-47p0"&gt;Every URL in your error messages is a promise: a 10-point checklist for links baked into software&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/head-vs-get-for-link-checking-head-is-faster-and-it-lies-more-than-youd-think-2fmd"&gt;HEAD vs GET for link checking: HEAD is faster, and it lies more than you'd think&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/case-study-hypothetical-the-view-receipt-links-in-payment-emails-that-expired-after-7-days-elm"&gt;Case study (hypothetical): the 'View receipt' links in payment emails that expired after 7 days&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Have you ever shipped a package with a broken README link and had to publish a patch release just to fix it? I suspect most maintainers have.&lt;/p&gt;

</description>
      <category>npm</category>
      <category>javascript</category>
      <category>node</category>
      <category>opensource</category>
    </item>
    <item>
      <title>Moving docs from a GitHub wiki to a docs site? An 11-point checklist so the links survive</title>
      <dc:creator>linkbruno.com</dc:creator>
      <pubDate>Sun, 15 Nov 2026 14:04:30 +0000</pubDate>
      <link>https://dev.to/linkbruno/moving-docs-from-a-github-wiki-to-a-docs-site-an-11-point-checklist-so-the-links-survive-130f</link>
      <guid>https://dev.to/linkbruno/moving-docs-from-a-github-wiki-to-a-docs-site-an-11-point-checklist-so-the-links-survive-130f</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Scheduled broken-link and redirect checks that tell you the moment a link breaks — &lt;a href="https://linkbruno.com" rel="noopener noreferrer"&gt;linkbruno.com&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;A GitHub wiki is a fine place to start a project's documentation. It's right there, anyone with access can edit it, and it needs no build step. Eventually most projects outgrow it: they want versioned docs, search, review on pull requests, a custom domain. So they move to MkDocs, Docusaurus, Starlight, Sphinx or similar.&lt;/p&gt;

&lt;p&gt;The new docs site goes live, and the wiki becomes either a stale copy or a single page saying "We've moved". And the links people have been posting for years, in issues, in commit messages, in Stack Overflow answers, in blog posts and in tutorials, start leading nowhere useful.&lt;/p&gt;

&lt;p&gt;GitHub wikis don't support server-side redirects, so this needs more planning than a normal site move. Here's the checklist.&lt;/p&gt;

&lt;h2&gt;
  
  
  Before the move
&lt;/h2&gt;

&lt;h3&gt;
  
  
  1. Inventory every wiki page and its URL
&lt;/h3&gt;

&lt;p&gt;Clone the wiki. It's a separate Git repository:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;git clone https://github.com/OWNER/REPO.wiki.git
ls REPO.wiki
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Each file maps to a URL: &lt;code&gt;Getting-Started.md&lt;/code&gt; becomes &lt;code&gt;https://github.com/OWNER/REPO/wiki/Getting-Started&lt;/code&gt;. Make a spreadsheet or CSV of every page and its wiki URL. That list is the basis for everything that follows.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Find out which wiki pages people actually link to
&lt;/h3&gt;

&lt;p&gt;You can't get full analytics for a wiki, but you can find the links that matter most:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Search your own issues, pull requests and discussions for &lt;code&gt;/wiki/&lt;/code&gt;. GitHub's search does this across the repo.&lt;/li&gt;
&lt;li&gt;Search the code: &lt;code&gt;git grep -n "/wiki/"&lt;/code&gt; in the main repo, including READMEs, docs, error messages and CLI help text.&lt;/li&gt;
&lt;li&gt;Search the web for your wiki URL in quotes, and look at the repo's traffic page (Insights, Traffic) for referring sites.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The pages that show up here are the ones that need a perfect landing spot on the new site.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Map every page to a new URL before you start writing
&lt;/h3&gt;

&lt;p&gt;For each wiki page, decide the exact new URL. Keep the mapping one-to-one where you can. If several wiki pages merge into one new page, map each to the right section anchor (&lt;code&gt;/install/#linux&lt;/code&gt;), not just to the page.&lt;/p&gt;

&lt;p&gt;Write the mapping down as data, not prose:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;wiki_page,new_url
Getting-Started,https://docs.example.org/start/
Installation-on-Linux,https://docs.example.org/install/#linux
FAQ,https://docs.example.org/faq/
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You'll use it three times: to write the redirect pages, to update links in your repo, and to test.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. Convert the wiki's link syntax
&lt;/h3&gt;

&lt;p&gt;Wiki pages link to each other with &lt;code&gt;[[Page Name]]&lt;/code&gt; or &lt;code&gt;[[Text|Page-Name]]&lt;/code&gt;, which no docs generator understands. Convert these to normal Markdown links pointing at the new paths during migration, and run a link check on the converted docs before anything goes live.&lt;/p&gt;

&lt;h2&gt;
  
  
  During the move
&lt;/h2&gt;

&lt;h3&gt;
  
  
  5. Replace each wiki page with a pointer, not a deletion
&lt;/h3&gt;

&lt;p&gt;Since the wiki can't redirect, the next best thing is to keep every page and replace its content with a short note and a link to the exact new location:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight markdown"&gt;&lt;code&gt;This page has moved to &lt;span class="gs"&gt;**[Installation on Linux](https://docs.example.org/install/#linux)**&lt;/span&gt;.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Generate these from the mapping file with a short script, so every page gets one and none point at the docs homepage. A pointer to the right page is far more useful than a generic "the docs moved" note.&lt;/p&gt;

&lt;h3&gt;
  
  
  6. Decide whether to disable the wiki at all
&lt;/h3&gt;

&lt;p&gt;Turning off the wiki in repo settings makes every wiki URL redirect back to the repo's main page. That's tidy, but it throws away the pointer pages from the previous step. For most projects, keeping the wiki enabled, with editing restricted to collaborators and every page replaced by a pointer, is the better trade-off.&lt;/p&gt;

&lt;h3&gt;
  
  
  7. Update every link you control
&lt;/h3&gt;

&lt;p&gt;Use the mapping to rewrite links in the main repo: README, contributing guide, issue and PR templates, code comments, error messages, CLI &lt;code&gt;--help&lt;/code&gt; output, package metadata (&lt;code&gt;homepage&lt;/code&gt; and &lt;code&gt;documentation&lt;/code&gt; fields), and the repo's website field. Error messages and CLI help are the easy ones to forget, and they ship in installed copies for years.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Wiki moves are where links quietly die.&lt;/strong&gt; &lt;a href="https://linkbruno.com" rel="noopener noreferrer"&gt;linkbruno.com&lt;/a&gt; checks the new docs site on a schedule and tells you the moment a migrated link breaks.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  After the move
&lt;/h2&gt;

&lt;h3&gt;
  
  
  8. Check that every old URL leads to the right place
&lt;/h3&gt;

&lt;p&gt;Run through the mapping and verify that each wiki URL loads, contains the new link, and that the new link resolves. A short script that fetches each pointer page, extracts the link and requests it will catch typos in the mapping.&lt;/p&gt;

&lt;h3&gt;
  
  
  9. Check the new site for broken links on every build
&lt;/h3&gt;

&lt;p&gt;Docs generators can usually fail the build on broken internal links (MkDocs &lt;code&gt;--strict&lt;/code&gt;, Docusaurus &lt;code&gt;onBrokenLinks: 'throw'&lt;/code&gt;). Turn that on now, while the docs are fresh, rather than after the first pile of broken links appears.&lt;/p&gt;

&lt;h3&gt;
  
  
  10. Update the most important external links
&lt;/h3&gt;

&lt;p&gt;You can't fix every blog post, but you can fix the ones you own and the most visible ones you don't: your project's listing in package registries, awesome-lists that link the wiki, and highly voted Stack Overflow answers (an edit suggestion to update a moved link is usually accepted).&lt;/p&gt;

&lt;h3&gt;
  
  
  11. Recheck in a month
&lt;/h3&gt;

&lt;p&gt;Some links only surface over time, as people revisit old bookmarks. A month after the move, look at which wiki pages still get visits (if you can tell) and run a crawl of the new docs site plus the pointer pages. External checkers such as &lt;a href="https://linkbruno.com" rel="noopener noreferrer"&gt;linkbruno.com&lt;/a&gt; can do the crawl side of this on a schedule.&lt;/p&gt;

&lt;h2&gt;
  
  
  The short version
&lt;/h2&gt;

&lt;p&gt;Clone the wiki, map every page to an exact new URL, replace pages with precise pointers instead of deleting them, rewrite every link you control from the same mapping, and turn on broken-link checks in the new docs build from day one.&lt;/p&gt;

&lt;h2&gt;
  
  
  Related reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/how-to-fix-neterrblockedbyclient-when-an-ad-blocker-breaks-your-images-scripts-and-links-4hg3"&gt;How to fix net::ERR_BLOCKED_BY_CLIENT (when an ad blocker breaks your images, scripts and links)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/markdown-links-in-a-monorepo-7-things-id-do-differently-after-the-third-big-folder-move-20dp"&gt;Markdown links in a monorepo: 7 things I'd do differently after the third big folder move&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/your-redirects-file-should-be-append-only-deleting-a-redirect-is-a-breaking-change-6lh"&gt;Your redirects file should be append-only. Deleting a redirect is a breaking change.&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/how-to-find-broken-anchor-links-the-ones-every-link-checker-says-are-fine-4be2"&gt;How to find broken #anchor links, the ones every link checker says are fine&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Have you moved a project off a GitHub wiki? Did you keep the pointer pages, or disable the wiki and accept the broken links?&lt;/p&gt;

</description>
      <category>documentation</category>
      <category>github</category>
      <category>opensource</category>
      <category>webdev</category>
    </item>
    <item>
      <title>How to fix net::ERR_BLOCKED_BY_CLIENT (when an ad blocker breaks your images, scripts and links)</title>
      <dc:creator>linkbruno.com</dc:creator>
      <pubDate>Sun, 15 Nov 2026 14:04:03 +0000</pubDate>
      <link>https://dev.to/linkbruno/how-to-fix-neterrblockedbyclient-when-an-ad-blocker-breaks-your-images-scripts-and-links-4hg3</link>
      <guid>https://dev.to/linkbruno/how-to-fix-neterrblockedbyclient-when-an-ad-blocker-breaks-your-images-scripts-and-links-4hg3</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Scheduled broken-link and redirect checks that tell you the moment a link breaks — &lt;a href="https://linkbruno.com" rel="noopener noreferrer"&gt;linkbruno.com&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;You open the console on your own site and see a column of red:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;GET https://example.com/static/ads/hero-banner.webp net::ERR_BLOCKED_BY_CLIENT
GET https://example.com/js/analytics-consent.js net::ERR_BLOCKED_BY_CLIENT
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Nothing is wrong with your server. The request never left the browser. &lt;code&gt;ERR_BLOCKED_BY_CLIENT&lt;/code&gt; means something running inside the browser, almost always an extension, cancelled it. For your visitors who use ad blockers (a large share on developer-focused and tech sites), that image, script or link target simply doesn't exist.&lt;/p&gt;

&lt;p&gt;This matters most when the blocked thing isn't an ad at all. Here's how to find out why it's blocked and how to fix it without asking anyone to turn off their blocker.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 1: confirm it's an extension
&lt;/h2&gt;

&lt;p&gt;Open the same page in an Incognito or private window with extensions disabled (Chrome disables them in Incognito unless you've allowed them). If the errors disappear, an extension is the cause.&lt;/p&gt;

&lt;p&gt;To find which extension, re-enable them one at a time, or check the obvious candidates first: uBlock Origin, AdBlock Plus, AdGuard, Ghostery, Privacy Badger, and the blockers built into Brave and Opera. Brave's shields block requests without any extension installed, and show up the same way.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 2: find the rule that matched
&lt;/h2&gt;

&lt;p&gt;Content blockers use filter lists such as EasyList, EasyPrivacy and Fanboy's lists. Rules match on URL patterns, so the file name or path is usually what triggered the block, not the content.&lt;/p&gt;

&lt;p&gt;In uBlock Origin, open the dashboard's logger (the "list" icon in the popup, or the extension's logger page). Reload your page and find the red entries. Each one shows the rule that matched and the list it came from, for example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;||example.com^*/ads/       EasyList
/analytics-               EasyPrivacy
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;AdGuard and AdBlock Plus have similar request logs. Once you know the rule, you know what to rename.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 3: the usual innocent victims
&lt;/h2&gt;

&lt;p&gt;These are the patterns that most often catch legitimate files:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;/ads/&lt;/code&gt;, &lt;code&gt;/ad/&lt;/code&gt;, &lt;code&gt;-ad-&lt;/code&gt;, &lt;code&gt;_ad_&lt;/code&gt;, &lt;code&gt;adserver&lt;/code&gt;, &lt;code&gt;banner&lt;/code&gt;&lt;/strong&gt; in paths or file names. A folder called &lt;code&gt;static/ads/&lt;/code&gt; holding product images for your "advertise with us" page gets blocked wholesale.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;analytics&lt;/code&gt;, &lt;code&gt;tracking&lt;/code&gt;, &lt;code&gt;track&lt;/code&gt;, &lt;code&gt;pixel&lt;/code&gt;, &lt;code&gt;beacon&lt;/code&gt;, &lt;code&gt;collect&lt;/code&gt;, &lt;code&gt;stats&lt;/code&gt;.&lt;/strong&gt; Including your own first-party code with those words in it: &lt;code&gt;analytics-consent.js&lt;/code&gt;, &lt;code&gt;track-order.js&lt;/code&gt; (an order tracking page!), &lt;code&gt;stats-chart.js&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;share&lt;/code&gt;, &lt;code&gt;social&lt;/code&gt;, &lt;code&gt;facebook&lt;/code&gt;, &lt;code&gt;twitter&lt;/code&gt;, &lt;code&gt;linkedin&lt;/code&gt;&lt;/strong&gt; in file names, caught by social-widget and annoyance lists.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;popup&lt;/code&gt;, &lt;code&gt;modal-promo&lt;/code&gt;, &lt;code&gt;newsletter-overlay&lt;/code&gt;&lt;/strong&gt;, caught by annoyance lists.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Common third-party hosts&lt;/strong&gt;: tag managers, analytics endpoints, ad networks, many chat and feedback widgets, and some font and CDN hosts on stricter lists.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The track-order case is a good example of why this is a broken-link problem as much as a script problem. If a page's important link or button is rendered by a blocked script, that link doesn't exist for a large slice of visitors.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Tired of hunting blocked assets one URL at a time?&lt;/strong&gt; &lt;a href="https://linkbruno.com" rel="noopener noreferrer"&gt;linkbruno.com&lt;/a&gt; crawls the site and flags the images, scripts and links that fail to load.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Step 4: fix it
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Rename first-party files and folders
&lt;/h3&gt;

&lt;p&gt;For your own assets, the fix is boring and effective: rename them so they don't match ad or tracker patterns. &lt;code&gt;static/ads/&lt;/code&gt; becomes &lt;code&gt;static/promo-images/&lt;/code&gt; (check that one too; "promo" appears in some lists), &lt;code&gt;track-order.js&lt;/code&gt; becomes &lt;code&gt;order-status.js&lt;/code&gt;, &lt;code&gt;stats-chart.js&lt;/code&gt; becomes &lt;code&gt;dashboard-chart.js&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Then search your codebase and build output for the old names, so nothing still references them. And keep redirects from the old URLs if any of them were ever linked from outside.&lt;/p&gt;

&lt;h3&gt;
  
  
  Don't build core features on blocked third parties
&lt;/h3&gt;

&lt;p&gt;If your checkout, sign-up form or main navigation depends on a third-party script that's on a common block list, some users can't complete those flows. Load essential functionality from your own domain, and treat third-party widgets as optional enhancements: the page should work, and the links should be visible, if they fail to load.&lt;/p&gt;

&lt;h3&gt;
  
  
  Make blocked loads fail softly
&lt;/h3&gt;

&lt;p&gt;When you do load optional third-party scripts, handle the failure:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;document&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;createElement&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"script"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;src&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"https://widget.example-chat.com/loader.js"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;onerror&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="c1"&gt;// Blocked or unreachable: show a plain contact link instead&lt;/span&gt;
  &lt;span class="nb"&gt;document&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;querySelector&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"#chat-fallback"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;hidden&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="nb"&gt;document&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;head&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;appendChild&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A plain &lt;code&gt;&amp;lt;a href="/contact"&amp;gt;&lt;/code&gt; fallback means the visitor still has somewhere to go.&lt;/p&gt;

&lt;h3&gt;
  
  
  Don't try to evade blockers for tracking
&lt;/h3&gt;

&lt;p&gt;Renaming your own product images and order-status script is fixing a false positive. Disguising tracking scripts so blockers miss them is something else, it's against your visitors' explicit choice, and filter list maintainers usually catch up anyway.&lt;/p&gt;

&lt;h2&gt;
  
  
  Find blocked files across the whole site
&lt;/h2&gt;

&lt;p&gt;Checking pages one at a time in a browser is slow. If you have a list of your own asset URLs (from your build manifest, or a crawl), you can test them against a downloaded copy of EasyList and EasyPrivacy with a filter-matching library, such as the &lt;code&gt;adblock&lt;/code&gt; Python bindings for Brave's Rust engine or the &lt;code&gt;@cliqz/adblocker&lt;/code&gt; package in Node. Load the lists, run every URL through the matcher with the right request type (image, script, stylesheet), and print the ones that match a blocking rule along with the rule. It takes a few minutes to set up and turns "some users say the chart is missing" into a precise list of files to rename before the next release.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 5: test with a blocker on
&lt;/h2&gt;

&lt;p&gt;Add a blocker to your regular testing. A browser profile with uBlock Origin and default lists, used for a quick click-through before each release, catches most of these issues. For automated tests, you can load an unpacked extension in Playwright using a persistent Chromium context, which lets you run your link and smoke tests with the blocker active.&lt;/p&gt;

&lt;p&gt;Link checkers that request URLs directly, including &lt;a href="https://linkbruno.com" rel="noopener noreferrer"&gt;linkbruno.com&lt;/a&gt;, see these files as healthy, because the server returns them fine. The blocker-on browser test is the check that sees what those visitors see.&lt;/p&gt;

&lt;h2&gt;
  
  
  Related reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/how-to-find-and-fix-broken-images-on-a-website-54o1"&gt;How to find and fix broken images on a website&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/0-broken-links-found-is-a-myth-7-places-link-checkers-dont-look-4g45"&gt;'0 broken links found' is a myth: 7 places link checkers don't look&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/how-to-find-mixed-content-http-links-on-an-https-site-1d69"&gt;How to find mixed content (http:// links) on an HTTPS site&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/moving-docs-from-a-github-wiki-to-a-docs-site-an-11-point-checklist-so-the-links-survive-130f"&gt;Moving docs from a GitHub wiki to a docs site? An 11-point checklist so the links survive&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;What's the most innocent file you've had blocked? I've seen a recipe site lose its whole "Add to shopping list" feature because the script was called &lt;code&gt;list-tracker.js&lt;/code&gt;.&lt;/p&gt;

</description>
      <category>webdev</category>
      <category>javascript</category>
      <category>debugging</category>
      <category>frontend</category>
    </item>
    <item>
      <title>Should code comments link to Stack Overflow answers? Let's argue about it</title>
      <dc:creator>linkbruno.com</dc:creator>
      <pubDate>Sat, 14 Nov 2026 08:07:37 +0000</pubDate>
      <link>https://dev.to/linkbruno/should-code-comments-link-to-stack-overflow-answers-lets-argue-about-it-33nh</link>
      <guid>https://dev.to/linkbruno/should-code-comments-link-to-stack-overflow-answers-lets-argue-about-it-33nh</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Scheduled broken-link and redirect checks that tell you the moment a link breaks — &lt;a href="https://linkbruno.com" rel="noopener noreferrer"&gt;linkbruno.com&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Open any mature codebase and search for &lt;code&gt;stackoverflow.com&lt;/code&gt;. You'll find a few dozen comments like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// Safari doesn't fire 'change' on autofill, see&lt;/span&gt;
&lt;span class="c1"&gt;// https://stackoverflow.com/a/&amp;lt;answer-id&amp;gt;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Some teams love these. Some ban them in code review. Most have never discussed it. I think it's worth arguing about, because the answer affects how future maintainers understand the strangest parts of your code.&lt;/p&gt;

&lt;h2&gt;
  
  
  The case for linking
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;It explains the why in one line.&lt;/strong&gt; The hardest code to maintain is a workaround with no explanation. Why is there a &lt;code&gt;setTimeout(fn, 0)&lt;/code&gt; here? Why is this header set twice? A link to the answer where someone diagnosed the browser bug or the library quirk gives the next developer the full context, including the discussion, the edge cases, and the comments saying "this also happens on Firefox".&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;It's honest attribution.&lt;/strong&gt; If you copied or adapted code from an answer, the source should be recorded. Content on Stack Overflow is licensed under Creative Commons Attribution-ShareAlike (the exact version depends on when it was posted), and attribution is part of the licence. A link in the comment is the simplest way to provide it.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;It's a signal for later cleanup.&lt;/strong&gt; A comment pointing to an answer from 2014 about an Internet Explorer bug is a flag: this code may no longer be needed. Without the link, nobody knows the workaround was for IE.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Search engines and AI assistants don't replace it.&lt;/strong&gt; "Look it up if you need to" assumes the next person knows what to search for. The link removes that step.&lt;/p&gt;

&lt;h2&gt;
  
  
  The case against linking
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Links in code rot quietly.&lt;/strong&gt; Nobody runs a link checker over source files. Answers get deleted, accounts get removed, questions get closed and merged, and URLs change format. When the link dies, the comment becomes a riddle: there was an explanation, and now there isn't.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Answers change after you link them.&lt;/strong&gt; An answer can be edited, sometimes substantially. The version you read and adapted may not be the version a maintainer reads in three years. The comment says "see this", and "this" now says something different.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;It outsources understanding.&lt;/strong&gt; A link can become a way to avoid writing an explanation. "See SO" is not documentation. If you can't summarise why the workaround is needed in two sentences, maybe you don't fully understand it yet.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Some links are noise.&lt;/strong&gt; A link to "How do I reverse a string in Python" next to &lt;code&gt;s[::-1]&lt;/code&gt; adds nothing. A codebase full of trivial links teaches people to ignore all of them, including the important ones.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Comments aren't the only links that rot.&lt;/strong&gt; &lt;a href="https://linkbruno.com" rel="noopener noreferrer"&gt;linkbruno.com&lt;/a&gt; watches the links in your docs and site and alerts you when one of them breaks.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Where I've landed
&lt;/h2&gt;

&lt;p&gt;I'm in favour of links, with rules:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Explain first, link second.&lt;/strong&gt; The comment should make sense even if the link dies. One or two sentences on what the problem is and why this code works around it, then the link as evidence.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Link the specific answer, not the question.&lt;/strong&gt; Use the "Share" link on the answer (&lt;code&gt;/a/&amp;lt;id&amp;gt;&lt;/code&gt;), so it points to the content you actually used.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Include a date or a version.&lt;/strong&gt; "Safari 16 and earlier, Nov 2026" tells a future maintainer when to re-check. It also helps if the answer is later edited.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Use revision links when the exact text matters.&lt;/strong&gt; Each answer has a revision history; linking a specific revision pins the version you read.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Attribute copied code properly.&lt;/strong&gt; If you used a substantial piece of code from an answer, the link is the minimum. Check what your organisation's policy says about Creative Commons code in your codebase, because some legal teams have opinions.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Check links in source code occasionally.&lt;/strong&gt; Most link checkers can read source files if you tell them to. Running one over &lt;code&gt;**/*.{js,ts,py,go}&lt;/code&gt; once a quarter costs little, and it's the only way those links ever get checked. Tools such as &lt;a href="https://linkbruno.com" rel="noopener noreferrer"&gt;linkbruno.com&lt;/a&gt; and command-line checkers can both do this.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;A good version of the example above:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// Safari (&amp;lt;= 16) doesn't fire 'change' when it autofills a field, so we&lt;/span&gt;
&lt;span class="c1"&gt;// also listen for 'input' and compare values. Discussion and repro:&lt;/span&gt;
&lt;span class="c1"&gt;// https://stackoverflow.com/a/&amp;lt;answer-id&amp;gt; (checked Nov 2026)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  The cases that start arguments
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;GitHub issue links instead.&lt;/strong&gt; Linking the upstream bug report (for example, the browser's issue tracker) is often better than an answer about it, because it tells you when the bug is fixed. But issue trackers move too.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Internal wikis.&lt;/strong&gt; Some teams prefer to copy the explanation into an internal page and link that. The wiki then rots instead, with fewer people looking at it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;AI-generated code.&lt;/strong&gt; If a workaround came from an assistant, there's often nothing to link. Should the comment say where it came from anyway?&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Why this matters more than it looks
&lt;/h2&gt;

&lt;p&gt;Comments with links are some of the longest-lived links you'll ever write. A blog post gets rewritten or deleted. A function with a clever workaround can sit untouched for ten years, and the link is the only explanation it has. If that link dies, the next person either keeps the weird code forever because they're afraid to remove it, or removes it and reintroduces the bug.&lt;/p&gt;

&lt;p&gt;Either way, it's worth deciding as a team rather than leaving it to whoever happens to review the PR.&lt;/p&gt;

&lt;h2&gt;
  
  
  Related reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/how-to-check-links-with-lychee-in-github-actions-config-caching-and-the-false-positives-to-9bn"&gt;How to check links with lychee in GitHub Actions (config, caching, and the false positives to exclude)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/should-your-docs-link-to-githubs-main-branch-or-to-a-permalink-both-options-rot-4ne6"&gt;Should your docs link to GitHub's main branch or to a permalink? Both options rot.&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/should-you-fix-dead-links-in-old-blog-posts-or-leave-them-as-a-historical-record-lets-argue-16i3"&gt;Should you fix dead links in old blog posts, or leave them as a historical record? Let's argue about it&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/should-your-link-checker-obey-robotstxt-lets-argue-about-it-1ldo"&gt;Should your link checker obey robots.txt? Let's argue about it&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So: do you allow Stack Overflow links in your codebase, require them for copied code, or strip them in review? And has a dead one ever left you staring at a workaround with no idea why it exists?&lt;/p&gt;

</description>
      <category>discuss</category>
      <category>programming</category>
      <category>documentation</category>
      <category>webdev</category>
    </item>
    <item>
      <title>How to check links with lychee in GitHub Actions (config, caching, and the false positives to exclude)</title>
      <dc:creator>linkbruno.com</dc:creator>
      <pubDate>Sat, 14 Nov 2026 08:07:12 +0000</pubDate>
      <link>https://dev.to/linkbruno/how-to-check-links-with-lychee-in-github-actions-config-caching-and-the-false-positives-to-9bn</link>
      <guid>https://dev.to/linkbruno/how-to-check-links-with-lychee-in-github-actions-config-caching-and-the-false-positives-to-9bn</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Scheduled broken-link and redirect checks that tell you the moment a link breaks — &lt;a href="https://linkbruno.com" rel="noopener noreferrer"&gt;linkbruno.com&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;lychee is a link checker written in Rust. It's fast, it understands Markdown, HTML and plain text, and it has an official GitHub Action. That makes it one of the easiest ways to stop broken links landing in a docs repo or a static site.&lt;/p&gt;

&lt;p&gt;The default setup works, but it tends to produce a noisy first run: rate-limited sites, links that need a login, and example URLs that were never meant to resolve. This post gets you from zero to a quiet, useful check: one workflow for pull requests, one for a weekly sweep, a config file, caching, and the exclusions worth adding on day one.&lt;/p&gt;

&lt;h2&gt;
  
  
  The minimal workflow
&lt;/h2&gt;

&lt;p&gt;Create &lt;code&gt;.github/workflows/links.yml&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Links&lt;/span&gt;

&lt;span class="na"&gt;on&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;pull_request&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;schedule&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;cron&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"0 6 * * 1"&lt;/span&gt;   &lt;span class="c1"&gt;# Mondays 06:00 UTC
&lt;/span&gt;  &lt;span class="na"&gt;workflow_dispatch&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;

&lt;span class="na"&gt;jobs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;lychee&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;runs-on&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ubuntu-latest&lt;/span&gt;
    &lt;span class="na"&gt;steps&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;uses&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;actions/checkout@v4&lt;/span&gt;

      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Check links&lt;/span&gt;
        &lt;span class="na"&gt;uses&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;lycheeverse/lychee-action@v2&lt;/span&gt;
        &lt;span class="na"&gt;with&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;args&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;--config lychee.toml './**/*.md' './**/*.html'&lt;/span&gt;
          &lt;span class="na"&gt;fail&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;fail: true&lt;/code&gt; makes the job fail when broken links are found. Without it the action reports problems but the check stays green, which is fine for a first trial run and useless after that.&lt;/p&gt;

&lt;p&gt;Quote the glob patterns. If you don't, the shell expands them before lychee sees them, and you'll either check too little or hit an argument-length limit.&lt;/p&gt;

&lt;h2&gt;
  
  
  A config file instead of a long args line
&lt;/h2&gt;

&lt;p&gt;Put settings in &lt;code&gt;lychee.toml&lt;/code&gt; at the repo root, so they're the same in CI and on your laptop:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight toml"&gt;&lt;code&gt;&lt;span class="c1"&gt;# lychee.toml
&lt;/span&gt;&lt;span class="na"&gt;max_concurrency&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;16&lt;/span&gt;
&lt;span class="na"&gt;max_retries&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;
&lt;span class="na"&gt;retry_wait_time&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;
&lt;span class="na"&gt;timeout&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;20&lt;/span&gt;

&lt;span class="c1"&gt;# Treat these as success, not failure
&lt;/span&gt;&lt;span class="na"&gt;accept&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;200&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;204&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;206&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;429&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="c1"&gt;# Don't check these at all
&lt;/span&gt;&lt;span class="na"&gt;exclude&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
  &lt;span class="s1"&gt;'^https?://localhost'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="s1"&gt;'^https?://127\.0\.0\.1'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="s1"&gt;'^https?://(www\.)?example\.(com|org|net)'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="s1"&gt;'^https?://(www\.)?linkedin\.com'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="na"&gt;exclude_path&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"node_modules"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"vendor"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"CHANGELOG.md"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="c1"&gt;# Check mailto: addresses for valid syntax only
&lt;/span&gt;&lt;span class="na"&gt;include_mail&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two settings deserve a comment.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Accepting 429.&lt;/strong&gt; "Too Many Requests" means the server is up and has decided you're asking too often. In CI, that's not a broken link. Accepting it keeps the check from failing because a popular site rate-limited GitHub's shared runner IPs. If you'd rather be strict, keep 429 as a failure in the weekly run and accept it on pull requests.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Excluding LinkedIn and friends.&lt;/strong&gt; Some sites block automated requests entirely (LinkedIn's famous status 999, some sites behind bot protection that return 403 to anything without a real browser). Checking them in CI produces failures you can't fix. Exclude them, and check those links by hand occasionally, or with a separate tool that uses a real browser.&lt;/p&gt;

&lt;h2&gt;
  
  
  Checking only what changed on pull requests
&lt;/h2&gt;

&lt;p&gt;On a big docs repo, checking every file on every pull request is slow and fails on problems the PR didn't introduce. For PRs, check only the changed files:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Changed files&lt;/span&gt;
        &lt;span class="na"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;changed&lt;/span&gt;
        &lt;span class="na"&gt;run&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;|&lt;/span&gt;
          &lt;span class="s"&gt;git fetch origin "${{ github.base_ref }}" --depth=1&lt;/span&gt;
          &lt;span class="s"&gt;files=$(git diff --name-only --diff-filter=AM "origin/${{ github.base_ref }}" -- '*.md' '*.html' | tr '\n' ' ')&lt;/span&gt;
          &lt;span class="s"&gt;echo "files=$files" &amp;gt;&amp;gt; "$GITHUB_OUTPUT"&lt;/span&gt;

      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Check links in changed files&lt;/span&gt;
        &lt;span class="na"&gt;if&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;github.event_name == 'pull_request' &amp;amp;&amp;amp; steps.changed.outputs.files != ''&lt;/span&gt;
        &lt;span class="na"&gt;uses&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;lycheeverse/lychee-action@v2&lt;/span&gt;
        &lt;span class="na"&gt;with&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;args&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;--config lychee.toml ${{ steps.changed.outputs.files }}&lt;/span&gt;
          &lt;span class="na"&gt;fail&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Use &lt;code&gt;fetch-depth: 0&lt;/code&gt; on the checkout step (or fetch the base branch as above) so the diff has something to compare against. The scheduled run still checks everything, which is where links that rotted on their own get caught.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Want link checks that still run when CI is quiet?&lt;/strong&gt; &lt;a href="https://linkbruno.com" rel="noopener noreferrer"&gt;linkbruno.com&lt;/a&gt; schedules broken-link and redirect checks and tells you the moment a link breaks.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Caching results between runs
&lt;/h2&gt;

&lt;p&gt;External links rarely change from hour to hour. lychee can cache results so repeated runs skip URLs it checked recently:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Restore lychee cache&lt;/span&gt;
        &lt;span class="na"&gt;uses&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;actions/cache@v4&lt;/span&gt;
        &lt;span class="na"&gt;with&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;path&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;.lycheecache&lt;/span&gt;
          &lt;span class="na"&gt;key&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;cache-lychee-${{ github.sha }}&lt;/span&gt;
          &lt;span class="na"&gt;restore-keys&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;cache-lychee-&lt;/span&gt;

      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Check links&lt;/span&gt;
        &lt;span class="na"&gt;uses&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;lycheeverse/lychee-action@v2&lt;/span&gt;
        &lt;span class="na"&gt;with&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;args&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;--config lychee.toml --cache --max-cache-age 1d './**/*.md'&lt;/span&gt;
          &lt;span class="na"&gt;fail&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The key includes the commit SHA so a new cache is saved after every run, and &lt;code&gt;restore-keys&lt;/code&gt; loads the most recent one. A one-day maximum age is a good balance: PRs on the same day are fast, and nothing stays "known good" for long. If you don't want failures to be cached at all, so a link that was down an hour ago is always checked again, recent lychee versions can leave chosen status codes out of the cache (look for &lt;code&gt;cache_exclude_status&lt;/code&gt; in the docs for your version).&lt;/p&gt;

&lt;h2&gt;
  
  
  Opening an issue from the weekly run
&lt;/h2&gt;

&lt;p&gt;A failing scheduled job is easy to miss. Turn the report into an issue instead:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Check links&lt;/span&gt;
        &lt;span class="na"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;lychee&lt;/span&gt;
        &lt;span class="na"&gt;uses&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;lycheeverse/lychee-action@v2&lt;/span&gt;
        &lt;span class="na"&gt;with&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;args&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;--config lychee.toml './**/*.md'&lt;/span&gt;
          &lt;span class="na"&gt;output&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;./lychee/out.md&lt;/span&gt;
          &lt;span class="na"&gt;fail&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;

      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Create issue&lt;/span&gt;
        &lt;span class="na"&gt;if&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;steps.lychee.outputs.exit_code != 0 &amp;amp;&amp;amp; github.event_name == 'schedule'&lt;/span&gt;
        &lt;span class="na"&gt;uses&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;peter-evans/create-issue-from-file@v5&lt;/span&gt;
        &lt;span class="na"&gt;with&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;title&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Broken links report&lt;/span&gt;
          &lt;span class="na"&gt;content-filepath&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;./lychee/out.md&lt;/span&gt;
          &lt;span class="na"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;broken-links&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The job needs &lt;code&gt;issues: write&lt;/code&gt; permission for this. The issue becomes the to-do list; close it when the links are fixed.&lt;/p&gt;

&lt;h2&gt;
  
  
  Things lychee won't catch by default
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Fragments.&lt;/strong&gt; &lt;code&gt;page.html#install&lt;/code&gt; passes as long as &lt;code&gt;page.html&lt;/code&gt; exists, even if the &lt;code&gt;#install&lt;/code&gt; heading was renamed. Recent lychee versions can check fragments with &lt;code&gt;--include-fragments&lt;/code&gt;; try it, and expect a few false positives on pages that build headings with JavaScript.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Root-relative links in local files.&lt;/strong&gt; &lt;code&gt;/docs/setup&lt;/code&gt; in a Markdown file has no domain. Tell lychee what the site root is (the option name has changed between versions, so check &lt;code&gt;lychee --help&lt;/code&gt;), or those links get skipped or reported as errors.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Soft 404s.&lt;/strong&gt; A page that says "not found" with a 200 status passes every link checker that only looks at status codes.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For those gaps, and for the sites that block CI runners, a crawler that runs from outside your CI, such as &lt;a href="https://linkbruno.com" rel="noopener noreferrer"&gt;linkbruno.com&lt;/a&gt;, is a useful second layer.&lt;/p&gt;

&lt;h2&gt;
  
  
  Related reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/how-to-find-broken-links-with-playwright-a-test-you-can-drop-into-ci-3d4l"&gt;How to find broken links with Playwright (a test you can drop into CI)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/your-static-site-generator-can-catch-broken-links-at-build-time-heres-how-in-6-popular-ones-30l8"&gt;Your static site generator can catch broken links at build time. Here's how in 6 popular ones&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/should-a-broken-external-link-fail-your-build-comparing-four-ci-policies-and-the-one-id-pick-304o"&gt;Should a broken external link fail your build? Comparing four CI policies, and the one I'd pick&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/check-links-in-markdown-docs-with-a-60-line-python-script-4hg6"&gt;Check links in Markdown docs with a 60-line Python script&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/should-code-comments-link-to-stack-overflow-answers-lets-argue-about-it-33nh"&gt;Should code comments link to Stack Overflow answers? Let's argue about it&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Are you running lychee, or another checker, in CI? Which exclusions did you end up needing that I haven't listed?&lt;/p&gt;

</description>
      <category>github</category>
      <category>webdev</category>
      <category>testing</category>
      <category>tutorial</category>
    </item>
    <item>
      <title>Should you fix dead links in old blog posts, or leave them as a historical record? Let's argue about it</title>
      <dc:creator>linkbruno.com</dc:creator>
      <pubDate>Fri, 13 Nov 2026 08:09:21 +0000</pubDate>
      <link>https://dev.to/linkbruno/should-you-fix-dead-links-in-old-blog-posts-or-leave-them-as-a-historical-record-lets-argue-16i3</link>
      <guid>https://dev.to/linkbruno/should-you-fix-dead-links-in-old-blog-posts-or-leave-them-as-a-historical-record-lets-argue-16i3</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Scheduled broken-link and redirect checks that tell you the moment a link breaks — &lt;a href="https://linkbruno.com" rel="noopener noreferrer"&gt;linkbruno.com&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Every blog, engineering or personal, eventually hits this. You run a link checker for the first time in years and it finds 140 dead links, almost all of them in posts written before 2021. A tutorial links to a library that was renamed. A "further reading" list points to a conference talk whose video was taken down. A tweet you quoted now goes to a login wall.&lt;/p&gt;

&lt;p&gt;The obvious answer is "fix them". But some people argue, sincerely, that old posts should be left as they are: they're a record of what you thought and linked to at the time, and silently editing them is a kind of rewriting history.&lt;/p&gt;

&lt;p&gt;I don't think either camp is entirely right. Here's the case for each, and where I've ended up.&lt;/p&gt;

&lt;h2&gt;
  
  
  The case for fixing everything
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Readers don't care when you wrote it.&lt;/strong&gt; Search engines and social shares keep sending people to old posts. To a reader landing today, a dead link is simply broken, and a post full of them feels abandoned. They'll trust the rest of the advice less, and probably they should.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Dead links are a small cost of doing business.&lt;/strong&gt; If you keep a site online, you maintain it. Nobody argues that you should leave a broken image or a 500 error on an old page for historical reasons.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Some dead links are actively harmful.&lt;/strong&gt; Expired domains get bought and turned into spam or malware. A link from your 2018 post to a library's old homepage might now send your readers to a casino. Leaving that alone isn't preservation, it's negligence.&lt;/p&gt;

&lt;h2&gt;
  
  
  The case for leaving them alone
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Old posts are a record.&lt;/strong&gt; A post from 2019 describing how you set up a build pipeline is valuable partly &lt;em&gt;because&lt;/em&gt; it shows what the ecosystem looked like in 2019. Swapping in today's links quietly changes what the post said.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Replacements can be wrong.&lt;/strong&gt; A link to "the docs" in 2019 meant version 2. Pointing it to today's docs for version 7 might be more misleading than a dead link, because the reader now trusts a page that contradicts your instructions.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;It doesn't scale.&lt;/strong&gt; For a big archive, hand-fixing every link is weeks of work for posts that get a few visits a month. Some people would rather write new posts.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where I land: fix the link, keep the record
&lt;/h2&gt;

&lt;p&gt;The middle ground that works for me is to treat link fixes like errata, not edits:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Prefer an archived copy over a modern replacement.&lt;/strong&gt; If the original page is in the Wayback Machine from around the time you wrote the post, link to that snapshot. The reader sees exactly what you meant, and the record stays intact.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Replace only when the target clearly moved.&lt;/strong&gt; If a project moved from one domain to another and the same page exists, update the link. That's fixing an address, not changing the content.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Mark what you changed.&lt;/strong&gt; A short line at the end, such as "Updated in 2026: two links pointed to pages that no longer exist; they now go to archived copies", keeps you honest. Readers who care about history can see it, and everyone else gets working links.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Remove links to hijacked domains immediately.&lt;/strong&gt; No debate there. Unlink, and note it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Leave content alone.&lt;/strong&gt; Don't rewrite the advice while you're in there. If the advice is outdated, add a banner at the top pointing to a newer post rather than silently changing the old one.&lt;/li&gt;
&lt;/ol&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;If you do decide to fix them, you need to know which old URLs have died since the last pass.&lt;/strong&gt; &lt;a href="https://linkbruno.com" rel="noopener noreferrer"&gt;linkbruno.com&lt;/a&gt; rechecks the archive on a schedule and tells you when another historical link breaks.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  A rough triage for a big archive
&lt;/h2&gt;

&lt;p&gt;If the link checker hands you hundreds of results, you don't need to treat them equally:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Start with traffic.&lt;/strong&gt; Fix the posts that actually get visits first. Your analytics will show that a small number of old posts get most of the traffic.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Then by link type.&lt;/strong&gt; Links in the body of a tutorial matter more than links in a "further reading" list at the bottom.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Then by risk.&lt;/strong&gt; Domains that now redirect somewhere unrelated, or that resolve to parked pages, go to the top regardless of traffic.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Many teams also batch it: one afternoon a quarter, top 20 posts, done. That's better than a once-a-decade cleanup.&lt;/p&gt;

&lt;h2&gt;
  
  
  The parts I'm less sure about
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Quoted social posts.&lt;/strong&gt; If an embedded post is deleted, linking to an archive of it may preserve something the author chose to remove. I usually replace the embed with a short paraphrase and no link.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Your own old URLs.&lt;/strong&gt; If you changed your own slugs, the right fix is a redirect on your side, not editing every post that linked to the old one. But editing them too avoids a pointless hop.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Code examples.&lt;/strong&gt; Links to a specific line of a file on GitHub's main branch will drift. Pinning them to a commit is arguably the "historical record" option and the "working link" option at the same time.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Making the next cleanup smaller
&lt;/h2&gt;

&lt;p&gt;Whatever you decide about the archive, you can make future posts rot more slowly:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Link to stable things.&lt;/strong&gt; A project's docs landing page outlives a deep link to one heading. A release tag outlives a branch.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Archive as you publish.&lt;/strong&gt; Submitting the pages you cite to the Wayback Machine on the day you publish costs seconds, and gives you a snapshot from exactly the right date if the link dies later.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Quote the important bit.&lt;/strong&gt; If one sentence from a source is the reason for the link, quote it. The reader keeps the point even if the page goes.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Check on a schedule.&lt;/strong&gt; A monthly check of the posts that get traffic catches dead links while you still remember what they pointed to.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Whichever side you're on, the first step is knowing how many dead links you have and where they are, which is the part &lt;a href="https://linkbruno.com" rel="noopener noreferrer"&gt;linkbruno.com&lt;/a&gt; helps with.&lt;/p&gt;

&lt;h2&gt;
  
  
  Related reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/how-to-check-every-url-in-your-sitemapxml-for-404s-redirect-chains-and-timeouts-a-python-script-3dj3"&gt;How to check every URL in your sitemap.xml for 404s, redirect chains and timeouts (a Python script for CI)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/how-to-find-broken-outbound-links-and-replace-them-using-the-wayback-machine-45pl"&gt;How to find broken outbound links and replace them (using the Wayback Machine)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/should-your-docs-link-to-githubs-main-branch-or-to-a-permalink-both-options-rot-4ne6"&gt;Should your docs link to GitHub's main branch or to a permalink? Both options rot.&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/you-fixed-the-link-but-its-still-broken-how-cached-404s-and-redirects-outlive-your-fix-27o7"&gt;You fixed the link, but it's still broken: how cached 404s and redirects outlive your fix&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So where do you stand: fix silently, fix with a note, point to archives, or leave old posts untouched? And has anyone had a reader complain about either choice?&lt;/p&gt;

</description>
      <category>webdev</category>
      <category>discuss</category>
      <category>writing</category>
      <category>blogging</category>
    </item>
    <item>
      <title>How to check every URL in your sitemap.xml for 404s, redirect chains and timeouts (a Python script for CI)</title>
      <dc:creator>linkbruno.com</dc:creator>
      <pubDate>Fri, 13 Nov 2026 08:08:40 +0000</pubDate>
      <link>https://dev.to/linkbruno/how-to-check-every-url-in-your-sitemapxml-for-404s-redirect-chains-and-timeouts-a-python-script-3dj3</link>
      <guid>https://dev.to/linkbruno/how-to-check-every-url-in-your-sitemapxml-for-404s-redirect-chains-and-timeouts-a-python-script-3dj3</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Scheduled broken-link and redirect checks that tell you the moment a link breaks — &lt;a href="https://linkbruno.com" rel="noopener noreferrer"&gt;linkbruno.com&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;A sitemap is a list of URLs you're telling the world are live, canonical and worth visiting. In practice, sitemaps drift. A page gets deleted but the generator still lists it from a stale cache. A slug changes and the sitemap keeps the old one, which now redirects. A section moves behind a login. Nobody notices, because nobody reads sitemaps except crawlers.&lt;/p&gt;

&lt;p&gt;Checking a sitemap is also one of the easiest link checks you can automate. You don't need to crawl anything: the list of URLs is already there. This post builds a small Python script that reads a sitemap (including sitemap indexes), fetches every URL with sensible limits, and reports four things:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;URLs that return an error (4xx, 5xx).&lt;/li&gt;
&lt;li&gt;URLs that redirect at all (a sitemap should list final URLs).&lt;/li&gt;
&lt;li&gt;Redirect chains of more than one hop.&lt;/li&gt;
&lt;li&gt;URLs that time out or fail to connect.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;It exits non-zero when it finds problems, so you can run it in CI after a deploy.&lt;/p&gt;

&lt;h2&gt;
  
  
  Reading the sitemap
&lt;/h2&gt;

&lt;p&gt;Sitemaps come in two shapes: a &lt;code&gt;&amp;lt;urlset&amp;gt;&lt;/code&gt; with &lt;code&gt;&amp;lt;url&amp;gt;&amp;lt;loc&amp;gt;&lt;/code&gt; entries, or a &lt;code&gt;&amp;lt;sitemapindex&amp;gt;&lt;/code&gt; that points to more sitemaps. Large sites nearly always use an index. The XML namespace trips people up, so handle it explicitly:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;xml&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;etree&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;ElementTree&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="no"&gt;ET&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;gzip&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;

&lt;span class="no"&gt;NS&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"sm"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"http://www.sitemaps.org/schemas/sitemap/0.9"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;fetch_bytes&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;req&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Request&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"User-Agent"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"sitemap-check/1.0"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;urlopen&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;data&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;read&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;gzip&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;decompress&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;endswith&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;".gz"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;sitemap_urls&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;seen&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;seen&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;seen&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="kt"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;url&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;seen&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="n"&gt;seen&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;root&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="no"&gt;ET&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;fromstring&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;fetch_bytes&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;root&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;tag&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;endswith&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"sitemapindex"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;out&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;loc&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;root&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;findall&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"sm:sitemap/sm:loc"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="no"&gt;NS&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;out&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="nf"&gt;sitemap_urls&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;loc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;seen&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;out&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;loc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;loc&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;root&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;findall&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"sm:url/sm:loc"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="no"&gt;NS&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;seen&lt;/code&gt; set protects you from an index that accidentally lists itself, which happens more often than you'd think with plugins.&lt;/p&gt;

&lt;h2&gt;
  
  
  Checking each URL without following redirects blindly
&lt;/h2&gt;

&lt;p&gt;Most HTTP libraries follow redirects silently. For a sitemap check, you want to see every hop, so follow them yourself. This uses &lt;code&gt;requests&lt;/code&gt; because it makes that easy:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;check&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;max_hops&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;hops&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;current&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;url&lt;/span&gt;
    &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Session&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"User-Agent"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"sitemap-check/1.0 (+your contact URL)"&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;max_hops&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;current&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;allow_redirects&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;False&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;stream&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;close&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Timeout&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"url"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"timeout"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"hops"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;hops&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;ConnectionError&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"url"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"connect-error"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"hops"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;hops&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"detail"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;str&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;120&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;is_redirect&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status_code&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;301&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;302&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;303&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;307&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;308&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;nxt&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;compat&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;urljoin&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;current&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Location"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="n"&gt;hops&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status_code&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;nxt&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="n"&gt;current&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;nxt&lt;/span&gt;
            &lt;span class="k"&gt;continue&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"url"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status_code&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"hops"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;hops&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"final"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;current&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"url"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"too-many-redirects"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"hops"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;hops&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two details matter here. &lt;code&gt;stream=True&lt;/code&gt; plus &lt;code&gt;close()&lt;/code&gt; means you don't download a 40 MB PDF just to learn it exists. And using &lt;code&gt;GET&lt;/code&gt; rather than &lt;code&gt;HEAD&lt;/code&gt; avoids the servers that answer &lt;code&gt;HEAD&lt;/code&gt; differently, or not at all.&lt;/p&gt;

&lt;h2&gt;
  
  
  Running it in parallel, politely
&lt;/h2&gt;

&lt;p&gt;A few thousand URLs one at a time takes a while. A small thread pool is enough, and keeping it small stops your own check from looking like an attack on your own site:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;concurrent&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;futures&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;ThreadPoolExecutor&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;main&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sitemap&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;workers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;urls&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;sorted&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;sitemap_urls&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sitemap&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"{len(urls)} URLs in sitemap"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;t0&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;time&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="nc"&gt;ThreadPoolExecutor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;max_workers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;workers&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;pool&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;results&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kt"&gt;list&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pool&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;check&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;urls&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;problems&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;results&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;st&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;hops&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"hops"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;isinstance&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;st&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;st&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="mi"&gt;400&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"ERROR {st}  {r['url']}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;problems&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
        &lt;span class="k"&gt;elif&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="nb"&gt;isinstance&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;st&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"FAIL  {st}  {r['url']}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;problems&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
        &lt;span class="k"&gt;elif&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;hops&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"CHAIN {len(hops)} hops  {r['url']} -&amp;gt; {r['final']}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;problems&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
        &lt;span class="k"&gt;elif&lt;/span&gt; &lt;span class="n"&gt;hops&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"REDIR {hops[0][0]}  {r['url']} -&amp;gt; {r['final']}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;problems&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"checked {len(urls)} in {time.time()-t0:.1f}s, {problems} problems"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;problems&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;__name__&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;"__main__"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;exit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;main&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Run it with &lt;code&gt;python sitemap_check.py https://example.com/sitemap.xml&lt;/code&gt;.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;A sitemap script in CI is a great gate; URLs also fail between deploys.&lt;/strong&gt; &lt;a href="https://linkbruno.com" rel="noopener noreferrer"&gt;linkbruno.com&lt;/a&gt; keeps checking them on a schedule and flags new 404s, chains and dead destinations.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  What the results usually show
&lt;/h2&gt;

&lt;p&gt;On real sites, the first run tends to surface the same categories:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Single redirects.&lt;/strong&gt; By far the most common. &lt;code&gt;http://&lt;/code&gt; URLs in an otherwise HTTPS sitemap, missing or extra trailing slashes, and old slugs. These aren't broken for users, but a sitemap that lists redirecting URLs is telling crawlers the wrong canonical. Fix the generator, not the redirect.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Chains.&lt;/strong&gt; Usually &lt;code&gt;http://example.com/old&lt;/code&gt; to &lt;code&gt;https://example.com/old&lt;/code&gt; to &lt;code&gt;https://example.com/new/&lt;/code&gt;. Each migration added a hop. Point the first rule straight at the final URL.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;404s for deleted content.&lt;/strong&gt; The generator is reading from a cache, a search index or a table that doesn't respect the "deleted" flag.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Timeouts on a handful of heavy pages.&lt;/strong&gt; Often the slowest pages on the site, and worth knowing about regardless of the sitemap.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Putting it in CI
&lt;/h2&gt;

&lt;p&gt;Run it after deploy against production or a preview environment, not before, because you're testing what's actually served:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;sitemap-check&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;runs-on&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ubuntu-latest&lt;/span&gt;
  &lt;span class="na"&gt;needs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;deploy&lt;/span&gt;
  &lt;span class="na"&gt;steps&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;uses&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;actions/checkout@v4&lt;/span&gt;
    &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;run&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;pip install requests&lt;/span&gt;
    &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;run&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;python scripts/sitemap_check.py https://example.com/sitemap.xml&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If your sitemap is large, check a random sample on every deploy and the whole thing nightly. If single redirects are noisy at first, treat them as warnings for a week while you fix the generator, then make them fail the build.&lt;/p&gt;

&lt;h2&gt;
  
  
  Limits of this approach
&lt;/h2&gt;

&lt;p&gt;A sitemap check only sees URLs you've listed. It won't find broken links &lt;em&gt;inside&lt;/em&gt; your pages, links to pages missing from the sitemap, or broken outbound links. It's one useful slice. For the rest you still need a crawl or a dedicated tool; checking the links inside pages, and the external ones, is the part &lt;a href="https://linkbruno.com" rel="noopener noreferrer"&gt;linkbruno.com&lt;/a&gt; focuses on.&lt;/p&gt;

&lt;h2&gt;
  
  
  Related reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/redirect-chains-why-3-hops-is-2-too-many-and-a-script-that-finds-them-3lpn"&gt;Redirect chains: why 3 hops is 2 too many (and a script that finds them)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/head-vs-get-for-link-checking-head-is-faster-and-it-lies-more-than-youd-think-2fmd"&gt;HEAD vs GET for link checking: HEAD is faster, and it lies more than you'd think&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/how-to-find-broken-links-with-playwright-a-test-you-can-drop-into-ci-3d4l"&gt;How to find broken links with Playwright (a test you can drop into CI)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/should-you-fix-dead-links-in-old-blog-posts-or-leave-them-as-a-historical-record-lets-argue-16i3"&gt;Should you fix dead links in old blog posts, or leave them as a historical record? Let's argue about it&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Have you ever run a check like this against your own sitemap? What share of the URLs redirected on the first run?&lt;/p&gt;

</description>
      <category>python</category>
      <category>webdev</category>
      <category>testing</category>
      <category>tutorial</category>
    </item>
    <item>
      <title>7 tools for testing iOS Universal Links and Android App Links before your users find the bugs</title>
      <dc:creator>linkbruno.com</dc:creator>
      <pubDate>Thu, 12 Nov 2026 08:34:28 +0000</pubDate>
      <link>https://dev.to/linkbruno/7-tools-for-testing-ios-universal-links-and-android-app-links-before-your-users-find-the-bugs-k29</link>
      <guid>https://dev.to/linkbruno/7-tools-for-testing-ios-universal-links-and-android-app-links-before-your-users-find-the-bugs-k29</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Scheduled broken-link and redirect checks that tell you the moment a link breaks — &lt;a href="https://linkbruno.com" rel="noopener noreferrer"&gt;linkbruno.com&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;App links are links that should open your mobile app when it's installed and fall back to the website when it isn't. When they work, nobody notices. When they don't, users land on a web page asking them to log in again, or in the app's home screen instead of the order they tapped, or in a browser tab that loops back to the App Store.&lt;/p&gt;

&lt;p&gt;The frustrating part is that these failures are hard to see from a desktop link checker. The URL returns 200. The page exists. The link is "fine", it just doesn't do what it's supposed to.&lt;/p&gt;

&lt;p&gt;Here are seven tools for testing them properly. None are paid, and most you already have.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. &lt;code&gt;curl&lt;/code&gt;, for the association files
&lt;/h2&gt;

&lt;p&gt;Both platforms start by fetching a file from your domain:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;iOS: &lt;code&gt;https://example.com/.well-known/apple-app-site-association&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;Android: &lt;code&gt;https://example.com/.well-known/assetlinks.json&lt;/code&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Many app-link failures are simply these files being wrong: redirected, served as HTML, blocked by a WAF, or missing on one hostname.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl -sS -D - -o /dev/null https://example.com/.well-known/apple-app-site-association
curl -sS https://example.com/.well-known/assetlinks.json &lt;span class="o"&gt;|&lt;/span&gt; python3 -m json.tool
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Check for:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A &lt;strong&gt;200&lt;/strong&gt; status, directly, without redirects.&lt;/li&gt;
&lt;li&gt;A JSON content type.&lt;/li&gt;
&lt;li&gt;Valid JSON (the second command will complain if not).&lt;/li&gt;
&lt;li&gt;The same result on &lt;strong&gt;every&lt;/strong&gt; hostname you use in links, including &lt;code&gt;www&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  2. Apple's CDN copy of your association file
&lt;/h2&gt;

&lt;p&gt;iOS devices typically don't fetch your association file directly; Apple's CDN fetches and caches it. So what &lt;em&gt;you&lt;/em&gt; serve and what devices &lt;em&gt;see&lt;/em&gt; can differ for a while after a change.&lt;/p&gt;

&lt;p&gt;You can request the cached copy from Apple's CDN endpoint (the path format is &lt;code&gt;https://app-site-association.cdn-apple.com/a/v1/&amp;lt;your-domain&amp;gt;&lt;/code&gt;). If it differs from your live file, devices are working from the old version. This explains a lot of "I fixed it but it still doesn't work" moments.&lt;/p&gt;

&lt;p&gt;For development, Apple supports a developer mode for associated domains that bypasses the CDN on test devices; check Apple's current documentation for how to enable it in your entitlements.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. &lt;code&gt;xcrun simctl openurl&lt;/code&gt;, for the iOS Simulator
&lt;/h2&gt;

&lt;p&gt;To test how the app handles a specific link without tapping through Notes or Messages:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;xcrun simctl openurl booted &lt;span class="s2"&gt;"https://example.com/orders/123"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This is quick for checking that your routing code sends each URL pattern to the right screen. Note that the Simulator isn't a perfect stand-in for association behaviour on real devices, so use it for routing tests and confirm association on hardware.&lt;/p&gt;

&lt;h2&gt;
  
  
  4. &lt;code&gt;adb&lt;/code&gt;, for Android devices and emulators
&lt;/h2&gt;

&lt;p&gt;Android gives you more visibility. To open a link:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;adb shell am start -a android.intent.action.VIEW \
  -c android.intent.category.BROWSABLE \
  -d &lt;span class="s2"&gt;"https://example.com/orders/123"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And to see whether Android has actually verified your domains for the app:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;adb shell pm get-app-links com.example.app
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The output shows each domain and its verification state. If a domain isn't verified, Android will treat your links as ordinary web links (or show a chooser), regardless of how good your routing code is. On recent Android versions you can also ask the system to re-run verification after fixing your &lt;code&gt;assetlinks.json&lt;/code&gt;; see the developer docs for the exact &lt;code&gt;pm verify-app-links&lt;/code&gt; options for your target version.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;App-link configuration is one test; the https URLs behind it still need to stay alive.&lt;/strong&gt; &lt;a href="https://linkbruno.com" rel="noopener noreferrer"&gt;linkbruno.com&lt;/a&gt; watches those links and tells you if a destination starts to 404 or redirect somewhere unexpected.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  5. Google's Digital Asset Links tools
&lt;/h2&gt;

&lt;p&gt;Google provides a web-based Statement List Generator and Tester for Digital Asset Links. Give it your domain, package name and signing certificate fingerprint, and it checks whether your &lt;code&gt;assetlinks.json&lt;/code&gt; grants the association.&lt;/p&gt;

&lt;p&gt;The most common failure it catches: the &lt;strong&gt;certificate fingerprint&lt;/strong&gt;. If you use Play App Signing, the fingerprint in the file must match the key Google uses to sign the app distributed through the Play Store, not just your local upload key. Debug builds signed with a debug key need their own entry if you want to test them.&lt;/p&gt;

&lt;h2&gt;
  
  
  6. Android Studio's App Links Assistant
&lt;/h2&gt;

&lt;p&gt;If your team uses Android Studio, the App Links Assistant (under the Tools menu) helps you add intent filters, generate the association file and test URLs against your manifest. It's especially useful for checking that the &lt;code&gt;pathPattern&lt;/code&gt; or &lt;code&gt;pathPrefix&lt;/code&gt; rules in your intent filters match the URLs you actually publish, because a link outside those patterns just opens the browser.&lt;/p&gt;

&lt;h2&gt;
  
  
  7. A plain list of real links, tested by hand
&lt;/h2&gt;

&lt;p&gt;The least glamorous tool is the most useful: a list of real URLs from real places your links appear (marketing emails, push notifications, QR codes, support macros, social posts, SMS) and a habit of tapping each one on a real iOS and Android device before a release.&lt;/p&gt;

&lt;p&gt;Two things this catches that the tools above miss:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Tracking wrappers.&lt;/strong&gt; Email platforms often rewrite links through a click-tracking domain. The user taps a link on &lt;code&gt;click.mailer.example.net&lt;/code&gt;, which redirects to your domain. Depending on platform behaviour, that redirect can stop the app from opening. Test links from actual sent emails, not from your source templates.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;In-app browsers.&lt;/strong&gt; Links tapped inside some social and messaging apps open in their own browser views, which may not hand off to your app the way the system browser does.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  A short release checklist
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;[ ] Association files return 200, JSON, no redirects, on every hostname.&lt;/li&gt;
&lt;li&gt;[ ] Apple's CDN copy matches the live file.&lt;/li&gt;
&lt;li&gt;[ ] &lt;code&gt;pm get-app-links&lt;/code&gt; shows domains as verified on a test device.&lt;/li&gt;
&lt;li&gt;[ ] Each URL pattern routes to the right screen, including logged-out states.&lt;/li&gt;
&lt;li&gt;[ ] Links from real emails, QR codes and notifications open the app.&lt;/li&gt;
&lt;li&gt;[ ] Fallback to the website works when the app isn't installed.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Regular checks that the web side of these links still resolves, including the association files, are part of what &lt;a href="https://linkbruno.com" rel="noopener noreferrer"&gt;linkbruno.com&lt;/a&gt; handles.&lt;/p&gt;

&lt;h2&gt;
  
  
  Related reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/experiment-what-share-of-the-links-in-your-github-readmes-still-work-measure-every-repo-you-own-kl2"&gt;Experiment: what share of the links in your GitHub READMEs still work? Measure every repo you own in 15 minutes&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/0-broken-links-found-is-a-myth-7-places-link-checkers-dont-look-4g45"&gt;'0 broken links found' is a myth: 7 places link checkers don't look&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/every-url-in-your-error-messages-is-a-promise-a-10-point-checklist-for-links-baked-into-software-47p0"&gt;Every URL in your error messages is a promise: a 10-point checklist for links baked into software&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/quick-tip-your-link-checker-probably-skips-mailto-and-tel-links-check-them-yourself-in-40-lines-4bjg"&gt;Quick tip: your link checker probably skips mailto: and tel: links. Check them yourself in 40 lines&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Which app-link failure has cost you the most debugging time: association files, signing fingerprints, or tracking redirects?&lt;/p&gt;

</description>
      <category>mobile</category>
      <category>ios</category>
      <category>android</category>
      <category>webdev</category>
    </item>
    <item>
      <title>Experiment: what share of the links in your GitHub READMEs still work? Measure every repo you own in 15 minutes</title>
      <dc:creator>linkbruno.com</dc:creator>
      <pubDate>Thu, 12 Nov 2026 08:33:13 +0000</pubDate>
      <link>https://dev.to/linkbruno/experiment-what-share-of-the-links-in-your-github-readmes-still-work-measure-every-repo-you-own-kl2</link>
      <guid>https://dev.to/linkbruno/experiment-what-share-of-the-links-in-your-github-readmes-still-work-measure-every-repo-you-own-kl2</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Scheduled broken-link and redirect checks that tell you the moment a link breaks — &lt;a href="https://linkbruno.com" rel="noopener noreferrer"&gt;linkbruno.com&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;A README gets written once, usually the week a project is created, and then it sits there. The project evolves; the README's links don't. Badges point at retired CI services. "See the docs" links go to a docs site that moved. The demo link is a free hosting app that was deleted years ago.&lt;/p&gt;

&lt;p&gt;I don't know what share of your README links are broken, and I'm not going to guess. Instead, here's an experiment that measures it for every public repo you own, in about fifteen minutes. You'll end up with a per-repo table, and probably a few surprises.&lt;/p&gt;

&lt;h2&gt;
  
  
  What you need
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;The GitHub CLI (&lt;code&gt;gh&lt;/code&gt;), logged in to your account&lt;/li&gt;
&lt;li&gt;Python 3.10+ with &lt;code&gt;requests&lt;/code&gt; (&lt;code&gt;pip install requests&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;A cup of coffee while it runs&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Step 1: list your repos and fetch their READMEs
&lt;/h2&gt;

&lt;p&gt;The GitHub CLI can list repos and call the API. The README endpoint returns the file's content base64-encoded:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;gh repo list YOUR_USERNAME --limit 200 --no-archived --source \
  --json nameWithOwner -q &lt;span class="s1"&gt;'.[].nameWithOwner'&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; repos.txt
wc -l repos.txt
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;--source&lt;/code&gt; skips forks, so you only measure READMEs you wrote. Remove it if you want forks included. Then fetch each README:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;mkdir -p readmes
&lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="nb"&gt;read&lt;/span&gt; -r repo&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
  &lt;span class="nv"&gt;safe&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;${repo//\//__}&lt;/span&gt;
  gh api &lt;span class="s2"&gt;"repos/$repo/readme"&lt;/span&gt; -q .content 2&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null &lt;span class="o"&gt;|&lt;/span&gt; base64 -d &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="s2"&gt;"readmes/$safe.md"&lt;/span&gt; \
    &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"no README: $repo"&lt;/span&gt;
&lt;span class="k"&gt;done&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; repos.txt
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Step 2: extract the links
&lt;/h2&gt;

&lt;p&gt;READMEs mix Markdown links, bare URLs, image badges and HTML. A pragmatic regex catches the vast majority:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;pathlib&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;

&lt;span class="no"&gt;URL&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;compile&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="s1"&gt;'https?://[^\s&amp;lt;&amp;gt;()"\'\]]+'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;links&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;found&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kt"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="no"&gt;URL&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;finditer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;url&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;group&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;rstrip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'.,;:!?*_`'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;found&lt;/span&gt;

&lt;span class="n"&gt;data&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;pathlib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Path&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"readmes"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;glob&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"*.md"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;stem&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"__"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"/"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;sorted&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;links&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;read_text&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;errors&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"ignore"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dump&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"links.json"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"w"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;indent&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;values&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"links in"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"READMEs"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This deliberately ignores relative links (like &lt;code&gt;docs/setup.md&lt;/code&gt;), which point inside the repo. Checking those is a separate exercise.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 3: check them, politely
&lt;/h2&gt;

&lt;p&gt;We'll use HEAD first and fall back to GET, because some servers handle HEAD badly. We'll also limit concurrency per host so we don't hammer anyone.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;concurrent&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;futures&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;cf&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;parse&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;urlsplit&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;collections&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;defaultdict&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;threading&lt;/span&gt;

&lt;span class="n"&gt;data&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"links.json"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;session&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Session&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;session&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"User-Agent"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"readme-link-experiment/1.0"&lt;/span&gt;
&lt;span class="n"&gt;host_locks&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;defaultdict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;threading&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Lock&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;check&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;host&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;urlsplit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;netloc&lt;/span&gt;
    &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;host_locks&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;host&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;            &lt;span class="c1"&gt;# one request at a time per host&lt;/span&gt;
        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;method&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"HEAD"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"GET"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;session&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;request&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;method&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;15&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;allow_redirects&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
                &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status_code&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;400&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="n"&gt;method&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;"GET"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status_code&lt;/span&gt;
            &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;RequestException&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;method&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;"GET"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kd"&gt;type&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;__name__&lt;/span&gt;

&lt;span class="n"&gt;unique&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;sorted&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;u&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;urls&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;values&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;u&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;urls&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;cf&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;ThreadPoolExecutor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;max_workers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;16&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;pool&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;results&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kt"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pool&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;check&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;unique&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dump&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;results&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"results.json"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"w"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;indent&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Checking unique URLs once (rather than once per repo) matters: badge URLs from the same service often appear in dozens of READMEs.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;A 15-minute audit is a snapshot; README links also die the week after you measure them.&lt;/strong&gt; &lt;a href="https://linkbruno.com" rel="noopener noreferrer"&gt;linkbruno.com&lt;/a&gt; keeps checking those URLs and alerts you when one of them breaks.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Step 4: score each repo
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;
&lt;span class="n"&gt;data&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"links.json"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;results&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"results.json"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;broken&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="nb"&gt;isinstance&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;404&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;410&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="mi"&gt;500&lt;/span&gt;

&lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;repo&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;urls&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;items&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;urls&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;continue&lt;/span&gt;
    &lt;span class="n"&gt;bad&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;u&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;u&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;urls&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nf"&gt;broken&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;results&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;u&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;bad&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;urls&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;bad&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;urls&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;repo&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;bad&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;share&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;nbad&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;total&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;repo&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;bad&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;sorted&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;reverse&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"{share:6.0%}  {nbad:3}/{total:&amp;lt;3}  {repo}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Notice what's &lt;em&gt;not&lt;/em&gt; counted as broken: 401, 403 and 429. Those usually mean "this site doesn't want to talk to a script", not "this page is gone". Count them separately as "couldn't verify", or you'll overstate the problem. Treat connection errors with some caution too, and re-run anything suspicious once before concluding it's dead.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 5: look for patterns
&lt;/h2&gt;

&lt;p&gt;Once you have the table, the interesting part is &lt;em&gt;why&lt;/em&gt; links broke. Group the broken URLs by host:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;collections&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;Counter&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;parse&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;urlsplit&lt;/span&gt;
&lt;span class="n"&gt;bad_hosts&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Counter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;urlsplit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;u&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;netloc&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;u&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;results&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;items&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nf"&gt;broken&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;host&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;bad_hosts&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;most_common&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;15&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;host&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Typical patterns worth looking for:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Retired badge and CI services.&lt;/strong&gt; A single dead service can break badges across every repo you own.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Free app hosts.&lt;/strong&gt; Demo links to free tiers that were later discontinued or put to sleep.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Your own old domains.&lt;/strong&gt; Personal sites and project domains that lapsed.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Moved documentation.&lt;/strong&gt; A docs site that changed its URL structure.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Branch names.&lt;/strong&gt; Links to &lt;code&gt;/blob/master/...&lt;/code&gt; in repos that renamed their default branch. GitHub redirects many of these, but not every pattern, so check yours.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Interpreting your number
&lt;/h2&gt;

&lt;p&gt;Some questions to ask about your results:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Is rot concentrated in old repos?&lt;/strong&gt; Plot broken share against repo creation date (available from &lt;code&gt;gh repo list --json createdAt&lt;/code&gt;). If older READMEs are worse, that's link rot. If a new repo is bad, that's a copy-paste template problem.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Is it one host or many?&lt;/strong&gt; One dead host means a quick find-and-replace across repos. Many hosts means ongoing rot that needs a recurring check.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Do the broken links matter?&lt;/strong&gt; A dead badge is cosmetic. A dead "Getting started" link loses users.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Fixing it efficiently
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Fix the top hosts first; they give the biggest win per edit.&lt;/li&gt;
&lt;li&gt;Remove badges for services you no longer use.&lt;/li&gt;
&lt;li&gt;Replace links to moved docs with their new locations, or link to an archived copy if the content is truly gone.&lt;/li&gt;
&lt;li&gt;Consider adding a scheduled link check workflow to repos that matter, so the next rot gets caught early.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;If you'd rather not maintain the scripts, &lt;a href="https://linkbruno.com" rel="noopener noreferrer"&gt;linkbruno.com&lt;/a&gt; checks links on a schedule and tells you when one stops working.&lt;/p&gt;

&lt;h2&gt;
  
  
  Related reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/check-links-in-markdown-docs-with-a-60-line-python-script-4hg6"&gt;Check links in Markdown docs with a 60-line Python script&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/should-your-docs-link-to-githubs-main-branch-or-to-a-permalink-both-options-rot-4ne6"&gt;Should your docs link to GitHub's main branch or to a permalink? Both options rot.&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/markdown-links-in-a-monorepo-7-things-id-do-differently-after-the-third-big-folder-move-20dp"&gt;Markdown links in a monorepo: 7 things I'd do differently after the third big folder move&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/7-tools-for-testing-ios-universal-links-and-android-app-links-before-your-users-find-the-bugs-k29"&gt;7 tools for testing iOS Universal Links and Android App Links before your users find the bugs&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/how-to-check-every-link-in-your-npm-package-before-you-publish-readme-homepage-repository-and-h28"&gt;How to check every link in your npm package before you publish (README, homepage, repository and bugs)&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;What share of your README links turned out to be broken, and which host was the worst offender?&lt;/p&gt;

</description>
      <category>github</category>
      <category>opensource</category>
      <category>python</category>
      <category>documentation</category>
    </item>
    <item>
      <title>You fixed the link, but it's still broken: how cached 404s and redirects outlive your fix</title>
      <dc:creator>linkbruno.com</dc:creator>
      <pubDate>Wed, 11 Nov 2026 15:51:07 +0000</pubDate>
      <link>https://dev.to/linkbruno/you-fixed-the-link-but-its-still-broken-how-cached-404s-and-redirects-outlive-your-fix-27o7</link>
      <guid>https://dev.to/linkbruno/you-fixed-the-link-but-its-still-broken-how-cached-404s-and-redirects-outlive-your-fix-27o7</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Scheduled broken-link and redirect checks that tell you the moment a link breaks — &lt;a href="https://linkbruno.com" rel="noopener noreferrer"&gt;linkbruno.com&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;You get a report that a link is broken. You find the cause, restore the page or add a redirect, test it yourself and it works. An hour later, another report: still broken.&lt;/p&gt;

&lt;p&gt;You're not imagining it, and they're not either. HTTP responses get cached in several places between your server and the reader, and errors can be cached just like successful pages. So can redirects, sometimes for much longer than you'd expect.&lt;/p&gt;

&lt;p&gt;This post walks through each layer that can remember a broken response, why it does, and how to work out which one is doing it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Layer 1: the browser cache
&lt;/h2&gt;

&lt;p&gt;Browsers cache responses according to HTTP caching rules (defined in RFC 9111). The headers that matter most are &lt;code&gt;Cache-Control&lt;/code&gt; and, for older setups, &lt;code&gt;Expires&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;A 404 response is "cacheable by default" under those rules, meaning a cache may store it and reuse it using heuristics even if there are no explicit caching headers. In practice, browsers tend to be conservative with error responses, but if your server sends something like &lt;code&gt;Cache-Control: public, max-age=86400&lt;/code&gt; on &lt;em&gt;every&lt;/em&gt; response, including 404s (a common result of a blanket config rule), the browser can legitimately keep showing the 404 for a day.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How to tell:&lt;/strong&gt; ask the reader to try a private window, or test with &lt;code&gt;curl&lt;/code&gt;, which has no cache. If curl sees the fixed page and the browser doesn't, it's local.&lt;/p&gt;

&lt;h2&gt;
  
  
  Layer 2: cached redirects (the long-lived one)
&lt;/h2&gt;

&lt;p&gt;Permanent redirects are the most surprising case. A 301 or 308 is cacheable by default, and browsers often cache them aggressively. If you once had a mistaken 301 from &lt;code&gt;/pricing&lt;/code&gt; to &lt;code&gt;/plans&lt;/code&gt; and later remove it, a browser that saw the 301 may keep jumping straight to &lt;code&gt;/plans&lt;/code&gt; without ever asking your server about &lt;code&gt;/pricing&lt;/code&gt; again, potentially for a long time.&lt;/p&gt;

&lt;p&gt;That's why "temporarily" using a 301 during a migration can be painful to undo. If you're unsure whether a redirect is permanent, a 302 or 307 is easier to take back.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How to tell:&lt;/strong&gt; open the browser's network panel with "Disable cache" ticked and compare. A redirect served "from disk cache" or "from memory cache" is the giveaway.&lt;/p&gt;

&lt;h2&gt;
  
  
  Layer 3: the CDN or reverse proxy
&lt;/h2&gt;

&lt;p&gt;If your site sits behind a CDN or a caching proxy (Varnish, nginx &lt;code&gt;proxy_cache&lt;/code&gt;, a platform's edge network), that layer may have cached the 404. Many CDNs have a separate setting for how long to cache error responses, sometimes called "negative caching" or error caching, and the default varies between providers and plans. Some cache 404s for a short time by default to protect your origin from floods of requests for missing URLs.&lt;/p&gt;

&lt;p&gt;This is the most common cause of "fixed but still broken for everyone": your origin is now correct, but the edge keeps serving the stored error until its TTL expires.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How to tell:&lt;/strong&gt; look at the response headers. Most CDNs add headers that reveal cache status, often something like &lt;code&gt;Age&lt;/code&gt;, &lt;code&gt;X-Cache: HIT&lt;/code&gt; or a provider-specific header. An &lt;code&gt;Age&lt;/code&gt; value larger than the time since your fix is proof you're seeing an old copy.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl -sI https://example.com/docs/install &lt;span class="o"&gt;|&lt;/span&gt; grep -iE &lt;span class="s1"&gt;'^(HTTP|age|cache-control|x-cache|cf-cache-status)'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;How to fix:&lt;/strong&gt; purge that URL from the CDN, and review your error caching settings.&lt;/p&gt;

&lt;h2&gt;
  
  
  Layer 4: the application's own cache
&lt;/h2&gt;

&lt;p&gt;Frameworks and CMSs often have their own page caches: static HTML generated at build time, a full-page cache plugin, an in-memory route cache. If the 404 was generated and cached at that level, fixing the content in the database won't change what's served until the cache is rebuilt.&lt;/p&gt;

&lt;p&gt;Static site generators are a special case: if the page was missing at build time, it stays missing until the next build and deploy, however many times you fix the source.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How to tell:&lt;/strong&gt; request the origin directly (bypassing the CDN, for example via an internal hostname or &lt;code&gt;curl --resolve&lt;/code&gt;) and see whether the origin itself still returns 404.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;After you fix a link, you still want a later check that the live URL actually recovered.&lt;/strong&gt; &lt;a href="https://linkbruno.com" rel="noopener noreferrer"&gt;linkbruno.com&lt;/a&gt; rechecks it on a schedule, so a cached 404 or a leftover redirect doesn't hide that the fix didn't stick.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Layer 5: DNS (when the link points at a hostname that changed)
&lt;/h2&gt;

&lt;p&gt;If the broken link was caused by a hostname problem (a missing DNS record for &lt;code&gt;docs.example.com&lt;/code&gt;, say) then DNS caching applies. Resolvers cache answers for the record's TTL, and many also cache "this name doesn't exist" responses for a period derived from the zone's settings. A newly added record may not be visible to everyone until those cached negative answers expire.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How to tell:&lt;/strong&gt; query a few public resolvers directly with &lt;code&gt;dig&lt;/code&gt; and compare.&lt;/p&gt;

&lt;h2&gt;
  
  
  Layer 6: search engines and link previews
&lt;/h2&gt;

&lt;p&gt;Finally, things outside your control. A search engine that crawled the broken URL may show it as an error or drop it from results until it's recrawled. Chat apps and social platforms cache link previews, sometimes for a long time, so a shared link may keep showing an error card even after the page is back. Many platforms offer a tool to refresh a preview.&lt;/p&gt;

&lt;h2&gt;
  
  
  A decision tree for "still broken"
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Does &lt;code&gt;curl&lt;/code&gt; against the public URL return the fixed response?&lt;/strong&gt;

&lt;ul&gt;
&lt;li&gt;No → the problem is server-side: CDN, app cache or origin. Check &lt;code&gt;Age&lt;/code&gt; and cache headers, then the origin directly.&lt;/li&gt;
&lt;li&gt;Yes → the problem is client-side or downstream: browser cache, cached redirect, DNS on their network, or a cached preview.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Is it a redirect?&lt;/strong&gt; Suspect browser-cached 301s first.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Is it a hostname?&lt;/strong&gt; Suspect negative DNS caching.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Prevention
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Don't send long &lt;code&gt;max-age&lt;/code&gt; on error responses.&lt;/strong&gt; Set short or no caching for 4xx and 5xx explicitly.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Use temporary redirects until you're sure.&lt;/strong&gt; Switch to 301 or 308 once a redirect is definitely permanent.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Know your CDN's error caching settings&lt;/strong&gt; and how to purge a single URL quickly.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Check links against the public URL&lt;/strong&gt;, the way readers see it, not just against your origin or source files.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That last habit is what &lt;a href="https://linkbruno.com" rel="noopener noreferrer"&gt;linkbruno.com&lt;/a&gt; does: it checks the links on your pages from the outside, through all the same caches your readers go through.&lt;/p&gt;

&lt;h2&gt;
  
  
  Related reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/should-your-link-checker-obey-robotstxt-lets-argue-about-it-1ldo"&gt;Should your link checker obey robots.txt? Let's argue about it&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/postmortem-the-deploy-deleted-last-weeks-javascript-bundles-and-cached-pages-kept-asking-for-them-393b"&gt;Postmortem: the deploy deleted last week's JavaScript bundles, and cached pages kept asking for them&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/your-redirects-file-should-be-append-only-deleting-a-redirect-is-a-breaking-change-6lh"&gt;Your redirects file should be append-only. Deleting a redirect is a breaking change.&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/redirect-chains-why-3-hops-is-2-too-many-and-a-script-that-finds-them-3lpn"&gt;Redirect chains: why 3 hops is 2 too many (and a script that finds them)&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Which layer has kept a broken link alive the longest for you: the browser, the CDN, or something stranger?&lt;/p&gt;

</description>
      <category>webdev</category>
      <category>http</category>
      <category>caching</category>
      <category>cdn</category>
    </item>
    <item>
      <title>Should your link checker obey robots.txt? Let's argue about it</title>
      <dc:creator>linkbruno.com</dc:creator>
      <pubDate>Wed, 11 Nov 2026 15:46:43 +0000</pubDate>
      <link>https://dev.to/linkbruno/should-your-link-checker-obey-robotstxt-lets-argue-about-it-1ldo</link>
      <guid>https://dev.to/linkbruno/should-your-link-checker-obey-robotstxt-lets-argue-about-it-1ldo</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Scheduled broken-link and redirect checks that tell you the moment a link breaks — &lt;a href="https://linkbruno.com" rel="noopener noreferrer"&gt;linkbruno.com&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Every link checker eventually has to answer a question its authors probably didn't expect: when a site's robots.txt says "please don't fetch this path", and your page links to that path, should the checker fetch it anyway to see if the link works?&lt;/p&gt;

&lt;p&gt;It sounds like a niche config flag. In practice, it decides whether your tool reports some links as broken, some as "skipped", or quietly doesn't check them at all. And people have strong feelings about it.&lt;/p&gt;

&lt;p&gt;Here are both sides, as fairly as I can put them, and then where I land.&lt;/p&gt;

&lt;h2&gt;
  
  
  A quick refresher on what robots.txt actually is
&lt;/h2&gt;

&lt;p&gt;robots.txt is a plain text file at the root of a site that tells automated clients which paths the site owner would rather they didn't fetch. It's a convention that was eventually written up as an internet standard (the Robots Exclusion Protocol, RFC 9309). Two things are worth remembering:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;It's voluntary.&lt;/strong&gt; Nothing technically stops a client from ignoring it. Well-behaved bots follow it; badly behaved ones don't.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;It's about crawling, not access.&lt;/strong&gt; It doesn't password-protect anything. A disallowed URL is still public. The file says "don't crawl", not "don't look".&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The standard talks about "crawlers", meaning automated clients that discover and fetch URLs. Whether a link checker counts is exactly the argument.&lt;/p&gt;

&lt;h2&gt;
  
  
  The case for obeying robots.txt
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;"It's an automated client. Automated clients follow robots.txt. End of story."&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The purist position is simple: your link checker is software making requests without a human clicking anything. That's the definition of a bot. If you're writing a bot, you follow the rules bots follow.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;"Site owners use it to protect fragile endpoints."&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Some paths are disallowed for good reasons: search pages that are expensive to render, calendar pages that generate infinite URLs, admin panels, faceted filters. A link checker that fans out across hundreds of documentation pages and hits those paths on every CI run is adding real load to someone else's server, every time you push.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;"If you ignore it, you'll get blocked, and then your results are worse anyway."&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Sites that notice bots ignoring robots.txt tend to respond with rate limits, 403s or outright IP blocks. If your link checker runs from a shared CI provider, you may be getting the whole IP range blocked for everyone. Politeness isn't just ethics; it's self-interest.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;"It's the norm in the ecosystem."&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Several well-known crawlers and SEO tools respect robots.txt by default. Following the same norm makes your tool predictable.&lt;/p&gt;

&lt;h2&gt;
  
  
  The case for ignoring it (for links you already have)
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;"A link checker isn't discovering URLs. It's verifying ones a human chose."&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;This is the strongest argument on the other side. A crawler starts somewhere and follows everything it finds, which is exactly what robots.txt was designed to manage. A link checker, at least for external links, does something narrower: it requests specific URLs that an author deliberately put on a page, usually once. That's closer to a reader clicking the link than to a crawler indexing a site.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;"robots.txt doesn't say whether a page exists."&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;If you skip disallowed URLs, you can't tell your users whether those links work. And plenty of important pages live behind disallow rules: login pages, search result links, account pages, product filter URLs, even some documentation sections that sites didn't want indexed. "Skipped" is not the same as "fine", but it often gets read that way.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;"Browsers don't read robots.txt, and link previews mostly don't either."&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;When someone pastes a link into a chat app, the app fetches the page to build a preview. That's automated too. The web already accepts that "fetching a specific URL on behalf of a person" is a different category from crawling.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;"The load is tiny."&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;One HEAD or GET per external link per run, with caching, is negligible for most sites. If you're checking a link to someone's search page once a day, that's not what robots.txt is defending against.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Whether you obey robots.txt or not, the links you do check still need a recurring run.&lt;/strong&gt; &lt;a href="https://linkbruno.com" rel="noopener noreferrer"&gt;linkbruno.com&lt;/a&gt; schedules those checks and tells you the moment a URL starts to fail.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  The awkward middle cases
&lt;/h2&gt;

&lt;p&gt;The argument gets messier when you look at what link checkers actually do:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Internal links on your own site.&lt;/strong&gt; If you're checking your own pages, your own robots.txt is about &lt;em&gt;search engines&lt;/em&gt;, not about you. Ignoring it for your own domain seems uncontroversial.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Recursive checking.&lt;/strong&gt; Some link checkers follow links found on the pages they check, sometimes across domains. At that point it really is crawling, and the "it's just one URL" defence evaporates.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Scale.&lt;/strong&gt; A docs site with 5,000 pages that links to the same third-party API reference 5,000 times, rechecked on every pull request, can generate meaningful traffic if the tool doesn't deduplicate and cache.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;User agent honesty.&lt;/strong&gt; Some people sidestep the debate by pretending to be a browser. That's the one option I think is clearly wrong: if you're going to fetch something a site asked bots not to fetch, at least say who you are.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Where I land
&lt;/h2&gt;

&lt;p&gt;I think the right default is a split:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Your own domain: ignore robots.txt.&lt;/strong&gt; You own both the links and the rules. Check everything.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;External links you've explicitly written: check them, once, politely&lt;/strong&gt;, even if robots.txt disallows the path. Identify your tool honestly in the User-Agent, deduplicate URLs across the whole run, cache results for a sensible period, and respect rate limits and Retry-After headers.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Anything that looks like crawling (following links you didn't write, recursing into external sites): obey robots.txt&lt;/strong&gt;, no exceptions.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Report skipped links as skipped&lt;/strong&gt;, never as passing. If a user configures the tool to obey robots.txt everywhere, the report should say clearly how many links were not verified and why.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Make it a visible setting&lt;/strong&gt;, not a hidden behaviour, so teams can pick the stricter option if their policy requires it.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The principle underneath is the one robots.txt itself is built on: it's about managing automated &lt;em&gt;discovery&lt;/em&gt; and &lt;em&gt;load&lt;/em&gt;. A tool that verifies a short list of author-chosen URLs, honestly and gently, is doing something different from a crawler. A tool that wanders, hammers or hides its identity is not.&lt;/p&gt;

&lt;p&gt;That's roughly how we think about checks at &lt;a href="https://linkbruno.com" rel="noopener noreferrer"&gt;linkbruno.com&lt;/a&gt;, but I'm genuinely not sure the line is in exactly the right place, and I know some people feel strongly that any automated request should obey the file, full stop.&lt;/p&gt;

&lt;h2&gt;
  
  
  A quick way to see how this affects you
&lt;/h2&gt;

&lt;p&gt;If you want to know whether this debate matters for your site, it's easy to measure:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Export all the unique external URLs from your site or docs.&lt;/li&gt;
&lt;li&gt;For each domain, fetch its robots.txt once.&lt;/li&gt;
&lt;li&gt;Count how many of your links fall under a Disallow rule for a generic user agent.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;On most content sites the number is small. On docs sites that link heavily to search pages, dashboards or app routes, it can be surprisingly large, and those are exactly the links that break when products change.&lt;/p&gt;

&lt;h2&gt;
  
  
  Related reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/429-403-999-a-link-checkers-faq-for-sites-that-dont-want-to-talk-to-you-1kjc"&gt;429, 403, 999: a link checker's FAQ for sites that don't want to talk to you&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/linkedin-returns-999-cloudflare-returns-403-why-your-link-checker-reports-links-that-arent-broken-48mm"&gt;LinkedIn returns 999, Cloudflare returns 403: why your link checker reports links that aren't broken&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/0-broken-links-found-is-a-myth-7-places-link-checkers-dont-look-4g45"&gt;'0 broken links found' is a myth: 7 places link checkers don't look&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/linkbruno/you-fixed-the-link-but-its-still-broken-how-cached-404s-and-redirects-outlive-your-fix-27o7"&gt;You fixed the link, but it's still broken: how cached 404s and redirects outlive your fix&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So where do you stand: is a link checker a crawler that must obey robots.txt, or a reader's stand-in that may politely check the links it was given?&lt;/p&gt;

</description>
      <category>webdev</category>
      <category>crawling</category>
      <category>robots</category>
      <category>discuss</category>
    </item>
  </channel>
</rss>
