I used to have a really bad habit. After deploying a major update to a client's site or migrating a blog to a new CMS, I would click through a few pages, assume everything worked, and close my laptop.
Then, a week later, Google Search Console would light up with 404 errors, or a user would email me: "Hey, the link to your pricing page is broken."
Broken links are a silent killer. They destroy your SEO, frustrate users, and make your site look unmaintained. But here's the thing: Manually checking links is a waste of your time as a developer.
Here is how I stopped clicking links manually and started automating the 404 hunting process.
The Problem with "Simple" Link Checkers
If you Google "broken link checker," you'll find hundreds of tools. But most of them have fatal flaws for developers:
They require installing a massive desktop app (looking at you, Screaming Frog). It's great for deep audits, but I don't want to launch a heavy Java app just to check 20 links.
They are filled with ads and require sign-ups. I don't want to hand over my email address just to check an HTTP status code.
The free tier limits you to 5 URLs.
Approach 1: The cURL Bash Script (For Server-Side Checking)
If you have terminal access, you can write a quick Bash script to check a list of URLs using curl. Here is a simple script that reads a text file of URLs (urls.txt) and spits out the HTTP status code for each:
bash
!/bin/bash
Read from urls.txt
while read -r url; do
# -o /dev/null discards the HTML body
# -s silences the progress bar
# -w writes out the HTTP status code
# -I fetches only the headers
status=$(curl -o /dev/null -s -w "%{http_code}" -I "$url")
if [ "$status" -ne 200 ]; then
echo "❌ [$status] $url"
else
echo "✅ [200] $url"
fi
done < urls.txt
This is fast and doesn't require any third-party API. However, if you are checking URLs that are behind Cloudflare or have strict WAF rules, curl might get blocked (returning a 403 instead of the real status).
Approach 2: The Client-Side Browser Check (For Quick Audits)
Sometimes I just want to check a handful of links on the fly without opening a terminal. But doing this in the browser has a massive hurdle: CORS (Cross-Origin Resource Sharing). Browsers block JavaScript from making arbitrary HTTP requests to external domains.
To bypass this, you can use a public CORS proxy like allorigins.win to fetch the URL and return the status code via frontend JS:
javascript
async function checkLink(url) {
const proxyUrl = https://api.allorigins.win/get?url=${encodeURIComponent(url)};
try {
const response = await fetch(proxyUrl);
const data = await response.json();
const status = data.status?.http_code || 0;
return status;
} catch (err) {
return 'Error/Timeout';
}
}
Approach 3: Just Use a Fast, No-Signup Tool
Writing scripts is fun, but when I'm in the middle of a deployment, I just want a clean interface to do a quick check.
I got tired of navigating through ad-ridden sites, so I actually built a couple of lightweight tools for my own workflow and put them on Site Tools Box (a directory of tools with absolutely no sign-up required).
For checking a single page's assets (CSS, JS, images), I use their Dead Link Checker. (Link: https://sitetoolsbox.com/tool.php?slug=http_status)
For checking a batch of URLs at once (up to 20 URLs), I use their Bulk HTTP Checker. (Link: https://sitetoolsbox.com/tool.php?slug=http_status_batch)
Best Practices for Handling the Results
Once you have a list of 404s, don't just delete the links blindly.
If the page was deleted: Set up a 301 Redirect to the closest relevant page. This preserves your SEO juice.
If the page is temporarily down (500/502): Don't touch it yet. Wait a day and check again. It might just be a server hiccup.
If the external site is permanently dead: Remove the link and replace it with plain text, or find an alternative source.
Don't let broken links sit on your site. Write a script, run a weekly check, and keep your site's SEO healthy.
How do you handle broken links in your workflow? Do you use a CLI tool or a browser extension? Let me know in the comments below! 💬
Top comments (0)