Stop Using Slow APIs: The Right Way to Bulk Check HTTP Statuses (50+ URLs in Seconds)

Last week, I found myself staring at a dead-links report for an old client project. Over 300 pages. I needed to know which URLs were returning 404s, which ones were silently redirecting, and which ones had just given up entirely.

My first instinct was to write a quick loop in Node.js using fetch.

I hit two massive walls immediately:

1. The Rate Limit / IP Ban wall: After about 15 requests, the target servers started dropping my connection.
2. The Timeout wall: My script hung for 5 minutes because three random URLs just never responded.

If you've ever tried to bulk check HTTP statuses at scale, you know the pain. So, let me save you a few hours of frustration.

Here is the breakdown of how this should work, and a free tool I started using to bypass the headache entirely.

The Naive Approach (And Why It Fails)

Here’s how most of us start writing a bulk HTTP checker:

javascript
// DON'T DO THIS FOR BULK CHECKS
async function checkUrls(urls) {
  for (const url of urls) {
    try {
      const res = await fetch(url, { method: 'HEAD' });
      console.log(url, res.status);
    } catch (e) {
      console.log(url, 'Error');
    }
  }
}

This is a disaster for a few reasons:

HEAD isn't always supported: Some servers block HEAD requests and only return a valid 200 for GET.

No Concurrency Control: If you use Promise.all(), you’ll fire 50 requests at once and your IP will get blocked by Cloudflare instantly.

No Timeouts: A single hanging URL will block your entire script.

The "Developer" Way to Fix It
To do this properly in a script, you need to implement:

Connection Pooling / Semaphores (to limit concurrency to ~5 requests at a time).

Custom AbortControllers (to kill requests after 5 seconds).

User-Agent spoofing (so it looks like a real browser).

It’s a fun technical exercise, but when you’re on a deadline and just need the data, writing this whole infrastructure is a waste of time.

The Alternative: Automate It Without the Code
When I realized I was spending more time debugging my scraper than actually auditing the site, I switched to using a bulk HTTP checker tool I found online.

I’ve been using Bulk HTTP Checker on Site Tools Box.

It’s a lightweight, backend-optimized tool that handles all the heavy lifting for you. You can paste up to 50 URLs, hit run, and it handles the concurrency, redirects, and timeouts instantly.

It returns a clean list of:

Status Codes (200, 301, 404, 500)

Redirect chains (where it actually goes)

Response times

And because it’s built for developers, it doesn't try to sell you a $29/month subscription for a simple check.

One Last Pro-Tip for SEO / Auditing
If you have a sitemap and want to find out which specific pages on your site are broken, you don't even need to copy-paste manually. Just use the Dead Link Checker on Site Tools Box. It scans a page, crawls the outgoing links, and highlights any broken ones in red. I used it to find 3 dead links in my footer that had been silently 404-ing for months.

Don't waste your weekends writing semaphores for a script you'll only use once. Optimize your workflow, and get back to writing actual code.

Have you ever accidentally DoS'd a server with a Promise.all() loop? Let me know in the comments. We've all been there.