Checking a single URL's HTTP status is trivial. One curl request, done. But when you need to check 50, 100, or 500 URLs at once — for example, when auditing a client's site for broken links — the naive approach falls apart fast.
I ran into this while building Site Tools Box, a free toolbox for webmasters. The bulk HTTP checker was one of the first tools I built, and I got the concurrency wrong three times before landing on the right pattern.
Here's what actually works.
Why Sequential Requests Are Too Slow
The obvious approach: loop through URLs, call curl_exec on each, collect results.
foreach ($urls as $url) {
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_NOBODY, true);
curl_exec($ch);
$status = curl_getinfo($ch, CURLINFO_HTTP_CODE);
curl_close($ch);
$results[$url] = $status;
}
For 50 URLs averaging 300ms response time, that's 15 seconds. For 500 URLs, it's 2.5 minutes. Users won't wait that long.
The Fix: curl_multi
curl_multi lets you run multiple requests in parallel from a single PHP process. Same 50 URLs, but all fired at once:
function checkBatch($urls, $concurrency = 10) {
$results = [];
for ($i = 0; $i < count($urls); $i += $concurrency) {
$chunk = array_slice($urls, $i, $concurrency);
$mh = curl_multi_init();
$handles = [];
foreach ($chunk as $url) {
$ch = curl_init();
curl_setopt_array($ch, [
CURLOPT_URL => $url,
CURLOPT_NOBODY => true,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_MAXREDIRS => 3,
CURLOPT_TIMEOUT => 8,
CURLOPT_CONNECTTIMEOUT => 5,
CURLOPT_USERAGENT => 'Mozilla/5.0',
CURLOPT_SSL_VERIFYPEER => false,
]);
curl_multi_add_handle($mh, $ch);
$handles[] = ['ch' => $ch, 'url' => $url];
}
$running = null;
do {
curl_multi_exec($mh, $running);
if ($running > 0) curl_multi_select($mh, 0.5);
} while ($running > 0);
foreach ($handles as $h) {
$status = curl_getinfo($h['ch'], CURLINFO_HTTP_CODE);
$results[$h['url']] = $status;
curl_multi_remove_handle($mh, $h['ch']);
curl_close($h['ch']);
}
curl_multi_close($mh);
}
return $results;
}
Now 50 URLs finish in about 1 second instead of 15.
The Concurrency Trap
My first version used $concurrency = 50. It worked... until a user pasted 500 URLs, and my server returned a 502 because the PHP process ran out of memory.
Three lessons:
1. Cap concurrency at 10-15. Higher than that, you're not gaining speed — you're gaining timeout errors. Some servers block burst connections, and cURL doesn't retry.
2. Process in chunks. Don't fire 500 requests at once. Slice the array, process each chunk, move on. This keeps memory bounded.
3. Set CURLOPT_NOBODY to true. For status checks, you don't need the response body. HEAD requests are 10x faster and use almost no bandwidth.
The Timeout Sweet Spot
Too short, and slow sites falsely report as "timeout." Too long, and one bad URL blocks the whole batch.
I landed on:
-
CURLOPT_CONNECTTIMEOUT = 5— how long to wait for the TCP handshake -
CURLOPT_TIMEOUT = 8— total request time -
CURLOPT_MAXREDIRS = 3— follow up to 3 redirects
For sites behind heavy Cloudflare protection, these numbers cause occasional false negatives. For 95% of URLs, they're perfect.
Handle Edge Cases Properly
A few things that bit me:
Redirects. A URL that 301s to a 404 needs to show the final status, not the redirect. Hence CURLOPT_FOLLOWLOCATION.
SSL errors. Old sites often have expired certificates. Setting CURLOPT_SSL_VERIFYPEER = false means you still get the status code instead of a hard failure.
Invalid URLs. Always validate before sending to curl. A malformed URL hangs the request in some curl builds.
Empty responses. Code 0 means curl failed entirely — server down, DNS failure, or connection reset. Handle it separately from HTTP error codes.
What I Actually Built
The production version handles 500 URLs in about 15-20 seconds with concurrency 10. Users see a live progress bar as each batch completes.
Try it here: Site Tools Box - Bulk HTTP Checker
It's free, no sign-up. Paste up to 500 URLs, one per line, and get the HTTP status for each.
If you're building something similar, the key takeaway is: curl_multi with a concurrency cap is the right pattern. Everything else is details.
Happy to answer questions about the implementation or performance tuning.
Top comments (2)
One thing I didn't cover in the article: handling servers that aggressively rate-limit HEAD requests.
Some sites (especially those behind certain CDNs) block HEAD entirely and only respond to GET. For those, you need a fallback that sends a full GET request when HEAD returns 405 or 403.
The check takes about 3x longer for those URLs, but it's the only way to get accurate results.
I ended up adding this as a second pass in my checker — HEAD first, GET fallback. Would be curious if others have run into the same issue.
tr.ee/dev-to