DEV Community: laing lii

How to Check HTTP Status Codes in Bulk with PHP and curl_multi

laing lii — Sat, 17 Oct 2026 01:12:21 +0000

Checking a single URL's HTTP status is trivial. One curl request, done. But when you need to check 50, 100, or 500 URLs at once — for example, when auditing a client's site for broken links — the naive approach falls apart fast.

I ran into this while building Site Tools Box, a free toolbox for webmasters. The bulk HTTP checker was one of the first tools I built, and I got the concurrency wrong three times before landing on the right pattern.

Here's what actually works.

Why Sequential Requests Are Too Slow

The obvious approach: loop through URLs, call curl_exec on each, collect results.

foreach ($urls as $url) {
    $ch = curl_init($url);
    curl_setopt($ch, CURLOPT_NOBODY, true);
    curl_exec($ch);
    $status = curl_getinfo($ch, CURLINFO_HTTP_CODE);
    curl_close($ch);
    $results[$url] = $status;
}

For 50 URLs averaging 300ms response time, that's 15 seconds. For 500 URLs, it's 2.5 minutes. Users won't wait that long.

The Fix: curl_multi

curl_multi lets you run multiple requests in parallel from a single PHP process. Same 50 URLs, but all fired at once:

function checkBatch($urls, $concurrency = 10) {
    $results = [];

    for ($i = 0; $i < count($urls); $i += $concurrency) {
        $chunk = array_slice($urls, $i, $concurrency);
        $mh = curl_multi_init();
        $handles = [];

        foreach ($chunk as $url) {
            $ch = curl_init();
            curl_setopt_array($ch, [
                CURLOPT_URL => $url,
                CURLOPT_NOBODY => true,
                CURLOPT_RETURNTRANSFER => true,
                CURLOPT_FOLLOWLOCATION => true,
                CURLOPT_MAXREDIRS => 3,
                CURLOPT_TIMEOUT => 8,
                CURLOPT_CONNECTTIMEOUT => 5,
                CURLOPT_USERAGENT => 'Mozilla/5.0',
                CURLOPT_SSL_VERIFYPEER => false,
            ]);
            curl_multi_add_handle($mh, $ch);
            $handles[] = ['ch' => $ch, 'url' => $url];
        }

        $running = null;
        do {
            curl_multi_exec($mh, $running);
            if ($running > 0) curl_multi_select($mh, 0.5);
        } while ($running > 0);

        foreach ($handles as $h) {
            $status = curl_getinfo($h['ch'], CURLINFO_HTTP_CODE);
            $results[$h['url']] = $status;
            curl_multi_remove_handle($mh, $h['ch']);
            curl_close($h['ch']);
        }

        curl_multi_close($mh);
    }

    return $results;
}

Now 50 URLs finish in about 1 second instead of 15.

The Concurrency Trap

My first version used $concurrency = 50. It worked... until a user pasted 500 URLs, and my server returned a 502 because the PHP process ran out of memory.

Three lessons:

1. Cap concurrency at 10-15. Higher than that, you're not gaining speed — you're gaining timeout errors. Some servers block burst connections, and cURL doesn't retry.

2. Process in chunks. Don't fire 500 requests at once. Slice the array, process each chunk, move on. This keeps memory bounded.

3. Set CURLOPT_NOBODY to true. For status checks, you don't need the response body. HEAD requests are 10x faster and use almost no bandwidth.

The Timeout Sweet Spot

Too short, and slow sites falsely report as "timeout." Too long, and one bad URL blocks the whole batch.

I landed on:

CURLOPT_CONNECTTIMEOUT = 5 — how long to wait for the TCP handshake
CURLOPT_TIMEOUT = 8 — total request time
CURLOPT_MAXREDIRS = 3 — follow up to 3 redirects

For sites behind heavy Cloudflare protection, these numbers cause occasional false negatives. For 95% of URLs, they're perfect.

Handle Edge Cases Properly

A few things that bit me:

Redirects. A URL that 301s to a 404 needs to show the final status, not the redirect. Hence CURLOPT_FOLLOWLOCATION.

SSL errors. Old sites often have expired certificates. Setting CURLOPT_SSL_VERIFYPEER = false means you still get the status code instead of a hard failure.

Invalid URLs. Always validate before sending to curl. A malformed URL hangs the request in some curl builds.

Empty responses. Code 0 means curl failed entirely — server down, DNS failure, or connection reset. Handle it separately from HTTP error codes.

What I Actually Built

The production version handles 500 URLs in about 15-20 seconds with concurrency 10. Users see a live progress bar as each batch completes.

Try it here: Site Tools Box - Bulk HTTP Checker

It's free, no sign-up. Paste up to 500 URLs, one per line, and get the HTTP status for each.

If you're building something similar, the key takeaway is: curl_multi with a concurrency cap is the right pattern. Everything else is details.

Happy to answer questions about the implementation or performance tuning.

How I Built a Domain Availability Checker with PHP and RDAP

laing lii — Fri, 16 Oct 2026 12:09:58 +0000

Domain availability checking looks simple: type a domain, get a "yes" or "no." But when you need to check hundreds of domains across dozens of TLDs — reliably, quickly, and without paying for a commercial API — the problem gets interesting.

I recently built Site Tools Box, a free online toolbox for webmasters that includes a bulk domain availability checker. Here's how the backend works, and why I chose RDAP over raw WHOIS.

Why Raw WHOIS Doesn't Scale

Traditional WHOIS is a plain-text protocol over port 43. It works, but it has real problems for a modern tool:

No standard format. Every registry returns text differently. Parsing "No match for" vs "Not found" vs "Status: available" becomes a fragile regex nightmare.
Rate limiting. Some WHOIS servers simply stop responding if you query too fast. There's no standard Retry-After header.
No concurrency support. You open a raw TCP socket for each query. That doesn't scale.

For a bulk checker that handles up to 200 domains per request, this approach falls apart quickly.

The Solution: RDAP with WHOIS Fallback

RDAP (Registration Data Access Protocol) is the modern replacement for WHOIS, mandated by ICANN. It's JSON-based, supports HTTP status codes, and most major registries now support it.

The idea is simple: use RDAP first, fall back to WHOIS only when RDAP isn't available.

Step 1: Map TLDs to RDAP Endpoints

Each registry has its own RDAP endpoint. I built a simple lookup table:

function domainRdapEndpoint($tld) {
    $map = [
        'com' => 'https://rdap.verisign.com/com/v1/domain/',
        'io'  => 'https://rdap.identitydigital.services/rdap/domain/',
        'ai'  => 'https://rdap.identitydigital.services/rdap/domain/',
        'xyz' => 'https://rdap.centralnic.com/xyz/domain/',
        // ... and more
    ];
    return $map[$tld] ?? null;
}

If the TLD isn't in the map, the tool falls back to a WHOIS socket connection for that domain.

Step 2: Parse the RDAP Response

RDAP returns JSON. A 200 status means the domain is registered, and a 404 means it's available. The JSON body also contains registrar info, registration date, and expiration date:

function parseRdapBody($domain, $body, $source = 'RDAP') {
    $d = json_decode($body, true);
    if (!$d) return null;

    $registrar = '';
    if (!empty($d['entities'])) {
        foreach ($d['entities'] as $entity) {
            if (!empty($entity['roles']) && in_array('registrar', $entity['roles'])) {
                foreach ($entity['vcardArray'][1] as $v) {
                    if ($v[0] === 'fn') { $registrar = $v[3]; break; }
                }
            }
        }
    }

    return [
        'domain' => $domain,
        'registered' => true,
        'registrar' => $registrar ?: '-',
    ];
}

Step 3: Handle Bulk Checks with curl_multi

For bulk requests, I use curl_multi to fire up to 30 concurrent RDAP queries. This is significantly faster than a sequential loop, and it respects the rate limits of most registries:

function domainCheckBatch($domains, $concurrency = 30) {
    $rdapDomains = [];
    foreach ($domains as $d) {
        $tld = substr(strrchr($d, '.'), 1);
        if (domainRdapEndpoint($tld)) $rdapDomains[] = $d;
    }

    for ($i = 0; $i < count($rdapDomains); $i += $concurrency) {
        $chunk = array_slice($rdapDomains, $i, $concurrency);
        $mh = curl_multi_init();
        // Add handles, execute, collect results
    }
}

Domains without an RDAP endpoint (like .cn or .uk) go through the WHOIS fallback, which uses a raw socket connection on port 43.

Three Things I Learned

1. RDAP is far more reliable than WHOIS. The JSON structure means no more parsing fragile text. HTTP status codes are unambiguous.

2. curl_multi is the right tool for bulk checks. Sequential queries make the tool painfully slow. A concurrency of 30 is the sweet spot before registries start throttling.

3. Always have a fallback. Not every TLD supports RDAP. The WHOIS fallback handles the rest, even if it's slower and messier.

Bonus: Scoring Available Domains

Since I was already building the checker, I added a small scoring algorithm for available domains. It looks at five dimensions:

Extension (.com scores highest)
Length (shorter is better)
Meaning (real English words or pinyin)
Pronounceability (vowel-consonant alternation)
Character quality (pure letters, no numbers)

It's not a professional valuation — just a rough starting point. But it helps users quickly filter hundreds of available domains.

Try It Out

You can try the domain availability checker here: Site Tools Box

It supports single and bulk checks, and gives available domains a rough value score.

If you're building something similar and have questions about the RDAP implementation, drop a comment below. I'm happy to share more of the code.

Show HN: Site Tools Box – 20+ free tools for webmasters

laing lii — Thu, 15 Oct 2026 07:43:41 +0000

Hi HN, I built a free toolbox with 20+ tools for webmasters: domain availability check, WHOIS, DNS, HTTP status, dead link scanner, JSON formatter, and more. No sign-up, no ads. Most tools run in your browser. Feedback welcome: https://sitetoolsbox.com

Hello everyone, I wanted to share

laing lii — Thu, 15 Oct 2026 07:41:31 +0000