Building a Proxy Health Checker in Python

If your scraper depends on proxies, you eventually learn a hard lesson: a proxy that worked an hour ago can be slow, blocked, or dead now, and your code will happily keep sending requests into the void. A small health checker that tests addresses before and during a run saves you from silent failures and skewed data. Here is a practical one in Python, plus what actually matters to measure.

What "healthy" really means

A proxy is not simply up or down. There are degrees, and each affects your run differently.

Reachable. It accepts a connection at all. The bare minimum.
Fast enough. Latency under a threshold you care about. A working but slow proxy quietly wrecks throughput.
Not blocked on your target. The important one. A proxy can be perfectly healthy in general and blocked on the specific site you are scraping.
Returning real content. It responds with 200 and actual data, not a captcha or a challenge page dressed up as success.

A good checker tests the last two, not just the first, because "connects fine but gets a captcha" is the failure that corrupts your data silently.

A minimal async checker

import asyncio, aiohttp, time

TEST_URL = "https://httpbin.org/ip"

async def check(session, proxy):
    start = time.perf_counter()
    try:
        async with session.get(TEST_URL, proxy=proxy, timeout=10) as r:
            body = await r.text()
            ok = r.status == 200 and "origin" in body
            return {"proxy": proxy, "ok": ok, "ms": int((time.perf_counter() - start) * 1000)}
    except Exception as e:
        return {"proxy": proxy, "ok": False, "error": str(e)}

async def run(proxies):
    async with aiohttp.ClientSession() as s:
        return await asyncio.gather(*(check(s, p) for p in proxies))

Point TEST_URL at something that echoes the exit address, so you also confirm the request actually went through the proxy and not your own connection.

Check against your real target, not just a generic URL

The trap is testing against a neutral endpoint and assuming that means the proxy works on your target. It does not. A proxy can pass httpbin and still get a challenge page on the site you care about. For a run that matters, health-check against the actual target, or a cheap page on it, and treat a captcha or challenge as unhealthy even when the status is 200.

Rotate out the bad ones, but do not overbuild

The pragmatic pattern is to check addresses at the start, drop the unhealthy ones, and re-check periodically during a long run, retiring any that start failing. Keep it simple. If you are spending more time maintaining a proxy scoreboard than scraping, the real problem is a low-quality pool, and the fix is a better pool, not more checker code.

Where a clean pool cuts the work

Most health-check complexity exists to route around bad shared proxies. WinGate reduces that by giving you private proxies that are yours alone, not a shared list where half the addresses are already burned. The rotating pool hands you fresh exits from a worldmix automatically, so instead of scoring a static list you let the pool cycle, traffic is unlimited, and it speaks HTTP, HTTPS, and SOCKS5 so your checker tests the same protocols you scrape with. There is a free 2 hour test, so run your health checker against a clean pool and see how much of your scoreboard logic you actually still need.

An honest note: a health checker tells you an address works now, it does not keep it working, and it does not exempt you from a site's terms. Pacing and rotation still do the anti-blocking work. What the checker buys you is honest data, because you stop trusting responses from proxies that were silently failing.

The takeaway: measure reachable, fast, unblocked, and returning-real-content, test against your actual target, drop and re-check during long runs, and start from a private rotating pool so you spend your time scraping instead of babysitting dead addresses.

DEV Community