DEV Community

Devil Scrapes
Devil Scrapes

Posted on

Our proxy pool reported 0 available IPs. It was lying — for weeks.

#webscraping #python #devops #debugging

Short version: our proxy pool reported availableCount: 0. We believed it, wrote that belief into our QA tooling, and spent weeks failing 38 scrapers against the wrong network. The pool was never empty — a dynamic residential pool can't be enumerated, so the API reports zero. The fix wasn't a plan upgrade. It was one billing field that proved the tier had been working the whole time.

Here's the trap, why a capability check has to be an observation and not a metadata read, and the two other bugs it was hiding.

🕳️ The field that lied

We run our scrapers on Apify. Asking the API which proxy groups an account has gives you something like this:

group availableCount residential
BUYPROXIES94952 5 false
GOOGLE_SERP 0 false
RESIDENTIAL 0 false

Read that table cold and the conclusion writes itself: we have five datacenter IPs and no residential access. That's what we concluded. It went into a comment in our QA tool, which is the worst place for a wrong belief to land, because a comment makes it look researched:

# `RESIDENTIAL` has availableCount 0 on this account, so any Actor whose schema
# prefills it fails QA with `CONNECT tunnel failed, response 590` ...
UNAVAILABLE_PROXY_GROUPS = {"RESIDENTIAL", "GOOGLE_SERP"}
Enter fullscreen mode Exit fullscreen mode

That constant did something quietly destructive. Before every cloud QA run, it rewrote the scraper's proxyConfiguration to point at the datacenter group. 38 of our scrapers declare a residential default. All 38 were being silently downgraded onto five shared datacenter IPs, and then failed, and then recorded a verdict — "target blocks us", "infra-blocked, needs a paid proxy tier" — against code that had never once run on the network it was written for.

We wrote off targets on that evidence. That's the part that stings.

🔬 How we caught it: bill, don't ask

The tell was a contradiction in our own accounting. The usage breakdown for the billing cycle showed this line:

PROXY_RESIDENTIAL_TRANSFER_GBYTES     0.0470 GB     $0.3759
Enter fullscreen mode Exit fullscreen mode

Forty-seven megabytes of residential traffic on an account with "no residential access." Both facts came from the same API. One of them was wrong.

So we stopped asking and ran the thing. One scraper, pinned to RESIDENTIAL with a Spanish exit, against a target that hard-localizes by IP. Then we pulled the run's per-service usage — not the logs, not the group metadata, the bill:

ACTOR_COMPUTE_UNITS                   0.00244333    $0.000489
PROXY_RESIDENTIAL_TRANSFER_GBYTES     0.00000712    $0.000057
Enter fullscreen mode Exit fullscreen mode

7.12 kilobytes over residential proxy. The tier had been available the entire time. availableCount: 0 doesn't mean "you have none" — a rotating residential pool is dynamic and unenumerable, so the honest answer to "how many IPs?" is zero. It's a cardinality field being read as an entitlement field.

A capability is proven by an observed effect, not by a field that claims to describe it. Metadata describes intent; billing records what happened. When they disagree, billing wins.

That's now a rule for us, and run_usage.py <runId> — print a run's per-service usage — is a permanent part of the toolkit. It's the only way to answer "which proxy tier did this run actually use?", because logs don't say, and a proxy-config helper that fails silently degrades to a direct connection without telling you.

🚧 The local probe that can't answer the question

The obvious next move is to probe every tier from your laptop and record what works. We wrote that script. It returns this:

datacenter (BUYPROXIES94952)    403 Forbidden
auto (no group pinned)          403 Forbidden
RESIDENTIAL (no country)        403 Forbidden
RESIDENTIAL country-ES          403 Forbidden
Enter fullscreen mode Exit fullscreen mode

Four tiers, four identical failures — including the datacenter group we know works, because it's been returning rows in production for weeks. A uniform failure across a tier you've independently proven is not a result. It's the measurement apparatus failing.

Apify's free plan gates proxy access to runs on the platform; from outside, every CONNECT gets a 403 regardless of tier. So a local sweep can only ever produce a flat "everything is blocked," which is indistinguishable from real, catastrophic breakage.

We made the script say so out loud rather than return its own confusion as data:

INCONCLUSIVE — every tier returned CONNECT 403, including datacenter, which is
known to work from Apify cloud. That is the FREE plan's LOCAL-only 'Proxy
external access' gate, not a tier verdict. Do NOT record a NO-GO from this output.
Enter fullscreen mode Exit fullscreen mode

A tool that can't answer the question should say "I can't answer that." Returning a confident wrong answer is worse than returning nothing, and this is a general principle for scraper diagnostics: classify your own failure modes, or you will file them as the target's.

💸 The cost objection, which was also wrong

The other reason residential stayed off was price. Residential transfer bills at $8.00/GB, which sounds ruinous against a small monthly budget — the whole allowance is about 0.6 GB.

Per gigabyte, yes. Per page, no. A search page is kilobytes. That probe cost $0.000057. You could run ten thousand of them for under a dollar. We'd been avoiding a tier on a unit-price headline without ever multiplying it by the quantity we actually use — and the real spend driver turned out to be something else entirely. Here's where the cycle's money was going:

service share
external data transfer 46.4%
actor compute 40.2%
residential proxy 12.8%
everything else 0.6%

Residential was never the problem. And while we're on measurement errors: we'd been reading a cycle-average burn rate that said "you'll run out of money in 1.4 days," computed by dividing total spend by days elapsed. Two of those days contained deliberate QA sweeps. Actual spend for the current day was $0.07. Averaging a spike into a forecast produces a panic, and the panic had been shaping release decisions for two days.

🐛 The bug that was hiding behind the wrong network

Once the scraper finally ran on residential, it failed in nine seconds with a real HTTP 403 from the target. Progress — an honest answer at last. But nine seconds is too fast, and the log explained why:

ES search page=1 HTTP 403
idealista ES: zero listings — failing loud
Enter fullscreen mode Exit fullscreen mode

One attempt. No rotation. The retry logic looked fine:

RETRY_STATUSES = (408, 429, HTTP_UNAVAILABLE)
MAX_RETRIES = 5
Enter fullscreen mode Exit fullscreen mode

408, 429, 503 — timeouts and throttles. Sensible, and wrong for this job. A 403 from a bot-management layer isn't a logical error, it's that exit IP being refused. When you're on a rotating pool, a fresh IP is the entire remedy — and it was the one case that fell through to "give up immediately":

if outcome.status not in RETRY_STATUSES and outcome.status != HTTP_SOFT_BLOCK:
    return None  # a genuine other-4xx logical failure — never retried
Enter fullscreen mode Exit fullscreen mode

Correct for a 404. Backwards for a 403. The distinction that matters isn't retryable-vs-not, it's is this response about the request, or about the connection it arrived on?

# Anti-bot refusals of *this exit IP*, not logical errors. Rotating the session
# picks a new residential exit, which is the whole point of a rotating pool.
BLOCK_STATUSES = (403, 451)
Enter fullscreen mode Exit fullscreen mode

Two regression tests pin it now: a 403 must rotate and retry, and a target that refuses every exit must still exhaust MAX_RETRIES rather than quit after one. That second test is the one that would have caught this originally — the bug wasn't "it gave up", it was "it gave up too early", and only an attempt count can tell those apart.

🧾 What we'd tell you to copy

Four things, none of them about proxies specifically:

  1. Never encode a platform belief as a constant without an observation attached. If you must, put the run ID that proves it in the comment. Ours now carries one.
  2. Prefer billing and side effects over descriptive metadata. availableCount describes a pool. The invoice describes reality.
  3. Make diagnostics classify their own blind spots. "Inconclusive, and here's why" beats a false negative every time.
  4. A "this target is unscrapable" verdict is only valid if it names every tier you probed. We had that rule already. What we didn't have was any way to check which tier a run actually used — so the rule was unenforceable, and quietly ignored.

The scrapers this affected are the ones that hit hard targets, where transport choice is the whole game:

We handle the blocks, the rotation, the retries and the tier selection so you get rows instead of a 403. Sometimes that means we find out our own tooling was the thing doing the blocking.

Top comments (0)