DEV Community

Devil Scrapes
Devil Scrapes

Posted on

A 25%-failing test is not flaky. It's a bug reporting itself honestly.

Quick answer: A test that fails ~25% of the time is not a flaky test — it's a probabilistic bug reporting itself honestly, and the tempting move is to re-run it until it goes green. pick_profile() in our HTTP client picked a browser-impersonation profile with a bare random.choice(BROWSER_PROFILES), including on a retry right after a block. One in four times, it re-picked the identical profile it had just been blocked on. "Rotate on retry" was a comment, not a contract.

The test that wouldn't sit still

test_fetch_vehicle_page_retries_429_then_succeeds_rotating_profile does what the name says: feed the client a 429, then a 200, and assert the second attempt used a different impersonation profile than the first. Straightforward, and it failed about one run in four.

The natural next move — the one every one of us has made at least once — is to call it flaky and hit re-run. A test that sometimes fails for no obvious reason feels like CI noise: a race condition in the test harness, a mock that isn't quite deterministic, a sleep that's too tight. None of that was true here.

The actual line

def pick_profile() -> str:
    return random.choice(BROWSER_PROFILES)
Enter fullscreen mode Exit fullscreen mode

BROWSER_PROFILES is a 4-tuple: chrome131, chrome124, firefox147, safari180. Every call is an independent draw. Call it once after a block to pick the retry's fingerprint, and there's a flat 1-in-4 chance you land on the exact profile that just got you blocked.

That's not a cosmetic detail. The entire point of rotating impersonation on retry is to present a different TLS/HTTP2 fingerprint after a block — same reasoning as rotating the proxy session_id. Retry with the identical fingerprint and you've retried nothing; you've just asked the same question again and hoped for a different answer. A quarter of our retries were doing exactly that, silently, and only a test that happened to assert on the outcome ever noticed.

Why "flaky" was the wrong diagnosis

A genuinely flaky test fails for reasons unrelated to the code under test — timing, shared global state, network calls that shouldn't be in a unit test. This one failed because the code under test had a bug, and the test was correctly, if noisily, exposing it. The failure rate — one in four — is a direct fingerprint of the bug's own math: 1 collision out of 4 uniformly random choices.

That's the tell worth remembering: if a test's failure rate lines up suspiciously well with "1 divided by the size of some pool," you're probably not looking at flakiness, you're looking at a sampling bug with a pool that size.

The fix

def pick_profile(*, previous: str | None = None) -> str:
    """Pick a browser-impersonation profile, excluding ``previous`` when possible.

    A retry after a block should present a genuinely different fingerprint,
    not risk re-picking the same one — ``random.choice`` alone can repeat.
    """
    choices = [p for p in BROWSER_PROFILES if p != previous] or list(BROWSER_PROFILES)
    return random.choice(choices)
Enter fullscreen mode Exit fullscreen mode

The retry loop now threads the previous attempt's profile through and excludes it from the next draw. We ran the test 20 consecutive times after the fix — zero failures, deterministic. The or list(BROWSER_PROFILES) guard is there for the degenerate case of a single-entry pool, where excluding "previous" would leave nothing to choose from.

The second thing the same pass caught

Separately, and less dramatically: the README advertised $3.00 per 1,000 results. That number only counted the per-row charge ($0.003 x 1,000). It left out the flat $0.20 actor-start warm-up fee every run pays once — which is a small, easy-to-drop term when you're doing pricing math by hand instead of running the actual events through the formula. A pricing gate we run before any Actor ships caught the discrepancy before a customer could, and the README now states the real effective cost: $3.20 per 1,000 results ($0.20 flat start + 1,000 x $0.003 per row).

Neither bug was exotic. Both were the kind of thing that passes a quick read and fails the moment you do the arithmetic — or run the test enough times to see the pattern instead of the individual failure.

What a row looks like

A real cloud run pulled 26 rows across three vehicles — honda/civic/2020 (16 trims), ford/f-150/2022 (4 trims), toyota/camry/2021 (6 trims) — covering both ways you can pass vehicles into this Actor. One row:

{
  "make": "honda",
  "model": "civic",
  "year": 2020,
  "trim": "LX Sedan 4D",
  "fair_purchase_price": 16600.0,
  "original_msrp": 21755.0,
  "currency": "USD",
  "source_url": "https://www.kbb.com/honda/civic/2020/"
}
Enter fullscreen mode Exit fullscreen mode

🚗 Kelley Blue Book Valuation Scraper turns KBB's own vehicle valuation pages into typed rows — Fair Purchase Price, Fair Market Price low/high, and original MSRP, one row per trim, for any make/model/year you batch in. We rotate curl-cffi browser fingerprints on every attempt (never repeating the one that just got blocked), retry with exponential backoff, and rotate Apify Proxy sessions on failure, so you get a clean dataset instead of a half-finished run. $3.20 per 1,000 results, and you only pay for rows that land.

FAQ

Why did the retry test fail ~25% of the time instead of consistently?
random.choice over a 4-item pool re-picks the same item roughly 1 time in 4. The failure rate was a direct signature of the bug's own math, not test-harness noise.

How do you avoid re-picking a blocked fingerprint on retry?
pick_profile() now takes the previous attempt's profile and excludes it from the candidate pool before drawing — verified deterministic over 20 consecutive test runs.

Why did the README say $3.00 and not $3.20?
The original figure counted only the per-row PPE charge and omitted the flat $0.20 actor-start fee every run pays once. A pricing gate we run before publish caught it and the README now states the real effective cost.

Is a probabilistic test failure always a bug?
Not always, but it's worth treating as one until proven otherwise — especially when the failure rate lines up with "1 over the size of some pool" in your own code, which is exactly what a uniform-random collision looks like.

Top comments (0)