Quick answer: A test that fails ~25% of the time is not a flaky test — it's a probabilistic bug reporting itself honestly, and the tempting move is to re-run it until it goes green. pick_profile() in our HTTP client picked a browser-impersonation profile with a bare random.choice(BROWSER_PROFILES), including on a retry right after a block. One in four times, it re-picked the identical profile it had just been blocked on. "Rotate on retry" was a comment, not a contract.
The test that wouldn't sit still
test_fetch_vehicle_page_retries_429_then_succeeds_rotating_profile does what the name says: feed the client a 429, then a 200, and assert the second attempt used a different impersonation profile than the first. Straightforward, and it failed about one run in four.
The natural next move — the one every one of us has made at least once — is to call it flaky and hit re-run. A test that sometimes fails for no obvious reason feels like CI noise: a race condition in the test harness, a mock that isn't quite deterministic, a sleep that's too tight. None of that was true here.
The actual line
def pick_profile() -> str:
return random.choice(BROWSER_PROFILES)
BROWSER_PROFILES is a 4-tuple: chrome131, chrome124, firefox147, safari180. Every call is an independent draw. Call it once after a block to pick the retry's fingerprint, and there's a flat 1-in-4 chance you land on the exact profile that just got you blocked.
That's not a cosmetic detail. The entire point of rotating impersonation on retry is to present a different TLS/HTTP2 fingerprint after a block — same reasoning as rotating the proxy session_id. Retry with the identical fingerprint and you've retried nothing; you've just asked the same question again and hoped for a different answer. A quarter of our retries were doing exactly that, silently, and only a test that happened to assert on the outcome ever noticed.
Why "flaky" was the wrong diagnosis
A genuinely flaky test fails for reasons unrelated to the code under test — timing, shared global state, network calls that shouldn't be in a unit test. This one failed because the code under test had a bug, and the test was correctly, if noisily, exposing it. The failure rate — one in four — is a direct fingerprint of the bug's own math: 1 collision out of 4 uniformly random choices.
That's the tell worth remembering: if a test's failure rate lines up suspiciously well with "1 divided by the size of some pool," you're probably not looking at flakiness, you're looking at a sampling bug with a pool that size.
The fix
def pick_profile(*, previous: str | None = None) -> str:
"""Pick a browser-impersonation profile, excluding ``previous`` when possible.
A retry after a block should present a genuinely different fingerprint,
not risk re-picking the same one — ``random.choice`` alone can repeat.
"""
choices = [p for p in BROWSER_PROFILES if p != previous] or list(BROWSER_PROFILES)
return random.choice(choices)
The retry loop now threads the previous attempt's profile through and excludes it from the next draw. We ran the test 20 consecutive times after the fix — zero failures, deterministic. The or list(BROWSER_PROFILES) guard is there for the degenerate case of a single-entry pool, where excluding "previous" would leave nothing to choose from.
The second thing the same pass caught
Separately, and less dramatically: the README advertised $3.00 per 1,000 results. That number only counted the per-row charge ($0.003 x 1,000). It left out the flat $0.20 actor-start warm-up fee every run pays once — which is a small, easy-to-drop term when you're doing pricing math by hand instead of running the actual events through the formula. A pricing gate we run before any Actor ships caught the discrepancy before a customer could, and the README now states the real effective cost: $3.20 per 1,000 results ($0.20 flat start + 1,000 x $0.003 per row).
Neither bug was exotic. Both were the kind of thing that passes a quick read and fails the moment you do the arithmetic — or run the test enough times to see the pattern instead of the individual failure.
What a row looks like
A real cloud run pulled 26 rows across three vehicles — honda/civic/2020 (16 trims), ford/f-150/2022 (4 trims), toyota/camry/2021 (6 trims) — covering both ways you can pass vehicles into this Actor. One row:
{
"make": "honda",
"model": "civic",
"year": 2020,
"trim": "LX Sedan 4D",
"fair_purchase_price": 16600.0,
"original_msrp": 21755.0,
"currency": "USD",
"source_url": "https://www.kbb.com/honda/civic/2020/"
}
🚗 Kelley Blue Book Valuation Scraper turns KBB's own vehicle valuation pages into typed rows — Fair Purchase Price, Fair Market Price low/high, and original MSRP, one row per trim, for any make/model/year you batch in. We rotate curl-cffi browser fingerprints on every attempt (never repeating the one that just got blocked), retry with exponential backoff, and rotate Apify Proxy sessions on failure, so you get a clean dataset instead of a half-finished run. $3.20 per 1,000 results, and you only pay for rows that land.
FAQ
Why did the retry test fail ~25% of the time instead of consistently?
random.choice over a 4-item pool re-picks the same item roughly 1 time in 4. The failure rate was a direct signature of the bug's own math, not test-harness noise.
How do you avoid re-picking a blocked fingerprint on retry?
pick_profile() now takes the previous attempt's profile and excludes it from the candidate pool before drawing — verified deterministic over 20 consecutive test runs.
Why did the README say $3.00 and not $3.20?
The original figure counted only the per-row PPE charge and omitted the flat $0.20 actor-start fee every run pays once. A pricing gate we run before publish caught it and the README now states the real effective cost.
Is a probabilistic test failure always a bug?
Not always, but it's worth treating as one until proven otherwise — especially when the failure rate lines up with "1 over the size of some pool" in your own code, which is exactly what a uniform-random collision looks like.
Top comments (0)