HTTP 200 Is Not Success: Write a Contract Per Host

#webscraping #proxy #python #tutorial

The last two posts were about spending an IP well: pace the requests, and hold a sticky session exactly as long as one unit of work. Both assume something I never stated out loud — that you can tell when a request worked.

Most pipelines can't. They check the status code, see 200, and move on. And 200 is exactly what a target returns when it has decided to stop giving you real answers.

The failure mode nobody alerts on

Hard failures are easy. A 403, a connection reset, a captcha page that throws on parse — your retry logic catches those, your dashboard turns red, you go look.

Soft degradation is the expensive one:

The page returns 200 with a challenge shell and none of the content.
The search endpoint returns 200 with a well-formed, empty result list.
The API returns 200 with a cached response from six hours ago.
The listing returns 200 with 3 rows where it used to return 50.

Every one of those sails through a status-code check. Your scraper reports a clean run. Your dataset quietly rots, and you find out days later when someone downstream asks why a whole region went empty.

Worse, you keep spending the IP. The target has already decided you're not a person; you just haven't been told. Every request after that point is paid for and worthless — and it's building exactly the kind of profile that turns a soft block into a hard one.

Write the contract before you write the parser

For each host you scrape, write down what a good response looks like. Three assertions is usually enough:

Shape — a structural element that only exists on a real page. Not <title>, which survives on challenge pages. Pick the container that holds the thing you came for: the results table, the JSON key with the records in it.
Volume — a plausible range, not a minimum of one. If a category page has returned between 20 and 60 rows every day for a month, then 3 rows is a failure even though it parsed fine.
Identity — proof the response is about what you asked for. The canonical URL, the product ID, the echoed query string. This is what catches stale caches and silent redirects to a generic page.

A response that fails any of the three is a failure, no matter what the status line says.

from dataclasses import dataclass
from typing import Callable

@dataclass
class Contract:
    host: str
    shape: Callable[[dict], bool]      # the container exists
    volume: range                      # plausible record count
    identity: Callable[[dict, str], bool]  # response matches request

    def check(self, parsed: dict, requested_id: str) -> str | None:
        if not self.shape(parsed):
            return "shape"
        if len(parsed.get("records", [])) not in self.volume:
            return "volume"
        if not self.identity(parsed, requested_id):
            return "identity"
        return None  # passed

Writing this down takes ten minutes per host and changes what "success rate" means in your metrics. Most people discover their real success rate was never the number on the dashboard.

The canary tells you whose fault it is

A contract tells you a response was bad. It doesn't tell you why, and the difference decides what you do next:

If that page is broken, skip it and move on. Rotating your IP is a waste.
If you are blocked, moving on is the worst possible move. Every subsequent request digs the hole deeper.

You separate the two with a canary: one URL per host whose correct answer you already know and that essentially never changes. A stable category page, a documented API example, an "about" endpoint with fixed fields. Run it through the same session, same headers, same exit IP as your real traffic, and check it against the same contract.

Now the diagnosis is mechanical:

Canary	Target page	Diagnosis	Action
pass	pass	healthy	continue
pass	fail	that page is genuinely bad or changed	log it, skip, don't rotate
fail	fail	you're degraded	stop the unit of work, rotate at the seam
fail	pass	flaky canary — fix your canary	investigate offline

That third row is the one that saves money. A canary failure is the earliest honest signal that your session has stopped being trusted — usually well before anything returns a 403.

Run the canary like a visitor, not like a monitor

Two rules, both inherited from the earlier posts.

Don't run it per request. One canary per unit of work — at the start, so you don't waste a journey, and once more before you commit expensive writes. That's a 1–2% overhead, not 50%.

Don't run it on a timer. A request to the same URL every 60.0 seconds, forever, from an IP that otherwise browses like a human is the single most machine-shaped pattern you can emit. The canary exists to detect that you look like a bot; it must not be the reason you do. Same jitter, same pacing, same session as everything else.

What this buys you

The contract turns "did it 200?" into "did I get what I came for?". The canary turns a bad response into an attributable one. Together they change the failure from silent, days-long data rot into an event you can act on in the same minute it happens.

And it changes what proxy quality even means to you. Once every response is checked against a contract, you stop arguing about pool sizes and start measuring the thing that matters: how many units of work complete, per IP, before the canary goes quiet. That number is comparable across providers, across weeks, and across your own config changes — which is the whole point.

I work on Roam, residential and static residential proxies billed per GB. The measurement habits above are how we evaluate our own pools; they apply no matter whose IPs you're renting.

DEV Community