A Retry on a New Proxy Exit Can Double-Submit: Idempotency for Scrapers That POST

Most scraping advice assumes you only ever GET. And for a lot of collection work, that is true: you fetch a page, you parse it, and if the request dies halfway you just fire it again. GET is idempotent by contract, so a duplicate costs you nothing but bandwidth.

Then one day your job has to POST. Maybe you are submitting a search form that a site only answers with JSON after a real POST body. Maybe you are driving an authenticated internal API where each call enqueues a task, files a claim, places a bid, or writes a row. Maybe you are posting to a moderation queue. The moment the request has a side effect, the retry loop you copied from a GET tutorial stops being free. A timed-out POST is not "probably didn't happen". It is "I have no idea whether it happened, and finding out wrong costs money."

This post is about that one failure mode, sharpened by a wrinkle most idempotency write-ups ignore: when you rotate through a residential or datacenter proxy pool, your retry may leave from a completely different exit IP than the original attempt. The origin's own dedup logic, if it keys on anything IP- or session-bound, silently stops working exactly when you need it most.

The two failure classes people conflate

When a POST over a proxy fails, the exception is the only evidence you have about whether the origin saw your request. There are, in practice, two very different buckets hiding behind "it errored":

1. Definitely-not-sent. The TCP connect to the proxy (or the proxy to the origin), the TLS handshake, or the write of the request line failed. Nothing reached the application layer on the far side. Retrying is safe from a side-effect standpoint.
2. Unknown-completion. The bytes left your socket, and then a read timed out, or the connection dropped, or you got a 5xx from an intermediary. The origin may have fully processed the request and committed its side effect, and you simply never saw the response. Retrying blindly here risks a double-submit.

The trap is that most HTTP clients throw a single generic "connection error" and most people's except Exception: retry() treats both classes as one. In requests, ConnectTimeout and most ConnectionError instances are (usually) class 1, while ReadTimeout and a dropped-connection-after-send are class 2. That distinction is worth encoding on purpose, not by accident.

Rotation makes class 2 worse in a specific way. Say attempt 1 goes out over exit IP 203.0.113.7, the origin commits the record, and the response is lost to a flapping proxy hop. Your retry handler rotates to a fresh random exit, 198.51.100.42, and re-POSTs. If the origin dedupes on (source_ip, payload_hash) or binds a submission token to the session that also carried the IP, the second POST now looks like a brand-new request from an unrelated client. You get two records. The thing that was supposed to protect you from a duplicate - the same sticky identity - is the thing rotation just removed.

What actually fixes it, in rough order of preference

The correct fix is a server-side idempotency key: you mint a unique token per logical operation, send it as a header (and/or embed it in the body), and the origin guarantees "run this at most once; replay the stored result if I've seen this key before." Stripe does exactly this with Idempotency-Key. If the endpoint you are POSTing to supports it, use it and stop reading; everything below is for the far more common case where it does not.

When the endpoint has no idempotency support, you build the guarantee yourself with three ideas working together:

• A client-generated nonce embedded in the payload, so you can later ask "did a record with my nonce get created?" even if the create endpoint gave you no key back.
• A verify-before-replay step for class-2 failures: before you dare POST again, do a read (GET) that can see the side effect, filtered by your nonce. Only if the verify says "not there" do you re-send.
• Pinning the exit for the duration of one logical operation: while an attempt is unresolved, do NOT rotate. Send the retry and the verify down the same sticky session you used for the first POST, so any IP-bound dedup on the origin still lines up. Rotation is per-operation, not per-request.

That last point is the one that quietly requires a proxy feature: you need a sticky session whose exit you control and reuse across the retry, rather than a gateway that hands you a new random IP on every single call. That is exactly the session mode on a residential pool, which is where this whole dance is most relevant.

A runnable harness

The code below is self-contained. It targets httpbin.org, a public request-echo service, so you can run it unchanged and watch the classification and the pin/verify logic behave. Swap the URL and payload shape for your real endpoint. It uses requests only.

import hashlib
import time
import uuid

import requests
from requests.exceptions import ConnectTimeout, ConnectionError, ReadTimeout

# A public echo service stands in for your real POST endpoint.
BASE = "https://httpbin.org"

# In production this dict is your durable in-flight ledger (a table, Redis, etc.),
# NOT a process-local dict. It must survive a crash, or the whole scheme collapses.
INFLIGHT = {}

def classify(exc):
    """Map an exception to 'not_sent' (safe to replay) or 'unknown' (verify first)."""
    if isinstance(exc, (ConnectTimeout, ConnectionError)):
        return "not_sent"
    if isinstance(exc, ReadTimeout):
        return "unknown"
    # A dropped connection mid-read is also class 2. When in doubt, treat as unknown.
    return "unknown"

def build_payload(nonce):
    # The nonce rides inside the body so the origin can be queried by it later.
    return {"nonce": nonce, "value": "whatever-you-are-submitting"}

def verify_exists(nonce, session):
    """Ask the target whether a record for this nonce already got created.

    With httpbin we fake a real lookup: replay a GET that 'remembers' the nonce.
    Against a real API you would GET /records?nonce=... here.
    """
    resp = session.get(f"{BASE}/get", params={"q": nonce}, timeout=15)
    resp.raise_for_status()
    # Replace this echo-based stub with a genuine existence check on your data.
    return nonce in getattr(verify_exists, "_seen", set())

def post_with_retry(session, url, payload, max_attempts=3):
    nonce = payload["nonce"]
    for attempt in range(1, max_attempts + 1):
        try:
            resp = session.post(url, json=payload,
                                headers={"X-Idempotency-Key": nonce}, timeout=15)
            if resp.status_code in (200, 201):
                verify_exists._seen = getattr(verify_exists, "_seen", set()) | {nonce}
                INFLIGHT.pop(nonce, None)
                return {"ok": True, "status": resp.status_code, "attempt": attempt}
            # 4xx/5xx are responses, so the request was understood. A 5xx from an
            # intermediary is still class 2 for side effects: verify, then decide.
            if 500 <= resp.status_code < 600:
                kind = "unknown"
            else:
                # 4xx means it did NOT commit (bad request / auth). Do not replay blindly.
                return {"ok": False, "status": resp.status_code, "attempt": attempt}
        except Exception as e:
            kind = classify(e)
            print(f"attempt {attempt}: {type(e).__name__} -> class={kind}")

        if kind == "unknown":
            # Do NOT rotate. Reuse the same pinned session to re-send or verify.
            if verify_exists(nonce, session):
                print("verify says it already landed; not re-POSTing")
                INFLIGHT.pop(nonce, None)
                return {"ok": True, "deduped": True, "attempt": attempt}
            # Not there yet: stay on the SAME session (same exit) and retry the POST.
            continue
        else:
            # not_sent: safe to try again. With no proxy this is just a loop.
            # With a real pool you may rotate here, because nothing reached origin.
            time.sleep(0.3 * attempt)
            continue
    return {"ok": False, "reason": "exhausted"}

if __name__ == "__main__":
    # One session == one sticky proxy identity for this logical operation.
    # Wire your gateway's sticky-session params onto this session's proxies/auth.
    op_session = requests.Session()
    # op_session.proxies = {"https": "http://user-zone-US-session-abc123:pass@gateway:port"}

    nonce = uuid.uuid4().hex
    payload = build_payload(nonce)
    INFLIGHT[nonce] = time.time()
    result = post_with_retry(op_session, f"{BASE}/post", payload)
    print("RESULT:", result)

The shape that matters is not httpbin; it is the three decisions: classify() separating not-sent from unknown, verify_exists() gating the class-2 retry, and the single Session standing in for a pinned exit. In production, INFLIGHT and the nonce must live somewhere durable (a DB row or Redis key with a TTL), because if the process dies mid-flight, an in-memory dict restores nothing and you are back to guessing.

Results and the traps that bit me

Run it as-is and the happy path returns in one attempt. To see class 2, force it: point verify_exists() at a stub that lies "already landed" on the second call, and watch the loop dedupe instead of double-posting. The lesson the demo cannot fake is the failure that actually hurts:

• The gateway that retries for you. Some proxies, on an upstream timeout, transparently re-send the request to the origin before returning an error to you. You never saw a duplicate, but two POSTs left the proxy. If you are on such a gateway, your client-side dedup is defending against a double-submit that already happened upstream. Know whether your provider's gateway auto-retries on timeout. On most scraper-facing gateways it does not, but "most" is not "never".
• Sticky sessions that quietly expire mid-operation. Stickiness is usually bounded by time AND by request count. If your retry lands after the gateway silently rolled the session to a new exit (max-use cap), you pinned to "the same session" and got a different IP anyway. Re-check the exit between the failed POST and the verify if the identity matters - hit an IP echo once per operation and assert it stayed put.
• Verify endpoints that are eventually consistent. "Not there yet" from a read replica ten seconds after the write does not mean "never happened." If your only proof-of-absence races replication, you will re-POST into a duplicate. Bound the verify with a real retry budget and a ceiling, and prefer a strong-read path.
• The 407 auth blip that is neither class. A 407 Proxy Authentication Required or a gateway 502 means the proxy never forwarded your request - that's class 1 (not-sent), even though you got an HTTP response back. Don't let "I received a status code" trick you into the unknown bucket for proxy-side errors.

Disclosure

I write about web data collection for Thordata, a proxy provider, so the sticky-session and gateway-retry points above come from that side of the fence; the failure taxonomy and the harness are mine. If your pool supports per-session exit pinning with a window you can observe, that is what makes the class-2 path tractable - residential sessions here: https://www.thordata.com/?ls=dev&lk=DEV (from $0.65/GB as of 2026-09-22; confirm on the page before you budget against it).