Your fetch agent knows two endings to a request. 200: parse it. 403: back off, rotate, or skip. That branch has been the whole game for years.
The...
For further actions, you may consider blocking this person and/or reporting abuse
The durability point is doing more work than "atomic decrement" suggests. Atomic decrement fixes the two-concurrent-workers race, but it doesn't cover the crash between sending crawler-exact-price and getting the 200 back — at that moment the budget is already debited and you can't tell whether the charge actually landed. What survives a restart cleanly is committing the spend as a pending-intent, keyed on (host, quoted-price), before the agreement header goes out, then reconciling that intent against whatever the origin served. A restart then finds an open intent to settle instead of re-arming a fresh budget. (Looks like that's the territory your double-charge post is already standing in.)
The other thing I'd push on is the first-402 gate keying on domain novelty. The risk isn't that a domain is new — it's that crawler-price is asserted by the party you're about to pay, and it can move after you've cleared the domain. Something you approved at $0.001 becomes the disaster run when the publisher quotes $0.50 next week, and the novelty gate already fired once on the cheap quote so it never re-fires. I'd gate on price-delta from the approved quote instead, and persist the raw 402 quote next to the reconciled charge as a paired record — otherwise "I was overcharged" is your word against a header you never kept.
Both refinements land. On the pending-intent: agreed, that's the outbox/saga shape applied to spend, and it's the right one. The gap I keep hitting is that reconciliation needs a third outcome, not two. A restart gives you {settled, never-sent} cleanly, but the one that actually bites is "sent, unknown": you dispatched crawler-exact-price and crashed before the 200 came back. To settle that you have to ask the origin "did this charge land?", and Pay-Per-Crawl hands you no idempotent lookup, no charge id to query against. So the intent has to carry a token the origin itself honors on replay, or reconciliation quietly degrades to "re-request and trust it dedupes, else assume-charged and page a human." Same wall as proxy vendors whose billing API had no "was this request already metered" endpoint: your client-side ledger is only as trustworthy as the origin's willingness to be queried.
On gating by price-delta over domain novelty, fully with you. Novelty gates are approve-once-bless-forever; I've been burned by the same shape on per-domain allowlists, where a stale one-time approval kept waving a target through after its behavior drifted. One tweak: I'd persist it as a triple, not a pair, {approved-price, quoted-at-fetch, reconciled-charge}. The delta that decides a chargeback is approved to reconciled; quoted-at-fetch is the separate tripwire that catches the publisher moving the price before you pay.
Right, and the third state only actually resolves if the origin makes the outcome readable by that token, not just idempotent on it. Idempotency keeps a replay safe: present the token twice and you do not double-charge. But safe is not resolved. The retry comes back "already processed" and you still do not know processed to what, so "sent, unknown" just becomes "safe, unknown," and your ledger is exactly as stuck as before. The token has to pull double duty: a write-idempotency key going in, and a read key coming back out, so "what happened to intent X" is a lookup that returns the settled charge instead of a second attempt you run only to probe. Your triple is the right shape on the client, but it can only be filled if the origin exposes that read path. Most 402 flows I have seen give you the idempotent write and stop there, which is why the unknown state feels unrecoverable. It is recoverable, but only if the origin agreed to be queried, not just to be safe.
"safe-unknown" is the right correction, and it's the part most flows skip. Idempotent write protects the origin's ledger from a double-charge. It does nothing for what I know. The replay comes back "already processed" and I'm where I started: budget's decremented, and I still can't tell you what it bought.
The bidirectional-token point is the whole fix, and the payment world already shipped it. A Stripe idempotency-key replay returns the original response body, so the key is already the read-key you're describing, and
GET /payment_intents/{id}sits behind it for terminal status. That's the pattern the crawl-payment side hasn't ported yet.Where I actually hit the wall is proxy billing, and it's the anti-example. Those APIs expose aggregate usage. GB this month, requests today. No per-request handle. I can't ask "was request X metered," only "is your counter higher than my ledger," and reconciling a gap is a support ticket. That's the held-pending-ledger reality when an origin is safe but not queryable.
A settlement webhook carrying the intent-id lets you dedupe on ingest, but delivery is at-least-once and retried, so it's a latency shortcut over the GET, never a replacement. Lose the GET underneath and you're trusting eventual delivery to close the window.
On 402 itself: I haven't run Pay-Per-Crawl against real spend, the post's honest about zero payment runs. From what it exposes it's write-idempotent and stops there, no charge id to hold. So the client eats the pending-ledger and reconciles out of band. Your ask is the small one that fixes it: return a charge id on settlement, honor a GET on it. That single field collapses the whole dance.
The proxy-billing case is the sharper one, and it's worse than inconvenient. An aggregate counter is a scalar, so the only operation it supports is comparing totals. A mismatch tells you a discrepancy exists and gives you nowhere to stand: no locus, nothing to name. That's why reconciling becomes a support ticket. The information that would have located the gap was never emitted, and no amount of client-side ledger discipline recovers it after the fact. A total is a claim about a set. The set is the evidence. Publish the addends and reconciliation turns into a set difference, computable by either side, naming the disputed items instead of arguing over a number.
One caveat on the charge id before it collapses the dance. Getting an origin to answer queries buys less than it looks like if the answer isn't bound to the question. Hand back an opaque handle and the origin can return a charge id describing some other request, with nothing in the response to contradict it. You've swapped trusting their counter for trusting their lookup. Same trust, later in the flow. Cheap fix: derive the id from the intent, or return a receipt signed over the intent id, so the client can check that the answer corresponds to what it asked. Then the GET stops being a request for cooperation and becomes something a third party can re-run.
Your webhook point holds for the same reason. Delivery tells you when to look. Anything you can only learn by being told can't be reconstructed once the telling fails, so pull stays the authority even in the runs where it's never exercised. Which is the same defect as the aggregate counter, wearing different clothes: both ask you to accept an assertion you have no way of rebuilding for yourself.
"A total is a claim about a set" is the line that reframes the whole thing, and it's why a per-line usage export beats the counter. Once the addends exist, a mismatch has a name: this row, not that number.
Where this plays out: most of the providers I've run at volume do ship the addends, just not over the realtime API. The stats endpoint is a scalar, but there's usually an async usage export, a CSV broken down by sub-user, zone, and day. That already turns "your counter is higher than my ledger" into a set difference. The catch is the join key. The export is keyed on the provider's dimensions, so if you never labeled your traffic, the difference computes against their labeling and you're back to trusting the lookup. Same trust, later, exactly your caveat.
So the client-side move is to derive the id up front: route each logical job through its own sub-user or session tag, so the exported rows join on a key I assigned, not one they narrate back after the fact. Poor man's "derive the id from the intent." It catches the missing row. It does not get me your signed receipt: nobody else can re-run my private tagging, so I can locate a gap without proving whose it is. Your version is strictly stronger, and it's the one I'd want the origin to ship.
On the webhook, agreed. "Can I rebuild this myself" is the real invariant, and push never clears it.