The machine inventory point is the one to underline for clients. Most teams review what they type into a chatbot and never review what the client sends before the first word: OS, paths, env names, sometimes key material sitting in the environment. A useful habit for any agency running LLM tools on client work: run the proxy once per tool, keep the capture, and add "what does this client volunteer about our machine" to the same checklist as "what do we paste into it". The second list is where the surprises live.
Senior Lead Tech/Dev, 20+ years. Building NoireBox — the flight recorder for your of AI agents. Proof, not promises. Rust · TS · Python · PHP. I write about trust in AI.
Both habits are the right ones, and they compound: the capture you keep from running the proxy once per tool is an engagement-opening receipt — "here is what this tool volunteered about our machine, on record before any work started" — which answers the client question nobody thinks to ask until the incident. And the spend-alert point is the buyer-side floor of the metering thread running this week: the seller's contract can require idempotency keys and per-run identities, but the ceiling only exists when the buyer carries one too — that thread's seller-side fields and your $5 alert are the two halves. The checklist line is the post's thesis in agency clothing: the paste-list is what you know you told it; the volunteer-list is what it said about you. Both belong in writing, and only one of them existed before the proxy.
The two-halves framing is the keeper: keys prevent, alerts detect, and they still leave a gap - the duplicate charge neither side noticed. That is where the capture habit earns its keep a second time: it is what gives the buyer standing in the refund conversation. "Your wire shows two POSTs" is a negotiation. "Here is our proxy capture, run before the engagement started, showing one call from our side" is a short email. Prevention belongs to the seller, detection to the buyer, and proof to whoever kept the receipt - which is why the receipt habit is the one piece neither side can outsource to the other.
GMT strategy and custom B2B outreach lists on request: people and companies by ICP, domain, profile URL, email or name. 1B+ profiles. Claude Code plugins, API, MCP. kevin@outricher.com
We watched the same retry from the far end of the wire. Our API is metered per returned row, and one morning an agent ran it 103 times in parallel: the server log showed 35,342 responses with status 429, the client's own log showed one "rate limited, retrying" line per run. Every retry resent the full request body, so the count on our side and the count on the app side differed by two orders of magnitude. A retry-count header like the one in your capture is the only thing that lets an access log tell a retry from new work, and most clients send nothing of the kind. I wrote the server-side capture up in my profile posts, if the mirror image is useful. One question: did the proxy see an idempotency key on the retried request? That field decides whether the second attempt is a duplicate charge or a free one.
GMT strategy and custom B2B outreach lists on request: people and companies by ICP, domain, profile URL, email or name. 1B+ profiles. Claude Code plugins, API, MCP. kevin@outricher.com
The server-side capture I mentioned, with the 429 counts per minute and what the client log showed for the same window here: dev.to/kevinhall/what-a-metered-da...
Senior Lead Tech/Dev, 20+ years. Building NoireBox — the flight recorder for your of AI agents. Proof, not promises. Rust · TS · Python · PHP. I write about trust in AI.
Read it whole — and the lost-rows finding is the sharpest thing on either side of the wire this week. "A row that looks like no number and never know the question was not asked" is the not-stated-versus-measured-zero failure wearing a metering API: error and no_phone serialize identically, so a refusal masquerades as a fact. Your backoff fix names it correctly — a storm should be slow, not lossy — and it's the same law the receipt threads keep landing on: an undecidable outcome reads as a named state, never as a plausible one. Two rows in a hundred wearing the wrong label is a small number with a long blast radius downstream, because every consumer of that dataset inherits the lie.
And your closing ask deserves an answer from every side of the wire. We can't post a worst-day refusal share — the receipt layer isn't a metered API — but the proxy capture now exists on this end, your access log exists on yours, and the two together are the bilateral record: what left, what arrived, and the delta that is the retries. Your four-line harness rule — sample first, stop on the hit rate, no parallel runs, one run per list — is the buyer-side contract; the idempotency-key requirement is the seller-side one. Both halves published is how a morning like 9 October becomes measurable from both ends instead of invisible from both.
GMT strategy and custom B2B outreach lists on request: people and companies by ICP, domain, profile URL, email or name. 1B+ profiles. Claude Code plugins, API, MCP. kevin@outricher.com
Here are the fields, since you asked for the contract. Per request the access log keeps four client-supplied values: X-Client-Name, X-Client-Instance (the run), X-Client-Version (the buyer), and Idempotency-Key, which becomes the request id and is echoed back as X-Request-Id. A retry with the same key re-bills nothing; the same key with a different payload is refused with IDEMPOTENCY_CONFLICT instead of silently replayed. The server half of a 429 is a Retry-After in seconds. Where I agree with you most: the key has to be required on metered endpoints, and on 9 October it was optional, which is why the meter counted what the client's silence sent. On error and no_phone serialising identically: that is fixed in the row contract, every empty row now carries its own status, and I counted the five kinds over 30 days today: dev.to/kevinhall/four-kinds-of-emp...
Senior Lead Tech/Dev, 20+ years. Building NoireBox — the flight recorder for your of AI agents. Proof, not promises. Rust · TS · Python · PHP. I write about trust in AI.
That contract is the real deliverable of the thread, and the load-bearing line is IDEMPOTENCY_CONFLICT: same key, different payload → refused, never silently replayed. The common shortcut is "same key → replay whatever the first attempt did," which is exactly the hole a mutated retry walks through — your version turns the fingerprint mismatch into a named error, which makes the key a receipt instead of a cache tag. Required, echoed, conflict-refused: the three words most metered endpoints are missing.
And the five-kinds post is the vocabulary at production scale: 310,550 lookups, five named empties with distinct latency signatures and exactly one retry that pays. "The harness that treats them all the same is making a mistake on four out of five requests" is the not-stated-versus-measured-zero law with a routing table. The bilateral record is now complete — the client capture on this end, the access log and the field contract on yours — which is more than most standards can say about themselves.
Senior Lead Tech/Dev, 20+ years. Building NoireBox — the flight recorder for your of AI agents. Proof, not promises. Rust · TS · Python · PHP. I write about trust in AI.
Re-ran the capture tonight rather than answer from memory — same SDK version, fresh run, full headers this time: no idempotency key on either attempt. The retry is a byte-identical duplicate POST; the only byte that differs between the two requests is the retry-count header itself. So on a row-metered API, the answer to your question is the expensive one: the second attempt is a duplicate charge by default — the client sends nothing to key it on, and the chat-completion contract offers nothing to key it with. The idempotency convention that payments learned the hard way never made it to this wire.
Which puts the defense on your side of the fence: on a row-metered API, the idempotency key has to be a server requirement, not a client courtesy — required on metered endpoints, echoed in the response, duplicates free or billed once. Your 35,342 429s were the server paying for the client's silence: every retry resent the full body, and the meter counted rows the application never asked for twice. An API that prices per row without requiring a dedup key is volunteering to absorb its clients' retry storms.
And your capture is the other half of mine: the proxy counts what left the client, the access log counts what arrived at the server, and the delta between the two streams IS the retries — invisible from either side alone. One log line against 35,342 server responses is the reconciliation gap standing in the wild. If your writeup names the fields your access log would need from clients — a retry marker, a request identity, anything durable — post them: that's the contract the client side should be shipping, and the two captures together are the whole picture.
The silent retry is the nasty part: the wire saw two requests, the app saw one call, and the conversation - system prompt included - crossed twice. x-stainless-retry-count means the provider can attribute the second call to a retry, but nothing surfaces it on your side. Same blind spot as egress bills: the party metering the traffic is the one who benefits from it.
Senior Lead Tech/Dev, 20+ years. Building NoireBox — the flight recorder for your of AI agents. Proof, not promises. Rust · TS · Python · PHP. I write about trust in AI.
Agreed — the counter is the sensor and the enabled button is its costume: "looks clickable" is what the DOM claims, "the route changed and the counter moved" is what the app did. Making success observable turns the fixture from a stopwatch into a verdict. And logging missed and duplicate actions beside the readiness time is the metric that actually matters: a fast return that costs the agent three duplicate clicks is slower in the only clock that counts — wall-clock-to-task-done. The cross-engine table then reads itself: readiness time, misses, duplicates — three columns, and whichever engine wins has to win at keeping the agent's hands off the retry button.
Senior Cloud and AI Engineer in Dallas, TX. 8+ years across Azure, GCP, and AWS. Leading API architecture currently. I write about LLM platforms, agentic AI, and production ML systems.
The silent retry is the finding worth the run. We log at the gateway for exactly this: the app sees one call and the wire shows two, and without your own capture you bill usage you never knew happened.
We're a place where coders share, stay up-to-date and grow their careers.
The machine inventory point is the one to underline for clients. Most teams review what they type into a chatbot and never review what the client sends before the first word: OS, paths, env names, sometimes key material sitting in the environment. A useful habit for any agency running LLM tools on client work: run the proxy once per tool, keep the capture, and add "what does this client volunteer about our machine" to the same checklist as "what do we paste into it". The second list is where the surprises live.
Both habits are the right ones, and they compound: the capture you keep from running the proxy once per tool is an engagement-opening receipt — "here is what this tool volunteered about our machine, on record before any work started" — which answers the client question nobody thinks to ask until the incident. And the spend-alert point is the buyer-side floor of the metering thread running this week: the seller's contract can require idempotency keys and per-run identities, but the ceiling only exists when the buyer carries one too — that thread's seller-side fields and your $5 alert are the two halves. The checklist line is the post's thesis in agency clothing: the paste-list is what you know you told it; the volunteer-list is what it said about you. Both belong in writing, and only one of them existed before the proxy.
The two-halves framing is the keeper: keys prevent, alerts detect, and they still leave a gap - the duplicate charge neither side noticed. That is where the capture habit earns its keep a second time: it is what gives the buyer standing in the refund conversation. "Your wire shows two POSTs" is a negotiation. "Here is our proxy capture, run before the engagement started, showing one call from our side" is a short email. Prevention belongs to the seller, detection to the buyer, and proof to whoever kept the receipt - which is why the receipt habit is the one piece neither side can outsource to the other.
We watched the same retry from the far end of the wire. Our API is metered per returned row, and one morning an agent ran it 103 times in parallel: the server log showed 35,342 responses with status 429, the client's own log showed one "rate limited, retrying" line per run. Every retry resent the full request body, so the count on our side and the count on the app side differed by two orders of magnitude. A retry-count header like the one in your capture is the only thing that lets an access log tell a retry from new work, and most clients send nothing of the kind. I wrote the server-side capture up in my profile posts, if the mirror image is useful. One question: did the proxy see an idempotency key on the retried request? That field decides whether the second attempt is a duplicate charge or a free one.
The server-side capture I mentioned, with the 429 counts per minute and what the client log showed for the same window here: dev.to/kevinhall/what-a-metered-da...
Read it whole — and the lost-rows finding is the sharpest thing on either side of the wire this week. "A row that looks like
no numberand never know the question was not asked" is the not-stated-versus-measured-zero failure wearing a metering API:errorandno_phoneserialize identically, so a refusal masquerades as a fact. Your backoff fix names it correctly — a storm should be slow, not lossy — and it's the same law the receipt threads keep landing on: an undecidable outcome reads as a named state, never as a plausible one. Two rows in a hundred wearing the wrong label is a small number with a long blast radius downstream, because every consumer of that dataset inherits the lie.And your closing ask deserves an answer from every side of the wire. We can't post a worst-day refusal share — the receipt layer isn't a metered API — but the proxy capture now exists on this end, your access log exists on yours, and the two together are the bilateral record: what left, what arrived, and the delta that is the retries. Your four-line harness rule — sample first, stop on the hit rate, no parallel runs, one run per list — is the buyer-side contract; the idempotency-key requirement is the seller-side one. Both halves published is how a morning like 9 October becomes measurable from both ends instead of invisible from both.
Here are the fields, since you asked for the contract. Per request the access log keeps four client-supplied values: X-Client-Name, X-Client-Instance (the run), X-Client-Version (the buyer), and Idempotency-Key, which becomes the request id and is echoed back as X-Request-Id. A retry with the same key re-bills nothing; the same key with a different payload is refused with IDEMPOTENCY_CONFLICT instead of silently replayed. The server half of a 429 is a Retry-After in seconds. Where I agree with you most: the key has to be required on metered endpoints, and on 9 October it was optional, which is why the meter counted what the client's silence sent. On error and no_phone serialising identically: that is fixed in the row contract, every empty row now carries its own status, and I counted the five kinds over 30 days today: dev.to/kevinhall/four-kinds-of-emp...
That contract is the real deliverable of the thread, and the load-bearing line is
IDEMPOTENCY_CONFLICT: same key, different payload → refused, never silently replayed. The common shortcut is "same key → replay whatever the first attempt did," which is exactly the hole a mutated retry walks through — your version turns the fingerprint mismatch into a named error, which makes the key a receipt instead of a cache tag. Required, echoed, conflict-refused: the three words most metered endpoints are missing.And the five-kinds post is the vocabulary at production scale: 310,550 lookups, five named empties with distinct latency signatures and exactly one retry that pays. "The harness that treats them all the same is making a mistake on four out of five requests" is the not-stated-versus-measured-zero law with a routing table. The bilateral record is now complete — the client capture on this end, the access log and the field contract on yours — which is more than most standards can say about themselves.
Re-ran the capture tonight rather than answer from memory — same SDK version, fresh run, full headers this time: no idempotency key on either attempt. The retry is a byte-identical duplicate POST; the only byte that differs between the two requests is the retry-count header itself. So on a row-metered API, the answer to your question is the expensive one: the second attempt is a duplicate charge by default — the client sends nothing to key it on, and the chat-completion contract offers nothing to key it with. The idempotency convention that payments learned the hard way never made it to this wire.
Which puts the defense on your side of the fence: on a row-metered API, the idempotency key has to be a server requirement, not a client courtesy — required on metered endpoints, echoed in the response, duplicates free or billed once. Your 35,342 429s were the server paying for the client's silence: every retry resent the full body, and the meter counted rows the application never asked for twice. An API that prices per row without requiring a dedup key is volunteering to absorb its clients' retry storms.
And your capture is the other half of mine: the proxy counts what left the client, the access log counts what arrived at the server, and the delta between the two streams IS the retries — invisible from either side alone. One log line against 35,342 server responses is the reconciliation gap standing in the wild. If your writeup names the fields your access log would need from clients — a retry marker, a request identity, anything durable — post them: that's the contract the client side should be shipping, and the two captures together are the whole picture.
The silent retry is the nasty part: the wire saw two requests, the app saw one call, and the conversation - system prompt included - crossed twice. x-stainless-retry-count means the provider can attribute the second call to a retry, but nothing surfaces it on your side. Same blind spot as egress bills: the party metering the traffic is the one who benefits from it.
Agreed — the counter is the sensor and the enabled button is its costume: "looks clickable" is what the DOM claims, "the route changed and the counter moved" is what the app did. Making success observable turns the fixture from a stopwatch into a verdict. And logging missed and duplicate actions beside the readiness time is the metric that actually matters: a fast return that costs the agent three duplicate clicks is slower in the only clock that counts — wall-clock-to-task-done. The cross-engine table then reads itself: readiness time, misses, duplicates — three columns, and whichever engine wins has to win at keeping the agent's hands off the retry button.
The silent retry is the finding worth the run. We log at the gateway for exactly this: the app sees one call and the wire shows two, and without your own capture you bill usage you never knew happened.