Before the model said a word, an LLM client told the endpoint all of this about my machine:
x-stainless-arch: arm64
x-stainless-async: false
x-stainless-lang: python
x-stainless-os: MacOS
x-stainless-package-version: 3.35.1
x-stainless-read-timeout: 600
x-stainless-retry-count: 0
x-stainless-runtime: CPython
x-stainless-runtime-version: 3.14.7
That block is not from a document. It is from a capture — I put a logging proxy between a real OpenAI client and an endpoint, made one call, and read the wire. Two other things were in the capture: the client silently retried after a 500, resending the entire conversation including the system prompt, and stamped the retry with a header counter. The proxy saw two requests. The application saw one call. Nobody's dashboard shows you that — because the dashboards belong to the party on the other side of the wire.
The rant is right, and the thread proves the demand
Yesterday a post by Jared Siegel made the front page of Hacker News with a title that is its own executive summary: If you don't run your own LLM proxy, how the fuck do you even know whats going on?. The post is a rant with receipts of a different kind — screenshots of his own proxy dashboards — and its sharpest lines are about what the provider's analytics actually answer: "I can't make a decision with this bullshit." About the pretty usage graphs: "That is a distraction. That is not token usage statistics with time series data meant for engineers." His prescription is one sentence: "I'm not going to suggest a proxy provider for you to use, only that you run it yourself somewhere you control."
The discussion under it is the demand in the wild. One commenter built their own agent harness "to have actual full observability of what it is doing and why it is doing what it is doing." Another asked the question the whole thread orbits: "Seeing some good stuff in the comments, but how are people doing this?" The practical answers name real tools — LiteLLM for virtual keys, so "provider API keys never go near the sandbox" — and the skeptical ones land on the same worry from the other side: the proxies themselves are young, and "with the velocity of updates I feel there has to be vulnerabilities there."
What nobody in the thread posted was the smallest thing that answers the question: a capture. Not a dashboard, not a product, not a config guide — the actual bytes. So here is one, runnable on your machine in two commands, no API key, and what it caught surprised even me.
The capture: one file, two servers, zero keys
The setup is one Python file. Two servers: a mock provider that speaks just enough OpenAI to answer a chat completion (and fails the first attempt on purpose — you will see why), and a logging reverse proxy that forwards every byte and keeps a verbatim copy. The client is the real OpenAI SDK — the same package your scripts and agents import — pointed at the proxy instead of the internet.
#!/usr/bin/env python3
"""proxydemo.py — see what actually leaves your machine when you call a model.
Two servers, one file, zero API keys:
:8081 a mock OpenAI-compatible provider (it fails the first attempt on purpose)
:8080 a logging reverse proxy that tees every byte to stdout
The client is the real OpenAI Python SDK, pointed at the proxy.
Run it:
pip install openai
python3 proxydemo.py
"""
import json
import threading
import urllib.error
import urllib.request
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
CAPTURE = []
class MockProvider(BaseHTTPRequestHandler):
"""Pretends to be api.example.com. The first request fails with a 500."""
attempts = 0
def do_POST(self):
self.rfile.read(int(self.headers.get("content-length", 0)))
MockProvider.attempts += 1
if MockProvider.attempts == 1:
self.send_response(500)
self.send_header("content-length", "0")
self.end_headers()
return
out = json.dumps({
"id": "cmpl-mock", "object": "chat.completion", "model": "mock-1",
"choices": [{"index": 0,
"message": {"role": "assistant", "content": "MOCK REPLY"},
"finish_reason": "stop"}],
"usage": {"prompt_tokens": 9, "completion_tokens": 2, "total_tokens": 11},
}).encode()
self.send_response(200)
self.send_header("content-type", "application/json")
self.send_header("content-length", str(len(out)))
self.end_headers()
self.wfile.write(out)
def log_message(self, *a):
pass
class LoggingProxy(BaseHTTPRequestHandler):
"""Forwards to the mock and keeps a verbatim copy of everything."""
def do_POST(self):
body = self.rfile.read(int(self.headers.get("content-length", 0)))
CAPTURE.append({"kind": "request", "method": self.command, "path": self.path,
"headers": dict(self.headers), "body": body.decode()})
fwd = {k: v for k, v in self.headers.items()
if k.lower() not in ("host", "content-length", "connection")}
req = urllib.request.Request("http://127.0.0.1:8081" + self.path,
data=body, headers=fwd, method="POST")
try:
with urllib.request.urlopen(req) as r:
status, rbody, ctype = r.status, r.read(), r.headers.get("content-type", "application/json")
except urllib.error.HTTPError as e: # pass provider errors through, untouched
status, rbody, ctype = e.code, e.read(), e.headers.get("content-type", "text/plain")
CAPTURE.append({"kind": "response", "status": status, "body": rbody.decode()})
self.send_response(status)
self.send_header("content-type", ctype)
self.send_header("content-length", str(len(rbody)))
self.end_headers()
self.wfile.write(rbody)
def log_message(self, *a):
pass
def main():
for srv, port in ((MockProvider, 8081), (LoggingProxy, 8080)):
httpd = ThreadingHTTPServer(("127.0.0.1", port), srv)
threading.Thread(target=httpd.serve_forever, daemon=True).start()
from openai import OpenAI # the real client, pointed at the proxy
client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="sk-demo-not-a-real-key")
reply = client.chat.completions.create(
model="mock-1",
messages=[{"role": "system", "content": "You are a test harness."},
{"role": "user", "content": "Reply with: MOCK REPLY"}])
print("client received:", reply.choices[0].message.content, "\n")
n = 0
for e in CAPTURE:
if e["kind"] == "request":
n += 1
print(f"=================== request #{n} -> provider ({e['method']} {e['path']}) ===================")
for k in sorted(e["headers"]):
print(f" {k}: {e['headers'][k]}")
print(f" body: {json.dumps(json.loads(e['body']))[:220]}")
else:
print(f"------------------- provider answered: {e['status']} -------------------\n")
print(f"the proxy saw {n} requests. your application saw one call.")
if __name__ == "__main__":
main()
I ran it. The output below is verbatim from that run, trimmed only where the JSON repeats:
client received: MOCK REPLY
=================== request #1 -> provider (POST /v1/chat/completions) ===================
Accept-Encoding: gzip, deflate, zstd
Connection: keep-alive
Content-Length: 136
Host: 127.0.0.1:8080
accept: application/json
authorization: Bearer sk-demo-not-a-real-key
content-type: application/json
user-agent: OpenAI/Python 3.35.1
x-stainless-arch: arm64
x-stainless-async: false
x-stainless-lang: python
x-stainless-os: MacOS
x-stainless-package-version: 3.35.1
x-stainless-read-timeout: 600
x-stainless-retry-count: 0
x-stainless-runtime: CPython
x-stainless-runtime-version: 3.14.7
body: {"model": "mock-1", "messages": [{"role": "system", "content": "You are a test harness."}, {"role": "user", "content": "Reply with: MOCK REPLY"}]}
------------------- provider answered: 500 -------------------
=================== request #2 -> provider (POST /v1/chat/completions) ===================
x-stainless-arch: arm64
x-stainless-os: MacOS
x-stainless-package-version: 3.35.1
x-stainless-read-timeout: 600
x-stainless-retry-count: 1
x-stainless-runtime-version: 3.14.7
body: {"model": "mock-1", "messages": [{"role": "system", "content": "You are a test harness."}, {"role": "user", "content": "Reply with: MOCK REPLY"}]}
------------------- provider answered: 200 -------------------
the proxy saw 2 requests. your application saw one call.
I ran it twice and diffed: the captures are byte-identical. This is not an anecdote, it is an instrument — anyone with Python can produce the same two requests on their own machine in under a minute.
What the capture actually shows
The machine inventory, volunteered on every call. The x-stainless-* block is the SDK's telemetry: your operating system, your CPU architecture, your runtime and its exact version, the package version, whether the call was async, the read timeout. It is not hidden — it is on the wire, which is different: it is invisible to anyone who does not capture. Your client is a sensor reporting on your environment, and the sensor's dashboard is on their side, not yours.
The silent retry. The mock fails the first attempt with a 500, and the SDK does what SDKs do: it retried, resending the identical conversation — system prompt included — and stamped the second attempt with x-stainless-retry-count: 1. Your application called the API once. The conversation crossed the wire twice. On the provider's side this can look like one successful call and some noise; on your side, without a capture, it looks like nothing at all. Every "why did my token bill double" question starts here.
Everything is readable on the path. The system prompt you consider private, the whole conversation, and the bearer token all travel as plaintext JSON between your process and the endpoint — through every proxy, gateway, and middleware that sits between them anyway. The question is never whether intermediaries can read it. It is whether the one who keeps the log is you.
The rung this sits on
There is a pattern underneath, and the authority ladder this series has been building names it: evidence you merely get to see is a caller label; evidence that lives on infrastructure you control is the floor. The provider's usage dashboard is their receipt about you. Jared's one-line prescription — run it yourself somewhere you control — is the same move as pinning a registrar or keeping your own journal: the log moves to the side of the party who needs to audit. A proxy you run is not a better dashboard. It is your dashboard.
What to steal
- The provider's dashboard answers their questions, not yours. Token graphs are marketing-shaped. A capture answers engineering questions: what left, how many times, with what attached.
- Virtual keys are the grown-up version of this demo. LiteLLM (61k+ stars) sits between your code and every provider, issues per-agent keys, and keeps the real ones out of your sandboxes — the thread's most repeated piece of advice, and it is the same proxy position with more features.
- Any control that touches your traffic should log, and the log should be yours. A gate that emits to someone else's analytics is a control you cannot audit — the bypass-without-a-log problem from yesterday's Sphinx debate, one layer down.
Two honest limits
The provider side of this demo is a mock, because there is no API key on this machine. Everything the client emits — headers, retry, plaintext conversation — is real and captured verbatim; the model's answer is canned. The mock is also the point: the capture works identically against the real endpoint, and the two commands below let anyone with a key produce the real version in minutes.
A 90-line stdlib proxy is an instrument, not infrastructure. It speaks plain HTTP; real agents speak TLS, and for those the tool is mitmproxy (45k+ stars) with your own CA — same receipts, one config file away. I did not run a frontier coding agent against this capture; the reader who does will find more headers than I did, and that finding belongs in the comments.
Your turn
Run the two commands — pip install openai, save the script, python3 proxydemo.py — and read your own capture. Then point a real agent at the same proxy (OPENAI_BASE_URL is one environment variable on most tools) and count three things: how many requests one user action produces, what headers your client volunteers about your machine, and how many retries your application never mentioned. Post the three numbers. That is the state of your agent's traffic, measured instead of assumed.
Top comments (7)
We watched the same retry from the far end of the wire. Our API is metered per returned row, and one morning an agent ran it 103 times in parallel: the server log showed 35,342 responses with status 429, the client's own log showed one "rate limited, retrying" line per run. Every retry resent the full request body, so the count on our side and the count on the app side differed by two orders of magnitude. A retry-count header like the one in your capture is the only thing that lets an access log tell a retry from new work, and most clients send nothing of the kind. I wrote the server-side capture up in my profile posts, if the mirror image is useful. One question: did the proxy see an idempotency key on the retried request? That field decides whether the second attempt is a duplicate charge or a free one.
The server-side capture I mentioned, with the 429 counts per minute and what the client log showed for the same window here: dev.to/kevinhall/what-a-metered-da...
Read it whole — and the lost-rows finding is the sharpest thing on either side of the wire this week. "A row that looks like
no numberand never know the question was not asked" is the not-stated-versus-measured-zero failure wearing a metering API:errorandno_phoneserialize identically, so a refusal masquerades as a fact. Your backoff fix names it correctly — a storm should be slow, not lossy — and it's the same law the receipt threads keep landing on: an undecidable outcome reads as a named state, never as a plausible one. Two rows in a hundred wearing the wrong label is a small number with a long blast radius downstream, because every consumer of that dataset inherits the lie.And your closing ask deserves an answer from every side of the wire. We can't post a worst-day refusal share — the receipt layer isn't a metered API — but the proxy capture now exists on this end, your access log exists on yours, and the two together are the bilateral record: what left, what arrived, and the delta that is the retries. Your four-line harness rule — sample first, stop on the hit rate, no parallel runs, one run per list — is the buyer-side contract; the idempotency-key requirement is the seller-side one. Both halves published is how a morning like 9 October becomes measurable from both ends instead of invisible from both.
Re-ran the capture tonight rather than answer from memory — same SDK version, fresh run, full headers this time: no idempotency key on either attempt. The retry is a byte-identical duplicate POST; the only byte that differs between the two requests is the retry-count header itself. So on a row-metered API, the answer to your question is the expensive one: the second attempt is a duplicate charge by default — the client sends nothing to key it on, and the chat-completion contract offers nothing to key it with. The idempotency convention that payments learned the hard way never made it to this wire.
Which puts the defense on your side of the fence: on a row-metered API, the idempotency key has to be a server requirement, not a client courtesy — required on metered endpoints, echoed in the response, duplicates free or billed once. Your 35,342 429s were the server paying for the client's silence: every retry resent the full body, and the meter counted rows the application never asked for twice. An API that prices per row without requiring a dedup key is volunteering to absorb its clients' retry storms.
And your capture is the other half of mine: the proxy counts what left the client, the access log counts what arrived at the server, and the delta between the two streams IS the retries — invisible from either side alone. One log line against 35,342 server responses is the reconciliation gap standing in the wild. If your writeup names the fields your access log would need from clients — a retry marker, a request identity, anything durable — post them: that's the contract the client side should be shipping, and the two captures together are the whole picture.
The silent retry is the nasty part: the wire saw two requests, the app saw one call, and the conversation - system prompt included - crossed twice. x-stainless-retry-count means the provider can attribute the second call to a retry, but nothing surfaces it on your side. Same blind spot as egress bills: the party metering the traffic is the one who benefits from it.
Agreed — the counter is the sensor and the enabled button is its costume: "looks clickable" is what the DOM claims, "the route changed and the counter moved" is what the app did. Making success observable turns the fixture from a stopwatch into a verdict. And logging missed and duplicate actions beside the readiness time is the metric that actually matters: a fast return that costs the agent three duplicate clicks is slower in the only clock that counts — wall-clock-to-task-done. The cross-engine table then reads itself: readiness time, misses, duplicates — three columns, and whichever engine wins has to win at keeping the agent's hands off the retry button.
The silent retry is the finding worth the run. We log at the gateway for exactly this: the app sees one call and the wire shows two, and without your own capture you bill usage you never knew happened.