The change went out at 12:00 on a Monday. Nothing paged. Our spend-rate alert fires at 3x the trailing hourly median, which is built for a tenant running away with the bill, and this was a steady 50 percent above normal. It surfaced at 16:36 because somebody was reading a per-model cost breakdown for an unrelated reason. We reverted and the deploy was green four minutes later.
Our normal token spend is about $1,420 a day. Monday finished about $480 over. Tuesday came in $674 over. Wednesday was $136 over. It was flat again on Thursday, and only because we shipped something else on Wednesday night, not because of the revert.
Nothing was broken. The revert did exactly what a revert does. My mistake was in what I thought that guaranteed.
The single largest term was not in the repository at all, so git revert was never going to touch it. That is the part I want to spend this post on, because it is the one I did not have a name for and now check first.
The rule that was not in the deploy
The Monday change was two things shipped together. A prompt edit, which added a few examples to a system prompt. And a routing change, moving one class of requests to a larger model.
The prompt lives in the repository. The routing rule does not. It lives in a runtime config store, so we can change routing without a deploy, which is a feature we asked for and use constantly. Steady-state we run several deliberate non-default routes.
So the revert took back the prompt and left the routing rule pointing at the larger model.
That class is about 6.3 percent of our calls, and the larger model is roughly 7.4 times the unit price, so the class goes from about $89 a day to about $660: call it $570 a day extra while the rule is live. (I am treating its share of calls and its share of spend as the same, which flatters me slightly, since a class routed to a bigger model is usually the heavier work.) It stayed live for another 19 hours, until someone looking at a per-model cost breakdown for an unrelated reason asked why that route was still there. Across the whole incident that rule cost about $562, more than any other single term, and just the Tuesday portion of it is $277, which is 41 percent of Tuesday's overage on its own.
I had confirmed the diff. I had watched the deploy. Neither of those was ever going to show me a rule that lives in a different system.
Three smaller terms, and why they mattered more than their size
The other $397 of Tuesday is three things, and I list them because each one had already committed work that the revert could not recall.
The queue was already full. The same deploy kicked off a re-summarisation pass over documents touched in the previous 30 days, because the new prompt changed the summary format and we wanted consistency. It enqueued its whole work list, 61,000 documents, in the first few minutes. Our summarisation workers are a separate deployment from the API, and the revert pipeline does not roll them, so they kept running with the new prompt held in memory. The pass cost $336, of which $214 landed on Tuesday.
Those summaries are an input cost now. This is the term I had never thought about. The new format produced longer summaries, roughly 520 tokens against the 180 we had before. Those summaries are stored, and they get retrieved as context by downstream requests. So 61,000 documents now carry an extra 340 tokens each into every request that touches them, and about 142,000 retrievals a day land on one of those documents, because the recent set is what people actually work on. That is roughly $145 a day. A revert does not shorten text that is already written to a table. This term had no expiry at all: it was going to keep billing until somebody re-summarised those documents, which is what we shipped on Wednesday night. Left alone it would have been about 10 percent of our token spend in perpetuity, near $53,000 a year, which is a much larger number than the Tuesday everybody was looking at.
A dead-letter queue kept firing the old prompt. Failed calls go to a dead-letter queue with a scheduled re-drive that runs every six hours for two days. The payload we persist includes the rendered prompt rather than a pointer to a prompt version, which is a reasonable choice for reproducibility and means a re-drive fires the expensive prompt long after the revert. $38 on Tuesday, $9 on Wednesday. Trivial money. It mattered because a thin tail of expensive calls kept the graph from going cleanly flat, and for most of Tuesday afternoon I read that as the revert not having worked, which sent me looking in the wrong place.
Wednesday is the detail I would want a reader to take away. It was 10 percent over baseline, and of the two terms still running, the one that mattered had no expiry at all. Nobody escalates a 10 percent day, which is why this one ran until Wednesday night.
Reversibility has to be designed in
Here is the question I now ask before shipping anything that touches prompts, models or routing. Not "can we roll this back", because the answer is always yes and it is not the useful question. Instead: what will this change write, enqueue, or set outside the repository, and how long does each of those live?
That has a concrete answer every time, and it is usually a number of hours.
For any such change I write down four things:
- What it writes, with a lifetime. Stored summaries, tags, embeddings, conversation state. Anything with no lifetime, like our summaries, gets a corrective backfill planned before the change ships, not after.
- What it enqueues, and how long that queue takes to drain if we stop feeding it. Including which deployment owns the consumers, because that is what decides whether a revert stops them.
- What lives outside the deploy. Feature flags, routing rules, model aliases, rate-limit tiers. This is the list I did not have.
- The longest of the above, in the change ticket, instead of "revertible: yes".
For that Monday an honest entry would have read: revertible in minutes for the prompt, 19 hours for the routing rule if nobody checks it, about a day for the queue, and open-ended for the stored summaries. Writing that down would have changed how we shipped it. Probably behind a percentage rollout, which caps every one of those terms proportionally, including the one with no expiry.
The cheap version of the fix
Two things, both small.
The first is stamping the version into everything the pipeline writes and enqueues. Not because stamping makes the revert clean, it does not, but so that afterwards I can answer how much of what is being served or queued came from the version I just removed. Before this I had no way to answer that except by watching the spend graph and guessing, which is how Tuesday afternoon went.
import hashlib
cache_key = f"{hashlib.sha256(normalised_request.encode()).hexdigest()}:{prompt_version}"
retry_payload["prompt_version"] = prompt_version
summary_row["written_by_prompt_version"] = prompt_version
hashlib rather than the builtin hash() matters here: hash() on a string is salted per process, so it is silently wrong for anything shared across workers or restarts.
Then the query that tells you whether the revert actually landed, which is the one I did not have:
-- how much of what we are still serving came from the version we removed?
-- No time window on purpose: derived rows have no TTL, so a trailing-7-day
-- filter goes blind on exactly the thing this query exists to find.
SELECT written_by_prompt_version,
count(*) AS rows_total,
sum(summary_tokens) AS tokens_carried,
max(written_at) AS last_written
FROM document_summaries
GROUP BY 1
ORDER BY tokens_carried DESC;
If last_written for the reverted version is later than your revert timestamp, something is still running. That one column would have caught the workers on Monday evening.
The second is an inventory of every LLM-affecting setting that is not in the repository, with an owner. Ours had eleven entries when we wrote it down, about eight more than I would have guessed. Three of them can change which model serves a request.
What I'd page on
Roughly in the order they would have helped that Monday:
- Live routing config diverging from a declared snapshot. Check the running config against a snapshot committed to the repository, and alert on any rule that has been live longer than four hours without a matching commit. This is the one that catches the $277. Note that alerting on "traffic to a non-default model" would be useless for us, because deliberate non-default routes are our steady state; the divergence from a declared expectation is the signal, not the routing itself.
- Any row written by a prompt version that is no longer deployed. A single counter off the query above. This is the alert that would have told me the workers were still going while I was staring at the graph.
- Stored-context token volume per retrieval, as a trend. We already watch average input tokens per request, and this is the same signal from the artifact side rather than the request side. The term with no expiry was invisible in every cost view we had, because it does not show up as a spike. It shows up as slightly worse unit economics forever. Watch the size of what you retrieve, not only the count.
- Queue depth on any job a deploy can trigger, weighted by mean cost per item rather than depth alone. Six hundred expensive items and sixty thousand cheap ones want different responses, and in this incident the sixty thousand were the expensive ones.
- Age of the oldest item in the dead-letter queue. A re-drive carrying a rendered prompt is invisible in every other view, and it is the thing most likely to make you distrust a revert that worked.
We already page on spend rate against the same hour last week. That alert is not on this list, and not because it did its job. An alert tuned for a tenant spiking 70x does not fire on a routing change worth 50 percent, and a human reading an unrelated breakdown beat it. Everything above exists because of the four and a half hours that cost us, and the two days after the revert that cost more.
The thing I got wrong was smaller than any of this. I had a rollback plan, and I had never asked which parts of the system it covered.
Top comments (0)