I wrote about our AI pipeline costs a while back. The comments were better than the post.
Valentin Monteiro made the point that cache alerts should be keyed per prompt version rather than on a global ratio, because a global number moves for boring reasons. Tae Kim pointed at how prompt caching fails silently, where a breakpoint on anything per-request gives you a miss on every call that logs exactly like a hit. Both need the same thing underneath, and I said I'd get cache metrics into our provenance records.
That turned into a bigger job than I expected. Here's what came out of it, what it costs, and the two parts I still haven't worked out.
Why store more than the output?
Because the output can't tell you whether something was always broken or broke last Tuesday, whether it's one record or ten thousand, or whether you changed something or the vendor did.
A total_amount column has the value. Nothing about where it came from.
Where do you record it?
At the provider, not in the feature.
I did it in the feature first. It worked fine. But then every new AI feature has to remember to do the same thing, and eventually one won't. So it moved down to where providers get built:
function withProvenance(provider, meta) {
return async (request) => {
const startedAt = performance.now()
try {
const response = await provider(request)
await recordAiCall(buildAiCallPayload({ meta, request, response, startedAt }))
return response
} catch (error) {
await recordAiCall(buildAiCallPayload({ meta, request, error, startedAt }))
throw error
}
}
}
Nothing opts in because nothing gets asked.
One detail: this write goes outside the handler's transaction. If the business transaction rolls back, the call still happened and you still paid for it.
What's in the payload?
{
"providerId": "anthropic",
"handlerName": "invoice-extract",
"requestedModel": "claude-sonnet-5",
"respondedModel": "claude-sonnet-5-20260514",
"promptVersion": "3f9a1c0e77b2",
"inputHash": "9c4e1ab77f30d552",
"latencyMs": 1180,
"usage": {
"inputTokens": 2140,
"outputTokens": 318,
"cacheCreationInputTokens": 0,
"cacheReadInputTokens": 1890
},
"reportedCostUsd": 0.0042,
"stopReason": "end_turn"
}
The two cache fields are there because of Valentin's comment. cacheReadInputTokens against inputTokens is your hit ratio, and now it's per call rather than a monthly average, so you can key an alert on it per prompt version like he suggested instead of watching one global number.
Requested and responded model are separate fields. Looks pedantic until a vendor routes you somewhere else and they don't match.
No prompt text, no output. Only hashes. The log lives forever and prompts are full of customer invoices.
Failures get a row too, with error kind and HTTP status. Most setups only log the successes.
How do you version a prompt?
I planned to put a version in the prompt file and bump it by hand. Then I didn't, because hand-maintained versions rot.
It hashes the stable, model-visible part of the request at runtime instead: system instructions, corpus, and the tool schemas. Twelve hex characters, and nobody bumps anything. Messages stay out on purpose, because if they were in it every call would get its own version and the thing would group nothing.
The "and the tool schemas" part is newer than this post. It used to hash the cacheable prefix only. Our extraction handler builds its tool from the caller's output schema, so you could change that schema, send the model a demonstrably different request, and promptVersion wouldn't move. inputHash saw it, but that one is unique per call, so it identifies without grouping. Which is the missed bump the runtime hash was supposed to make impossible. Found it writing this, wrote an issue, fixed it, merged it before the post went out. toolChoice went in at the same time, the field Tae flagged in the last thread ;)
What's left after that is key order, and it's a real one. JSON.stringify hashes the serialisation, so reordering properties in a tool schema flips the version without changing anything semantically.
The obvious move is to sort keys and hash a canonical form. We decided against it, and the reason is the interesting bit: the model sees the prompt as serialised text, and property order in a JSON schema affects the order an LLM generates fields in. Two differently sorted schemas are genuinely two different prompts. Sorting them into one bucket would rebuild exactly the missed bump we just removed, only invisibly. So the trade isn't false splits against churn. It's a visible false split against an invisible missed bump, and visible wins.
The other thing is on purpose. The hash is one-way, so you get 3f9a1c0e77b2 and no route back to the prompt. You can tell which calls are affected but not what changed in them. Reconstructing means git plus rehashing against the corpus from that day. The hand-bumped file version would have handled that with git blame. I didn't think about it until after I'd shipped the hash.
We have a prompt store with proper revision history sitting in the same codebase, connected to none of this. That's probably the answer and I haven't wired it up.
What does it cost?
Postgres 16, synthetic events, shape checked against real ones from the provider path:
- 513 bytes per payload at the median, 517 at p95
- about 1,036 bytes per call with row overhead and the event store's three indexes
- at 10k calls a day that's roughly 296 MB a month, 3.5 GB a year
Same per-call number at 10k, 100k and 1M rows. Storage isn't where the money goes.
Can you query it?
This is the part where measuring changed my mind.
"Every call with prompt version X" is what you run when something is wrong. An event store indexes tenant, aggregate type and time. Not the inside of a JSONB payload.
Scoped to one tenant and a time window: 162 ms over a million events. The index picks ~67,000 candidate rows and the payload filter narrows those to 8,490, so you pay for hauling 67,000 rows out of the heap.
Globally it's a seq scan, so you add an expression index on the payload field. At a million rows:
| distinct prompt versions | no index | with index | plan |
|---|---|---|---|
| 8 | 115 ms | 281 ms | ignored it, seq scan |
| 50 | 107 ms | 61 ms | bitmap heap scan |
| 500 | 107 ms | 1.02 ms | bitmap heap scan |
Index costs 6.9 MB in all three.
With eight versions one version is 12.5% of the table, so Postgres scans and is right to. The index just sits there.
The problem is that day one is when you benchmark this, see nothing, and decide the index isn't worth it. Six months in you've edited prompts a few hundred times, one version is 0.2% of the table, and that same index is a hundred times faster. Add it once you're past single digits.
Do I need event sourcing?
No. An append-only ai_calls table gets you nearly all of it. We already had the event log so it came out of that for free.
The bit worth copying either way is where you put the recording.
What would you change before this goes live?
Asking for real. It's built and merged but not out to a live tenant yet, so this is still a good moment to hear that it's wrong.
Which is also why there are no cost or latency numbers from real traffic in here. We don't have them yet. In two months I will.
If you've run something like this for a while: what's in your call log that isn't in mine? And where does this fall apart at a volume I haven't hit?
The benchmark SQL is in the repo if you'd rather measure your own database than trust mine. It's self-contained, so psql -v rows=1000000 -v prompt_versions=500 -f ai-call-provenance-benchmark.sql against a throwaway database reproduces the table above.
I build Kumiko, an event sourced framework for multi-tenant B2B systems in TypeScript.
Top comments (0)