Log every LLM call as one structured event: prompt version, model id, sampling settings, token counts, time to first token and total latency, tool calls made, validation outcome, retries, and an estimated cost. Redact user content by default and keep a sampled, access-controlled raw store for debugging. Without that, the one question that matters after an incident, what the model actually saw, has no answer.
What goes in the event?
One event per LLM call, written when the call completes, holding roughly ten fields about the call and nothing about its content. It has to answer two questions later: how the call behaved, and what it cost, without storing what the user typed. Here's the field list we start from, boring on purpose.
- A trace id that ties together every call made for one user action.
- The prompt template version and a hash of the rendered prompt.
- Model id, temperature, max tokens, and any provider-specific flags.
- Input and output token counts, taken from the provider response, never estimated.
- Time to first token and total duration.
- Every tool call: name, arguments hash, duration, success or failure.
- Validation result against the output schema, plus the retry count.
- The route taken, if you use model routing, and the feature-flag state.
- A cost estimate computed from the token counts.
- A hashed user or tenant id.
Notice what isn't in there: the prompt, the response, the user's message. That omission is the whole design.
Why redact by default?
Because prompts are where the sensitive data ends up. Contract text, medical notes, salaries, customer messages: whatever your product touches flows into the prompt in plain form. If your log pipeline stores prompts raw, your logging system has quietly become your most sensitive database, usually with the weakest access controls.
The moment that made this personal for us: an engineer debugging a bad response pasted a full log line into a Slack channel to ask for help. The line contained the entire prompt, and the prompt contained a customer's message with their phone number and account details. Nobody meant harm. The system had made it the path of least resistance.
So we redact at ingestion, and we split the storage in two.
| What you store | Default structured event | Sampled raw store |
|---|---|---|
| Token counts, latency, cost | 100 percent of calls | Not needed |
| Prompt version and hash | Yes | Yes |
| Full prompt and response text | Never | 1 to 5 percent, plus all failures |
| Retention | Normal log retention | 7 to 30 days, TTL enforced |
| Who can read it | Anyone who reads logs | Row-level access, reviewed |
A regex pass catches the obvious patterns (emails, phone numbers, card-like sequences, national id formats), a small classifier catches names and free-text PII, and the redacted event stays the default view.
How do you keep logs useful as they grow?
Sample raw content at one to five percent for healthy traffic, and keep one hundred percent of failures: validation errors, timeouts, tool errors, user thumbs-down. The failures are what you'll be reading. Emit spans through OpenTelemetry so an LLM call sits inside the same trace as the database query before it.
Then build four dashboards and resist building more: latency (p50 and p95) by prompt version, validation-failure rate by model, cost per successful task, and escalation or fallback rate. Every incident I've worked on an AI feature was visible in one of those four before a human noticed.
What breaks when you don't do this?
Model upgrades break first. A provider ships a new version, your prompt behaves slightly differently, and with no logged baseline there is nothing to compare against and no way to reproduce the old behaviour. The team blames the model, the feature loses trust, and nobody can prove what changed.
Teams whose AI projects survive tend to have this instrumentation in place before launch, which is one of the habits shared by AI projects that don't get cancelled. At Shanti Infosoft this logging schema ships in the first sprint of every AI build, before the prompt is even good, because you can't improve what you didn't record. It's the same schema we drop into an AI development engagement that arrives with a working feature and no telemetry, which is most of them.
If a customer told you their answer yesterday at 3pm was wrong, how long would it take you to see exactly what the model saw?
Sagar Jain is the technical co-founder of Shanti Infosoft, a CMMI Level 5 company, and has spent more incident calls than he'd like reading LLM logs that weren't there.
Top comments (0)