Friday AI Question. This week's question is one every team shipping an LLM feature has to answer, usually after the first weird bug report:
Do you log your LLM prompts and responses?
- A. No. We don't store prompts or outputs at all.
- B. Metadata only. Model, token counts, latency, cost, finish reason, user/session IDs. No text.
- C. Full text, redacted, short retention. Prompts and outputs are kept, with obvious personal data masked, deleted after a fixed window, and access is restricted.
- D. Everything, forever. Raw prompts and outputs in the same log pipeline as everything else.
Drop your letter in the comments. Here's how I think about each one.
Why this question matters
An LLM feature fails in ways a status code won't show you. The request succeeds, the answer is wrong, and the only evidence is the text that went in and the text that came out. If you never kept that text, you can't reproduce the bug, you can't build an eval set from real traffic, and you can't explain to a user why the bot said what it said.
But prompts are also where users paste the things they shouldn't: names, emails, account numbers, contract text, health details. The OWASP Top 10 for LLM Applications lists Sensitive Information Disclosure as one of its top risks, and your logs are one more place that information can leak from. Privacy laws such as the GDPR also include a storage limitation principle: personal data shouldn't be kept longer than you need it for its purpose.
So the real question isn't "log or don't log". It's "what do you keep, for how long, and who can read it".
A. No logging
Good: nothing to leak, nothing to delete, an easy privacy story.
Bad: you're debugging blind. Every "the bot gave a weird answer" ticket becomes guesswork, and you have no real-world examples to turn into evals. Fine for a prototype; hard to defend for a feature people depend on.
B. Metadata only
Good: you can still watch cost, latency, error rates and stop reasons, and spot that something changed. This is a solid baseline for every LLM app.
Bad: you'll know that quality dropped but not why. Metadata tells you a request was long and expensive; it doesn't tell you the retrieved context was the wrong document.
C. Full text, redacted, short retention (my pick)
Good: you can reproduce real failures, read traces end to end, and sample production traffic into an eval set, while limiting the blast radius.
What "done properly" looks like:
- Redact before you store. Mask emails, phone numbers, IDs and other obvious personal data at write time, not at read time.
- Set a retention window and enforce it. Pick a number of days that matches how long you actually debug, and delete automatically.
- Restrict access. Prompt logs shouldn't be readable by everyone who can read app logs. Keep them in a separate store with their own permissions and an access trail.
- Promote, don't hoard. When a logged example is useful, copy it (cleaned up) into your eval set on purpose. Don't keep the whole stream "just in case".
- Tell users. Say in your privacy notice that conversations may be reviewed to improve quality, and offer an opt-out where you can.
Bad: it takes real work: a redaction step, a retention job, separate access controls. Redaction is never perfect, so treat the store as sensitive anyway.
D. Everything, forever
Good: maximum debuggability, zero effort.
Bad: every customer secret anyone ever pasted now lives in your log platform, readable by anyone with log access, kept long after it's useful. That's the option that turns a small incident into a big one.
Where I land
Start with B on day one, because cost and latency metrics are cheap and always useful. Move to C as soon as the feature has real users, because you can't improve what you can't read. Avoid D.
LLM observability tools (Langfuse, for example, which I covered in an earlier post) make the "read traces end to end" part easy. The policy part (what to mask, how long to keep it, who can see it) is still your decision.
References
- OWASP Top 10 for LLM Applications: https://genai.owasp.org/llm-top-10/
- GDPR Article 5, principles relating to processing of personal data (including storage limitation): https://gdpr-info.eu/art-5-gdpr/
- Langfuse documentation (LLM tracing): https://langfuse.com/docs
- Microsoft Presidio, open-source PII detection and anonymization: https://microsoft.github.io/presidio/
So, A, B, C or D? And if you picked C, how many days do you keep prompt logs?
Top comments (6)
C is useful only if redaction preserves enough relationships to reproduce the failure. Replacing every email with the same placeholder can merge two different people in the trace and hide the exact attribution bug being investigated.
I would test a conversation containing two identities plus a later reference to the first one. Trace-local placeholders should preserve that distinction without creating a durable identifier across unrelated traces. The cleaned example promoted into an eval set then needs the same check, since a privacy transformation that changes the expected answer no longer reproduces the original failure.
The two-identities test is the right shape. One implementation note that satisfies it: derive the placeholder from a keyed hash of (trace id + original value). The same person maps to the same token inside one trace, but the token differs across traces, so you keep attribution without minting a durable cross-trace identifier - and there is no mapping table sitting next to the logs for someone to reverse. The collision failure you describe, two originals collapsing into one placeholder, is what a per-trace salt rules out. Your eval-promotion point generalizes too: any transformation that changes the expected answer turns that eval into a test of the redactor rather than a test of the model.
Good test case: two identities plus a later back-reference catches exactly the merge bug a single placeholder hides. I'd add the cleaned trace to the eval set only after checking that the expected answer is unchanged by the masking.
C, with one addition to step 1: 'mask obvious personal data at write time' has a silent-failure mode - when the recognizer misses something, nothing errors, and the raw text lands in the store anyway. Worth testing the redaction step itself like any other control: run planted synthetic PII (data you own, so no real exposure) through it continuously and measure recall per type, because one average hides a bad category. One non-obvious miss class that came up in a recent experiment thread here: a recognizer registered for one language can skip the same PII type in another language entirely, so the canary set wants type crossed with language, not type alone. The retention window then caps whatever still slips through, which is the right way to frame C's real cost.
Testing the redactor with planted synthetic PII is the right move, and crossing type with language is a good catch. I'd report recall per (type, language) pair in CI and fail the build when any single cell drops, since the average would hide it.
Redacting at write time is the detail that makes option C workable. Without a retention window that actually deletes, it is just storage with extra steps.