How do you monitor an LLM system in production?

What the interviewer is testing

Whether you can define request-level logging, multi-step tracing, operational alerts, quality monitoring, and safe handling of production data.

Answer framework

• Log each request with its prompt version, model, token usage, latency, cost, and outcome.
• Trace multi-step calls so the order of model, retrieval, and tool operations is visible.
• Build dashboards and alerts for error rate, p95 latency, cost spikes, and quality signals.
• Sample conversations for review and track user feedback to identify quality problems.
• Redact PII before information is written to logs and restrict access to sensitive records.

Sample answer

I would monitor an LLM system at three levels: individual requests, multi-step traces, and aggregate service health. For every request, I would log a request identifier, the prompt or template version, model name, token counts, latency, cost, and a structured outcome. Those fields let me connect a user report to the exact call that produced it. I would avoid storing unnecessary prompt content, and I would redact personally identifiable information before any text reaches a log or observability tool.

For an agent or multi-step workflow, request logs are not enough. I would add distributed traces that show each model call, tool call, retrieval step, input, output, and error in the order they occurred. Dashboards would track error rate, p95 latency, token usage, and cost, while alerts would fire for sustained error increases, p95 latency regressions, and abnormal cost spikes. I would be careful to make alerts actionable so responders know which service and version are affected.

Operational logs and traces alone do not establish whether an output was useful. I would sample conversations for periodic human review, track thumbs-up, thumbs-down, and other user feedback, and add quality signals such as schema validity, refusal rates, or retrieval hit rates where they apply. I would segment these metrics by prompt version and model so I can detect a quality regression that averages hide. Finally, I would document retention limits, verify that redaction works, and restrict access to sensitive logs.

Likely follow-ups

Which production signals should trigger alerts? I would prioritize sustained error-rate increases, p95 latency regressions, abnormal cost spikes, and sharp changes in quality signals. Each alert should identify the affected service and version.

How do I monitor quality when there are too many conversations to review? I would combine sampled human review with user feedback and targeted automatic checks such as schema validity, refusal rates, or retrieval hit rates, then segment the results by model and prompt version.

How should monitoring handle sensitive user data? I would redact PII before logging, minimize the content retained, restrict access, and define retention limits. I would also test that the redaction works across prompts, traces, and error messages.

Common mistakes

• Logging only final responses without prompt version, model, token usage, latency, cost, or outcome.
• Using aggregate dashboards without request-level logs and traces for multi-step workflows.
• Sending an alert for every isolated fluctuation instead of sustained changes that need attention.
• Storing raw PII in logs because the observability tool is considered internal.

This article was prepared with AI assistance and reviewed before publishing. The original lives on the free question bank:

https://johnnnmai.github.io/ai-engineer-interview-questions/questions/how-to-monitor-llm-in-production/?utm_source=devto&utm_medium=community&utm_campaign=how-to-monitor-llm-in-production