The demo looked ready.
The application could search company documents, answer support questions, classify requests, and create tickets through an API. It handled every test prompt we gave it.
Then real users arrived.
They asked incomplete questions. They pasted entire email threads. They used terms that did not exist in our documentation. Some requests matched several policies, while others required information the system could not access.
The LLM was still producing fluent answers. The workflow around it was falling apart.
The Output Was Valid Until It Wasn’t
During testing, the model returned predictable JSON:
{
"category": "billing",
"priority": "high",
"requires_human": true
}
Production inputs were less predictable. Sometimes the model changed a field name, returned an unsupported category, or added an explanation outside the JSON object.
A response that looks reasonable to a person can still break an application.
The first improvement was to treat model output as untrusted input. Every response had to pass schema validation before the application could use it.
from typing import Literal
from pydantic import BaseModel
class TicketAction(BaseModel):
category: Literal["billing", "technical", "account"]
priority: Literal["low", "medium", "high"]
requires_human: bool
def validate_action(model_output: dict):
return TicketAction.model_validate(model_output)
Structured output reduced parsing failures, but it did not prove that the selected category was correct. Format validation and business validation became separate steps.
RAG Retrieved Something Relevant but Not Correct
The knowledge base contained current policies, archived documents, internal notes, and several pages describing similar processes.
The retrieval system often returned a document related to the question. That did not mean it returned the document needed to answer it.
One customer asked about cancelling a subscription. The system retrieved an older cancellation policy because it shared more words with the question than the current policy did.
The solution was not simply increasing the number of retrieved chunks.
Documents needed version metadata, ownership, effective dates, and access rules. Archived content had to be removed from normal retrieval. The application also needed to recognise conflicting evidence and stop instead of asking the LLM to choose a convenient answer.
RAG improved the model’s access to information. It did not remove the need to manage that information.
Tool Calling Turned Small Errors Into Real Actions
A wrong answer is harmful. A wrong action can be worse.
Once the LLM could create tickets, update CRM records, and trigger notifications, every uncertain decision had operational consequences.
Retries created duplicate records. Incorrect parameters sent requests to the wrong queue. A broad service account gave the agent access to functions it never needed.
We reduced this risk by giving each tool one narrow purpose. Tool arguments were validated outside the model, write operations used idempotency keys, and high-impact actions required confirmation.
This follows the principle behind OWASP’s guidance on excessive agency: limit available tools, permissions, functionality, and autonomy.
The model could recommend an action. The application decided whether that action was allowed.
One User Request Became Eight Model Calls
The original demo made one request to an LLM.
The production version classified the user’s intention, rewrote the search query, retrieved documents, reranked the results, generated an answer, checked the answer, selected a tool, and summarized the result.
Each step appeared reasonable on its own. Together, they created noticeable latency and unpredictable costs.
Agents made the problem harder because the number of steps could change for every request. A simple question might finish immediately, while an ambiguous request could enter a loop of repeated searches and tool calls.
We added limits for execution time, model calls, retries, retrieved context, and total tokens. Smaller models handled basic classification, while stronger models were reserved for decisions that needed deeper reasoning.
The goal was not to minimize every model call. It was to ensure that each call justified its cost and delay.
Our Logs Said Everything Was Successful
The API returned 200. The workflow was still wrong.
Traditional logs showed that the request completed, but they did not explain which documents were retrieved, why a tool was selected, or where the answer changed.
Production debugging required a trace of the complete workflow. We recorded the prompt version, model version, retrieved document IDs, tool arguments, tool results, latency, token usage, validation failures, and final outcome.
Sensitive customer data was removed or masked before storage. Observability should help investigate failures without creating a new privacy problem.
The NIST Generative AI Profile also emphasizes ongoing monitoring, documented responsibilities, incident handling, and human oversight for generative AI systems.
The Evaluation Set Was Too Polite
Our early tests asked clear questions with known answers.
Real users did not behave like an evaluation dataset.
They submitted vague instructions, conflicting requests, spelling mistakes, old account details, pasted web content, and questions requiring permissions they did not have.
The evaluation set had to include those conditions. We added cases involving stale documents, conflicting sources, API timeouts, repeated actions, missing information, prompt injection, and requests that should be refused or escalated.
Every production failure became a new regression test.
That changed evaluation from a task completed before launch into a process that continued throughout the product’s life.
The LLM Was Only One Part of the Product
The biggest lesson was simple: connecting an LLM to an application is not the same as building a production AI system.
Reliable AI requires structured outputs, managed knowledge, secure tools, permission checks, human approvals, observability, cost controls, and realistic evaluations.
This is also what teams should examine when they plan to hire AI engineers. Prompting skills matter, but production AI development also requires backend engineering, API design, security, data management, testing, and operational thinking.
The LLM did not suddenly become less capable after deployment. Production simply exposed every assumption that the demo never tested.
What broke first when you moved an LLM application into production?
Top comments (0)