How would you take an LLM app from prototype to production?

The interviewer nods at your working chatbot demo, then asks how you would put it in front of real users. You have one whiteboard.

A strong answer starts from the fact that a production LLM app has failure modes a demo hides. An LLM, or large language model, is a model that generates text from a prompt. The answer then names the controls that address each failure mode: evals to measure behaviour, retrieval to supply knowledge, guardrails sized to the risk, and continuous evaluation after launch. Fine-tuning comes last and only when the evals call for it. This article gives that answer, then works through the follow-up questions interviewers tend to ask and the mistakes that weaken a response.

Production fails in ways the demo never shows

Open by explaining why the prototype is not most of the work. Moving generative AI products from proof-of-concept to production has proven hard for engineers, largely because teams treat them as extensions of traditional transactional or analytical systems. These systems bring new problems, including hallucination, unbounded data access and non-determinism.

Then describe the baseline you are moving away from. In "direct prompting", users type straight to an off-the-shelf model with no intermediate steps. It has three weaknesses:

• Stale knowledge. The model only knows its training data, so it knows nothing after training and nothing outside that set.
• Manipulation. It can be tricked by malicious prompts into revealing confidential information.
• Confident errors. It tends to sound confident when its knowledge is weak and to invent plausible but nonsensical answers.

That last trait is a serious liability when the model speaks for an organisation.

Think of it as hiring a junior researcher who is articulate and well-read, but overconfident and unfamiliar with your details. The rest of the answer is about giving that researcher the right documents, checking their work, and stopping them from saying things they shouldn't.

A production request passes through rewriting, hybrid retrieval, reranking and guardrails around the LLM

Evals come before any architectural choice

The first step in a strong answer is measurement, not model selection. Traditional software is checked with tests, but an LLM system is non-deterministic and can give different outputs for the same input. Evals are the replacement: evaluations of the model's responses for a specific task, usually across a range of scenarios rather than a single output.

Describe how scoring works. A scorer is a component that gives each output a numerical score on metrics such as relevance, coherence, factuality, or semantic similarity to an expected answer. Then say who acts as the judge. There are three options:

• Self-evaluation. The model grades itself. This is risky, because a flawed self-assessment can reinforce its own errors.
• LLM as a judge. A different model scores the output. Two models are unlikely to share the same errors.
• Human evaluation. People check tone and intent by hand. It is hard to scale but catches qualitative issues that automation misses.

The pattern's authors report that combining an LLM judge with human evaluation works better than either alone.

Each eval judge trades scale against blind spots; pairing LLM judge with humans covers both

Benchmarks belong here too, but as a different tool. Benchmarking compares models on standardised datasets and tasks, which helps you decide whether to upgrade to a new model version. It does not tell you whether the model suits your problem. That needs task-specific evals built on a dataset you create, because no industry dataset exists for your use case.

Retrieval is where most of the architecture lives

Next, explain how the model gets knowledge it was not trained on. Retrieval augmented generation (RAG) fetches relevant document fragments and includes them in the prompt. To make documents searchable, you:

1. Split them into chunks.
2. Turn each chunk into an embedding.
3. Store the chunks and embeddings in a vector database.

An embedding is a numeric vector produced so that related concepts sit near each other.

At query time, you embed the user's question and run an approximate nearest-neighbour search to find matching fragments. A prompt template then combines those fragments with the question. The template should tell the model to use the supplied context and to admit when the context is insufficient.

What separates a strong candidate is admitting that basic RAG is not enough. The pattern's authors found that a working RAG needed many enhancements before it could handle a serious problem. Each gap has a matching pattern.

Each limitation of basic RAG maps to the pattern that mitigates it

Query rewriting has a cost you should mention unprompted. Each rewritten query adds an LLM call plus extra retriever calls, which raises both resource costs and latency. Its authors suggest experimenting to see whether the gain is worth it, and using evals to track progress as you tune the rewrites.

Guardrails are sized to who can type into the box

Guardrails need their own place in the design, because a conversational interface raises the risk well above a form-based one. Users can type anything, including "ignore previous instructions". There are two kinds:

• Input guardrails screen the query for malicious or badly worded prompts before it reaches the conversational model.
• Output guardrails scan the response for information that should not be there.

There are three common ways to implement them. One uses a separate LLM call wrapped in a block-or-allow template. Another uses embeddings to route on-topic queries and reject off-topic ones. A third applies predefined rules, such as filtering personally identifiable information.

Guardrails sit on both sides of the conversational LLM

The judgement that matters is proportion. Anything connected to the general public must have guardrails. A small, logged group of employees needs them less, though it still needs protection from inappropriate content, misinformation and bias that the model itself produces. Guardrails are not free: the extra LLM calls add cost and latency, and someone has to set them up and monitor them.

Operations are judged across six pillars

Close the main answer by showing you can operate the system, not just build it. A recent reference point is the generative AI lens of a widely used cloud well-architected framework, current in its November 19, 2025 publication. It organises guidance into six pillars. It also spans the lifecycle stages of scoping, model selection, customization, development, deployment and continuous improvement.

Six pillars give an operating checklist beyond getting answers right

The same guidance treats responsible AI as a responsibility shared between model producers, providers and consumers. A sensible interview move is to name which of those roles your app occupies, since that decides what you must build yourself.

Follow-up questions and where weak answers slip

"Why not just fine-tune the model on our data?"

Model answer: Fine-tuning requires significant skill, compute, expense and time, so it is wise to try other techniques first. Those techniques usually prove enough. The order is:

1. Improve the prompt.
2. Add RAG.
3. Fine-tune, only if eval metrics remain unsatisfactory after RAG has been optimised.

Fine-tuning earns its place when the task needs the model to work in a different style, not just to know more documents. If you do fine-tune, the data matters most. In the legal fine-tuning project the authors describe, nearly half the effort went into data preparation and curation.

Weak-candidate mistake: Treating fine-tuning as the default way to "teach" the model, with no eval evidence that it is needed.

Escalate to fine-tuning only when evals stay unsatisfactory after RAG

"How do you know a change made things better?"

Model answer: Run evals in the build pipeline. They are not binary pass or fail: you set thresholds and check that scores do not decline, much as you would with performance testing. Evaluate each LLM-bearing component on its own as well. For example, guardrails and query rewriting can be scored separately and as part of the whole request flow.

Weak-candidate mistake: Relying on the model's own self-assessment, which can reinforce its errors, or on a handful of manual spot checks.

"Isn't vector search enough for retrieval?"

Model answer: Vector search alone is useful but not very effective, because compressing paragraphs into one dense vector loses detail. Combining it with keyword techniques such as BM25 surfaces candidates that vector search misses. For relationships, exact matches or numerical comparisons, a traditional database query beats embeddings. One team found a graph more effective than embeddings for code dependencies.

Weak-candidate mistake: Assuming embeddings are the only knowledge base a retriever can use.

"The model has a long context window. Why rerank?"

Model answer: Models struggle to use information in the middle of long inputs, even models designed for large contexts. A reranker, typically a cross-encoder, scores only the candidates a fast search returned. That keeps the prompt focused. It adds cost and latency, which makes it less suitable for high-traffic applications.

Weak-candidate mistake: Stuffing every retrieved chunk into the prompt because it fits.

"What changes after launch?"

Model answer: A live system can change its performance in production, so evaluation continues on the deployed system, again looking for declining scores. Reliability guidance adds observability, graceful failure handling, versioned artifacts and distributed inference.

Weak-candidate mistake: Treating evals as a one-time gate before deployment.

Key takeaways

• Start by naming the failure modes the demo hides: hallucination, unbounded data access and non-determinism.
• Put evals first. Combine an LLM judge with human review, and run evals both in the pipeline and in production.
• Present RAG as the default, then pair each of its limitations with a fix: hybrid retrieval, query rewriting, reranking and guardrails.
• Size guardrails to how far you can trust the users, and state the cost and latency each added call brings.
• Escalate to fine-tuning only when evals stay unsatisfactory after RAG.

This answer covers architecture and operating concerns. It does not settle concrete latency or cost budgets, which depend on your traffic and models. It also leaves open the best mechanisms for scoring and judging, which remain an area the field is still learning about.

Links

• Full article: https://confidentprep.com/blog/how-would-you-take-an-llm-app-from-prototype-to-production/ <!-- cf-origin:end -->