Production LLM Application Architecture in 2026: An Engineering Checklist Beyond the Demo

The demo worked. That is the problem.

A demo usually proves one narrow path under friendly conditions. Production exposes the same feature to unfamiliar inputs, real load, dependency failures and changes you did not schedule.

The gap is not only a model problem. Much of it is ordinary application, security and operations engineering.

This is a checklist for closing it. It is organised by how things fail, not by vendor feature, because the vendors keep renaming everything and the failure classes stay put.

Why the demo proves nothing

This checklist groups the main production failure modes into seven classes. A demo usually exercises only a small part of them.

Failure class	The question nobody asked during the demo
Input	What happens when the input is hostile, enormous, or in the wrong language?
Retrieval	What happens when the right document is not found, or the wrong user's document is?
Model	What happens when the provider is slow, down, or retires the model?
Output	What happens when the model returns something your code cannot parse?
Action	What happens when the model does something real and wrong?
Cost	What happens when usage grows ten times?
Operations	What happens when it breaks and nobody knows why?

Work through them and you have a useful production readiness baseline.

The shape that survives contact with users

A useful production architecture can be thought of as six core pieces.

A gateway. One place where requests enter, get authenticated, rate limited and tagged. Without this you cannot answer "who spent that money" or "which tenant caused that spike".

An orchestrator. The thing that decides what happens in what order: classify, retrieve, call, validate, respond. This is your application logic. It belongs in code you own.

Retrieval. Whatever finds the facts. Its most important property is not similarity scoring. It is knowing what the current user is allowed to see.

A model adapter. One internal interface that keeps provider specific API details in a single place. It should make model migrations testable, let you pin a production model or snapshot where the provider supports it, and give the team a documented migration or fallback path.

A second active provider is optional. It can be worth the cost when business continuity requirements justify it, but it also brings a second set of behaviours to evaluate and more surface to keep current. Decide it on requirements rather than by default.

A policy layer. Schema validation, content checks, permission checks, approval gates. Everything that decides whether the model's output gets to become an action.

Telemetry. Traces, costs and evaluation results. If you cannot see it, you cannot fix it.

Notice what is not on that list: a framework. You may use one. You will still need all six.

The flow through them is straightforward:

Client
  |
Gateway .......... authentication, rate limits, request ID
  |
Orchestrator
  |---- authorization scoped retrieval
  |---- model adapter
  |
Output validation and policy
  |
Approval when required
  |
Tools and side effecting actions
  |
Response

Telemetry, cost, security and evaluation metadata wrap the full path.

Read only work such as retrieval and lookups can happen earlier in that path. The ordering that matters is that a high impact action does not execute before the policy and approval gate it requires.

Controls that earn their place

Authorize before you retrieve

One serious class of RAG failure is applying authorization only after retrieval.

The rule worth holding onto: restricted content should not become model context unless the current principal is authorized to access it. Prefer authorization aware or pre retrieval filtering where your datastore and access model support it.

When the retrieval layer can apply the authorization filter efficiently, it can also reduce the candidate set and the amount of search work. Treat tenant isolation as a first class concern rather than a configuration detail, because a leak across tenants is the kind of incident that ends contracts.

Treat external content as data, not instructions

Anything the model reads can try to instruct it. That includes the obvious case of a user typing an attack into a chat box, and the less obvious case of instructions hidden inside a retrieved document, a scraped web page, an uploaded file or a tool response. The second kind is harder to spot, because the text arrives through a channel you trusted.

Design on the assumption that some of it gets through. Enforce authorization and tool permissions in application code rather than trusting a prompt to hold the line. Give each tool the narrowest scope that lets it work. Keep tenant boundaries enforced at the data layer.

Be deliberate about what leaves the system too. Secrets and sensitive data should not end up in prompts, logs or traces without a reason, and captured prompts, documents and outputs need retention and redaction rules that match the product.

Finally, bound consumption. Put limits on tokens, tool calls, agent steps and expensive operations, so a hostile or malformed input cannot turn into an unbounded bill or an infinite loop. OWASP's GenAI security material is the practical reference here, and its current edition is worth reading end to end.

Treat model output as untrusted input

Your code is going to parse this. Validate it against a schema before anything downstream touches it.

OpenAI and Gemini both support schema constrained structured outputs. When the feature is supported and the response completes normally, it can enforce the expected structure far more reliably than prompting for JSON alone. You still need to handle refusals, incomplete responses and application level validation.

Schema compliance is not semantic correctness. The provider can constrain the shape; your code still has to check the values, the permissions and the business rules. A perfectly valid JSON object can still contain a refund amount nobody is authorised to issue.

Put explicit approval in front of high impact actions

Not everything needs approval. Reading a document does not.

Explicit user confirmation or human approval earns its friction when an action is destructive, externally visible, financially significant, privilege changing, legally or compliance sensitive, or difficult to reverse. Reading and retrieval normally need far less.

Design can lower the bar. Staged operations, reversible changes and compensating actions sometimes reduce what needs approving in the first place, which is usually better than adding another confirmation dialog.

Whatever you gate, log it: who or what requested the action, what was approved, the resulting operation, and the final state.

Reliability is mostly boring

Timeouts on every external call. Bounded retries, not infinite ones. A fallback that degrades honestly rather than pretending.

Five failure modes catch people out repeatedly:

• Partial streams. The connection drops halfway through a response. Your UI shows half an answer and your database saves it as complete.
• Stale caches. You cached an answer that was true last week.
• Schema drift. Your code, SDK or provider response evolves and an old parser assumption quietly becomes wrong.
• Retry storms. The provider slows down, every client retries, and you complete the outage for them.
• Unsafe retries. Retries around side effecting operations need idempotency or state reconciliation. A timeout does not prove the original operation failed, and repeating it can create a second charge, a second booking or a second email.

None of these are AI problems. They are the same distributed systems problems that have always existed, which is good news, because the fixes are well understood.

Cost, latency and the thing you cannot see

Attribute cost per request, per tenant and per feature from day one. Retrofitting this is miserable, and without it your only lever when the bill arrives is panic.

For latency, measure the median and the 95th percentile separately. Averages hide exactly the users who are having the worst time and telling other people about it.

One example from a client system we rebuilt. A document assistant was answering in roughly 10 to 30 seconds depending on the query, across a knowledge base of more than 120 GB and approximately 40,000 PDF documents. The rebuild put dynamic intent detection and routing in front of retrieval, made document retrieval authorization aware, and added caching and optimised context selection. First token response came down to under three seconds in typical interactions. This project illustrates the production AI engineering work JarvisBitz Engineering focuses on.

The transferable lesson is about method rather than numbers: measure latency by stage before blaming the model.

Evaluation is a release gate, not a research project

Start with a small representative set covering common requests, known failure cases, edge cases and adversarial inputs.

Not every task has one known good answer. Depending on the task, use reference answers, expected outcomes, or explicit acceptance criteria. A summarisation feature and a booking agent need different definitions of correct.

Add real production failures to the set as they appear. The bugs your users find are the most valuable test cases you will ever get for free.

Run the relevant evaluations before any change to prompts, retrieval, models, tools or workflow logic that could affect behaviour.

Make every change reversible

Version your prompts. Record the exact model or model snapshot where the provider exposes one. Record the retrieval or index version, and the application configuration that shaped the request.

For risky changes, use feature flags, a staged rollout or a canary deployment, and compare behaviour against your evaluation gates before going wide. Keep a rollback path you have actually tested, not one you assume works.

The goal is easy to state and easy to neglect: when a bad response turns up in a support ticket, you should be able to trace it to the exact model, prompt, retrieval and configuration versions that produced it. Without that, every investigation starts from guesswork.

Make traces answer questions

A useful production trace should be able to identify, where relevant: a request or trace ID, a tenant or principal reference that does not expose unnecessary personal data, the model and version, the prompt or configuration version, the retrieval or index version, tool calls with their results or status, retries, fallbacks, token usage, cost, per stage latency, and the final outcome or evaluation signal.

That is not a licence to log everything. Do not indiscriminately capture raw prompts, retrieved documents or secrets. Apply redaction and retention controls appropriate to the product and the data it handles.

The moving parts are genuinely moving

This year made the case better than any argument could.

OpenAI's Assistants API shut down on 26 August 2026, one year after notice, with the Responses and Conversations APIs as the replacement direction. Its transcription models were deprecated the same day, with removal set for 26 February 2027. A batch of legacy audio and realtime models goes on 20 January 2027. All of those dates are on OpenAI's deprecations page, which is worth a bookmark.

Google made its Interactions API generally available in June 2026 and recommends it for all new projects, with the original generateContent API now considered legacy but still fully supported. Separately, Vertex AI Platform became Gemini Enterprise Agent Platform, taking most of the surrounding product names with it, which Google documents in a name changes table.

OWASP published a new Top 10 for LLM Applications on 3 August 2026, with updated rankings and research drawn from real-world AI security incidents.

Two conclusions follow. Keep a deprecation calendar with real dates and a named owner, because these announcements do not arrive when it is convenient. And keep provider specific code behind your adapter, because "we will migrate later" ages badly when later has a published date attached.

The checklist

Area	Ready when
Input	Length limits, content checks, and a known answer for hostile input
Retrieval	Authorization applied before content becomes context; tenant isolation enforced; a defined behaviour for "nothing found"
Model	Provider specific code isolated; production model or snapshot pinned where possible; migration or fallback path tested; deprecation calendar owned
Output	Schema enforced by the provider and validated again by your code, including values and business rules
Action	Risk based approval for destructive, externally visible, financially significant, privilege changing or hard to reverse actions, with an audit trail
Reliability	Timeouts, bounded retries, honest degradation, idempotency or state reconciliation on writes
Security	Prompt injection has a threat model; tool scopes are least privilege; tenant and data boundaries enforced; sensitive data has retention and redaction rules; token, tool and step consumption bounded
Cost	Attributed per request, tenant and feature; alerts on the trend
Latency	Median and 95th percentile per stage, not just end to end
Evaluation	A representative set that gates changes to prompts, retrieval, models, tools and workflow logic
Release	Prompt, model and configuration versions recorded; evaluations gate changes; risky changes can be staged; rollback tested
Operations	Searchable traces, alerts and SLOs, version metadata, and an owner with a runbook who can diagnose failures

What you can safely leave until later

Not everything needs doing before launch, and pretending otherwise is how features never ship.

Fine tuning can often wait until evaluation evidence shows that prompting, retrieval, model choice or workflow changes are not closing the gap. A multi agent architecture can wait until a concrete need appears and one agent is demonstrably not enough. Self hosting a model can wait until you have a specific reason: cost at scale, data residency, a control requirement, or a capability no provider offers.

For a production feature handling real users, data or actions, each checklist area should have an explicit answer before launch, even if the first answer is deliberately simple. "We cap input length at 4,000 characters and reject the rest" is a real answer. "We have not thought about it" is not.

Take an existing feature and run it through the failure class table before you build anything new. The gaps you find there are usually worth more than the next feature on the roadmap.

Drafted with AI assistance and reviewed, edited and approved by the author.