What the operations layer around a production language model actually contains, and what it costs to skip it.
Picture the failure. Support deflection has sat at 61% for six weeks. On a Tuesday it reads 48%. Your repo has no new commits. Latency is flat, the error rate is zero, every HTTP response is a 200. The provider rolled a point release behind the model identifier you pinned loosely, and the summarization step now writes three sentences where it wrote one, so the routing logic downstream stopped matching.
You find out from a customer.
That gap between “the service is up” and “the answers are right” is the entire subject of LLMOps. It is the engineering between a model endpoint and the people depending on it, and in 2026 it is the part of the stack most teams shipped without.
The thing you are operating is a configuration, not a model
Here is the distinction that makes LLMOps its own discipline rather than MLOps with new vocabulary.
In MLOps you version training data, features and weights. You own the artifact. When it degrades, the cause is data drift or a broken feature pipeline, and the fix is a retrain on your schedule.
In LLMOps the artifact belongs to a vendor. What you own is everything wrapped around it: prompt versions, a retrieval corpus, tool contracts, a routing config, a provider and model version. A regression comes from a version bump you did not schedule, a corpus update someone ran at 4pm, or a one-word prompt edit that passed review because nobody had a way to test it.
Rollback means restoring a previous combination of prompt, corpus and provider. There is no artifact to roll back to. That difference cascades into five components, and most engagements start with whichever one is missing.
Evaluation suites. A versioned case set covering the paths users take, run in CI against every prompt edit, corpus change and provider bump, with a pass threshold gating release.
Tracing. Every model call, retrieval and tool invocation, captured at the request level.
Versioning. Prompts, corpora and tool contracts under change control, the same way code is.
Cost telemetry. Spend attributed per request across a multi-step path.
Release control. A documented rollback path someone has actually used.
Evaluation is the control plane, and logging is not
The industry spent two years buying tracing tools and calling it observability. Tracing tells you what happened after it happened. An evaluation suite stops the bad change from shipping.
A working suite is a versioned case set with graded expectations, run in the same pipeline as your unit tests. Four kinds of case earn their place:
Golden cases with known-correct answers.
Adversarial cases covering ambiguous and out-of-scope input.
Multi-step cases that exercise the tool path, including the loop that runs twice.
Regressions pulled from real traces, added the week each one broke.
The pass threshold blocks the merge. That single mechanic turns a prompt edit from a judgment call into a diff someone reviews on evidence, and it turns a provider release from an incident into a report: the suite runs against the new version, the delta comes back case by case, and the routing config either moves or stays.
Now the honest part. Real evaluation adds weeks to a first release and returns nothing on that release. It pays back on the first model change and every one after. A team shipping one prompt into a low-volume internal tool should skip it and spend the time elsewhere.
What a trace has to carry before it is useful
Most LLM tracing captures the model call and stops. That covers maybe a third of your failure modes. A trace that answers real questions has four layers.
Retrieval. Which documents came back, what they scored, and whether the answer actually cited them. Most quality complaints resolve here, and they resolve fast when the retrieved chunks sit next to the output.
The model call. Provider, model identifier, prompt version, tokens in and out, time to first token, total latency.
Tool calls. Which tool the model picked, the arguments it passed, what came back, and how many times the loop ran before it stopped.
Outcome. Whether the request resolved, escalated to a person, or got retried.
Join that last layer to cost and the number changes meaning. Spend per million tokens is an accounting figure. Spend per resolved request is an operating one, and it is the only version of the metric that tells you whether a cheaper model is actually cheaper once it escalates twice as often.
In a regulated deployment, the ops layer is the evidence layer
If you run in fintech, insurance or health, the auditor will ask for records rather than architecture. Four artifacts carry that weight, and each is a design decision taken before launch rather than a report generated after one.
Version lineage per decision. For any output that affected a customer: the prompt version, model identifier, retrieval snapshot and tool responses behind it. Under SR 11–7 model risk guidance, this is what makes a language model component reviewable in the same terms as a credit scorecard.
Human review records. Which outputs a person approved, changed or rejected, timestamped and attributable. Where a fully automated decision would sit in scope under the EU AI Act, this is the control teams most often rely on to stay outside it.
Retention and residency. Traces hold customer text. Retention windows, redaction and inference region are GDPR questions rather than infrastructure preferences, and DORA adds concentration risk where one provider serves every route.
Change control. SOC 2 change management applied to prompts and corpora, not only to code, because both move model behavior.
Teams that treat these as a compliance workstream separate from engineering build the system twice.
Where the classical model still wins
One more thing worth saying out loud, because vendors rarely say it.
On a stable, high-volume classification step, a small trained classifier or a deterministic rule beats a model call on cost, latency and predictability. Every time. If the step is “route this ticket to one of nine queues” and you have labeled history, the language model is the expensive option with the worse audit story.
Teams already running classical models are in the best position here anyway. The registry, the deployment pipeline, the monitoring backbone and the incident rota serve both disciplines. Splitting them doubles the operational surface for no return. What diverges is the clock: a classical model gets retrained on your data, a model route gets reconfigured on a vendor’s release schedule.
Four questions for whoever you hire
Apply these to us as readily as to anyone else.
Who writes the evaluation suite, and when? If the answer is “after the first release,” evaluation is an add-on rather than part of the engineering.
Where does inference run and who holds the keys? Your cloud account, a vendor platform, or a mix, and the data path in each case.
What is the handover artifact? A running system with traces, a documented case set and an on-call rota is a different deliverable from an architecture diagram.
What did the team do the week a provider last deprecated a model?
This one is hard to fake.
We wrote up how we run this work, including a 3–5 day readiness audit that leaves you with a written gap list you can act on with any partner: LLMOps consulting services.
Or take the shorter path. A 15-minute call with our CTO about where your model runs, what your traces show today, and what breaks the next time a provider ships a version bump.
Top comments (0)