LLMOps: Shipping AI to Production

LLMOps: Shipping AI to Production

In November 2022, Jake Moffatt asked the chatbot on Air Canada's website about bereavement fares after his grandmother died. The bot said he could claim a reduced fare within 90 days of the ticket being issued, even after travel. Air Canada's own policy page, linked inside the same chatbot answer, said the opposite. Moffatt booked two full-fare flights on the strength of that answer. When he claimed the discount, Air Canada refused and argued it could not be held liable for what its chatbot said. On 14 February 2024, British Columbia's Civil Resolution Tribunal disagreed. In Moffatt v. Air Canada, 2024 BCCRT 149, the tribunal called that argument "a remarkable submission", found negligent misrepresentation, and ordered the airline to pay C$812.02.

The amount is small. The principle is not. A chatbot is part of your website, and its words are your words. That is LLMOps in one case: production is where a demo becomes a liability. Everything in this article is about surviving that moment.

This is Part 8, the final part of the AI Engineer Series. Part 7 connected agents to real tools. This part ships the result: deployment, monitoring, cost control, and the failure modes that only appear when real users and real traffic show up. As with every part in this series, I ran the code and measured the numbers instead of quoting someone else's slides.

LLMOps is not MLOps with a new name

MLOps is about models you train: datasets, training runs, model registries, GPU clusters. LLMOps is about a model you did not train and do not own, reached through an API that changes under you. Your job shifts from training to operating: prompts, context, cost, latency, evals, and rollouts. The model is a vendor dependency, like a database or a payment provider, except this dependency invents facts and its bill grows with every word your users type.

That shift changes what shipping means. A traditional deploy is done when the code runs. An LLM deploy is done when you can answer four questions about it in production, with numbers. If you cannot, you have a demo with users, not a product.

The four numbers that decide whether you are in production

Before any tooling, decide the four numbers you will watch. Everything else in this article exists to move one of them.

• Latency, p95. Not the average. The average hides the slow requests, and slow requests are the ones users remember. Measure the 95th percentile end to end, from request to final token.

• Error rate. Failed requests over total requests. Include model timeouts, provider 5xx errors, and your own guardrail blocks. A 2% error rate is a good starting gate for a canary. Pick a number and write it down before you need it.

• Cost per 1,000 requests. Model bills are priced per million tokens, which tells you nothing about your business. Cost per 1,000 requests is the number you can budget, alert on, and explain to a founder.

• Eval pass rate. The share of a fixed test set your app answers correctly, measured on every deploy. Part 5 showed how to build one and why it matters. Latency, errors, and cost tell you the system is up. Only an eval tells you it is still good.

If you can quote all four for your app right now, you are operating it. If you cannot, the rest of this article is the shortest path to being able to.

The experiment: a gateway you can build in an afternoon

Theory is cheap here, so I built the smallest honest version of a production LLM service: a FastAPI gateway that takes a question, calls a model, logs everything to SQLite, and returns the answer with its cost attached. The model call in my test is a stub with seeded latency between 650 and 1,050 ms, because I am measuring the gateway, not a vendor's network. Every token count below is real, counted with tiktoken, and every dollar figure is that count times the published GPT-4o-mini rates of $0.15 per million input tokens and $0.60 per million output tokens. Check the current price page before you budget off these numbers. Rates move.

The gateway is short enough to read in full. The logging is the whole point.

from fastapi import FastAPI, Request
import sqlite3, time, tiktoken

enc = tiktoken.get_encoding("cl100k_base")
IN_RATE, OUT_RATE = 0.15, 0.60  # USD per 1M tokens, GPT-4o-mini

app = FastAPI()

@app.post("/chat")
async def chat(req: Request):
    t0 = time.perf_counter()
    body = await req.json()
    # call the model here with the system prompt + user question
    in_tokens = len(enc.encode(SYSTEM_PROMPT + body["q"]))
    out_tokens = len(enc.encode(answer))
    cost = (in_tokens * IN_RATE + out_tokens * OUT_RATE) / 1_000_000
    db = sqlite3.connect("llmops.db")
    db.execute(
        "INSERT INTO req_log (model, in_tokens, out_tokens, latency_ms, cost_usd, status)"
        " VALUES (?,?,?,?,?,?)",
        ("gpt-4o-mini", in_tokens, out_tokens,
         (time.perf_counter() - t0) * 1000, cost, "ok"))
    db.commit()
    return {"answer": answer, "cost_usd": round(cost, 6)}

Six fields per request: model, input tokens, output tokens, latency, cost, status. That is the entire observability stack for a small app. You do not need a tracing platform on day one. You need a table you can query. When something goes wrong at 2am, the difference between a guess and an answer is this table.

What the gateway measured

I pointed the gateway at a realistic support agent: a system prompt carrying the shop's return, shipping, warranty, order, and privacy policies, plus one customer question about a broken tent zipper on a sale item. Here are the measured numbers.

MeasurementValue

System prompt + policy tokens531
User question tokens38
Answer tokens75
Input tokens per request569
Cost per request, no caching$0.000130
Cost per 1,000 requests$0.13
Cost per 100,000 requests$13.04
Cost per request, cache write$0.000150
Cost per request, cache hit$0.000059
Monthly bill at 10,000 requests/day, no caching$39.10
Monthly bill at 10,000 requests/day, 95% cache hits$18.97

Two things stand out. First, the system prompt is 531 of the 569 input tokens. The question is almost free; the instructions are the bill. That is normal for RAG and agent apps, and it is why caching matters so much. Second, a cache hit cut the per-request cost by 55%, and the write premium paid for itself on the very first hit. Over a month at 10,000 requests a day, caching saved $20.13, or 51.5% of the bill, with the model and the tokens completely unchanged.

Then I load-tested the gateway itself: 200 requests at 20 concurrent, through the real HTTP server, with the stubbed model latency. Wall time 9.29 seconds, throughput 21.5 requests per second, 0 errors. Latency came back at p50 868 ms, p95 1,036 ms, p99 1,052 ms. The SQLite log agreed: 300 requests logged across both tests, mean gateway latency 850 ms, total logged model cost $0.0391, or $0.13 per 1,000 requests. Remember the caveat: the model call was a stub, so these latency numbers describe the gateway under load, not a real provider. The cost numbers are real token counts times published rates, and they are the numbers that decide your bill.

Prompt caching: the cheapest line on your bill

Prompt caching is the single biggest cost lever most teams never turn on. The idea is simple: the provider stores the processed form of a stable prompt prefix, and later requests that reuse it pay a fraction of the price. On Anthropic's API a cache read costs 0.1x the base input price, a 90% discount, while writing the cache costs 1.25x with the default 5-minute window, or 2x for a 1-hour window. OpenAI applies caching automatically on older models at a 50% discount on cached input. The exact numbers vary by model and change over time, so verify them against the provider docs for the model you call. The shape of the deal does not change: reads are nearly free, writes cost a little extra.

Three rules decide whether you actually get the discount. First, stable content goes first and variable content goes last. A cache hit needs a byte-identical prefix, so a timestamp or a session id at the top of the prompt kills the cache for everything after it. Put the system prompt, the policies, and the tool definitions first, and the user question last. Second, the prefix has to be long enough. Minimums vary by model: 1,024 tokens on Claude Sonnet, higher on some models, so check the docs and count your prefix with a tokenizer before assuming it caches. Third, mind the time window. The default cache lives about 5 minutes. If your traffic is sparse, the cache expires between requests and you pay the write premium every time. Caching pays on steady traffic, not on a trickle.

The scale of the win is real. Anthropic's published case study on Notion reports API costs cut by 90% and response latency cut by up to 85% after restructuring prompts around caching. My own test is smaller but the same shape: a 531-token stable prefix, 55% cheaper per cache hit, 51.5% cheaper for the month. Before you shop for a cheaper model or trim your context, check whether you are paying full price for the same 500 tokens on every single request.

Canary deploys: let the gate say no

A canary deploy sends a small share of traffic, often 5 to 10%, to the new version and compares it against the old one. If it breaks a gate, the rollout stops and traffic goes back. No meeting, no judgment call at midnight. The gate decides.

I tested the gate by shipping a bad version on purpose. Version 2 of my gateway carried a 10% injected failure rate, a timeout bug of the kind that ships more often than anyone admits. I sent it 100 requests at 10 concurrent. It failed 13 of them, a 13.0% error rate against a 2% gate. The verdict was automatic: block the rollout. A human watching the same deploy might have shipped it anyway. Thirteen failures in a hundred looks small in a dashboard, and timeouts feel like someone else's problem. The gate does not feel anything. It compares two numbers and says no.

Error rate is a crude gate, and it is only the first one. The gate that matters more is the eval pass rate from Part 5. A new prompt can answer every request on time and still be worse: more hallucinations, more refusals, more wrong refunds. Run the eval set against the canary before you promote it, and treat a drop in pass rate the same way you treat a spike in errors. The rollout that never happens is cheap. The rollback after a bad prompt reaches all your users is not.

Rollbacks need to be boring. Pin the model version and the SDK version in config, keep the previous prompt in version control, and make the rollback one flag flip, not a redeploy. If rolling back takes longer than five minutes, you do not have a rollback. You have a hope.

What to log, and what to wake up for

The six fields in the gateway log cover the basics. As traffic grows, add three more: a request id you can trace across services, the eval score for sampled requests, and the user or tenant id so one abusive customer cannot hide in the average. Tracing tools such as LangSmith and Langfuse sit on top of this same idea: every request becomes a trace you can open, filter, and replay. They are worth it once you have more than one service. They are not a substitute for the log itself.

Alerts should be few and loud. Four are enough to start: error rate above 2% for five minutes, p95 latency above 3 seconds, cost per 1,000 requests up 50% day over day, and eval pass rate down on the latest deploy. Everything else is a dashboard you check in the morning, not a page at night. And one rule about logging itself: never log full prompts that contain personal data, secrets, or API keys. Log token counts and hashes, redact the rest. Your log is a data store with the same privacy duties as your database, and a leaked prompt log is a breach.

Cost control beyond caching

Caching is the first lever, not the only one. The second is routing: send easy requests to a cheap model and hard ones to a strong model. My earlier article on model routing with a cheap-first cascade measured exactly that pattern, and five measured techniques to cut the LLM bill covers the rest of the toolkit, from shorter outputs to batching. The third lever is a hard budget: a per-user or per-day cap with a kill switch that stops paid calls instead of letting a bug or an attacker run up the bill. The fourth is output discipline. Output tokens cost four times input tokens at GPT-4o-mini rates, so a 120-word cap on support answers is not a style choice. It is a cost control.

Put the budget in the same place as the log. A cost spike you discover at the end of the month is a loss. A cost spike your log shows within the hour is a bug you can fix.

The failure modes that only show up in production

Local testing hides a whole class of failures. Here are the ones I would plan for before launch.

• Provider outage. The API will go down. Retry with backoff, fail over to a second model, and degrade honestly: tell the user the service is slow, do not invent an answer while you wait.

• Context blowup. Agent loops re-send history on every turn, so a long conversation quietly grows past the context window. Part 6 showed the agent loop; in production, cap the history, summarise it, or drop old tool results before the bill and the latency both explode.

• Injection through user input. A user can paste instructions into a support ticket, a document, or a tool result, and the model may follow them. Part 7 and the MCP hardening guide cover the gate that blocks this, and the tool poisoning demo shows what happens without it. Production needs the same gate on every untrusted input, not just tool descriptions.

• Version drift. Providers deprecate models and change defaults. Pin the model version and the SDK version, test the exact versions you ship, and read the changelog before you upgrade. A silent model swap is a silent behavior change.

• Bad retrieval, confidently delivered. A RAG app with stale or wrong documents answers wrong with full confidence. Chunking tests, hybrid search, and the RAG eval guide with Ragas are the defenses. Re-run the eval when the documents change, not just when the code changes.

• Runaway agents. An agent that can call tools can also loop, retry forever, or call an expensive tool a thousand times. Cap steps per task, cap tool calls per session, and put a cost ceiling on every agent run. Sandboxing keeps the blast radius small when a cap fails.

My opinionated take

Most LLMOps advice is written for teams with fifty engineers and a platform group. If that is not you, here is the short version. Ship behind a flag to 5% of traffic. Log six fields per request. Run an eval before every deploy. Turn on prompt caching. Set a budget with a kill switch. Do all five and you are ahead of most production LLM apps I have seen, including some run by large companies. Skip them and you will learn the same lessons from an incident, a bill, or a tribunal decision, and all three cost more.

Two more opinions, earned the hard way. First, do not buy a GPU to host your own model until the API bill justifies it. Most small apps spend under $50 a month on tokens, and a GPU costs more than that before it answers a single request. Second, LLMOps is mostly boring engineering: logs, gates, budgets, rollbacks. The teams that treat it as boring ship reliably. The teams that treat it as magic ship incidents.

Production readiness checklist

Run through this list before you call anything production.

• Four numbers on a dashboard: p95 latency, error rate, cost per 1,000 requests, eval pass rate.

• Every request logged with model, tokens, latency, cost, and status.

• Prompt caching on, with the stable prefix first and the variable content last.

• Canary deploy with an error-rate gate and an eval gate, and a rollback that takes minutes.

• Model and SDK versions pinned, and the previous prompt in version control.

• Per-user or per-day budget with a kill switch.

• Step and tool-call caps on every agent run.

• No personal data, secrets, or keys in logs.

• A fallback model and an honest degraded message for provider outages.

• An eval set that runs in CI, so a bad prompt cannot ship silently.

What is Next in This Series

This is Part 8, the final part of the AI Engineer Series. The full series:

• Part 1: AI Engineer Roadmap 2026

• Part 2: Python for AI Engineers: Asyncio, APIs, and What Tutorials Skip

• Part 3: How LLMs Actually Work: Tokens, Context, and Temperature

• Part 4: RAG from Zero: Embeddings, Vector DBs, and Retrieval

• Part 5: Evals: The Skill 56% of Jobs Want

• Part 6: Building Your First AI Agent with LangGraph

• Part 7: MCP: Connecting Agents to Real Tools Safely

• Part 8: LLMOps: Shipping AI to Production (this article)

The series ends where real work starts. You now have the whole path: a roadmap, the Python to build with, how the model works, retrieval, evals, agents, tools, and shipping. The next step is not another article. It is a small app in production with a log, a gate, and a budget. If you want to go deeper on any single piece, the standalone guides on this site cover the ground in more detail: evaluating a RAG system with Ragas, sandboxing agents, cutting the LLM bill, and routing between models. Build something small, ship it behind a flag, and measure it. That is the whole series in one sentence.

---

Originally published on LogicStack. Run the code examples right in your browser with our free interactive Python and SQL playgrounds — no setup, no sign-up.