DEV Community: Mridul Nagpal

Machine learning in production: the model is the easy part

Mridul Nagpal — Wed, 24 Jun 2026 04:09:32 +0000

A model that scores 95% on your test set feels like the finish line. Then you ship it, and you find out it was the starting line. The model was maybe 10% of the work; everything that makes it survive production is the other 90%.

We deploy machine-learning systems for companies, and the projects that stall almost never stall on model accuracy. They stall on the engineering around the model. Here's what actually breaks.

1. Training-serving skew

Your model was trained on clean, batch-computed features. In production it gets features computed by different code, at request time, sometimes from a slightly different source. The distributions drift apart and accuracy quietly craters — with no error, no crash, just worse predictions.

The fix is sharing one feature-computation path between training and serving (a feature store or shared library), and logging production features so you can compare them against training. If training and serving don't compute features the same way, nothing else matters.

2. The model rots and nobody notices

The world changes — user behaviour, pricing, seasonality, an upstream schema. A model frozen at launch slowly decays against a moving target. Without monitoring, the first signal you get is a business metric dropping months later.

Monitor input drift and prediction distributions, not just uptime. Alert when the live data stops looking like the training data. Treat "the model still works" as a claim that needs evidence, not an assumption.

3. There's no retraining pipeline

"We'll retrain when it degrades" usually means a person manually re-running a notebook they half-remember. Build the retraining path early: reproducible data snapshots, an automated training run, evaluation against a held-out set, and a gate that blocks a worse model from shipping. Retraining should be a button, not an archaeology project.

4. The offline winner loses online

The model with the best test-set score is often too slow or too expensive to serve at real traffic. Latency and cost are product features. Sometimes a smaller, simpler model that answers in 50ms beats a heavyweight that needs 2 seconds — because users feel the 2 seconds and the heavyweight quietly blows the budget.

5. Accuracy is not the business metric

A fraud model at 99% accuracy is worthless if it flags so many false positives that support drowns. Define the metric that maps to the actual outcome — caught fraud net of review cost, handle-time saved, conversion lifted — and optimise that. Accuracy is a proxy, and proxies get gamed.

The pattern underneath all of these

None of this is exotic ML. It's data engineering, monitoring, CI/CD, and clear success metrics — applied to a component that happens to be statistical and non-deterministic. The teams whose models create value aren't the ones with the fanciest architecture; they're the ones who treat the model as one part of a production system that has to be observed, retrained, and held to a real metric.

That's the lens we bring from running production systems at scale before this wave. If you have models that work in a notebook but stall on the way to production, that's the kind of machine-learning deployment and consulting work we do at Krazimo.

What's killed an ML project for you on the way to production? I'll get into specifics in the comments.

RAG in production: the failure modes nobody warns you about

Mridul Nagpal — Wed, 24 Jun 2026 03:55:15 +0000

Retrieval-augmented generation looks trivial in a tutorial: embed some documents, drop them in a vector database, stuff the top matches into a prompt, done. Then you point it at real company data and real users, and you discover that the demo was the easy 10%.

We build RAG systems over private knowledge for companies, and almost every painful bug traces back to the same handful of failure modes. Here they are, and what actually fixes them.

1. Retrieval returns the wrong chunks — and the model uses them anyway

The single biggest source of "wrong" RAG answers isn't the LLM. It's retrieval handing it irrelevant or partial context, which the model then summarizes with total confidence. Naive fixed-size chunking splits a table from its header, or a clause from the sentence that negates it.

The fix is unglamorous data engineering: chunk on semantic boundaries, not character counts; add a reranking step so the top-k you actually pass is the top-k by relevance, not by raw vector distance; and store enough metadata to filter before you search. Retrieval quality sets the ceiling on everything downstream.

2. The model answers beyond its context

Even with perfect retrieval, an LLM will happily fill gaps with plausible invention. In a RAG system that's worse than no answer, because it looks sourced.

Force grounding: instruct the model to answer only from the retrieved context and to say "I don't know" when the context doesn't cover it — then verify that with citations that point back to specific chunks. If you can't trace a sentence to a source, treat it as a hallucination, not an answer.

3. The knowledge goes stale, and nobody notices

RAG is only as good as the index behind it. Documents change, get duplicated, get deleted — and a pipeline that ingested once at launch quietly serves last quarter's truth. The unsexy work is the ingestion pipeline: incremental re-indexing, de-duplication, and a freshness signal so old content can be down-weighted or expired.

4. You have no evals, so you're guessing

"The new embedding model feels better" is not an engineering statement. Without a held-out set of real questions with known-good answers, every change is a coin flip — you fix one query and silently break five. Build an eval set early, measure retrieval hit-rate and answer faithfulness on every change, and treat a regression like a failing test.

5. Latency and cost sneak up on you

Embedding the query, searching, reranking, and stuffing a large context into a big model adds up — in both seconds and dollars. Cache embeddings and frequent queries, retrieve fewer-but-better chunks rather than dumping everything, and reserve the largest model for the steps that genuinely need it.

The pattern underneath all of these

Notice what's missing from that list: clever prompting. Production RAG is a data and retrieval engineering problem wearing an AI costume. The teams whose RAG holds up aren't the ones with the fanciest prompt — they're the ones who treat ingestion, chunking, retrieval, and evaluation as real systems with real tests.

That's the lens we bring from years of building data systems at scale before this wave. If you're moving a RAG prototype toward something users can actually trust in production, that's the kind of RAG and knowledge-AI work we do at Krazimo.

What's bitten you hardest in a production RAG system? I'll dig into specifics in the comments.

What actually breaks when you put AI agents in production

Mridul Nagpal — Tue, 23 Jun 2026 08:18:04 +0000

Demos lie. An AI agent that books a meeting, queries an API, and summarizes the result in a slick demo is maybe 20% of the work. The other 80% is everything that happens when the same agent meets a real user, real data, and a Tuesday afternoon when an upstream API is having a bad day.

We build multi-agent systems for companies for a living, and the gap between "works in the notebook" and "works in production" is where most AI projects quietly die. Here are the failure modes we see most often — and what we actually do about them.

1. The agent is confidently wrong, and nothing catches it

A single LLM call has no idea when it's hallucinating. Chain three of them together and the errors compound: agent A invents a customer ID, agent B dutifully looks it up, agent C writes a confident summary about a customer who doesn't exist.

The fix isn't a better prompt. It's treating every agent output as untrusted input — the same discipline you'd apply to a form field from the public internet. Validate structured outputs against a schema. Make tools return typed results, not prose. And put a deterministic check between "the model decided X" and "X happened in your database."

[REAL EXAMPLE → a time a validation layer caught a bad agent action before it hit prod, and what the check was]

2. There's no boundary between "thinking" and "doing"

The scariest agent bugs aren't wrong answers — they're wrong actions. An agent with write access to your systems and a fuzzy objective will eventually do something irreversible.

We draw a hard line between read and write. Agents can plan, retrieve, and propose freely. Anything that mutates state — sends an email, charges a card, updates a record — goes through a narrow, audited gate with explicit guardrails, and often a human approval step for the high-stakes ones. It's the difference between an assistant and a loose cannon.

3. You can't see what the agent did

When a multi-agent run fails, "the AI messed up" is not a debuggable statement. If you can't replay exactly which agent called which tool with which arguments and got which result, you're flying blind — and you will be flying blind at 2am when a customer complains.

Tracing is not optional. Every agent step, every tool call, every input and output gets logged and is replayable. This one decision — building observability before you scale the agents — saves more production incidents than any model upgrade.

4. Cost and latency are a feature, not an afterthought

A multi-agent system that calls the biggest model for every step is both slow and expensive, and users feel both. The engineering work is matching the model to the job: a small fast model for routing and classification, the big one only where reasoning genuinely pays for itself. Cache aggressively. Set hard timeouts. Budget tokens like you'd budget any other resource.

[REAL EXAMPLE → a before/after on latency or cost when you right-sized models for a real client]

5. "Done" is undefined, so it's never done

The most common reason an AI project stalls isn't technical — it's that nobody agreed what success looks like before building. "Add AI agents" is not a spec. "Cut average handle time on tier-1 tickets by 30% without raising escalations" is.

We define the success metric before writing a line of code. It tells you which failures matter, when to ship, and whether the thing is actually working — instead of shipping an impressive demo that moves no real number.

The pattern underneath all of these

None of this is exotic AI knowledge. It's ordinary production engineering — input validation, least-privilege access, observability, resource budgets, clear requirements — applied to a new and unusually unpredictable component. The teams that ship reliable agents aren't the ones with the cleverest prompts. They're the ones who treat the LLM as one more untrusted, expensive, non-deterministic dependency and engineer around it accordingly.

That's the lens we bring from years of building systems at scale before this wave: the AI is new, but the discipline that makes it production-grade is not. If you're moving an agent system from demo to production and want a second set of eyes from a team that does this daily, that's the kind of multi-agent AI work we do at Krazimo.

What failure modes have bitten you in production? I'll answer questions in the comments.