Jupyter notebooks do not serve users. That sounds obvious, but it describes the most common failure mode in ML projects: a well-trained model that never reaches production because the team underestimated the engineering distance between a working notebook and a working web app. O'Reilly's 2025 AI and Data Show found that 57 percent of ML practitioners identify deployment and integration as the primary bottleneck to production, ahead of both data quality and model accuracy. The model is often the easy part. The three layers that wrap it are where the actual engineering work lives.
The frontend never talks to the model directly, and the model never talks to the browser directly. Everything flows through the API layer, which is what makes each piece independently replaceable: swap the model without touching the frontend, swap the frontend framework without touching inference code.
API layer: FastAPI is the default for a reason
FastAPI plus Uvicorn is the correct default for most business ML web apps. Its async-first design keeps the server thread free during inference, unlike Flask, where a 200ms transformer call blocks the thread and caps throughput near five requests per second per worker:
# FastAPI: the thread stays free during inference
@app.post("/predict")
async def predict(payload: PredictRequest):
cached = await cache.get(payload.hash())
if cached:
return cached
result = await run_in_threadpool(model.infer, payload.features)
await cache.set(payload.hash(), result, ttl=3600)
return result
If the use case is a churn dashboard updated nightly rather than a real-time prediction, skip the API layer's real-time path entirely: a batch pipeline needs no synchronous inference, and infrastructure cost drops 80 to 90 percent versus equivalent real-time serving for identical business value.
Model serving layer
: packaging and monitoring are not details
A model pickled in Python is tied to the exact Python and library versions used during training. A model exported to ONNX runs on any ONNX-compatible runtime, CPU, GPU via TensorRT, even WASM in-browser, decoupled entirely from the training environment. For anything expected to run past 12 months, ONNX is the right call.
Monitoring is the piece most often skipped at launch and most regretted six months later. An unmonitored model degrades silently: accuracy drops as input distribution shifts, and nothing alerts because nobody instrumented it. Evidently AI, running as a sidecar to the inference endpoint, compares incoming feature distributions against the training baseline and flags drift past a defined threshold.
Three latency traps that kill production ML apps
Cold start on serverless inference: Lambda and Cloud Run spin down idle containers, and the first request after idle takes 2 to 8 seconds, worse than useless for a checkout flow. Provisioned concurrency adds 15 to 60 dollars a month per endpoint and eliminates it entirely.
No prediction cache for repeated inputs: identical queries re-run inference every time when they don't need to, as the code sample above shows. Hashing input features and caching in Redis with a sensible TTL cuts inference cost 30 to 60 percent for high-repeat patterns.
Oversized model for the task:
a GPT-4-class model handling simple binary classification that a fine-tuned DistilBERT would handle at a fraction of the cost and several times the speed. This one compounds, hitting latency and cost at the same time.
The model you choose sets the accuracy ceiling. The three-layer architecture above determines whether that accuracy ever reaches a user reliably, cheaply, and without silent degradation. The notebook is the start of the work. The stack is the deliverable.
This piece was originally published in longer form on shreyans.tech, where it includes the full model-integration walkthrough, the complete tech stack table, and an FAQ section.
About the author: Shreyans Padmani is a freelance AI and ML developer with a 100 percent Upwork job success score and 12 published case studies with quantified business outcomes. He writes about production ML engineering at shreyans.tech. If you're moving a model from notebook to production, his machine learning development services page covers architecture selection, API construction, and deployment across hourly, monthly, and fixed-price engagement models.
Top comments (0)