Hello, I'm Shrijith Venkatramana. I'm building git-lrc, an AI code reviewer that runs on every commit. Star Us to help devs discover the project. Do give it a try and share your feedback for improving the product.
Training a model and deploying it are two very different problems.
After completing a PyTorch tutorial, it's easy to think of the resulting .pt file as the artifact that matters. In production, however, the challenge shifts from training to serving: accepting requests, loading models, scheduling inference, managing GPUs, and returning responses with predictable latency.
As LLMs have grown larger, serving infrastructure has become a significant engineering problem in its own right. John Hennessy remarked in 2023 that running an LLM request could be around ten times more expensive than a traditional keyword search, making inference costs an important consideration for production systems.
This article walks through the fundamentals of model serving, from what a deployed model actually consists of to why frameworks like vLLM and Triton exist.
1. A Model Is More Than a Weight File
It's common to think of a trained model as simply a large collection of learned weights.
In practice, a deployed model consists of three distinct parts:
- Model architecture – the code that defines the structure of the neural network.
- Model data – the learned weights, biases, and configuration.
- Execution code – the runtime responsible for loading the model and executing inference.
The book describes models as executable programs rather than passive data files. The weights alone are not enough—they need an architecture and an execution runtime before they can produce predictions.
One practical consequence is that many projects store the architecture and weights separately. This makes it possible to evolve the architecture while partially loading compatible weights during deployment, rather than retraining the model from scratch.
2. Training and Serving Have Different Goals
Training and serving optimize for different workloads.
During training, the objective is to update model parameters using backpropagation. This usually involves very large batches running across multiple GPUs to maximize throughput.
Serving only performs forward propagation. Instead of optimizing for training speed, it focuses on:
- low latency
- high availability
- scalability
- predictable operational costs
The serving stack is expected to handle anything from a handful of requests to millions while maintaining consistent response times. Cost-to-serve becomes one of the primary metrics when evaluating different deployment approaches.
3. What Model Serving Actually Looks Like
Production model serving involves considerably more than exposing a generate() endpoint.
A typical request flows through several components before it reaches the GPU:
Client
↓
API Gateway
↓
Load Balancer
↓
Inference Server
↓
Continuous Batch Scheduler
↓
GPU
Traditional web infrastructure handles authentication, routing, and load balancing. ML-specific infrastructure is responsible for tokenization, scheduling, batching requests together, executing inference, and streaming responses back to clients. The architecture shown in the book separates these responsibilities explicitly.
4. Containerization Is the Foundation
Most production inference happens inside containers.
A serving container typically includes:
- the serving API
- model management
- the inference backend
- access to model storage
The container exposes an HTTP or gRPC interface while handling model initialization, execution, and resource management internally. This allows the same deployment to run consistently across development, testing, and production environments.
A simplified request flow looks like this:
Request
↓
Serving API
↓
Inference Backend
↓
Model Runtime
↓
Prediction
5. Why Specialized Serving Frameworks Exist
General-purpose ML libraries are designed primarily for experimentation and training. Serving large language models efficiently requires additional optimizations.
Frameworks such as vLLM, TensorRT-LLM, and SGLang are built specifically for inference workloads.
For example, a typical vLLM deployment exposes tuning parameters such as GPU memory utilization, maximum concurrent sequences, tensor parallelism, and maximum batched tokens:
python -m vllm.entrypoints.openai.api_server \
--model openai/gpt-oss-20b \
--dtype bf16 \
--gpu-memory-utilization 0.9 \
--max-num-seqs 16 \
--max-num-batched-tokens 16384 \
--tensor-parallel-size 2
These settings control how requests are scheduled and distributed across available hardware.
The benchmark reproduced in Hands-On LLM Serving and Optimization shows substantially higher serving throughput for vLLM than standard Hugging Face serving under the tested configurations.
6. Serving More Than One Model
Many production applications rely on multiple models rather than a single LLM.
A typical application might combine:
- an embedding model for semantic search
- a reranker
- a vision model
- a language model for generation
Managing each model independently quickly becomes operationally complex.
NVIDIA Triton Inference Server addresses this by providing a unified serving platform capable of loading multiple models in different formats, including PyTorch, ONNX, TensorRT, and TensorFlow. It also manages model loading, caching, and routing through a common API.
7. Why Serving Economics Matter
The cost of serving an LLM is determined by more than GPU prices.
A large part of the problem is utilization.
GPUs that spend significant time idle still incur the same infrastructure cost. Shared inference platforms can aggregate requests from many customers, keeping hardware utilization high and reducing the average cost per request.
The notes summarize the resulting cost hierarchy as:
- Model labs
- General-purpose inference providers
- Enterprise self-hosted deployments
The more effectively GPU capacity is utilized, the lower the cost of serving each token tends to be.
Closing Thoughts
Model serving sits at the intersection of machine learning and distributed systems.
The model itself is only one component of a production deployment. The serving stack is responsible for scheduling requests, managing hardware, exposing APIs, handling failures, and keeping inference costs under control.
Understanding these pieces makes it easier to evaluate serving frameworks, deployment architectures, and the trade-offs between latency, throughput, and operational cost.
What serving framework or deployment architecture are you using today, and what trade-offs led you to that choice?
*AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.
git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.*
Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.
HexmosTech
/
git-lrc
Free, Micro AI Code Reviews That Run on Git Commit
| 🇩🇰 Dansk | 🇪🇸 Español | 🇮🇷 Farsi | 🇫🇮 Suomi | 🇯🇵 日本語 | 🇳🇴 Norsk | 🇵🇹 Português | 🇷🇺 Русский | 🇦🇱 Shqip | 🇨🇳 中文 | 🇮🇳 हिन्दी |
git-lrc
Free, Micro AI Code Reviews That Run on Commit
GenAI today is a race car without brakes. It accelerates fast -- you describe something, and large blocks of code appear instantly. But AI agents silently break things: they remove logic, relax constraints, introduce expensive cloud calls, leak credentials, and change behavior -- without telling you. You often find out in production.
git-lrc is your braking system. It hooks into git commit and runs an AI review on every diff before it lands. 60-second setup. Completely free.
In short, git-lrc helps Prevent Outages, Breaches, and Technical Debt Before They Happen
At a glance: 10 risk categories · 100+ failure patterns tracked · every commit…
Top comments (0)