Loading a model with a native Hugging Face pipeline (AutoModelForCausalLM) inside a web server is simple, and it is inefficient: you carry a heavy Python interpreter, get no real hardware scheduling and tend to fall over under concurrent traffic. Production uses dedicated inference engines instead. This lesson covers the two that matter: Ollama for development and edge compute, and vLLM for datacentre concurrency.
Ollama: development and edge
Ollama wraps llama.cpp and GGUF weights behind a small daemon.
Diagram: FIG 1 · Ollama wraps llama.cpp and GGUF weights behind a simple daemon and an OpenAI-compatible API. Good for development and edge machines. See the animated version.
- Memory efficiency. It can split a model's layers between CPU RAM and GPU memory, so models larger than your VRAM still run.
- A Modelfile describes a reproducible model setup.
-
OpenAI-compatible endpoints, such as
/v1/chat/completions, so existing clients work by changing the base URL.
# Modelfile
FROM gemma2:2b
PARAMETER temperature 0.2
PARAMETER num_ctx 4096
SYSTEM """You are an enterprise infrastructure diagnostic assistant. Output structured JSON only."""
ollama create infra-diag -f Modelfile
ollama run infra-diag "Summarise: link flap on port 12, CRC errors rising"
vLLM: datacentre concurrency
vLLM is built to serve many users at once from a GPU.
Diagram: FIG 2 · vLLM's scheduler interleaves requests at the token level and stores the KV-cache in blocks, so one GPU serves many users at once. See the animated version.
PagedAttention. A standard engine allocates the KV-cache in one contiguous block per request, sized for the maximum context, and much of it stays empty. The vLLM authors measured 60 to 80% of KV-cache memory wasted in earlier systems. vLLM instead stores the cache in non-contiguous blocks, like virtual-memory paging in an operating system, and reports under 4% waste.
Diagram: FIG 3 · PagedAttention applies operating-system paging to the KV-cache. Figures come from the vLLM paper and vary with workload. See the animated version.
Continuous batching. Rather than waiting for a whole batch to finish, vLLM interleaves incoming prompts into the running batch at the token level, which keeps the GPU busy and raises throughput.
Deploying vLLM
docker run -d --name enterprise-vllm \
--gpus all \
--shm-size="4g" \
-p 8000:8000 \
-v /opt/models/gemma-2-9b-awq:/model \
vllm/vllm-openai:latest \
--model /model \
--quantization awq \
--max-model-len 4096 \
--gpu-memory-utilization 0.90
Diagram: FIG 4 · gpu-memory-utilization sets how much VRAM vLLM may claim for weights and its KV-cache pool, and max-model-len limits the context per request. See the animated version.
-
--gpu-memory-utilization 0.90lets vLLM claim 90% of VRAM for the weights and the PagedAttention cache pool. It is a starting point, so leave headroom if other processes share the GPU. -
--max-model-len 4096caps the context per request, so the cache pool can serve more users at once. -
--quantization awqmatches the AWQ weights from Day 4.
vLLM exposes the same OpenAI-compatible API, so your client code does not change.
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
reply = client.chat.completions.create(
model="/model", # the name vLLM was started with
messages=[{"role": "user", "content": "List three causes of CRC errors."}],
)
print(reply.choices[0].message.content)
Pick Ollama while you build and for small or edge machines. Move to vLLM when concurrency, not convenience, is the problem.
Run it yourself
This is the exact example from the video. A NumPy simulation of the idea, not vLLM itself, so use it to understand the effect and measure your own engine for real numbers.
import numpy as np
rng = np.random.default_rng(0)
MAX, BLOCK = 4096, 16
used = rng.integers(100, 1500, 1000) # tokens really used
contig = np.full(used.size, MAX) # reserve the max
paged = np.ceil(used / BLOCK) * BLOCK # whole blocks
def waste(reserved):
return 100 * (1 - used.sum() / reserved.sum())
print(f"contiguous waste: {waste(contig):.0f}%")
print(f"paged waste: {waste(paged):.1f}%")
slots = 10**6
print("fit in 1M:", slots // MAX, "vs", int(slots // paged.mean()))
Run it:
python paged.py
Output I got while recording the video:
contiguous waste: 80%
paged waste: 0.9%
fit in 1M: 244 vs 1205
Coming Up Next
Day 6: Containerizing AI microservices: building and packaging APIs with FastAPI.
LocalAI #vLLM #Ollama #ModelServing #DevOps #DataSovereignty #HighPerformanceComputing #OpenSource
Originally published at https://sureshpallapothu.in/blog/ai-int-day-5-local-runtimes, where this post includes animated diagrams.
Top comments (0)