DEV Community

Suresh Kumar Pallapothu
Suresh Kumar Pallapothu

Posted on Originally published at sureshpallapothu.in

Day 5: Local AI Runtimes — Deploying Ollama and vLLM for Bare-Metal Execution

Loading a model with a native Hugging Face pipeline (AutoModelForCausalLM) inside a web server is simple, and it is inefficient: you carry a heavy Python interpreter, get no real hardware scheduling and tend to fall over under concurrent traffic. Production uses dedicated inference engines instead. This lesson covers the two that matter: Ollama for development and edge compute, and vLLM for datacentre concurrency.

Ollama: development and edge

Ollama wraps llama.cpp and GGUF weights behind a small daemon.

Diagram: FIG 1 · Ollama wraps llama.cpp and GGUF weights behind a simple daemon and an OpenAI-compatible API. Good for development and edge machines. See the animated version.

  • Memory efficiency. It can split a model's layers between CPU RAM and GPU memory, so models larger than your VRAM still run.
  • A Modelfile describes a reproducible model setup.
  • OpenAI-compatible endpoints, such as /v1/chat/completions, so existing clients work by changing the base URL.
# Modelfile
FROM gemma2:2b
PARAMETER temperature 0.2
PARAMETER num_ctx 4096
SYSTEM """You are an enterprise infrastructure diagnostic assistant. Output structured JSON only."""
Enter fullscreen mode Exit fullscreen mode
ollama create infra-diag -f Modelfile
ollama run infra-diag "Summarise: link flap on port 12, CRC errors rising"
Enter fullscreen mode Exit fullscreen mode

vLLM: datacentre concurrency

vLLM is built to serve many users at once from a GPU.

Diagram: FIG 2 · vLLM's scheduler interleaves requests at the token level and stores the KV-cache in blocks, so one GPU serves many users at once. See the animated version.

PagedAttention. A standard engine allocates the KV-cache in one contiguous block per request, sized for the maximum context, and much of it stays empty. The vLLM authors measured 60 to 80% of KV-cache memory wasted in earlier systems. vLLM instead stores the cache in non-contiguous blocks, like virtual-memory paging in an operating system, and reports under 4% waste.

Diagram: FIG 3 · PagedAttention applies operating-system paging to the KV-cache. Figures come from the vLLM paper and vary with workload. See the animated version.

Continuous batching. Rather than waiting for a whole batch to finish, vLLM interleaves incoming prompts into the running batch at the token level, which keeps the GPU busy and raises throughput.

Deploying vLLM

docker run -d --name enterprise-vllm \
  --gpus all \
  --shm-size="4g" \
  -p 8000:8000 \
  -v /opt/models/gemma-2-9b-awq:/model \
  vllm/vllm-openai:latest \
  --model /model \
  --quantization awq \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.90
Enter fullscreen mode Exit fullscreen mode

Diagram: FIG 4 · gpu-memory-utilization sets how much VRAM vLLM may claim for weights and its KV-cache pool, and max-model-len limits the context per request. See the animated version.

  • --gpu-memory-utilization 0.90 lets vLLM claim 90% of VRAM for the weights and the PagedAttention cache pool. It is a starting point, so leave headroom if other processes share the GPU.
  • --max-model-len 4096 caps the context per request, so the cache pool can serve more users at once.
  • --quantization awq matches the AWQ weights from Day 4.

vLLM exposes the same OpenAI-compatible API, so your client code does not change.

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
reply = client.chat.completions.create(
    model="/model",                       # the name vLLM was started with
    messages=[{"role": "user", "content": "List three causes of CRC errors."}],
)
print(reply.choices[0].message.content)
Enter fullscreen mode Exit fullscreen mode

Pick Ollama while you build and for small or edge machines. Move to vLLM when concurrency, not convenience, is the problem.

Run it yourself

This is the exact example from the video. A NumPy simulation of the idea, not vLLM itself, so use it to understand the effect and measure your own engine for real numbers.

import numpy as np

rng = np.random.default_rng(0)
MAX, BLOCK = 4096, 16
used = rng.integers(100, 1500, 1000)   # tokens really used

contig = np.full(used.size, MAX)  # reserve the max
paged = np.ceil(used / BLOCK) * BLOCK  # whole blocks

def waste(reserved):
    return 100 * (1 - used.sum() / reserved.sum())

print(f"contiguous waste: {waste(contig):.0f}%")
print(f"paged waste: {waste(paged):.1f}%")
slots = 10**6
print("fit in 1M:", slots // MAX, "vs", int(slots // paged.mean()))
Enter fullscreen mode Exit fullscreen mode

Run it:

python paged.py
Enter fullscreen mode Exit fullscreen mode

Output I got while recording the video:

contiguous waste: 80%
paged waste: 0.9%
fit in 1M: 244 vs 1205
Enter fullscreen mode Exit fullscreen mode

Coming Up Next

Day 6: Containerizing AI microservices: building and packaging APIs with FastAPI.

LocalAI #vLLM #Ollama #ModelServing #DevOps #DataSovereignty #HighPerformanceComputing #OpenSource


Originally published at https://sureshpallapothu.in/blog/ai-int-day-5-local-runtimes, where this post includes animated diagrams.

Top comments (0)