Phil IPS

Posted on Oct 12

Deploying LLMs Locally: Ollama, llama.cpp, and vLLM Comparison

#llm #ollama #llamacpp #tutorial

Deploying LLMs Locally: Ollama, llama.cpp, and vLLM Comparison

Run any model on your hardware — no API keys, no cloud, complete privacy.

Introduction

Local LLMs have matured. You can now run 7B-70B models on consumer hardware with near-cloud performance. This guide covers the three main engines: Ollama (easiest), llama.cpp (most flexible), vLLM (highest throughput).

Quick Comparison

Feature	Ollama	llama.cpp	vLLM
Ease of use	⭐⭐⭐⭐⭐	⭐⭐⭐	⭐⭐
Model formats	GGUF	GGUF	HF/Safetensors
API compatible	✅ OpenAI	✅ OpenAI	✅ OpenAI
Multimodal	✅	✅	✅
Batching	❌	❌	✅ PagedAttention
Quantization	Auto	Manual	Auto (AWQ/GPTQ)
GPU/CPU	Both	Both	GPU (CUDA/ROCm)
Best for	Devs, local chat	Embedded, edge	Production serving

Hardware Requirements

Model	Quantization	VRAM/RAM	CPU Only
7B	Q4_K_M	6 GB	✅ Slow
7B	Q8_0	8 GB	✅ Usable
13B	Q4_K_M	10 GB	✅ Slow
33B	Q4_K_M	20 GB	❌
70B	Q4_K_M	40 GB	❌

Rule of thumb: VRAM ≈ params × 0.5 (Q4) to 1.0 (Q8) in GB.

Option 1: Ollama — Easiest Start

Install

# Linux
curl -fsSL https://ollama.com/install.sh | sh

# macOS
brew install ollama

# Windows
winget install Ollama.Ollama

Run a Model

# Pull and run (auto-downloads GGUF)
ollama run mistral:7b-instruct-q4_K_M

# List local models
ollama list

# Show model info
ollama show mistral:7b-instruct-q4_K_M

OpenAI-Compatible API

# Start server (default port 11434)
ollama serve

# Test API
curl -X POST http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mistral:7b-instruct-q4_K_M",
    "messages": [{"role": "user", "content": "Hello!"}],
    "stream": false
  }'

Python Client

import requests

def chat(messages, model="mistral:7b-instruct-q4_K_M"):
    response = requests.post(
        "http://localhost:11434/v1/chat/completions",
        json={"model": model, "messages": messages, "stream": False}
    )
    return response.json()["choices"][0]["message"]["content"]

print(chat([{"role": "user", "content": "Write a haiku about GPUs"}]))

Custom Models (Modelfile)

# Modelfile
FROM ./my-model.Q4_K_M.gguf
TEMPLATE "{{ .System }}\n\n{{ .Prompt }}"
PARAMETER temperature 0.7
PARAMETER top_p 0.9
PARAMETER num_ctx 8192
SYSTEM "You are a helpful coding assistant."

ollama create my-coder -f Modelfile
ollama run my-coder

Option 2: llama.cpp — Maximum Control

Build

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make LLAMA_CUBLAS=1  # For NVIDIA GPU
# or: make LLAMA_METAL=1  # For Apple Silicon

Convert HF → GGUF

# Download model (needs HF token for gated models)
huggingface-cli download meta-llama/Llama-3.1-8B-Instruct --local-dir ./llama-3.1-8b

# Convert
python convert-hf-to-gguf.py ./llama-3.1-8b \
  --outfile ./llama-3.1-8b.Q4_K_M.gguf \
  --outtype q4_k_m

Quantization Types

Type	Size (7B)	Quality	Speed
Q2_K	~2.8 GB	Low	Fastest
Q3_K_M	~3.3 GB	Medium	Fast
Q4_K_M	~3.8 GB	Good	Balanced
Q5_K_M	~4.7 GB	Better	Slower
Q6_K	~5.3 GB	High	Slower
Q8_0	~7.2 GB	Near fp16	Slowest

Run Server (OpenAI Compatible)

# Basic server
./llama-server -m ./model.Q4_K_M.gguf -c 8192 --port 8080

# With GPU offload (all layers)
./llama-server -m ./model.Q4_K_M.gguf -c 8192 --port 8080 -ngl 99

# With flash attention (faster)
./llama-server -m ./model.Q4_K_M.gguf -c 8192 --port 8080 -ngl 99 -fa

Python Client

from llama_cpp import Llama

llm = Llama(
    model_path="./model.Q4_K_M.gguf",
    n_ctx=8192,
    n_gpu_layers=-1,  # -1 = all layers on GPU
    verbose=False
)

# Chat completion
response = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Hello!"}],
    temperature=0.7,
    max_tokens=512
)
print(response["choices"][0]["message"]["content"])

# Streaming
for chunk in llm.create_chat_completion(
    messages=[{"role": "user", "content": "Write a story"}],
    stream=True
):
    delta = chunk["choices"][0]["delta"]
    if "content" in delta:
        print(delta["content"], end="", flush=True)

Advanced: Speculative Decoding

# Draft model (small) + target model (large)
./llama-server -m ./llama-3.1-8b.Q4_K_M.gguf \
  -md ./llama-3.1-1b.Q4_K_M.gguf \
  -c 8192 --port 8080 -ngl 99

Option 3: vLLM — Production Throughput

Install

# Requires CUDA 12.1+ and Python 3.10+
pip install vllm

# For ROCm (AMD)
pip install vllm --extra-index-url https://download.pytorch.org/whl/rocm6.0

Run Server

# HF model (auto-downloads)
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.9 \
  --max-model-len 8192 \
  --dtype auto \
  --api-key your-secret-key

Quantized Models (AWQ/GPTQ)

# AWQ (faster inference)
vllm serve TheBloke/Llama-3.1-8B-Instruct-AWQ \
  --quantization awq \
  --gpu-memory-utilization 0.9

# GPTQ
vllm serve TheBloke/Llama-3.1-8B-Instruct-GPTQ \
  --quantization gptq \
  --gpu-memory-utilization 0.9

Multi-GPU (Tensor Parallel)

# 2 GPUs
vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.9 \
  --max-model-len 4096

Python Client (OpenAI Compatible)

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="your-secret-key"
)

# Chat
response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Hello!"}],
    temperature=0.7,
    max_tokens=512
)
print(response.choices[0].message.content)

# Streaming
stream = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Write a poem"}],
    stream=True
)
for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

Benchmarks (vLLM vs Others)

Model: Llama-3.1-8B, Batch: 32, Context: 2048

vLLM (PagedAttention):     ~450 tok/s
llama.cpp (batch 32):      ~280 tok/s
Ollama (no batching):       ~120 tok/s

Model Recommendations (Oct 2024)

General Purpose

Size	Model	Best For
1B	SmolLM2-1.7B-Instruct	Edge, mobile
3B	Phi-3.5-mini-instruct	Fast, capable
7B	Mistral-Nemo-12B-Instruct	Best 7B-class
7B	Llama-3.1-8B-Instruct	Standard baseline
7B	Qwen2.5-7B-Instruct	Multilingual, code
13B	Nemotron-3-Ultra	Reasoning

Coding

Size	Model
7B	CodeQwen1.5-7B
7B	DeepSeek-Coder-V2-Lite
13B	CodeLlama-13B-Instruct
33B	DeepSeek-Coder-V2-33B

Reasoning

Size	Model
7B	Nemotron-3-Ultra
13B	QwQ-32B-Preview (distilled)
33B	Nemotron-3-Ultra-33B

Performance Tuning

llama.cpp

# Optimal thread count (physical cores)
./llama-server -m model.gguf -t $(nproc --all) -ngl 99

# Batch size for throughput
./llama-server -m model.gguf -b 512 -ub 512

# Flash attention (requires Hopper/Ada or Apple Silicon)
./llama-server -m model.gguf -fa

vLLM

# Increase batch size
vllm serve model --max-num-batched-tokens 8192 --max-num-seqs 256

# Chunked prefill (long contexts)
vllm serve model --enable-chunked-prefill --max-model-len 32768

# Prefix caching (shared prompts)
vllm serve model --enable-prefix-caching

Ollama

# Environment variables
OLLAMA_NUM_PARALLEL=4       # Concurrent requests
OLLAMA_MAX_LOADED_MODELS=2  # Keep models in VRAM
OLLAMA_FLASH_ATTENTION=1    # Enable flash attention

Cost Comparison (Monthly)

Setup	Hardware	Models	Est. Cost
Ollama + RTX 3090	$800 used	7B-33B Q4	$15/mo electricity
llama.cpp + RTX 4090	$1600	7B-70B Q4	$25/mo electricity
vLLM + 2×A100 80GB	$30k	70B fp8	$500/mo colocation
Cloud API (OpenRouter)	—	Any	$50-500/mo usage

Break-even: Local GPU pays for itself in 6-18 months vs cloud APIs.

Common Issues

Problem	Fix
OOM on GPU	Reduce `n_gpu_layers`, use smaller quantization, offload to CPU
Slow CPU inference	Use `Q4_K_M`, enable BLAS (`LLAMA_BLAS=1`), reduce context
Model not loading	Check GGUF compatibility, try `llama.cpp` convert script
API connection refused	Check port, firewall, `OLLAMA_HOST=0.0.0.0`
Poor quality	Increase temperature, try larger model, better quantization

Decision Guide

START: What's your use case?
│
├─→ "Just chat / quick tests" → **Ollama**
│
├─→ "Embed in app / edge device / custom quant" → **llama.cpp**
│
├─→ "Serve many users / high throughput / production" → **vLLM**
│
└─→ "Apple Silicon Mac" → **Ollama** or **llama.cpp (Metal)**

Conclusion

Local LLMs are production-ready. Choose:

Ollama for developer experience
llama.cpp for control and edge deployment
vLLM for serving scale

All three expose OpenAI-compatible APIs — swap providers without code changes.

Next: Add RAG (see previous tutorial), function calling, or fine-tune your own LoRA.

Part of the Local LLM series on Dev.to

DEV Community

Deploying LLMs Locally: Ollama, llama.cpp, and vLLM Comparison

Deploying LLMs Locally: Ollama, llama.cpp, and vLLM Comparison

Introduction

Quick Comparison

Hardware Requirements

Option 1: Ollama — Easiest Start

Install

Run a Model

OpenAI-Compatible API

Python Client

Custom Models (Modelfile)

Option 2: llama.cpp — Maximum Control

Build

Convert HF → GGUF

Quantization Types

Run Server (OpenAI Compatible)

Python Client

Advanced: Speculative Decoding

Option 3: vLLM — Production Throughput

Install

Run Server

Quantized Models (AWQ/GPTQ)

Multi-GPU (Tensor Parallel)

Python Client (OpenAI Compatible)

Benchmarks (vLLM vs Others)

Model Recommendations (Oct 2024)

General Purpose

Coding

Reasoning

Performance Tuning

llama.cpp

vLLM

Ollama

Cost Comparison (Monthly)

Common Issues

Decision Guide

Conclusion

Top comments (0)