DEV Community

Phil IPS
Phil IPS

Posted on

Deploying LLMs Locally: Ollama, llama.cpp, vLLM & Text Generation Inference

#ollama #llamacpp #vllm #tutorial

Deploying LLMs Locally: Ollama, llama.cpp, vLLM & Text Generation Inference

Complete guide to running LLMs locally — from quick experiments with Ollama to production-grade serving with vLLM and TGI.

Introduction

Local LLM deployment offers privacy, cost savings, and offline capability. This guide covers the four main approaches with production configurations.

Quick Comparison

Tool Best For Ease of Use Performance Hardware
Ollama Quick start, Mac/Win/Linux ⭐⭐⭐⭐⭐ Good CPU/GPU
llama.cpp CPU inference, quantization ⭐⭐⭐ Excellent (CPU) CPU/GPU
vLLM High-throughput serving ⭐⭐⭐ Best (GPU) GPU required
TGI Enterprise, OpenAI-compatible ⭐⭐⭐ Excellent GPU required

Part 1: Ollama — Easiest Start

Installation & Basics 

# macOS
brew install ollama

# Linux
curl -fsSL https://ollama.com/install.sh | sh

# Windows
# Download from https://ollama.com/download

# Start service
ollama serve

# Pull model
ollama pull llama3.1:8b
ollama pull codellama:13b
ollama pull nemotron-mini:4b
Enter fullscreen mode Exit fullscreen mode

Python Integration 

# ollama/client.py
import ollama
from typing import List, Dict, Optional

class OllamaClient:
    def __init__(self, host: str = "http://localhost:11434"):
        self.client = ollama.Client(host=host)

    def list_models(self) -> List[Dict]:
        return self.client.list()["models"]

    def pull(self, model: str) -> None:
        self.client.pull(model)

    def generate(
        self,
        model: str,
        prompt: str,
        system: str = None,
        temperature: float = 0.7,
        top_p: float = 0.9,
        num_predict: int = -1,
        stream: bool = False
    ) -> str:
        response = self.client.generate(
            model=model,
            prompt=prompt,
            system=system,
            options={
                "temperature": temperature,
                "top_p": top_p,
                "num_predict": num_predict,
            },
            stream=stream
        )
        return response["response"]

    def chat(
        self,
        model: str,
        messages: List[Dict],
        stream: bool = False
    ):
        return self.client.chat(model=model, messages=messages, stream=stream)

    def embeddings(self, model: str, prompt: str) -> List[float]:
        return self.client.embeddings(model=model, prompt=prompt)["embedding"]

# Usage
client = OllamaClient()
response = client.generate("llama3.1:8b", "Explain LoRA in 3 sentences")
print(response)

# Streaming chat
for chunk in client.chat("llama3.1:8b", [
    {"role": "user", "content": "Write a haiku about GPUs"}
], stream=True):
    print(chunk["message"]["content"], end="", flush=True)
Enter fullscreen mode Exit fullscreen mode

Custom Modelfiles 

# Modelfile for custom model
FROM llama3.1:8b

# System prompt
SYSTEM """You are a senior Python developer. Write clean, typed, documented code."""

# Parameters
PARAMETER temperature 0.3
PARAMETER top_p 0.9
PARAMETER num_ctx 8192
PARAMETER num_predict 2048

# Template (optional)
TEMPLATE """{{ .System }}User: {{ .Prompt }}Assistant: """

# Save and create
# ollama create python-expert -f Modelfile
Enter fullscreen mode Exit fullscreen mode

Ollama API Server 

# ollama/api_server.py
from fastapi import FastAPI
from pydantic import BaseModel
import ollama

app = FastAPI()

class GenerateRequest(BaseModel):
    model: str
    prompt: str
    system: str = None
    temperature: float = 0.7
    stream: bool = False

@app.post("/generate")
async def generate(req: GenerateRequest):
    if req.stream:
        return StreamingResponse(
            stream_generate(req),
            media_type="text/event-stream"
        )

    response = ollama.generate(
        model=req.model,
        prompt=req.prompt,
        system=req.system,
        options={"temperature": req.temperature}
    )
    return {"response": response["response"]}

async def stream_generate(req: GenerateRequest):
    stream = ollama.generate(
        model=req.model,
        prompt=req.prompt,
        system=req.system,
        options={"temperature": req.temperature},
        stream=True
    )
    for chunk in stream:
        yield f"data: {chunk['response']}\n\n"

# Run: uvicorn api_server:app --host 0.0.0.0 --port 8000
Enter fullscreen mode Exit fullscreen mode

Part 2: llama.cpp — CPU Optimization

Building from Source 

# Ubuntu/Debian
sudo apt update && sudo apt install -y build-essential cmake git

# Clone
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp

# CPU-only build (AVX2)
make LLAMA_CUBLAS=0

# With CUDA (NVIDIA GPU)
make LLAMA_CUBLAS=1

# With Metal (Apple Silicon)
make LLAMA_METAL=1

# With Vulkan (AMD/Intel GPU)
make LLAMA_VULKAN=1
Enter fullscreen mode Exit fullscreen mode

Model Conversion & Quantization 

# Download HF model
huggingface-cli download meta-llama/Meta-Llama-3.1-8B --local-dir llama3.1-8b

# Convert to GGUF
python convert_hf_to_gguf.py llama3.1-8b --outfile llama3.1-8b.f16.gguf --outtype f16

# Quantize (smaller, faster, slight quality loss)
# Q4_K_M = best balance (4-bit)
./llama-quantize llama3.1-8b.f16.gguf llama3.1-8b.Q4_K_M.gguf Q4_K_M

# Other quantization levels
# Q2_K  - smallest, lowest quality
# Q3_K_M - small, decent
# Q4_K_M - recommended default
# Q5_K_M - higher quality
# Q6_K  - near original
# Q8_0  - almost lossless
Enter fullscreen mode Exit fullscreen mode

Running llama.cpp Server 

# Basic server
./llama-server -m llama3.1-8b.Q4_K_M.gguf -c 8192 --host 0.0.0.0 --port 8080

# With GPU offloading (Metal)
./llama-server -m llama3.1-8b.Q4_K_M.gguf -c 8192 --host 0.0.0.0 --port 8080 -ngl 99

# With CUDA
./llama-server -m llama3.1-8b.Q4_K_M.gguf -c 8192 --host 0.0.0.0 --port 8080 -ngl 99 -mgpu 0

# Performance options
./llama-server \
  -m model.gguf \
  -c 8192 \           # Context size
  -b 512 \            # Batch size
  -ub 512 \           # UBatch size
  -ngl 99 \           # GPU layers (-1 = all)
  -t 8 \              # Threads
  --flash-attn \      # Flash attention
  --mlock \           # Lock in RAM
  --no-mmap \         # Don't mmap (for containers)
Enter fullscreen mode Exit fullscreen mode

Python Client 

# llama_cpp/client.py
from llama_cpp import Llama
from typing: List, Dict
import json

class LlamaCppClient:
    def __init__(
        self,
        model_path: str,
        n_ctx: int = 8192,
        n_gpu_layers: int = -1,
        n_threads: int = 8,
        verbose: bool = False
    ):
        self.llm = Llama(
            model_path=model_path,
            n_ctx=n_ctx,
            n_gpu_layers=n_gpu_layers,
            n_threads=n_threads,
            verbose=verbose,
            chat_format="llama-3",  # or "chatml", "mistral-instruct"
        )

    def generate(
        self,
        prompt: str,
        max_tokens: int = 512,
        temperature: float = 0.7,
        top_p: float = 0.9,
        stop: List[str] = None,
        stream: bool = False
    ):
        return self.llm(
            prompt=prompt,
            max_tokens=max_tokens,
            temperature=temperature,
            top_p=top_p,
            stop=stop,
            stream=stream
        )

    def chat(
        self,
        messages: List[Dict],
        max_tokens: int = 512,
        temperature: float = 0.7,
        stream: bool = False
    ):
        return self.llm.create_chat_completion(
            messages=messages,
            max_tokens=max_tokens,
            temperature=temperature,
            stream=stream
        )

    def embeddings(self, texts: List[str]) -> List[List[float]]:
        return [self.llm.embed(text) for text in texts]

# Usage
client = LlamaCppClient("./llama3.1-8b.Q4_K_M.gguf", n_gpu_layers=99)
response = client.generate("Explain quantization:")
print(response["choices"][0]["text"])
Enter fullscreen mode Exit fullscreen mode

Part 3: vLLM — High-Throughput Serving

Installation 

# Requires NVIDIA GPU with CUDA 11.8+
pip install vllm

# Or with specific CUDA
pip install vllm --extra-index-url https://download.pytorch.org/whl/cu118

# For AMD ROCm
pip install vllm --extra-index-url https://download.pytorch.org/whl/rocm5.6
Enter fullscreen mode Exit fullscreen mode

Basic Server 

# OpenAI-compatible server
vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 8000 \
  --dtype auto \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.9 \
  --tensor-parallel-size 1 \
  --enforce-eager

# With quantization (AWQ/GPTQ)
vllm serve TheBloke/Llama-3.1-8B-Instruct-AWQ \
  --quantization awq \
  --max-model-len 8192

# Multi-GPU tensor parallel
vllm serve meta-llama/Meta-Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.9
Enter fullscreen mode Exit fullscreen mode

Python Client (OpenAI Compatible) 

# vllm/client.py
from openai import OpenAI
from typing: List, Dict
import asyncio

class VLLMClient:
    def __init__(self, base_url: str = "http://localhost:8000/v1"):
        self.client = OpenAI(
            base_url=base_url,
            api_key="dummy"  # Not used by vLLM
        )

    def list_models(self):
        return self.client.models.list()

    def generate(
        self,
        model: str,
        prompt: str,
        max_tokens: int = 512,
        temperature: float = 0.7,
        top_p: float = 0.9,
        stop: List[str] = None,
        stream: bool = False
    ):
        return self.client.completions.create(
            model=model,
            prompt=prompt,
            max_tokens=max_tokens,
            temperature=temperature,
            top_p=top_p,
            stop=stop,
            stream=stream
        )

    def chat(
        self,
        model: str,
        messages: List[Dict],
        max_tokens: int = 512,
        temperature: float = 0.7,
        stream: bool = False
    ):
        return self.client.chat.completions.create(
            model=model,
            messages=messages,
            max_tokens=max_tokens,
            temperature=temperature,
            stream=stream
        )

    async def achat(self, *args, **kwargs):
        return await self.client.chat.completions.create(*args, **kwargs)

# Usage
client = VLLMClient()

# Chat completion
response = client.chat("meta-llama/Meta-Llama-3.1-8B-Instruct", [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Write a Python decorator for retry logic"}
])
print(response.choices[0].message.content)

# Streaming
for chunk in client.chat(model, messages, stream=True):
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
Enter fullscreen mode Exit fullscreen mode

Advanced vLLM Features 

# vllm/advanced.py
from vllm import LLM, SamplingParams
from vllm.lora.request import LoRARequest

# Offline inference (no server)
llm = LLM(
    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
    tensor_parallel_size=1,
    gpu_memory_utilization=0.9,
    max_model_len=8192,
    enable_lora=True,
    max_lora_rank=64,
)

sampling_params = SamplingParams(
    temperature=0.7,
    top_p=0.9,
    max_tokens=512,
    stop=["</s>", "<|eot_id|>"],
)

# Batch generation
prompts = [
    "Write a function for: ",
    "Explain the concept of: ",
    "Debug this code: ",
]
outputs = llm.generate(prompts, sampling_params)

# With LoRA adapter
lora_outputs = llm.generate(
    prompts,
    sampling_params,
    lora_request=LoRARequest("sql_adapter", 1, "./sql-lora-adapter")
)

# Structured output (guided decoding)
from vllm import SamplingParams
from pydantic import BaseModel

class CodeResponse(BaseModel):
    explanation: str
    code: str
    language: str

guided_params = SamplingParams(
    temperature=0.3,
    max_tokens=1024,
    guided_json=CodeResponse.model_json_schema()
)

output = llm.generate(["Write a binary search in Python"], guided_params)
print(output[0].outputs[0].text)  # Valid JSON matching schema
Enter fullscreen mode Exit fullscreen mode

vLLM Performance Tuning 

# vllm/tuning.sh

# Key parameters:
# --max-model-len: Context window (affects KV cache memory)
# --gpu-memory-utilization: Fraction of GPU memory for KV cache (0.8-0.95)
# --block-size: KV cache block size (16 or 32)
# --max-num-seqs: Max concurrent sequences
# --max-num-batched-tokens: Max tokens per batch
# --enable-chunked-prefill: For long contexts
# --disable-log-stats: Reduce logging overhead

# High-throughput config
vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.95 \
  --max-num-seqs 256 \
  --max-num-batched-tokens 8192 \
  --enable-chunked-prefill \
  --disable-log-stats \
  --block-size 16

# Low-latency config
vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.8 \
  --max-num-seqs 16 \
  --enforce-eager \
  --disable-log-stats
Enter fullscreen mode Exit fullscreen mode

Part 4: Text Generation Inference (TGI)

Installation 

# Docker (recommended)
docker run --gpus all --shm-size 1g \
  -p 8080:80 \
  -v $PWD/data:/data \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model-id meta-llama/Meta-Llama-3.1-8B-Instruct \
  --max-input-length 4096 \
  --max-total-tokens 8192 \
  --max-batch-prefill-tokens 4096

# With quantization
docker run --gpus all --shm-size 1g \
  -p 8080:80 \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model-id TheBloke/Llama-3.1-8B-Instruct-AWQ \
  --quantize awq

# Multi-GPU
docker run --gpus all --shm-size 1g \
  -p 8080:80 \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model-id meta-llama/Meta-Llama-3.1-70B-Instruct \
  --num-shard 4
Enter fullscreen mode Exit fullscreen mode

Python Client 

# tgi/client.py
from text_generation import Client
from typing: List, Dict
import asyncio

class TGIClient:
    def __init__(self, url: str = "http://localhost:8080"):
        self.client = Client(url, timeout=60)

    def generate(
        self,
        prompt: str,
        max_new_tokens: int = 512,
        temperature: float = 0.7,
        top_p: float = 0.9,
        stop_sequences: List[str] = None,
        stream: bool = False
    ):
        return self.client.generate(
            prompt,
            max_new_tokens=max_new_tokens,
            temperature=temperature,
            top_p=top_p,
            stop_sequences=stop_sequences,
            stream=stream
        )

    async def agenerate(self, *args, **kwargs):
        return await self.client.generate_async(*args, **kwargs)

# Usage
client = TGIClient()
response = client.generate("Explain RAG:")
print(response.generated_text)
Enter fullscreen mode Exit fullscreen mode

Part 5: Model Selection Guide

Recommended Models by Use Case 

# models/recommendations.py
RECOMMENDED_MODELS = {
    "general_chat": [
        "meta-llama/Meta-Llama-3.1-8B-Instruct",
        "mistralai/Mistral-7B-Instruct-v0.3",
        "google/gemma-2-9b-it",
        "Qwen/Qwen2.5-7B-Instruct",
    ],
    "coding": [
        "deepseek-ai/deepseek-coder-6.7b-instruct",
        "Qwen/Qwen2.5-Coder-7B-Instruct",
        "bigcode/starcoder2-7b",
        "codellama/CodeLlama-13b-Instruct-hf",
    ],
    "reasoning": [
        "deepseek-ai/DeepSeek-R1-Distill-Llama-8B",
        "Qwen/QwQ-32B-Preview",
    ],
    "multilingual": [
        "Qwen/Qwen2.5-7B-Instruct",
        "google/gemma-2-9b-it",
        "CohereForAI/aya-23-8B",
    ],
    "small_efficient": [
        "microsoft/Phi-3.5-mini-instruct",
        "google/gemma-2-2b-it",
        "Qwen/Qwen2.5-1.5B-Instruct",
    ],
    "long_context": [
        "gradientai/Llama-3-8B-Instruct-Gradient-1048k",
        "Qwen/Qwen2.5-7B-Instruct-1M",
    ],
}

# Quantization recommendations
QUANTIZATION_GUIDE = {
    "4bit": "Q4_K_M (llama.cpp) / AWQ (vLLM/TGI) — best balance",
    "8bit": "Q8_0 / GPTQ — near lossless",
    "2bit": "Q2_K — only for extreme constraints",
}
Enter fullscreen mode Exit fullscreen mode

Hardware Requirements 

# models/hardware.py
MODEL_REQUIREMENTS = {
    "7B_f16": {"vram_gb": 14, "ram_gb": 16, "disk_gb": 14},
    "7B_q4":  {"vram_gb": 6,  "ram_gb": 8,  "disk_gb": 4.5},
    "7B_q8":  {"vram_gb": 10, "ram_gb": 12, "disk_gb": 8},
    "13B_f16": {"vram_gb": 26, "ram_gb": 32, "disk_gb": 26},
    "13B_q4":  {"vram_gb": 10, "ram_gb": 16, "disk_gb": 8},
    "70B_f16": {"vram_gb": 140, "ram_gb": 160, "disk_gb": 140},
    "70B_q4":  {"vram_gb": 48, "ram_gb": 64, "disk_gb": 40},
}

def estimate_requirements(model_size: str, quantization: str) -> dict:
    """Estimate hardware needs."""
    key = f"{model_size}_{quantization}"
    return MODEL_REQUIREMENTS.get(key, {"error": "Unknown combination"})

# Apple Silicon unified memory
# M1/M2/M3 Max (96GB) → 70B q4
# M1/M2/M3 Pro (36GB) → 13B q4 or 7B f16
# M1/M2/M3 Base (8-24GB) → 7B q4
Enter fullscreen mode Exit fullscreen mode

Part 6: Production Deployment

Docker Compose Stack 

# deployment/docker-compose.yml
version: '3.8'

services:
  # vLLM for high throughput
  vllm:
    image: vllm/vllm-openai:latest
    runtime: nvidia
    environment:
      - HF_TOKEN=${HF_TOKEN}
    command: >
      --model meta-llama/Meta-Llama-3.1-8B-Instruct
      --max-model-len 8192
      --gpu-memory-utilization 0.9
      --tensor-parallel-size 1
      --host 0.0.0.0
      --port 8000
    ports:
      - "8000:8000"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

  # TGI for OpenAI compatibility
  tgi:
    image: ghcr.io/huggingface/text-generation-inference:latest
    runtime: nvidia
    command: >
      --model-id meta-llama/Meta-Llama-3.1-8B-Instruct
      --max-input-length 4096
      --max-total-tokens 8192
    ports:
      - "8080:80"
    shm_size: 1g

  # Ollama for local development
  ollama:
    image: ollama/ollama:latest
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

volumes:
  ollama_data:
Enter fullscreen mode Exit fullscreen mode

Kubernetes Deployment 

# deployment/k8s-vllm.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-llama3-8b
spec:
  replicas: 2
  selector:
    matchLabels:
      app: vllm
  template:
    metadata:
      labels:
        app: vllm
    spec:
      containers:
      - name: vllm
        image: vllm/vllm-openai:latest
        command: ["python", "-m", "vllm.entrypoints.openai.api_server"]
        args:
          - "--model=meta-llama/Meta-Llama-3.1-8B-Instruct"
          - "--max-model-len=8192"
          - "--gpu-memory-utilization=0.9"
          - "--host=0.0.0.0"
          - "--port=8000"
        ports:
        - containerPort: 8000
        resources:
          limits:
            nvidia.com/gpu: 1
          requests:
            nvidia.com/gpu: 1
            memory: "16Gi"
        env:
        - name: HF_TOKEN
          valueFrom:
            secretKeyRef:
              name: hf-token
              key: token
---
apiVersion: v1
kind: Service
metadata:
  name: vllm-service
spec:
  selector:
    app: vllm
  ports:
  - port: 8000
    targetPort: 8000
  type: LoadBalancer
Enter fullscreen mode Exit fullscreen mode

Load Balancing & Autoscaling 

# deployment/load_balancer.py
import httpx
from typing: List
import random

class LLMLoadBalancer:
    """Round-robin load balancer with health checks."""

    def __init__(self, endpoints: List[str]):
        self.endpoints = endpoints
        self.healthy = set(endpoints)
        self.current = 0

    async def health_check(self):
        """Periodic health check."""
        async with httpx.AsyncClient(timeout=5) as client:
            for ep in self.endpoints:
                try:
                    resp = await client.get(f"{ep}/health")
                    if resp.status_code == 200:
                        self.healthy.add(ep)
                    else:
                        self.healthy.discard(ep)
                except:
                    self.healthy.discard(ep)

    def get_endpoint(self) -> str:
        healthy = list(self.healthy)
        if not healthy:
            raise Exception("No healthy endpoints")
        ep = healthy[self.current % len(healthy)]
        self.current += 1
        return ep

    async def generate(self, *args, **kwargs):
        ep = self.get_endpoint()
        async with httpx.AsyncClient(base_url=ep, timeout=60) as client:
            return await client.post("/v1/completions", json=kwargs)
Enter fullscreen mode Exit fullscreen mode

Part 7: Monitoring & Observability

Prometheus Metrics 

# monitoring/prometheus_metrics.py
from prometheus_client import Counter, Histogram, Gauge
import time

LLM_REQUESTS = Counter(
    "llm_requests_total",
    "Total LLM requests",
    ["model", "status"]
)

LLM_LATENCY = Histogram(
    "llm_latency_seconds",
    "Request latency",
    ["model"],
    buckets=[0.1, 0.5, 1, 2, 5, 10, 30, 60]
)

LLM_TOKENS = Counter(
    "llm_tokens_total",
    "Tokens generated",
    ["model", "type"]  # prompt, completion
)

LLM_GPU_MEMORY = Gauge(
    "llm_gpu_memory_bytes",
    "GPU memory usage",
    ["model", "gpu_id"]
)

LLM_QUEUE_SIZE = Gauge(
    "llm_queue_size",
    "Request queue size",
    ["model"]
)

class MonitoredLLM:
    def __init__(self, client, model_name: str):
        self.client = client
        self.model_name = model_name

    def generate(self, prompt: str, **kwargs):
        start = time.time()
        try:
            result = self.client.generate(prompt, **kwargs)
            LLM_REQUESTS.labels(model=self.model_name, status="success").inc()
            return result
        except Exception as e:
            LLM_REQUESTS.labels(model=self.model_name, status="error").inc()
            raise
        finally:
            LLM_LATENCY.labels(model=self.model_name).observe(time.time() - start)
Enter fullscreen mode Exit fullscreen mode

Part 8: Benchmarking

Throughput Test 

# benchmark/throughput.py
import asyncio
import time
from statistics import mean, stdev
from vllm import LLM, SamplingParams

async def benchmark_throughput(
    model: str,
    num_requests: int = 100,
    prompt_len: int = 512,
    output_len: int = 256,
    concurrency: int = 10
):
    """Benchmark throughput and latency."""

    llm = LLM(model=model, gpu_memory_utilization=0.9)
    sampling_params = SamplingParams(
        max_tokens=output_len,
        temperature=0.7
    )

    prompt = "x" * prompt_len
    prompts = [prompt] * num_requests

    # Warmup
    llm.generate([prompt], sampling_params)

    # Benchmark
    start = time.time()
    outputs = llm.generate(prompts, sampling_params)
    total_time = time.time() - start

    total_tokens = sum(len(o.outputs[0].token_ids) for o in outputs)

    return {
        "total_time_sec": total_time,
        "requests_per_sec": num_requests / total_time,
        "tokens_per_sec": total_tokens / total_time,
        "avg_latency_ms": (total_time / num_requests) * 1000,
        "total_tokens": total_tokens,
    }

# Run
results = asyncio.run(benchmark_throughput("meta-llama/Meta-Llama-3.1-8B-Instruct"))
print(f"Throughput: {results['requests_per_sec']:.1f} req/s")
print(f"Tokens/sec: {results['tokens_per_sec']:.0f}")
Enter fullscreen mode Exit fullscreen mode

Part 9: Cost Analysis

Local vs Cloud 

# benchmark/cost_analysis.py
CLOUD_PRICING = {
    "gpt-4o": {"input": 2.50, "output": 10.00},      # per 1M tokens
    "gpt-4o-mini": {"input": 0.15, "output": 0.60},
    "claude-3.5-sonnet": {"input": 3.00, "output": 15.00},
}

GPU_COSTS = {
    "RTX 4090 (24GB)": {"purchase": 1600, "power_w": 450, "vram": 24},
    "RTX 3090 (24GB)": {"purchase": 800, "power_w": 350, "vram": 24},
    "A100 40GB": {"hourly": 1.50, "power_w": 400, "vram": 40},
    "A100 80GB": {"hourly": 3.00, "power_w": 400, "vram": 80},
    "H100 80GB": {"hourly": 8.00, "power_w": 700, "vram": 80},
}

def calculate_break_even(
    model: str,
    requests_per_day: int,
    avg_input_tokens: int,
    avg_output_tokens: int,
    gpu: str = "RTX 4090"
) -> dict:
    """Calculate when local becomes cheaper than cloud."""

    cloud = CLOUD_PRICING.get(model, {"input": 0, "output": 0})
    daily_cloud_cost = (
        requests_per_day * avg_input_tokens * cloud["input"] +
        requests_per_day * avg_output_tokens * cloud["output"]
    ) / 1_000_000

    gpu_info = GPU_COSTS[gpu]
    if "purchase" in gpu_info:
        # Electricity cost (assume $0.12/kWh)
        daily_power_cost = (gpu_info["power_w"] * 24 / 1000) * 0.12
        daily_local_cost = daily_power_cost
        break_even_days = gpu_info["purchase"] / (daily_cloud_cost - daily_local_cost)
    else:
        daily_local_cost = gpu_info["hourly"] * 24
        break_even_days = float('inf') if daily_local_cost >= daily_cloud_cost else 0

    return {
        "daily_cloud_cost": daily_cloud_cost,
        "daily_local_cost": daily_local_cost,
        "monthly_savings": (daily_cloud_cost - daily_local_cost) * 30,
        "break_even_days": max(0, break_even_days),
    }

# Example: 1000 requests/day, 1k in / 500 out tokens
print(calculate_break_even("gpt-4o-mini", 1000, 1000, 500, "RTX 4090"))
# {'daily_cloud_cost': 0.45, 'daily_local_cost': 1.30, 'monthly_savings': -25.5, 'break_even_days': 0}
# Local cheaper at scale, cloud cheaper for low volume
Enter fullscreen mode Exit fullscreen mode

Part 10: Troubleshooting

Common Issues 

# troubleshooting/common.py
TROUBLESHOOTING = {
    "OOM": {
        "vllm": [
            "Reduce --gpu-memory-utilization (0.8 → 0.7)",
            "Reduce --max-model-len",
            "Enable --enable-chunked-prefill",
            "Use quantization (AWQ/GPTQ)",
            "Reduce --max-num-seqs",
        ],
        "llama.cpp": [
            "Reduce -c (context size)",
            "Reduce -ngl (GPU layers)",
            "Use smaller quantization (Q4 → Q3)",
            "Add --mlock to prevent swap",
        ],
        "tgi": [
            "Reduce --max-total-tokens",
            "Reduce --max-batch-prefill-tokens",
            "Enable --quantize",
        ],
    },
    "Slow_Inference": {
        "vllm": [
            "Increase --gpu-memory-utilization",
            "Disable --enforce-eager",
            "Increase --max-num-batched-tokens",
            "Use tensor parallel for large models",
        ],
        "llama.cpp": [
            "Increase -ngl (more GPU layers)",
            "Use -ngl 99 (all layers GPU)",
            "Enable --flash-attn",
            "Increase -t (threads) for CPU",
        ],
    },
    "Quality_Loss": {
        "quantization": [
            "Try Q4_K_M instead of Q4_0",
            "Use Q5_K_M or Q6_K for better quality",
            "Avoid Q2_K/Q3_K for reasoning tasks",
            "Test with your specific prompts",
        ],
    },
}
Enter fullscreen mode Exit fullscreen mode

Resources

Start with Ollama for development, graduate to vLLM/TGI for production. The best local LLM setup is the one that runs on your hardware.

Top comments (0)