Cloud LLMs are amazing - until the bill arrives. Or until you need to process something you can't send to a third party. Or until you hit a rate limit at 3am.
This is the architecture I built to run any GGUF model locally with a drop-in OpenAI-compatible API, and the cool part: it picks the right model for your GPU automatically.
The setup
- llama.cpp for inference (compiled with CUDA or Metal)
- FastAPI for the server (OpenAI-compatible endpoints)
- GGUF models in a local directory
- NVML (NVIDIA) or Metal (Apple) for VRAM detection
User app (uses openai SDK)
v
http://localhost:8080/v1/chat/completions
v
FastAPI server
+- VRAM router (picks model by query)
+- llama.cpp subprocess (loads GGUF)
+- Stream tokens back
v
Your GPU
Step 1: Get the models
Pull GGUF models from HuggingFace. I keep a few sizes around for different hardware:
# 7B model - fits in 8GB VRAM
huggingface-cli download TheBloke/Llama-2-7B-Chat-GGUF llama-2-7b-chat.Q4_K_M.gguf
# 13B model - needs 12GB
huggingface-cli download TheBloke/Llama-2-13B-Chat-GGUF llama-2-13b-chat.Q4_K_M.gguf
# 70B model - needs 40GB (or 24GB with Q3)
huggingface-cli download TheBloke/Llama-2-70B-Chat-GGUF llama-2-70b-chat.Q3_K_M.gguf
mkdir -p ~/models
mv *.gguf ~/models/
Step 2: The server skeleton
# server.py
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from typing import List, Optional
import subprocess, json, asyncio
app = FastAPI(title="Local LLM Server")
class Message(BaseModel):
role: str
content: str
class ChatRequest(BaseModel):
model: str = "auto" # < magic
messages: List[Message]
max_tokens: int = 512
temperature: float = 0.7
stream: bool = False
class ChatResponse(BaseModel):
id: str
object: str = "chat.completion"
model: str
choices: list
usage: dict
Step 3: The VRAM-aware router
This is the magic. The router picks the right model based on:
- Available VRAM
- Requested
max_tokens - Whether the model is already loaded
# router.py
import pynvml
def get_available_vram_mb() -> int:
"""Returns free VRAM in MB. -1 if no NVIDIA GPU."""
try:
pynvml.nvmlInit()
handle = pynvml.nvmlDeviceGetHandleByIndex(0)
info = pynvml.nvmlDeviceGetMemoryInfo(handle)
return info.free // 1024 // 1024
except Exception:
return -1 # CPU-only or Apple Silicon
# Model VRAM requirements (rough, for Q4_K_M quantization)
MODELS = {
"llama-2-7b": {"file": "llama-2-7b-chat.Q4_K_M.gguf", "vram_mb": 6000},
"llama-2-13b": {"file": "llama-2-13b-chat.Q4_K_M.gguf", "vram_mb": 11000},
"llama-2-70b": {"file": "llama-2-70b-chat.Q3_K_M.gguf", "vram_mb": 28000},
"mistral-7b": {"file": "mistral-7b-instruct.Q4_K_M.gguf","vram_mb": 6000},
"mixtral-8x7b": {"file": "mixtral-8x7b-instruct.Q4_K_M.gguf","vram_mb": 30000},
}
def pick_model(requested: str, max_tokens: int) -> str:
free_vram = get_available_vram_mb()
needed = max_tokens * 2 # rough KV cache estimate
if requested == "auto":
# Pick the largest model that fits in VRAM
for name in sorted(MODELS.keys(), key=lambda m: -MODELS[m]["vram_mb"]):
if MODELS[name]["vram_mb"] + needed < free_vram:
return name
return "llama-2-7b" # fallback to smallest
return requested
Step 4: Run the inference
I use llama-cpp-python for the inference layer - it has Python bindings to llama.cpp with streaming support:
# inference.py
from llama_cpp import Llama
import asyncio
class ModelPool:
def __init__(self, model_dir: str = "~/models"):
self.pool: dict[str, Llama] = {}
self.model_dir = model_dir
def get(self, model_name: str) -> Llama:
if model_name not in self.pool:
cfg = MODELS[model_name]
self.pool[model_name] = Llama(
model_path=f"{self.model_dir}/{cfg['file']}",
n_ctx=4096,
n_gpu_layers=-1, # all layers on GPU
n_threads=8,
)
return self.pool[model_name]
pool = ModelPool()
Step 5: The OpenAI-compatible endpoint
This is where the magic happens - any OpenAI SDK call works:
# app.py
@app.post("/v1/chat/completions")
async def chat_completions(req: ChatRequest):
model_name = pick_model(req.model, req.max_tokens)
if model_name not in MODELS:
raise HTTPException(404, f"Model {model_name} not found")
llm = pool.get(model_name)
# Run inference (streaming or batch)
response = llm.create_chat_completion(
messages=[m.dict() for m in req.messages],
max_tokens=req.max_tokens,
temperature=req.temperature,
stream=req.stream,
)
return {
"id": f"chatcmpl-{hash(req.messages)}",
"object": "chat.completion",
"model": model_name,
"choices": response["choices"],
"usage": response["usage"],
}
Step 6: Use it with the OpenAI SDK
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8080/v1",
api_key="not-needed",
)
response = client.chat.completions.create(
model="auto", # router picks the right model
messages=[{"role": "user", "content": "Explain VRAM-aware routing"}],
max_tokens=512,
)
print(response.choices[0].message.content)
Zero code changes to switch from OpenAI to local. This means I can:
- Develop locally (no API costs)
- Test in CI (deterministic, no rate limits)
- Deploy to production (OpenAI for users who want it, local for those who need privacy)
Performance tips I learned the hard way
- Don't keep multiple models loaded. Each model eats VRAM for the KV cache. Load on demand, unload after N minutes of idle.
- Use Q4_K_M for most cases. Q5 is barely better, Q8 is 2x the size for marginal gains.
- Speculative decoding can 2-3x the speed: have a small model draft, the big model verifies.
-
Mlock the model (
use_mlock=True) to prevent swapping - massive speedup on machines with slow disk. - Apple Silicon users: build llama.cpp with Metal support, performance is excellent on M1/M2/M3.
The whole project
I've packaged this into Strata - a desktop app with a chat UI, model browser, and this exact server underneath.
[link] github.com/Omerfaruk-aydn (Strata repo)
The desktop app is built with Tauri 2 + React; the server is this Python + FastAPI + llama.cpp stack.
Originally published on omerfarukaydn.com - more on the desktop UI, model marketplace, and the WebSocket streaming implementation.
Top comments (0)