Building a Local LLM API Server with Ollama and FastAPI

Running a language model locally means you control the data, avoid per-token costs, and can tune latency to your use case. The problem is that Ollama's built-in HTTP API is minimal — no auth, no schema validation, no easy integration with your existing Python stack. Wrapping it with FastAPI fixes that in under 150 lines and gives you an endpoint you can hand to teammates or wire into a pipeline without exposing raw inference infrastructure.

Why Ollama + FastAPI

Ollama provides a dead-simple way to pull and serve open models (Llama 3, Mistral, Qwen, Phi, etc.) locally or on a private server. Its REST API is functional but basic: no middleware, no request validation, no usage tracking. FastAPI adds exactly what's missing:

• Pydantic models for input validation and clear error messages
• Async support compatible with Ollama's SSE streaming
• Auth middleware, rate limiting, structured logging
• OpenAPI docs out of the box at /docs

The result is a drop-in local replacement for cloud LLM APIs that you fully control. No data leaves your server, no per-token billing, no vendor rate limits during load spikes.

Setting Up Ollama

Install Ollama on Linux or macOS:

curl -fsSL https://ollama.com/install.sh | sh
ollama pull llama3.2
ollama serve  # starts on http://localhost:11434 by default

Verify the bare API responds:

curl http://localhost:11434/api/generate \
  -d '{"model": "llama3.2", "prompt": "Hello", "stream": false}'

Ollama exposes two main endpoints: /api/generate for single-turn completions and /api/chat for multi-turn conversations with message history. We'll proxy and enhance both.

The FastAPI Wrapper

Install dependencies:

pip install fastapi uvicorn httpx pydantic

Here's the full server:

import httpx
import json
import os
import time
import logging
from typing import Optional, List

from fastapi import FastAPI, HTTPException, Depends, Header, Request
from fastapi.responses import StreamingResponse
from pydantic import BaseModel

app = FastAPI(title="Local LLM API", version="1.0.0")
logger = logging.getLogger("llm_api")
logging.basicConfig(level=logging.INFO)

OLLAMA_BASE_URL = os.getenv("OLLAMA_URL", "http://localhost:11434")
API_KEY = os.getenv("API_KEY", "")

# --- Auth ---

async def verify_api_key(x_api_key: Optional[str] = Header(None)):
    if API_KEY and x_api_key != API_KEY:
        raise HTTPException(status_code=401, detail="Invalid API key")

# --- Request/Response models ---

class Message(BaseModel):
    role: str   # "user" or "assistant"
    content: str

class ChatRequest(BaseModel):
    model: str = "llama3.2"
    messages: List[Message]
    stream: bool = False
    temperature: Optional[float] = 0.7
    max_tokens: Optional[int] = None

# --- Middleware for latency logging ---

@app.middleware("http")
async def log_requests(request: Request, call_next):
    start = time.monotonic()
    response = await call_next(request)
    duration = time.monotonic() - start
    logger.info(
        "method=%s path=%s status=%d duration=%.3fs",
        request.method, request.url.path,
        response.status_code, duration,
    )
    return response

# --- Routes ---

@app.get("/health")
async def health():
    async with httpx.AsyncClient() as client:
        try:
            r = await client.get(f"{OLLAMA_BASE_URL}/api/tags", timeout=3.0)
            r.raise_for_status()
            models = [m["name"] for m in r.json().get("models", [])]
            return {"status": "ok", "models": models}
        except Exception:
            raise HTTPException(status_code=503, detail="Ollama unreachable")

@app.post("/v1/chat", dependencies=[Depends(verify_api_key)])
async def chat(req: ChatRequest):
    payload = {
        "model": req.model,
        "messages": [m.model_dump() for m in req.messages],
        "stream": req.stream,
        "options": {"temperature": req.temperature},
    }
    if req.max_tokens:
        payload["options"]["num_predict"] = req.max_tokens

    if req.stream:
        return StreamingResponse(
            _stream_ollama("/api/chat", payload),
            media_type="text/event-stream",
        )

    async with httpx.AsyncClient(timeout=120.0) as client:
        r = await client.post(f"{OLLAMA_BASE_URL}/api/chat", json=payload)
        if r.status_code != 200:
            raise HTTPException(status_code=r.status_code, detail=r.text)
        data = r.json()
        return {
            "model": req.model,
            "content": data["message"]["content"],
            "done": data.get("done", True),
        }

async def _stream_ollama(path: str, payload: dict):
    async with httpx.AsyncClient(timeout=120.0) as client:
        async with client.stream(
            "POST", f"{OLLAMA_BASE_URL}{path}", json=payload
        ) as r:
            async for line in r.aiter_lines():
                if line:
                    yield f"data: {line}\n\n"
    yield "data: [DONE]\n\n"

Run it:

API_KEY=mysecret uvicorn server:app --host 0.0.0.0 --port 8080

The /health endpoint tells you which models are loaded. Hit /docs for the auto-generated OpenAPI UI — useful when teammates are exploring what the server supports.

Consuming the Streaming Endpoint

For chat interfaces and long outputs, streaming matters. First-token latency is everything for perceived responsiveness. Here's a minimal Python client that prints tokens as they arrive:

import httpx, json

def stream_chat(prompt: str, api_key: str = "mysecret"):
    messages = [{"role": "user", "content": prompt}]
    with httpx.Client() as client:
        with client.stream(
            "POST",
            "http://localhost:8080/v1/chat",
            json={"messages": messages, "stream": True},
            headers={"x-api-key": api_key},
            timeout=120.0,
        ) as r:
            for line in r.iter_lines():
                if line.startswith("data: ") and line != "data: [DONE]":
                    chunk = json.loads(line[6:])
                    token = chunk.get("message", {}).get("content", "")
                    if token:
                        print(token, end="", flush=True)
    print()

stream_chat("Explain mTLS in two sentences")

The [DONE] sentinel keeps the client loop clean without relying on connection close detection, and it's easy to adapt for JavaScript EventSource clients if you're building a frontend.

Deploying as a Systemd Service

For a persistent local API, run both Ollama and FastAPI as systemd services. Create /etc/systemd/system/llm-api.service:

[Unit]
Description=Local LLM API (FastAPI)
After=ollama.service

[Service]
User=llm
WorkingDirectory=/opt/llm-api
ExecStart=/opt/llm-api/venv/bin/uvicorn server:app --host 127.0.0.1 --port 8080
Restart=on-failure
Environment=API_KEY=your_secret_here
Environment=OLLAMA_URL=http://127.0.0.1:11434

[Install]
WantedBy=multi-user.target

systemctl daemon-reload
systemctl enable --now llm-api

Add Nginx in front to terminate TLS and you have a production-grade local LLM endpoint that survives reboots. For hardening the API layer — authentication patterns, rate limit configuration, allowed model allowlists — the security hardening checklists we publish cover these patterns with concrete configs.

What to Build Next

A few natural extensions once the base is running:

• Request caching: hash the (model, messages, temperature) tuple and store responses in Redis or SQLite. Repeated identical prompts return instantly with zero inference cost — useful for FAQ-style bots.
• Model router: inspect prompt length and complexity, route short factual queries to a small fast model (Phi-3 mini), longer reasoning tasks to a larger one. Ollama's /api/tags tells you what's available.
• Token usage tracking: write a row to SQLite per request — model name, Ollama's eval_count and prompt_eval_count fields, latency. Gives you cost-equivalent data even with local inference, so you can benchmark models fairly.
• Rate limiting per API key: the current implementation has a single key. Add a key→tier mapping in a dict or SQLite table, and use a token-bucket decorator per key.

The Takeaway

Ollama handles model management and inference. FastAPI handles everything around it: validation, auth, streaming, observability. The combination is roughly 150 lines of Python and gives you a private, cost-free, API-compatible LLM endpoint you control end to end.

The bare Ollama API is fine for local tinkering. Once you need to share the endpoint with teammates, add auth, or integrate it into a production pipeline, wrapping it with FastAPI takes an afternoon and pays for itself immediately in clarity and maintainability.

---

I run AYI NEDJIMI Consultants, a cybersecurity consulting firm. We publish free security hardening checklists — PDF and Excel.