Running a language model locally and exposing it as a proper REST API has real practical value: no rate limits, no per-token billing, no data leaving your infrastructure. The problem is that Ollama's native HTTP interface is intentionally minimal — it works, but you'll want authentication, structured request validation, streaming support, and proper error handling before using it in any real workload.
This guide builds a thin FastAPI wrapper around Ollama that you'd actually deploy.
Prerequisites
You'll need Ollama installed and running, with at least one model pulled:
# Install Ollama (Linux)
curl -fsSL https://ollama.ai/install.sh | sh
# Pull a model
ollama pull llama3.2:3b
# Verify
ollama list
For the FastAPI layer:
pip install fastapi uvicorn httpx python-dotenv
The Architecture
Ollama runs its own HTTP server on localhost:11434. The strategy here is simple:
- FastAPI handles incoming requests from clients
- It validates the payload with Pydantic
- Forwards the request to Ollama
- Returns the response — either full JSON or a streaming SSE response
This keeps the Ollama process as a dumb backend and lets you layer authentication, logging, and request transformation at the FastAPI level without touching Ollama's configuration.
Building the Server
# server.py
import os
import httpx
import asyncio
from typing import AsyncGenerator
from fastapi import FastAPI, HTTPException, Depends, Header
from fastapi.responses import StreamingResponse
from pydantic import BaseModel, Field
OLLAMA_BASE_URL = os.getenv("OLLAMA_BASE_URL", "http://localhost:11434")
API_KEY = os.getenv("API_KEY", "change-me-in-production")
app = FastAPI(title="Local LLM API", version="1.0.0")
class ChatMessage(BaseModel):
role: str = Field(..., pattern="^(system|user|assistant)$")
content: str
class ChatRequest(BaseModel):
model: str = Field(default="llama3.2:3b")
messages: list[ChatMessage]
stream: bool = False
temperature: float = Field(default=0.7, ge=0.0, le=2.0)
max_tokens: int = Field(default=1024, ge=1, le=8192)
def verify_api_key(x_api_key: str = Header(...)):
if x_api_key != API_KEY:
raise HTTPException(status_code=401, detail="Invalid API key")
return x_api_key
async def stream_ollama_response(payload: dict) -> AsyncGenerator[str, None]:
async with httpx.AsyncClient(timeout=120) as client:
async with client.stream(
"POST",
f"{OLLAMA_BASE_URL}/api/chat",
json=payload,
) as response:
async for line in response.aiter_lines():
if line:
yield f"data: {line}\n\n"
@app.post("/v1/chat")
async def chat(request: ChatRequest, _: str = Depends(verify_api_key)):
payload = {
"model": request.model,
"messages": [m.model_dump() for m in request.messages],
"stream": request.stream,
"options": {
"temperature": request.temperature,
"num_predict": request.max_tokens,
},
}
if request.stream:
return StreamingResponse(
stream_ollama_response(payload),
media_type="text/event-stream",
)
async with httpx.AsyncClient(timeout=120) as client:
try:
response = await client.post(
f"{OLLAMA_BASE_URL}/api/chat",
json=payload,
)
response.raise_for_status()
except httpx.HTTPStatusError as e:
raise HTTPException(status_code=502, detail=f"Ollama error: {e.response.text}")
except httpx.ConnectError:
raise HTTPException(status_code=503, detail="Ollama is not running")
return response.json()
@app.get("/v1/models")
async def list_models(_: str = Depends(verify_api_key)):
async with httpx.AsyncClient(timeout=10) as client:
try:
resp = await client.get(f"{OLLAMA_BASE_URL}/api/tags")
resp.raise_for_status()
return resp.json()
except httpx.ConnectError:
raise HTTPException(status_code=503, detail="Ollama is not running")
@app.get("/health")
async def health():
try:
async with httpx.AsyncClient(timeout=5) as client:
resp = await client.get(f"{OLLAMA_BASE_URL}/api/tags")
return {"status": "ok", "ollama": resp.status_code == 200}
except httpx.ConnectError:
return {"status": "degraded", "ollama": False}
Start the server:
API_KEY=my-secret-key uvicorn server:app --host 0.0.0.0 --port 8000
Test it:
curl -X POST http://localhost:8000/v1/chat \
-H "x-api-key: my-secret-key" \
-H "Content-Type: application/json" \
-d '{
"model": "llama3.2:3b",
"messages": [{"role": "user", "content": "What is a JWT?"}]
}'
Handling Streaming Responses
Ollama returns newline-delimited JSON when streaming. The stream_ollama_response generator wraps each chunk as a Server-Sent Event. Here's a client that consumes it:
# client_stream.py
import json
import httpx
def stream_chat(prompt: str, api_key: str, base_url: str = "http://localhost:8000"):
payload = {
"model": "llama3.2:3b",
"messages": [{"role": "user", "content": prompt}],
"stream": True,
}
with httpx.Client(timeout=120) as client:
with client.stream(
"POST",
f"{base_url}/v1/chat",
json=payload,
headers={"x-api-key": api_key},
) as response:
for line in response.iter_lines():
if line.startswith("data: "):
chunk = json.loads(line[6:])
if not chunk.get("done"):
print(chunk["message"]["content"], end="", flush=True)
print()
if __name__ == "__main__":
stream_chat("Explain TLS handshake in 3 sentences.", "my-secret-key")
The output appears token by token — same behavior you'd expect from any hosted API.
Production Considerations
A few things that matter before this goes anywhere beyond localhost.
Model validation. The current code accepts any model string and lets Ollama fail with a 500. Better to fetch available models at startup and validate against that set on each request.
Concurrency control. Ollama processes one request at a time by default (unless your GPU supports true parallelism). Concurrent requests queue internally, but they can pile up fast under load. Cap concurrent inference calls with asyncio.Semaphore:
_sem = asyncio.Semaphore(2) # max 2 concurrent Ollama requests
@app.post("/v1/chat")
async def chat(request: ChatRequest, _: str = Depends(verify_api_key)):
async with _sem:
# ... forward to Ollama
pass
Timeouts. timeout=120 is a ceiling, not a guarantee. A 13B model generating a long response can blow past 2 minutes on consumer hardware. Tune per model, or expose timeout as an optional request field.
Logging. Add structured logging at the request level: model name, input token count (Ollama returns this), latency, and status code. Without it, debugging slow responses is painful.
Security headers. If this API is behind nginx, add X-Content-Type-Options, X-Frame-Options, and a narrow Content-Security-Policy. For a structured API hardening reference, this security checklist covers the gaps that tend to get overlooked at deployment time.
TLS. Do not put this directly on a public interface without TLS termination. Use Caddy or nginx in front of it, even for internal use.
The Takeaway
Running a language model behind a FastAPI layer costs you about 100 lines of code and gives you a self-hostable inference endpoint with proper validation, auth, and streaming. The key tradeoffs versus a hosted API: you own the hardware and the latency profile, model size is bounded by VRAM, and you're on the hook for updates and reliability.
For internal tooling, air-gapped environments, or cost-sensitive workloads where you're hitting the API thousands of times a day, this stack is genuinely competitive. Where it breaks down is high concurrent throughput — at that point, move to vLLM instead of Ollama, which is purpose-built around throughput over single-request latency.
I run AYI NEDJIMI Consultants, a cybersecurity consulting firm. We publish free security hardening checklists — PDF and Excel.
Top comments (0)