DEV Community

Carlos Oliva Pascual
Carlos Oliva Pascual

Posted on • Originally published at stacknotice.com

Ollama Complete Guide: Run LLMs Locally in 2026

#ai #tutorial #llm #opensource

Running LLMs locally has gone from a niche experiment to a legitimate production pattern. Privacy requirements, API cost at scale, and offline-first applications all push teams toward local inference. Ollama is the tool that made this practical — it wraps model management, a REST API, and OpenAI-compatible endpoints into a single binary.

Installing Ollama

macOS and Linux:

curl -fsSL https://ollama.com/install.sh | sh
Enter fullscreen mode Exit fullscreen mode

Windows: Download the installer from ollama.com. It runs as a system service automatically.

ollama --version
curl http://localhost:11434  # → "Ollama is running"
Enter fullscreen mode Exit fullscreen mode

Pulling and Running Models 

ollama pull llama3.3:8b
ollama run llama3.3:8b
echo "Explain async/await in one paragraph" | ollama run llama3.3:8b
ollama list
Enter fullscreen mode Exit fullscreen mode

Choosing the Right Model

Model Size VRAM Best for
llama3.3:70b 43GB 48GB+ Best quality
llama3.3:8b 4.7GB 8GB Most laptops
qwen2.5-coder:7b 4.7GB 8GB Fast coding
gemma3:27b 16GB 20GB Google's best local
deepseek-r1:8b 4.9GB 8GB Reasoning + CoT
phi4:14b 9.1GB 12GB Microsoft, reasoning

Rule: pick the largest model your VRAM can hold with ~20% headroom.

TypeScript Integration 

// lib/ollama.ts
interface OllamaMessage {
  role: 'system' | 'user' | 'assistant'
  content: string
}

const OLLAMA_URL = process.env.OLLAMA_URL ?? 'http://localhost:11434'

export async function chat(messages: OllamaMessage[], model = 'llama3.3:8b'): Promise<string> {
  const response = await fetch(`${OLLAMA_URL}/api/chat`, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ model, messages, stream: false })
  })

  if (!response.ok) throw new Error(`Ollama error: ${response.status}`)
  const data = await response.json()
  return data.message.content
}

export async function* streamChat(messages: OllamaMessage[], model = 'llama3.3:8b') {
  const response = await fetch(`${OLLAMA_URL}/api/chat`, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ model, messages, stream: true })
  })

  const reader = response.body!.getReader()
  const decoder = new TextDecoder()

  while (true) {
    const { done, value } = await reader.read()
    if (done) break
    const lines = decoder.decode(value).split('\n').filter(Boolean)
    for (const line of lines) {
      const data = JSON.parse(line)
      if (data.message?.content) yield data.message.content
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

OpenAI SDK Compatibility

Ollama exposes an OpenAI-compatible endpoint at /v1. Use the openai package with zero code changes:

import OpenAI from 'openai'

const client = new OpenAI({
  baseURL: 'http://localhost:11434/v1',
  apiKey: 'ollama'  // required by SDK, ignored by Ollama
})

const completion = await client.chat.completions.create({
  model: 'llama3.3:8b',
  messages: [
    { role: 'system', content: 'You are a code reviewer.' },
    { role: 'user', content: 'Review: SELECT * FROM users WHERE id = ' + userId }
  ]
})

console.log(completion.choices[0].message.content)
Enter fullscreen mode Exit fullscreen mode

Switching between cloud APIs and local Ollama is often just changing baseURL and model.

Python Integration 

import httpx
import json
from typing import Generator

def chat(messages: list[dict], model: str = "llama3.3:8b") -> str:
    response = httpx.post(
        "http://localhost:11434/api/chat",
        json={"model": model, "messages": messages, "stream": False},
        timeout=120.0
    )
    response.raise_for_status()
    return response.json()["message"]["content"]

def stream_chat(messages: list[dict], model: str = "llama3.3:8b") -> Generator[str, None, None]:
    with httpx.stream(
        "POST",
        "http://localhost:11434/api/chat",
        json={"model": model, "messages": messages, "stream": True},
        timeout=None
    ) as response:
        for line in response.iter_lines():
            if line:
                data = json.loads(line)
                if data.get("message", {}).get("content"):
                    yield data["message"]["content"]
Enter fullscreen mode Exit fullscreen mode

With the OpenAI Python SDK:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

response = client.chat.completions.create(
    model="qwen2.5-coder:7b",
    messages=[{"role": "user", "content": "Write a Python email validator"}]
)
print(response.choices[0].message.content)
Enter fullscreen mode Exit fullscreen mode

Custom Modelfiles

Lock in a system prompt and parameters for a specific use case:

# Modelfile.code-reviewer
FROM qwen2.5-coder:7b

SYSTEM """
You are a senior software engineer doing code reviews. Identify:
1. Bugs and security issues
2. SOLID principle violations
3. Performance problems

Format issues as: [SEVERITY] Description → fix
SEVERITY: CRITICAL | HIGH | MEDIUM | LOW
"""

PARAMETER temperature 0.2
PARAMETER num_ctx 8192
Enter fullscreen mode Exit fullscreen mode 
ollama create code-reviewer -f Modelfile.code-reviewer
ollama run code-reviewer
Enter fullscreen mode Exit fullscreen mode

Streaming in a Next.js API Route 

// app/api/chat/route.ts
export async function POST(request: Request) {
  const { messages } = await request.json()

  const ollamaResponse = await fetch('http://localhost:11434/api/chat', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ model: 'llama3.3:8b', messages, stream: true })
  })

  const stream = new ReadableStream({
    async start(controller) {
      const reader = ollamaResponse.body!.getReader()
      const decoder = new TextDecoder()

      while (true) {
        const { done, value } = await reader.read()
        if (done) { controller.close(); break }

        const lines = decoder.decode(value).split('\n').filter(Boolean)
        for (const line of lines) {
          const data = JSON.parse(line)
          if (data.message?.content) {
            controller.enqueue(
              new TextEncoder().encode(`data: ${JSON.stringify({ content: data.message.content })}\n\n`)
            )
          }
          if (data.done) { controller.close(); return }
        }
      }
    }
  })

  return new Response(stream, {
    headers: { 'Content-Type': 'text/event-stream', 'Cache-Control': 'no-cache' }
  })
}
Enter fullscreen mode Exit fullscreen mode

Production Setup 

# Ollama runs as a systemd service after install
systemctl status ollama
systemctl enable ollama

# Expose to network (only on private networks)
sudo systemctl edit ollama
# Add: Environment="OLLAMA_HOST=0.0.0.0:11434"
# Add: Environment="OLLAMA_NUM_PARALLEL=2"
Enter fullscreen mode Exit fullscreen mode

Ollama auto-detects NVIDIA (CUDA), AMD (ROCm), and Apple Silicon (Metal). No configuration — it picks the fastest backend.

Ollama vs Cloud APIs

Factor Ollama Cloud API
Cost at scale ~$0 (electricity) $0.01–$15/1M tokens
Privacy Data stays local Sent to provider
Quality ceiling Llama 3.3 70B ≈ GPT-4o Best-in-class
Offline ✅ ❌
Scaling Hardware limit Unlimited

Use Ollama for: sensitive data (PII, financial, medical), high-volume internal tools, offline/air-gapped deployment, development without burning API credits.

Use cloud APIs for: frontier model quality, unpredictable load, no GPU hardware.

The practical pattern: Ollama for dev and internal tools, cloud APIs (Claude Sonnet, GPT-4o) for customer-facing production features.

Full article at stacknotice.com/blog/ollama-complete-guide-2026

Top comments (0)