DEV Community

LeoJulieta
LeoJulieta

Posted on

Run LLMs Offline: Kodro, LocalAI‑Playground & More

#kodro #offlineai #privacyconcerns #english

Offline LLMs Are Booming: How Kodro, LocalAI‑Playground & Friends Let You Run Powerful Models on Your Own Hardware

Introduction

You’re tired of watching the token meter tick every time you call an API, and you’re nervous about sending sensitive data to the cloud. Running a large‑language model locally is no longer a geek‑only hobby – it’s a practical, cost‑effective way to keep data private and keep latency low.

In 2024 projects like Kodro, AI‑Lab, and LocalAI‑Playground have turned that possibility into a plug‑and‑play reality. This guide walks you through the whole offline‑AI ecosystem, compares the most popular toolkits, shows a complete Docker‑based installation, and drops ready‑to‑run code snippets for three real‑world use cases (chatbot, text‑to‑SQL, and document summarisation). By the end you’ll be able to spin up a LLaMA‑7B or Mistral‑7B model on a laptop or a modest server and start experimenting without ever touching the internet again.

Quick FAQ

# Question TL;DR
1 Can a 7‑billion‑parameter model run on a laptop? Yes – use 4‑bit GGUF quantisation with llama.cpp or vllm. A modern RTX 3060 (or even a recent AMD GPU) handles inference comfortably; CPU‑only is slower but still usable for prototyping.
2 Do I need internet after the first download? No. All binaries, model checkpoints and datasets are cached locally. You only reconnect to pull updates or new models.
3 Is offline inference cheaper than cloud APIs? After the initial hardware outlay, yes. No per‑token fees, zero latency from network hops, and full control over data handling.
4 What hardware do I really need? 16 GB RAM, 8 GB VRAM (or 12 GB for 8‑bit), and ~30 GB of storage for a 7B GGUF model.
5 Which open‑source stack should I pick? For maximum flexibility: Docker → kodro/ai‑lab → llama.cpp or vllm. For a UI‑first experience, try LocalAI‑Playground (Web UI + OpenAI‑compatible endpoint).

Why Offline LLMs Are Hot Right Now

  1. Privacy regulations – GDPR, CCPA, and the upcoming EU AI Act forbid sending personal data to third‑party services without explicit consent. Running the model locally guarantees that data never leaves your network.

  2. API cost explosion – OpenAI’s GPT‑4 pricing sits at $0.03 / 1 k tokens for completions and $0.06 for embeddings. A startup that processes 10 M tokens a month spends $300‑$600 on inference alone. A one‑time hardware purchase (≈$1 500) eliminates that recurring bill.

  3. Connectivity gaps – Rural regions in Africa, South America and parts of Asia still average <5 Mbps. Offline AI brings “AI on the edge” to classrooms, clinics and farms where cloud access is unreliable.

  4. Curriculum demands – Universities now require students to experiment with model weights, tokenisers and fine‑tuning. An offline sandbox satisfies ethics policies while giving hands‑on experience.

  5. Open‑source momentum – The release of LLaMA‑7B GGUF, Mistral‑7B, and the permissively‑licensed vLLM server has lowered the barrier to local deployment dramatically.

The Offline AI Toolbox

Tool Primary Language UI OpenAI‑compatible API Quantisation Support Typical Use‑Case
Kodro Python Minimal (CLI) ✅ (via kodro serve) 4‑bit GGUF, 8‑bit Research & custom pipelines
AI‑Lab Python + JS Jupyter‑style notebooks ✅ (REST) 4‑bit, 5‑bit Rapid prototyping & teaching
LocalAI‑Playground Go Full web UI ✅ (OpenAI‑compatible) 4‑bit GGUF, 8‑bit, 16‑bit End‑user apps & demos
vLLM Python None (server only) ✅ (OpenAI‑compatible) 4‑bit GGUF, 8‑bit High‑throughput inference
llama.cpp C++ (bindings for Python) None ✅ (via llama.cpp server) 4‑bit GGUF, 5‑bit, 8‑bit Ultra‑lightweight CPU inference

Bottom line: If you need a quick UI for non‑technical users, go with LocalAI‑Playground. If you want full control over the inference pipeline, pair Kodro or vLLM with llama.cpp for quantisation.

Step‑by‑Step Docker Setup (Works on Linux, macOS & Windows WSL)

Below is a single‑file Docker Compose that pulls the latest LocalAI‑Playground image, mounts a local models/ directory, and starts an OpenAI‑compatible endpoint on port 8080.

# docker-compose.yml
version: "3.9"

services:
  localai:
    image: ghcr.io/go-skynet/localai:latest
    container_name: localai
    restart: unless-stopped
    ports:
      - "8080:8080"
    environment:
      - MODELS_PATH=/models
      - DEBUG=true
    volumes:
      - ./models:/models        # <-- place GGUF files here
      - ./config:/config       # optional custom config.yaml
Enter fullscreen mode Exit fullscreen mode

Run it:

# 1️⃣ Create the folder structure
mkdir -p models config

# 2️⃣ Download a 4‑bit GGUF checkpoint (example: LLaMA‑7B)
wget -O models/llama-7b.gguf \
  https://huggingface.co/QuantFactory/llama-7b-gguf/resolve/main/llama-7b.Q4_K_M.gguf

# 3️⃣ Start the stack
docker compose up -d
Enter fullscreen mode Exit fullscreen mode

The service now listens at http://localhost:8080/v1/chat/completions and can be used with any OpenAI‑compatible client (Postman, cURL, or the openai Python library).

Three Ready‑to‑Run Projects

1️⃣ Simple Chatbot (Python) 

import openai

openai.api_base = "http://localhost:8080/v1"
openai.api_key = "any‑string"          # not checked locally

def ask(prompt: str) -> str:
    resp = openai.ChatCompletion.create(
        model="llama-7b",               # name matches the GGUF file (without extension)
        messages=[{"role": "user", "content": prompt}],
        temperature=0.7,
        max_tokens=256,
    )
    return resp.choices[0].message.content.strip()

print(ask("Explain quantum computing in two sentences."))
Enter fullscreen mode Exit fullscreen mode

Result: Instant reply with zero network latency and no token cost.

2️⃣ Text‑to‑SQL Generator (Node.js) 

// npm i openai
const { OpenAI } = require("openai");
const client = new OpenAI({
  baseURL: "http://localhost:8080/v1",
  apiKey: "dummy",
});

async function sqlFromQuestion(question) {
  const response = await client.chat.completions.create({
    model: "mistral-7b",
    messages: [
      { role: "system", content: "You are a helpful assistant that converts natural language to PostgreSQL queries." },
      { role: "user",   content: question },
    ],
    temperature: 0,
    max_tokens: 150,
  });
  console.log(response.choices[0].message.content);
}

sqlFromQuestion("Show the top 5 customers by total purchase amount for the last month.");
Enter fullscreen mode Exit fullscreen mode

Works offline, perfect for internal dashboards where data must never leave the premises.

3️⃣ Document Summarisation (Bash + curl) 

DOC=$(cat report.txt)

curl -s http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
        "model":"llama-7b",
        "messages":[
          {"role":"system","content":"Summarise the following text in three bullet points."},
          {"role":"user","content":"'"${DOC}"'"}
        ],
        "temperature":0.3,
        "max_tokens":200
      }' | jq -r '.choices[0].message.content'
Enter fullscreen mode Exit fullscreen mode

Great for on‑premise compliance teams that need quick extracts without uploading documents to a SaaS

Herramienta mencionada: Groq Cloud

Top comments (0)