Benchmarking Local AI: Building a llama.cpp vs. Ollama Comparison & Benchmarking App

#ollama #llamacpp #bob #localaisolution

Pros and cons of both tools provided by Bob!

Introduction

I’ve been reading and seeing blog posts regarding the virtues of llama.cpp versus Ollama. Myself, I'm an Ollama user since a long time. I decided to ask Bob to build a thorough comparison and if possible a benchmarking code. And amazingly, this is exactly what I got!

Bob didn’t just give me a static list of bullet points; he engineered a full-stack Flask application equipped with live environment probes, dynamic GGUF model discovery, and an asynchronous benchmarking engine to test real-time inference side by side.

System Architecture & Overview

To evaluate both backends objectively, Bob structured the system around a lightweight, single-page application (SPA) frontend paired with a Python/Flask REST backend.

┌─────────────────────────────────────────────────────┐
│  Browser (vanilla JS SPA)                           │
│  benchRun() → POST /api/bench → poll /api/bench/job │
└────────────────────┬────────────────────────────────┘
                     │ HTTP
┌────────────────────▼────────────────────────────────┐
│  Flask Backend (app.py)                             │
│  api_bench() → background worker thread              │
│  _run_bench_job() → _bench_ollama() / _bench_llamacpp│
└───────────┬───────────────────┬─────────────────────┘
            │ HTTP              │ Subprocess
┌───────────▼──────┐  ┌─────────▼────────────────────┐
│  Ollama Daemon   │  │  llama CLI / Dispatcher      │
│  localhost:11434 │  │  PATH / ~/.local/bin          │
│  /api/generate   │  │  llama cli -m … -p … -n …    │
└──────────────────┘  └──────────────────────────────┘

The application serves static comparison data, auto-detects installed local tools and .gguf model files, and executes isolated timing benchmarks.

# app.py: Core Flask Application Setup & Data Routes
from flask import Flask, jsonify, render_template, request
from flask_cors import CORS
import json, os, subprocess, threading, time

app = Flask(__name__)
CORS(app)

DATA_PATH  = os.path.join(os.path.dirname(__file__), "data", "comparison.json")
BENCH_PATH = os.path.join(os.path.dirname(__file__), "data", "bench_history.json")

@app.route("/")
def index():
    return render_template("index.html")

@app.route("/api/data")
def api_data():
    with open(DATA_PATH, "r", encoding="utf-8") as fh:
        return jsonify(json.load(fh))

Interface & Local Tool Integration

Ollama: Frictionless Desktop & Daemon Workflows

Ollama provides an out-of-the-box desktop UI and background daemon process (localhost:11434). It simplifies model management into one-liner commands like ollama run and abstract model tags (e.g., ibm/granite4:3b).

The Flask backend probes the live Ollama daemon to discover currently pulled models:

@app.route("/api/probe/ollama")
def probe_ollama():
    result = {"installed": False, "version": None, "models": [], "error": None}

    # Check CLI presence
    try:
        proc = subprocess.run(["ollama", "--version"], capture_output=True, text=True, timeout=5)
        if "version" in proc.stdout.lower() or "version" in proc.stderr.lower():
            result["installed"] = True
    except FileNotFoundError:
        result["error"] = "ollama binary not found in PATH"
        return jsonify(result)

    # Fetch models via Ollama API
    try:
        req = urllib.request.Request("http://localhost:11434/api/tags")
        with urllib.request.urlopen(req, timeout=5) as resp:
            data = json.loads(resp.read().decode())
            result["models"] = [
                {
                    "name": m.get("name", ""),
                    "size_gb": round(m.get("size", 0) / 1e9, 2),
                    "quantization": m.get("details", {}).get("quantization_level", ""),
                }
                for m in data.get("models", [])
            ]
    except Exception as exc:
        result["error"] = f"Ollama API unreachable: {exc}"

    return jsonify(result)

llama.cpp: Granular Binary & Hardware Optimization

llama.cpp focuses on high-performance C/C++ execution with raw hardware access, custom quantizations (such as mxfp4, Q8_0, Q4_0, Q4_K_M), and multi-command dispatchers (llama cli).

Bob built a scanner into app.py to auto-discover GGUF weights stored across local Hugging Face caches, custom folders, and system directories:

@app.route("/api/probe/llamacpp/models")
def probe_llamacpp_models():
    models = []
    search_dirs = [
        os.path.expanduser("~/.cache/huggingface/hub"),
        os.path.expanduser("~/models"),
        os.path.expanduser("~/Downloads"),
    ]

    for d in search_dirs:
        if not os.path.isdir(d):
            continue
        for dirpath, _, files in os.walk(d):
            for fname in files:
                if fname.lower().endswith(".gguf"):
                    full = os.path.join(dirpath, fname)
                    models.append({
                        "name": fname,
                        "path": full,
                        "size_gb": round(os.path.getsize(full) / 1e9, 2)
                    })
    return jsonify({"models": models})

Benchmarking Inference: Streaming vs. Subprocess Parsing

The primary technical challenge in comparing both systems is capturing accurate, equivalent performance metrics:

Tokens Per Second (TPS): Generation throughput.
Time to First Token (TTFT): Perceived initial latency.
Prompt Evaluation Time: Time required to ingest input tokens.

Ollama Streaming Runner

Bob used Ollama’s HTTP streaming API to capture the exact timestamp of the first token chunk received:

def _bench_ollama(model: str, prompt: str, n_predict: int) -> dict:
    url = "http://localhost:11434/api/generate"
    payload = {
        "model": model,
        "prompt": prompt,
        "stream": True,
        "options": {"num_predict": n_predict},
    }

    t_start = time.monotonic()
    t_first_token = None
    total_tokens = 0
    eval_duration_ns = 0

    resp = requests.post(url, json=payload, stream=True, timeout=300)
    for raw_line in resp.iter_lines():
        if not raw_line: continue
        chunk = json.loads(raw_line)

        # Capture TTFT on first non-empty response token
        if chunk.get("response") and t_first_token is None:
            t_first_token = time.monotonic()

        if chunk.get("done"):
            eval_duration_ns = chunk.get("eval_duration", 0)
            total_tokens = chunk.get("eval_count", 0)
            break

    t_end = time.monotonic()
    eval_ms = eval_duration_ns / 1_000_000 if eval_duration_ns else None
    tps = (total_tokens / (eval_ms / 1000)) if eval_ms else None

    return {
        "time_to_first_token_ms": round((t_first_token - t_start) * 1000, 2) if t_first_token else None,
        "total_time_ms": round((t_end - t_start) * 1000, 2),
        "tokens_per_second": round(tps, 2) if tps else None,
        "tokens_generated": total_tokens,
    }

llama.cpp Native Dispatcher Runner

For llama.cpp, the app executes the CLI binary via subprocess.Popen with safe pipe draining to avoid deadlock buffers, parsing standard timing blocks (llama_print_timings):

def _bench_llamacpp(model: str, prompt: str, n_predict: int) -> dict:
    cli_path, _ = _find_llama_cli()
    cmd = [
        cli_path, "cli",
        "-m", model,
        "-p", prompt,
        "-n", str(n_predict),
        "--no-conversation",
        "--single-turn",
        "--reasoning", "off",
        "--log-disable", "-e"
    ]

    t_start = time.monotonic()
    proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
    stdout, stderr = proc.communicate(timeout=300)
    t_end = time.monotonic()

    combined = stdout + "\n" + stderr

    # Extract timings from llama_print_timings log output
    load_ms   = _parse_ms(r"load time\s*=\s*([\d.]+)\s*ms", combined)
    prompt_ms = _parse_ms(r"prompt eval time\s*=\s*([\d.]+)\s*ms", combined)
    eval_tps  = _parse_tps(r"eval time.*?([\d.]+)\s*tokens per second", combined)

    ttft_ms = (load_ms + prompt_ms) if (load_ms and prompt_ms) else prompt_ms

    return {
        "time_to_first_token_ms": round(ttft_ms, 2) if ttft_ms else None,
        "total_time_ms": round((t_end - t_start) * 1000, 2),
        "tokens_per_second": round(eval_tps, 2) if eval_tps else None,
        "command": " ".join(cmd),
    }

Key Takeaways & Comparison Matrix

Here is how the two stack up across core technical categories:

## Key Takeaways & Comparison Matrix

Here is how the two stack up across core technical categories:

| Dimension               | llama.cpp                                                    | Ollama                                                    |
| ----------------------- | ------------------------------------------------------------ | --------------------------------------------------------- |
| **Architecture**        | C/C++ native backend engine                                  | Go wrapper daemon around custom `llama.cpp` builds        |
| **User Experience**     | CLI flags, config scripts, manual GGUF management            | Single binary / desktop app with REST API & model pulls   |
| **Model Discovery**     | Direct GGUF loading from disk or HF cache                    | Centralized model library registry (`ollama pull`)        |
| **Performance Control** | Unrestricted access to context sizing, GPU offloading, threads | Curated defaults optimized for ease of use                |
| **Ideal For**           | Custom pipelines, maximum hardware tuning, offline GGUFs     | Rapid prototyping, web wrappers, hassle-free agent setups |