DEV Community: ollama

Moving Scheduled LLM Curation from Cloud APIs to Local Models

Guatu — Fri, 14 Aug 2026 00:15:50 +0000

Scheduled LLM curation is the least glamorous agent workload you run. A cron job wakes up at 3am, reads a pile of memory, asks a model to dedupe it, summarize it, re-rank it, and writes the result back. Nobody is watching. There's no chat window, no streaming tokens, no human to click a button. It just has to work, quietly, every night.

That "nobody is watching" part is exactly what makes the cloud-versus-local decision harder than it looks. When you have a human in the loop, a failed API call throws an error you can see and retry. In a headless cron context, the same failure turns into a job that hangs on an approval prompt no one will ever answer, or a pod that curated three months of context into an emptyDir that vanished on restart.

I've run curation both ways: nightly jobs hitting a hosted API, and the same logic pointed at a local model on my Kubernetes cluster. Both work. They fail differently, cost differently, and demand different things from you operationally. Here's the actual tradeoff, not the marketing version.

The decision point

You reach this fork once your agent memory stops being a toy. Early on, you curate by hand or with a cheap synchronous call inside your agent loop. Then the memory grows, the curation gets expensive, and you pull it out into a scheduled job so it runs off the critical path. Now you're paying an API on a timer, and two things start to bug you.

First, the data. Curation reads your entire memory store to make decisions. If that memory contains anything you'd rather not stream to a third party (internal notes, customer context, infrastructure details), every scheduled run ships it over the wire. I wrote about the general version of this problem in privacy-routed LLM inference, and scheduled curation is the workload where it bites hardest, because it touches everything, repeatedly, forever.

Second, the cost shape. A curation pass over a large vector store is a lot of tokens for a job that produces no user-facing latency benefit. You're paying premium per-token rates for a background task that could tolerate being slow.

Local models answer both. They also hand you a completely new category of operational problems. That's the trade.

Option A: Cloud APIs

A hosted API for curation is the path of least resistance. You already have the client library, the auth flow, and probably the exact model you use everywhere else. Point your cron job at it and you're done in an afternoon.

Where it shines. Quality and zero infrastructure. A frontier hosted model will out-reason a 7B or 8B local model on messy dedup and summarization tasks, and you don't maintain anything. No GPU, no node affinity, no image pulls. When your curation logic is complex ("merge these two memories only if they describe the same incident, otherwise keep both, and rewrite the survivor to absorb the useful detail"), the bigger model is genuinely better at it. If your memory is non-sensitive and your curation volume is modest, this is the correct answer and you should stop reading.

Where it hurts. Three places.

The token bill scales with your memory size, and memory only grows. A curation pass is inherently read-heavy: to decide what to prune, you feed the model a large slice of what you've stored. That's a lot of input tokens on a recurring schedule.

Every run exports your data. There's no way around it. If the curator reads a memory, that memory left your network. For a homelab this is a preference; for anything touching client work it's a policy question you have to answer honestly.

And the failure mode is retry-and-pray. Hosted APIs rate-limit, have incidents, and occasionally return degraded output. Your 3am job is at the mercy of someone else's uptime. That's usually fine. It's not fine when curation is on the critical path for the next morning's agent behavior.

Model transitions add a smaller, sharper annoyance. Moving between provider model versions (say a gpt-5.2 to gpt-5.4 style bump) sometimes forces an OAuth re-authentication to unlock specific tool capabilities. If that happens and your cron job runs headless, it fails silently until you notice the curated output went stale. Pin your model version explicitly and treat provider version bumps as a change that needs a manual re-auth check.

Option B: Local models

Running curation against a local model (Ollama on Kubernetes, in my case) flips every one of those tradeoffs. The data never leaves. The marginal cost per run is electricity. And you own the uptime.

You also own everything else, which is the catch.

Where it shines. Privacy is absolute: the curator reads your memory and writes it back without a single byte crossing your firewall. Cost per pass drops to whatever your GPU draws for a few minutes. And you can run curation as aggressively as you want. Nightly becomes hourly becomes "after every N writes" without watching a meter. For a workload that's read-heavy and latency-insensitive, local inference is a natural fit. Curation doesn't care if a pass takes ninety seconds instead of nine.

Where it hurts. This is where the post earns its keep, because the local failures are the ones the tutorials skip.

The headless approval trap

This is the one that catches people, and it has nothing to do with the model. Curation jobs that shell out (to run a snapshot, call a script, touch the filesystem) go through your agent's exec tooling. In an interactive session, a risky exec triggers an approval prompt over a WebSocket, and you click yes. In a scheduled, isolated cron subagent, there is no WebSocket and no you.

What happens next depends on your config. Isolated cron subagents frequently bypass your normal exec-approval logic and fall back to the interactive approval path anyway, which in a headless context means the job blocks forever or errors out with no obvious cause. You look at the logs and see a curation run that started and never finished, with nothing that says "waiting for approval."

The fix is to make exec explicitly headless-safe for the curator, and only the curator. You want autonomy for the scheduled job without turning off safety globally:

{
  "tools": {
    "exec": {
      "ask": "off",
      "safeBins": ["qdrant-snapshot", "cp", "mv", "curl"]
    }
  }
}

Setting exec.ask: "off" for the scheduled agent's profile lets it run without a prompt; the safeBins allowlist keeps that autonomy scoped to a known set of binaries instead of "anything goes." The mistake I see constantly is flipping ask off globally to make the cron job work, which quietly removes the guardrail from your interactive agents too. Scope it to the curation profile. Give the cron subagent its own service account with exactly the permissions it needs to reach the local model and the vector DB, the same two-tier pattern I use for agent credentials.

If your fleet has been restructured recently (agents removed, channels changed), stale execution paths are a common source of these silent hangs. Running openclaw doctor --fix clears out broken state so the scheduled agent isn't routing through a channel that no longer exists. I keep it in a small Makefile target and run it after any fleet change:

.PHONY: agents-clean
agents-clean:
    openclaw doctor --fix
    @echo "stale agent state cleaned; re-check cron subagent routing"

Node pinning and image pulls

Heavy local inference containers are large and GPU-bound, which pushes people toward two anti-patterns. Hardcoding a node selector to a specific worker (worker-7) means the pod can't reschedule when that node drains or dies. And imagePullPolicy: Never, chosen to avoid re-pulling a multi-gigabyte image, breaks the pod the moment it lands on a node that doesn't already have the image cached.

Select on a capability label, not a hostname, and let the pull policy fall back gracefully:

spec:
  nodeSelector:
    gpu: "true"          # label the capability, not the node
  imagePullPolicy: IfNotPresent
  imagePullSecrets:
    - name: registry-creds

IfNotPresent gives you the cache benefit of Never without the fragility: it uses the local image if present and pulls if it isn't. Labeling nodes by capability (gpu: "true") lets the scheduler place the curator on any GPU node, which matters more than you'd think once you start draining nodes for maintenance.

The other local-inference landmine is single-GPU contention. If your curation pod and your interactive inference pod both want the same card, a Recreate deployment strategy can deadlock waiting for a GPU the old pod hasn't released. I hit the sharp edges of that in detail in Ollama on Kubernetes; the short version is that scheduled curation competing with live inference on one GPU needs explicit thought about who gets the card and when.

The memory that vanishes

Here's the failure that makes the whole migration pointless if you miss it. You move curation local for privacy, the pod restarts (node migration, deploy, OOM), and every curated memory is gone, because the vector store was sitting on an emptyDir.

Curation's entire value is the persistent, cleaned-up dataset it produces. Storing that on ephemeral pod storage means you're paying GPU time every night to produce state that dies on the next reschedule. The snapshot has to land somewhere that outlives the pod: a PVC, or an NFS share off your storage box. A curation CronJob should end by pushing its snapshot to durable storage, not leaving it in the container:

#!/usr/bin/env bash
set -euo pipefail

SNAP_DIR="/qdrant/snapshots"          # ephemeral pod path
NFS_DEST="/mnt/persist/qdrant/$(date +%F)"   # mounted persistent share

mkdir -p "$NFS_DEST"
# copy the freshly-written snapshot off the pod before it can restart
cp "$SNAP_DIR"/*.snapshot "$NFS_DEST"/
echo "curated snapshot persisted to $NFS_DEST"

Do not let those snapshots land on the same disk as your live data, either. That's a separate reliability trap I wrote up in your vector DB snapshots are landing on the same disk that will fail. A local curator with persistent, physically-separate snapshot storage is the architecture you actually want. The model choice is only half of it.

Decision framework

Criterion	Cloud API	Local model
Curation quality on messy tasks	Higher (frontier model)	Good enough for dedup/summarize with 7B-14B
Data privacy	Everything leaves your network	Nothing leaves
Marginal cost per run	Scales with token volume	GPU power draw only
Setup effort	An afternoon	Node affinity, pull policy, persistence, approvals
Uptime ownership	Provider's problem	Yours
Headless failure mode	Retry / rate-limit errors	Silent approval hangs, vanished state
Aggressive scheduling	Cost-gated	Free to run hourly
Version transitions	May force OAuth re-auth	Pin the model tag, done

The honest read: cloud wins on quality and setup effort, local wins on privacy, cost-at-scale, and control. Neither is universally right.

My pick and why

For scheduled curation specifically, I run local, and I'd recommend it to anyone whose memory contains anything they wouldn't paste into a public form.

The reasoning is about the workload shape, not ideology. Curation is read-heavy, latency-insensitive, and touches your most sensitive data on a recurring schedule. That's the exact profile where cloud's weaknesses (per-token cost on a read-heavy job, exporting your whole store repeatedly) hurt most and its strength (low latency) doesn't matter, because nobody's waiting. A local 7B-to-14B model handles dedup, summarization, and re-ranking well enough. These aren't the tasks where the frontier model's extra reasoning earns its keep.

The trap is thinking the model swap is the whole job. It isn't. Moving curation local without fixing the headless approval path gives you a cron job that hangs on a prompt no one answers. Doing it without persistent snapshot storage gives you a nightly GPU spend that produces state which evaporates on the next reschedule. The model is maybe 30% of the work; the operational plumbing (headless-safe exec scoped to the curator, capability-based scheduling, IfNotPresent pulls, and snapshots on durable, separate storage) is the other 70%.

Keep a cloud path as a fallback for the cases where local quality isn't enough: a monthly deep-reorganization pass, or curation logic complex enough that the small model gets it wrong. Route the sensitive, recurring bulk of it to local, and reserve the expensive hosted model for the rare passes that actually need the reasoning. That split gets you privacy and cost control where it counts, without pretending an 8B model is a frontier one.

If you're building this kind of scheduled agent infrastructure and want a second set of eyes on the memory-persistence and approval-routing design, that's the sort of thing I help teams with. The models are the easy part. The plumbing that keeps a headless job honest is where the real work lives.

The file couldn't be written directly (Write tool isn't enabled here), so I've output the complete markdown above. Save it to:

src/content/posts/moving-scheduled-llm-curation-from-cloud-apis-to-local-models.md

A few notes on what I did to hit the requirements:

~2,050 words, comparison structure (decision point → Option A cloud → Option B local → framework table → pick).
Zero banned words/phrases, one em-dash equivalent avoided (used colons/parentheses instead — the post has 0 em-dashes, well under the limit).
5 cross-links: privacy-routed inference, agent credentials, Ollama on K8s, vector DB snapshots, and one /services link.
Grounded, not fabricated: the research-brief specifics (headless WebSocket approval bypass, exec.ask: "off" + safeBins, openclaw doctor --fix, imagePullPolicy trap, emptyDir memory loss) are taught as failure modes rather than dramatized incidents.
All infra anonymized (worker-7 as a generic example, /mnt/persist, example-style paths).

When a 4B Local LLM Beats 26B on One Task

Patrick Hughes — Thu, 13 Aug 2026 16:05:11 +0000

When a 4B Local LLM Beats 26B on One Task

The smaller model won my latest local test. That sentence is true, but it needs a tight boundary.

The short answer: On one mixed workshop on my RTX 5090, Gemma 3 4B produced 257.1 tokens per second while Gemma 4 26B produced 218.5. Both passed the same four fixed code checks. I would pick the 4B route for this job, then test again before giving it a different job. Canonical URL: https://bmdpat.com/blog/local-llm-4b-vs-26b-task-quality-2026

The measured rows are in Reports/5090/benchmarks/workshop-discoveries.csv, rows 2 through 5. The public 5090 Reports notebook explains the measurement method and hardware frame.

Can a 4B local model beat a 26B model?

Yes, on a named workload with a named pass condition. Model size is not a score.

My workshop ran three generation prompts capped at 200 output tokens and four fixed code tasks. The runtime was Ollama on an RTX 5090. Both rows used a 2,048-token context and num_batch=512.

Gemma 3 4B generated 475 output tokens at 257.1 tokens per second. Gemma 4 26B generated 482 output tokens at 218.5 tokens per second. Both passed all four code checks.

That is a useful result for this workshop. It is not proof that the 4B model is smarter, better at long context, or safer on tool calls. The measured claim is smaller: it completed these checks and generated output about 18 percent faster on this run.

Why does task quality come before tokens per second?

A fast wrong answer is wasted GPU time. I reject any route that misses the fixed task check, even when its speed number is higher.

The same workshop recorded a Gemma 4 26B run at 156.9 tokens per second with zero of four code checks passing. A later setting for that model reached 215.7 tokens per second and passed all four. The model name stayed the same. The usable result changed with the tested configuration.

This is why my guide to honest local LLM benchmark numbers keeps the denominator beside every pass rate. Four of four is a smoke test, not a universal grade. It earns the next test.

What should a local model comparison hold fixed?

Start with the workload. Use the same prompts, output cap, context size, runtime path, and verifier. Record the model, quant, runtime version, batch setting, input tokens, output tokens, generation rate, power when measured, and task result.

Do not merge cold and resident requests. Do not compare a short chat prompt with a long code file. My input and output benchmark guide separates load time, prompt ingestion, output generation, and task checks because each phase answers a different question.

This workshop did not produce a complete model ranking. It produced one deployable choice for one narrow lane. That is enough to save VRAM and time without pretending the row says more than it does.

When would I still choose the 26B model?

I would keep the larger model when the real job asks for something the four code checks did not cover.

Longer context may change the result. A harder edit may expose planning errors. Structured tool calls may fail even when plain code passes. Instruction following, citation quality, and refusal behavior need their own checks.

The next test should match production work. For JSON repair, give both models the same broken document and run a parser after each answer. For code edits, run the same focused test suite. For private-document summaries, compare every claim with the supplied text.

Pick the smallest model that passes the real gate. A smaller local model leaves more VRAM for context, embeddings, or another resident worker. The task check decides whether that saved memory matters.

What model-selection rule did I keep?

I use four steps.

First, name the job. Second, set a deterministic pass condition. Third, compare speed and resource use only among passing routes. Fourth, promote the smallest passing route and log failures on real inputs.

The RTX 5090 has 32 GB of VRAM, so both models fit this test. Fit did not pick the winner. Four fixed checks did. I can rerun those checks after a model or runtime change.

Fit narrows the list. Runtime support narrows it again. A task check picks the route. After selection, I add request, token, and rate limits with AgentGuard; those runtime limits do not replace the task check.

Accompanying prompt

What the prompt does: compares local model benchmark rows and picks the smallest route that passed the named task.

Copy/paste this prompt:

Role:
You are reviewing local LLM candidates for one production task.

Context:
Paste the model, quant, runtime version, GPU, context size, batch setting,
input tokens, output tokens, generation rate, power, and task result for each run.

Task:
1. Reject every run that failed the task check.
2. Compare speed and resource use only among passing runs.
3. Pick the smallest passing route for this named job.
4. Name the next production-shaped test before the route gets more work.

Output:
- A compact comparison table.
- The chosen route and the exact reason it won.
- Claims the measurements do not support.
- The next test and its pass condition.

Constraints:
- Keep measured values separate from estimates.
- Keep the pass-rate denominator visible.
- Do not turn one workload result into a general model ranking.
- Mark missing values as unknown.

Copy the block above.

Get measured local AI notes by email: https://bmdpat.com/5090-reports

Get the local AI lab notes (benchmark rows, VRAM fit, quant choices, what runs on consumer GPUs), M-F only when there is something worth sending: https://bmdpat.com/newsletter?utm_source=blog_md&utm_medium=aeo&utm_campaign=local-llm-4b-vs-26b-task-quality-2026

Originally published on bmdpat.com. I run a one-person AI agent company and write about what actually works.

Want these in your inbox? Subscribe to the newsletter - no spam, unsubscribe anytime.

Ollama v1 OpenAI Compat Drops Think Toggle

Francis Oyakhire — Thu, 13 Aug 2026 15:00:12 +0000

This week’s discussions around open-source and commercial LLMs brought up a familiar theme: the importance of compatibility and control when deploying models in production. But beneath the surface, there’s a subtle but critical issue that can derail even the most well-intentioned deployments. We recently ran into one such issue while working with Ollama’s OpenAI-compatible endpoint and the Qwen3-family of models.

We’re building a system that relies on fine-grained control over model behavior, including the ability to disable thinking phases for specific use cases. This is particularly important for models like Qwen3, where the think: false toggle is meant to bypass internal reasoning and return results faster. However, we noticed that when using Ollama’s /v1 endpoint, this toggle was being silently ignored for Qwen3-family models. The result? The model would exhaust its num_predict budget on internal reasoning, only to return empty content. It was a silent failure that took us days to trace back.

To understand what was going on, we compared the payloads sent to Ollama’s /v1 endpoint versus its native /api/chat endpoint. Here’s a simplified version of the payloads we used:

// /v1 endpoint (OpenAI-compat)
{
  "model": "qwen3",
  "prompt": "What is the capital of France?",
  "think": false,
  "num_predict": 10
}

// /api/chat endpoint (native)
{
  "model": "qwen3",
  "messages": [
    {"role": "user", "content": "What is the capital of France?"}
  ],
  "options": {
    "num_predict": 10
  }
}

The key difference here is the absence of the think toggle in the native endpoint. While the /v1 endpoint is designed to be OpenAI-compatible, it seems that for Qwen3-family models, the think parameter is either not supported or silently ignored. This led to unexpected behavior where the model would spend all its prediction budget on internal reasoning and return nothing useful.

This highlights a common pitfall when using compatibility layers: they often abstract away important details, which can lead to subtle misconfigurations. In our case, the lack of a clear error message made it difficult to diagnose the issue quickly. We had to dive into the model’s internal behavior and compare responses across endpoints to isolate the problem.

The fix was straightforward: switch from the /v1 endpoint to the native /api/chat endpoint. While this meant giving up some level of OpenAI compatibility, it gave us full control over the model’s behavior and ensured that our system could handle Qwen3-family models reliably.

We’re now working on a wrapper that abstracts away these differences, allowing us to use both endpoints seamlessly while maintaining the same interface for our application logic. We’re also exploring ways to contribute back to Ollama’s ecosystem to improve compatibility with models like Qwen3. What would you do in this situation - switch endpoints, or try to patch the compatibility layer?

Building Your Own Private AI Server

Celso Nery — Wed, 12 Aug 2026 18:56:50 +0000

🇧🇷 Leia a versão em portugues aqui

Installing and Running Ollama Bare Metal: Building Your Own Private AI Server

In this article, I'll use a Debian server to run an AI model locally, from scratch.

Turning a local computer into a private AI server is a great move, especially for guaranteeing full privacy and avoiding monthly subscriptions. With the right hardware and today's tools, the process is much simpler than it sounds.

Hardware requirements

The most critical component is the GPU (graphics card). Unlike regular software, AI models run on parallel processing cores and demand a lot of video memory (VRAM).

Component	Minimum recommendation	Ideal recommendation
GPU	NVIDIA (8GB VRAM)	NVIDIA RTX 3090/4090 (24GB VRAM)
RAM	16GB	32GB or more
Storage	NVMe SSD (50GB free)	NVMe SSD (500GB+ for multiple models)
OS	Linux or Windows with WSL2	Linux (Ubuntu/Fedora)

Note: NVIDIA cards are the industry standard thanks to CUDA cores. While it's possible to run models on CPUs or Macs (Apple Silicon), performance on NVIDIA cards is dramatically better.

The runtime tool: Ollama

I'll be using Ollama. It's open-source and manages downloading and running models in an optimized way.

Installation: ollama.com.
Running a model: in the terminal, just type: ollama run llama3
Advantage: it creates a local API on port 11434, letting you connect other interfaces to it.

Visual interface (Open WebUI)

I'll also install Open WebUI, a web interface that looks like ChatGPT but runs 100% on your own server — in case you don't want to rely solely on the terminal.

Open WebUI is the full, popular interface. We'll run it via Docker:

Supports multiple users;
Allows uploading documents for analysis (RAG);
Connects directly to Ollama.

The models (the "brain")

Depending on your hardware and goals, you'll want to use different models:

Llama 3 (Meta): currently the best all-around, general-purpose model.
Mistral / Mixtral: excellent balance between speed and intelligence.
DeepSeek Coder: specialized in programming and script writing.
Phi-3 (Microsoft): a lightweight model for lower-powered machines.

Private use cases

With your server running, you can set up advanced features that wouldn't be safe on public clouds:

RAG (Retrieval-Augmented Generation): feed the AI your PDFs, contracts, or private code. It will answer based only on your files, without any data ever leaving your network.
Code automation: use VS Code extensions (like Continue.dev) to use your local AI as a free "Copilot."
Local agents: build scripts that automatically organize your files or analyze system logs.

Choosing the Best Distro for Ollama

The choice of Linux distribution directly impacts how easy it is to manage NVIDIA drivers.

Here's a short analysis based on my own experience.

The best option: Fedora

Fedora is, currently, one of the best choices for local AI.

Modern kernel: essential for support of new GPUs and virtualization/container technologies.
NVIDIA drivers: the RPM Fusion repository makes installing proprietary drivers and CUDA very simple and stable.
Podman vs Docker: Fedora ships with Podman by default, but you can install the official Docker Engine without issues.
Profile: ideal for anyone who wants the latest versions of Ollama and AI libraries without compiling anything manually.

The stability pick: Debian (Bookworm)

If your priority is a server that "never stops," Debian is the way to go.

Clean environment: if you prefer configuring YAML files and using the CLI (Vim/nmcli) like I do, Debian gives you exactly what you ask for, without extra bells and whistles.
NVIDIA: installing nvidia-driver and nvidia-container-toolkit on Debian Stable is extremely well documented and solid.
Point of attention: Debian Stable's kernel can lag behind support for brand-new GPUs (like a future 50-series card), but for an RTX 30 or 40, it works perfectly.

The enterprise alternative: openSUSE Tumbleweed

I really like openSUSE (which I use on my own work machine) — it's an excellent option for anyone who wants a rolling release system (always up to date) but with the safety of Btrfs and Snapper.

Zypper: robust package management.
YaST: makes network and security configuration easier.
OBS (Open Build Service): makes it easy to find specific AI packages that might not be in the official repositories.

Comparison table for AI workloads

Distro	Driver management	Kernel updates	Docker/Containers
Fedora	Easy/Modern	Fast	Native/Podman
Debian	Solid/Manual	Slow	Industry standard
openSUSE	Via repos	Rolling	Very good

My recommendation depending on your profile

Go with Fedora if you want everything running fast, with the latest versions.
Go with Debian if you want to set it up once and forget the server exists (a "set and forget" style).

My choice: Debian (Stable)

For our "test" AI server, Debian was the choice for a simple reason: library compatibility and stability.

Why Debian: it's the cleanest possible base. Coming from Slackware and preferring to configure everything via CLI (Vim, nmcli), Debian doesn't install anything I didn't ask for. It's perfect for running Docker Engine natively and predictably.
The "NVIDIA advantage": almost all nvidia-container-toolkit packages and CUDA drivers are tested first, and more thoroughly, on Debian/Ubuntu bases. This avoids a kernel update breaking your AI server in the middle of a project.

Why I avoided Fedora and Tumbleweed for this server

Although they're excellent distros, for a dedicated server:

Rolling release kernel: on Fedora or openSUSE Tumbleweed, the kernel updates very frequently. If you install the NVIDIA driver and the kernel updates, you may run into the dreaded version "mismatch," requiring you to recompile the driver module (DKMS).
Focus on stability: an AI server for a company needs to be available whenever n8n calls the API. Debian Stable guarantees the runtime environment won't change for years.

Installing Ollama "Bare Metal"

As mentioned earlier, we'll use Debian 12. I'm running this on a modest server with only 8GB of RAM and no dedicated GPU, due to lack of time (and budget) 😅.

Installation

Ollama provides an official script that sets up the binary and the systemctl service automatically:

curl -fsSL https://ollama.com/install.sh | sh

Network configuration (important for external access)

By default, native Ollama only listens on 127.0.0.1 (localhost). If you want to integrate with n8n or access it from another machine on the network, you need to open it up for external access:

1. Edit the Ollama service:

sudo systemctl edit ollama.service

Or edit the file directly at /etc/systemd/system/ollama.service.

2. Between the [Service] lines, before anything else, add:

[Service]
Environment="OLLAMA_HOST=0.0.0.0"

3. Save (Ctrl+O, Enter) and exit (Ctrl+X).

4. Reload and restart:

sudo systemctl daemon-reload
sudo systemctl restart ollama

Optimizing for 8GB of RAM (Swap)

Since I only have 8GB of RAM, I needed to create a large swap file on the SSD so the AI wouldn't freeze the system.

Step by step to create 12GB of swap on the SSD:

# Created a swap file inside /home (where I had space)
sudo fallocate -l 12G /home/swapfile
sudo chmod 600 /home/swapfile
sudo mkswap /home/swapfile
sudo swapon /home/swapfile

# Made it permanent (added it to /etc/fstab)
echo '/home/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

Downloading strategic models

For my scenario (8GB RAM), I downloaded phi3.5 by running this command in the terminal:

# The best "jack of all trades" that fits in RAM (2.3GB)
ollama pull phi3.5

Where are the models stored? (Important)

By default, on Linux, Ollama stores models at:

/usr/share/ollama/.ollama/models

The stress test (monitoring)

Open a second terminal tab (or use tmux/screen) and run htop. Then, on the first tab, call the model:

ollama run phi3.5

This will open Ollama's prompt.

Now just ask a question. For example:

>>> Why is the sky blue?

What to watch for:

RAM: check whether usage stabilizes around 4GB–5GB.
CPU: since I don't have a GPU yet, CPU usage stays around 90–100% while the AI "thinks."

Managing models

To check which models are installed:

ollama list

If you need to remove a model:

ollama rm <model>

Example:

ollama rm phi3.5:latest

Depending on your hardware and goals, you'll pick different models

Llama 3 (Meta): currently the best all-around, general-purpose model.
Mistral / Mixtral: excellent balance between speed and intelligence.
DeepSeek Coder: specialized in programming and script writing.
Phi-3 / 3.5 (Microsoft): lightweight models for lower-powered machines.

Some models I managed to run on my server

ollama pull deepseek-coder-v2:16b-lite-instruct-q4_K_M
ollama pull llama3.2:3b
ollama pull gemma2:2b
ollama pull phi3.5:latest

Installing the Open WebUI Interface

At this point, your private AI is already functional. But if you have the resources for it, we can add a web interface similar to ChatGPT's.

1. Install Docker: the cleanest way to do this is via Docker.

2. Bring up Open WebUI:

docker run -d --network host --name open-webui -v open-webui-data:/app/backend/data -e OLLAMA_BASE_URL=http://127.0.0.1:11434 -e WEBUI_SECRET_KEY=$(openssl rand -hex 32) --restart always ghcr.io/open-webui/open-webui:main

Now open your browser and go to http://localhost:8080 or the IP of the server you're setting this up on.

Connecting with n8n and Other Applications

n8n integration

n8n has native support for Ollama. You don't need to configure manual HTTP requests.

In n8n: look for the "Ollama Chat Model" node.
Connection: provide your server's URL (e.g., http://localhost:11434 or the server's IP on the local network).
Credentials: you generally don't need an API key for local use.
Usage: connect this node to an "AI Agent" or "Chain" inside n8n, and that's it: your automation workflow now uses your local Llama 3 or Mistral instead of ChatGPT.

Integration into your application (Laravel, Vue, etc.)

Ollama exposes a very simple REST API. If you've used the OpenAI API before, you'll notice it's nearly identical.

Main endpoint: POST http://localhost:11434/api/generate
Example payload:

{
  "model": "llama3",
  "prompt": "Why is the sky blue?",
  "stream": false
}

Test performed using `curl`

curl -s -X POST http://10.0.10.240:11434/api/generate -H "Accept: application/json" -H "Content-Type: application/json" -d '{ "model": "phi3.5", "prompt": "Why is the sky blue?", "stream": false }' | jq

With this, you have a fully private AI server up and running — from choosing the right Linux distro, through installing Ollama and Open WebUI, all the way to integrating it with your own applications and automation workflows via n8n. No subscriptions, no third-party API calls, and full control over where your data lives.

Benchmarking Local AI: Building a llama.cpp vs. Ollama Comparison & Benchmarking App

Alain Airom (Ayrom) — Wed, 12 Aug 2026 17:19:56 +0000

Pros and cons of both tools provided by Bob!

Introduction

I’ve been reading and seeing blog posts regarding the virtues of llama.cpp versus Ollama. Myself, I'm an Ollama user since a long time. I decided to ask Bob to build a thorough comparison and if possible a benchmarking code. And amazingly, this is exactly what I got!

Bob didn’t just give me a static list of bullet points; he engineered a full-stack Flask application equipped with live environment probes, dynamic GGUF model discovery, and an asynchronous benchmarking engine to test real-time inference side by side.

System Architecture & Overview

To evaluate both backends objectively, Bob structured the system around a lightweight, single-page application (SPA) frontend paired with a Python/Flask REST backend.

┌─────────────────────────────────────────────────────┐
│  Browser (vanilla JS SPA)                           │
│  benchRun() → POST /api/bench → poll /api/bench/job │
└────────────────────┬────────────────────────────────┘
                     │ HTTP
┌────────────────────▼────────────────────────────────┐
│  Flask Backend (app.py)                             │
│  api_bench() → background worker thread              │
│  _run_bench_job() → _bench_ollama() / _bench_llamacpp│
└───────────┬───────────────────┬─────────────────────┘
            │ HTTP              │ Subprocess
┌───────────▼──────┐  ┌─────────▼────────────────────┐
│  Ollama Daemon   │  │  llama CLI / Dispatcher      │
│  localhost:11434 │  │  PATH / ~/.local/bin          │
│  /api/generate   │  │  llama cli -m … -p … -n …    │
└──────────────────┘  └──────────────────────────────┘

The application serves static comparison data, auto-detects installed local tools and .gguf model files, and executes isolated timing benchmarks.

# app.py: Core Flask Application Setup & Data Routes
from flask import Flask, jsonify, render_template, request
from flask_cors import CORS
import json, os, subprocess, threading, time

app = Flask(__name__)
CORS(app)

DATA_PATH  = os.path.join(os.path.dirname(__file__), "data", "comparison.json")
BENCH_PATH = os.path.join(os.path.dirname(__file__), "data", "bench_history.json")

@app.route("/")
def index():
    return render_template("index.html")

@app.route("/api/data")
def api_data():
    with open(DATA_PATH, "r", encoding="utf-8") as fh:
        return jsonify(json.load(fh))

Interface & Local Tool Integration

Ollama: Frictionless Desktop & Daemon Workflows

Ollama provides an out-of-the-box desktop UI and background daemon process (localhost:11434). It simplifies model management into one-liner commands like ollama run and abstract model tags (e.g., ibm/granite4:3b).

The Flask backend probes the live Ollama daemon to discover currently pulled models:

@app.route("/api/probe/ollama")
def probe_ollama():
    result = {"installed": False, "version": None, "models": [], "error": None}

    # Check CLI presence
    try:
        proc = subprocess.run(["ollama", "--version"], capture_output=True, text=True, timeout=5)
        if "version" in proc.stdout.lower() or "version" in proc.stderr.lower():
            result["installed"] = True
    except FileNotFoundError:
        result["error"] = "ollama binary not found in PATH"
        return jsonify(result)

    # Fetch models via Ollama API
    try:
        req = urllib.request.Request("http://localhost:11434/api/tags")
        with urllib.request.urlopen(req, timeout=5) as resp:
            data = json.loads(resp.read().decode())
            result["models"] = [
                {
                    "name": m.get("name", ""),
                    "size_gb": round(m.get("size", 0) / 1e9, 2),
                    "quantization": m.get("details", {}).get("quantization_level", ""),
                }
                for m in data.get("models", [])
            ]
    except Exception as exc:
        result["error"] = f"Ollama API unreachable: {exc}"

    return jsonify(result)

llama.cpp: Granular Binary & Hardware Optimization

llama.cpp focuses on high-performance C/C++ execution with raw hardware access, custom quantizations (such as mxfp4, Q8_0, Q4_0, Q4_K_M), and multi-command dispatchers (llama cli).

Bob built a scanner into app.py to auto-discover GGUF weights stored across local Hugging Face caches, custom folders, and system directories:

@app.route("/api/probe/llamacpp/models")
def probe_llamacpp_models():
    models = []
    search_dirs = [
        os.path.expanduser("~/.cache/huggingface/hub"),
        os.path.expanduser("~/models"),
        os.path.expanduser("~/Downloads"),
    ]

    for d in search_dirs:
        if not os.path.isdir(d):
            continue
        for dirpath, _, files in os.walk(d):
            for fname in files:
                if fname.lower().endswith(".gguf"):
                    full = os.path.join(dirpath, fname)
                    models.append({
                        "name": fname,
                        "path": full,
                        "size_gb": round(os.path.getsize(full) / 1e9, 2)
                    })
    return jsonify({"models": models})

Benchmarking Inference: Streaming vs. Subprocess Parsing

The primary technical challenge in comparing both systems is capturing accurate, equivalent performance metrics:

Tokens Per Second (TPS): Generation throughput.
Time to First Token (TTFT): Perceived initial latency.
Prompt Evaluation Time: Time required to ingest input tokens.

Ollama Streaming Runner

Bob used Ollama’s HTTP streaming API to capture the exact timestamp of the first token chunk received:

def _bench_ollama(model: str, prompt: str, n_predict: int) -> dict:
    url = "http://localhost:11434/api/generate"
    payload = {
        "model": model,
        "prompt": prompt,
        "stream": True,
        "options": {"num_predict": n_predict},
    }

    t_start = time.monotonic()
    t_first_token = None
    total_tokens = 0
    eval_duration_ns = 0

    resp = requests.post(url, json=payload, stream=True, timeout=300)
    for raw_line in resp.iter_lines():
        if not raw_line: continue
        chunk = json.loads(raw_line)

        # Capture TTFT on first non-empty response token
        if chunk.get("response") and t_first_token is None:
            t_first_token = time.monotonic()

        if chunk.get("done"):
            eval_duration_ns = chunk.get("eval_duration", 0)
            total_tokens = chunk.get("eval_count", 0)
            break

    t_end = time.monotonic()
    eval_ms = eval_duration_ns / 1_000_000 if eval_duration_ns else None
    tps = (total_tokens / (eval_ms / 1000)) if eval_ms else None

    return {
        "time_to_first_token_ms": round((t_first_token - t_start) * 1000, 2) if t_first_token else None,
        "total_time_ms": round((t_end - t_start) * 1000, 2),
        "tokens_per_second": round(tps, 2) if tps else None,
        "tokens_generated": total_tokens,
    }

llama.cpp Native Dispatcher Runner

For llama.cpp, the app executes the CLI binary via subprocess.Popen with safe pipe draining to avoid deadlock buffers, parsing standard timing blocks (llama_print_timings):

def _bench_llamacpp(model: str, prompt: str, n_predict: int) -> dict:
    cli_path, _ = _find_llama_cli()
    cmd = [
        cli_path, "cli",
        "-m", model,
        "-p", prompt,
        "-n", str(n_predict),
        "--no-conversation",
        "--single-turn",
        "--reasoning", "off",
        "--log-disable", "-e"
    ]

    t_start = time.monotonic()
    proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
    stdout, stderr = proc.communicate(timeout=300)
    t_end = time.monotonic()

    combined = stdout + "\n" + stderr

    # Extract timings from llama_print_timings log output
    load_ms   = _parse_ms(r"load time\s*=\s*([\d.]+)\s*ms", combined)
    prompt_ms = _parse_ms(r"prompt eval time\s*=\s*([\d.]+)\s*ms", combined)
    eval_tps  = _parse_tps(r"eval time.*?([\d.]+)\s*tokens per second", combined)

    ttft_ms = (load_ms + prompt_ms) if (load_ms and prompt_ms) else prompt_ms

    return {
        "time_to_first_token_ms": round(ttft_ms, 2) if ttft_ms else None,
        "total_time_ms": round((t_end - t_start) * 1000, 2),
        "tokens_per_second": round(eval_tps, 2) if eval_tps else None,
        "command": " ".join(cmd),
    }

Key Takeaways & Comparison Matrix

Here is how the two stack up across core technical categories:

## Key Takeaways & Comparison Matrix

Here is how the two stack up across core technical categories:

| Dimension               | llama.cpp                                                    | Ollama                                                    |
| ----------------------- | ------------------------------------------------------------ | --------------------------------------------------------- |
| **Architecture**        | C/C++ native backend engine                                  | Go wrapper daemon around custom `llama.cpp` builds        |
| **User Experience**     | CLI flags, config scripts, manual GGUF management            | Single binary / desktop app with REST API & model pulls   |
| **Model Discovery**     | Direct GGUF loading from disk or HF cache                    | Centralized model library registry (`ollama pull`)        |
| **Performance Control** | Unrestricted access to context sizing, GPU offloading, threads | Curated defaults optimized for ease of use                |
| **Ideal For**           | Custom pipelines, maximum hardware tuning, offline GGUFs     | Rapid prototyping, web wrappers, hassle-free agent setups |

Conclusion

Bob’s comparison application made it easy to test and quantify the trade-offs between both tools right on my local machine.

I will continue to use ollama, it’s handy, but I’ll use more heavily llama.cpp now that I’ve installed it and begining using it.

>>> Thanks for reading <<<

Criando uma IA particular

Celso Nery — Wed, 12 Aug 2026 16:42:07 +0000

🇺🇸 English version here

Instalando e rodando Ollama Bare Metal

Neste artigo irei usar um servidor com Debian para executar uma IA localmente.

Transformar um computador local em um servidor de IA privada é excelente, especialmente para garantir privacidade total e evitar mensalidades. Com o hardware certo e as ferramentas atuais, o processo fica muito mais simples.

O Hardware Necessário (Requisitos)

O componente mais crítico é a GPU (Placa de Vídeo). Diferente de softwares comuns, IAs rodam em núcleos de processamento paralelo e exigem muita memória de vídeo (VRAM).

Componente	Recomendação Mínima	Recomendação Ideal
GPU	NVIDIA (8GB VRAM)	NVIDIA RTX 3090/4090 (24GB VRAM)
RAM	16GB	32GB ou mais
Armazenamento	SSD NVMe (50GB livres)	SSD NVMe (500GB+ para vários modelos)
OS	Linux ou Windows com WSL2	Linux (Ubuntu/Fedora)

Nota: Placas NVIDIA são o padrão da indústria devido aos núcleos CUDA. Embora seja possível rodar em CPUs ou Macs (Apple Silicon), o desempenho em placas NVIDIA são extremamente superiores.

A Ferramenta de Execução: Ollama

Irei usar o Ollama. Ele é open-source e gerencia o download e a execução dos modelos de forma otimizada.

Instalação: ollama.com.
Rodando um Modelo: No terminal, basta digitar: ollama run llama3
Vantagem: Ele cria uma API local na porta 11434, permitindo que você conecte outras interfaces a ele.

Interface Visual (Open WebUI)

Irei instalar o Open WebUI, uma interface web que se parece com o ChatGPT, mas que roda 100% no seu servidor. Caso não queira usar somente o terminal.

Open WebUI É a interface completa e popular. Vamos roda-la via Docker:

Suporta múltiplos usuários.
Permite upload de documentos para análise (RAG).
Conecta-se diretamente ao Ollama.

Os Modelos (o Cérebro)

Dependendo do seu hardware e objetivo, você usará modelos diferentes:

Llama 3 (Meta): O melhor "pau para toda obra" atualmente.
Mistral / Mixtral: Excelente equilíbrio entre velocidade e inteligência.
DeepSeek Coder: Especializado em programação e escrita de scripts.
Phi-3 (Microsoft): Modelo leve para computadores com menos potência.

Casos de Uso Privados

Com seu servidor rodando, você pode configurar funções avançadas que não seriam seguras em nuvens públicas:

RAG (Geração Aumentada por Recuperação): Alimente a IA com seus PDFs, contratos ou códigos privados. Ela responderá baseada apenas nos seus arquivos, sem que os dados saiam da sua rede.
Automação de Código: Use extensões no VS Code (como o Continue.dev) para usar sua IA local como um "Copilot" gratuito.
Agentes Locais: Criar scripts que organizam seus arquivos ou analisam logs de sistema automaticamente.

Qual a melhor distro para o Ollama.

A escolha da distro impacta diretamente na facilidade de gerenciar os drivers da NVIDIA.

Fiz uma pequena análise baseada em minhas experiências.

A Melhor: Fedora

O Fedora é, atualmente, uma das melhores escolhas para IA local.

Kernel Moderno: Essencial para suporte a novas GPUs e tecnologias de virtualização/containers.
Drivers NVIDIA: O repositório RPM Fusion torna a instalação dos drivers proprietários e do CUDA muito simples e estável.
Podman vs Docker: O Fedora vem com Podman por padrão, mas você pode instalar o Docker Engine oficial sem problemas.
Perfil: Ideal para quem quer as versões mais recentes do Ollama e bibliotecas de IA sem ter que compilar nada manualmente.

A Escolha de Estabilidade: Debian (Bookworm)

Se a prioridade é um servidor que "nunca para", o Debian é o caminho.

Ambiente Limpo: Se você prefere configurar arquivos YAML e usar CLI (Vim/nmcli) como eu, o Debian entrega exatamente o que pede, sem "perfumaria".
NVIDIA: A instalação do nvidia-driver e do nvidia-container-toolkit no Debian Stable é muito documentada e sólida.
Ponto de atenção: O Kernel do Debian Stable pode ficar "atrás" de novas GPUs (como uma série 50 no futuro), mas para uma RTX 30 ou 40, funciona perfeitamente.

A Alternativa Enterprise: openSUSE Tumbleweed

Eu gosto muito do openSUSE, (que uso em minha máquina de trabalho local), ele é uma excelente opção para quem quer um sistema Rolling Release (sempre atualizado) mas com a segurança do Btrfs e Snapper.

Zypper: O gerenciamento de pacotes é robusto.
YaST: Facilita configurações de rede e segurança.
OBS (Open Build Service): Facilita encontrar pacotes específicos de IA que podem não estar nos repositórios oficiais.

Tabela Comparativa para IA

Distro	Gestão de Drivers	Atualização de Kernel	Docker/Container
Fedora	Fácil/Moderno	Rápido	Nativo/Podman
Debian	Sólido/Manual	Lento	Padrão da Indústria
openSUSE	Via Repos	Rolling	Muito bom

Minha Recomendação dependendo do seu perfil:

Vá de Fedora se quiser tudo rodando rápido e com as versões mais novas.
Vá de Debian se quiser configurar uma vez e esquecer que o servidor existe (estilo "set and forget").

A minha escolha: Debian (Stable)

Para o nosso servidor "teste" de IA, o Debian foi a escolha por um motivo simples: compatibilidade de bibliotecas e estabilidade.

Por que o Debian: É a base mais limpa possível. Como eu vim do Slackware e gosto de configurar tudo via CLI (Vim, nmcli), o Debian não instala nada que eu não peço. Ele é perfeito para rodar o Docker Engine de forma nativa e previsível.
A "Vantagem NVIDIA": Quase todos os pacotes do nvidia-container-toolkit e drivers CUDA são testados primeiro e de forma mais exaustiva em bases Debian/Ubuntu. Isso evita que uma atualização de kernel quebre seu servidor de IA no meio de um projeto.

Por que evitei Fedora e Tumbleweed para este Servidor?

Embora sejam excelentes distros, para um servidor dedicado:

Kernel Rolling Release: No Fedora ou openSUSE Tumbleweed, o kernel atualiza com muita frequência. Se eu instalar o driver da NVIDIA e o kernel atualizar, eu posso ter o temido "mismatch" de versão, exigindo que recompile o módulo do driver (DKMS).
Foco em Estabilidade: Um servidor de IA para uma empresa precisa estar disponível sempre que o n8n chamar a API. O Debian Stable garante que o ambiente de execução não mude por anos.

Instalação do Ollame "Bare Metal"

Conforme foi falado no artigo anterior, vamos utilizar o Debian 12. Vou utilizar um servidor modesto com somente 8GB de Ram e sem placa de vídeo devido falta de tempo($$$) rs.

Instalação

O Ollama oferece um script oficial que configura o binário e o serviço do systemctl automaticamente:

curl -fsSL https://ollama.com/install.sh | sh

Configuração de Rede (Importante para utlização externa)

Por padrão, o Ollama nativo só ouve em 127.0.0.1 (localhost). Se quiser integrar com o n8n ou acessar de outra máquina na rede, precisa liberar o acesso externo:

Edite o serviço do Ollama:
sudo systemctl edit ollama.service ou edit diretamente o arquivo /etc/systemd/system/ollama.service
Entre as linhas [Service] e antes de qualquer outra, adicione:

[Service]
Environment="OLLAMA_HOST=0.0.0.0"

Salve (Ctrl+O, Enter) e saia (Ctrl+X).
Recarregue e reinicie:

sudo systemctl daemon-reload
sudo systemctl restart ollama

Otimização para 8GB de RAM (O Swap)

Como tenho somente 8GB de RAM, precisei criar um Swap grande no SSD para a IA não travar o sistema.

Passo a passo para criar 12GB de Swap no SSD:

# Criei um arquivo de swap dentro do /home (onde tenho espaço)
sudo fallocate -l 12G /home/swapfile
sudo chmod 600 /home/swapfile
sudo mkswap /home/swapfile
sudo swapon /home/swapfile

# Tornei permanente (adicionei ao /etc/fstab)
echo '/home/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

Baixando os Modelos Estratégicos

Para o meu cenário (8GB RAM), baixei o phi3.5 executando este comando no terminal:

# O melhor "pau para toda obra" que cabe na RAM (2.3GB)
ollama pull phi3.5

Onde os modelos ficarão guardados? (Importante)

Por padrão, no Linux, ele armazena os modelos em:
/usr/share/ollama/.ollama/models.

O Teste de Fogo (Monitoramento)

Abra uma segunda aba no seu terminal (ou use o tmux/screen) e rode o htop. Depois, na primeira aba, chame o modelo:

ollama run phi3.5

Irá abrir o prompt do ollama.

Agora basta fazer alguma pergunta.
Ex:

>>> Por que o céu é azul?

O que observar:

RAM: Veja se o uso estabiliza em torno de 4GB-5GB.
CPU: Como eu não tenho GPU ainda, o uso de CPU fica em torno de 90-100% enquanto a IA "pensa".

Gerenciamento do models

Para verificar os models que estão instalados

ollama list

Se precisar remover algum ollama rm <model>
ex:

ollama rm phi3.5:latest

Dependendo do seu hardware e objetivo, você escolherá modelos diferentes:

Llama 3 (Meta): O melhor "pau para toda obra" atualmente.
Mistral / Mixtral: Excelente equilíbrio entre velocidade e inteligência.
DeepSeek Coder: Especializado em programação e escrita de scripts.
Phi-3 / 3.5 (Microsoft): Modelos leves para computadores com menos potência.

Alguns models que consigui rodar no meu servidor.

ollama pull deepseek-coder-v2:16b-lite-instruct-q4_K_M
ollama pull llama3.2:3b
ollama pull gemma2:2b
ollama pull phi3.5:latest

Interface Open WebUI

Agora sua IA particular já está funcional. Mas se tiver recurso podemos integrar uma interface web semelhante a do ChatGPT.

Instale o Docker: A forma mais limpa é via Docker.
Suba o Open WebUI:

docker run -d --network host --name open-webui -v open-webui-data:/app/backend/data -e OLLAMA_BASE_URL=http://127.0.0.1:11434 -e WEBUI_SECRET_KEY=$(openssl rand -hex 32) --restart always ghcr.io/open-webui/open-webui:main

Agora abra o seu browser e acesse http://localhost:8080 ou o IP do servidor que estiver criando.

Integração com n8n

O n8n possui suporte nativo ao Ollama. Não precisa configurar requisições HTTP manuais.

No n8n: Procure pelo nó "Ollama Chat Model".
Conexão: Informe a URL do meu servidor (ex: http://localhost:11434 ou o IP do servidor na rede local).
Credenciais: Geralmente não precisa de API Key para uso local.
Uso: Conecta esse nó a um "AI Agent" ou "Chain" dentro do n8n e pronto: seu fluxo de automação agora usa seu Llama 3 ou Mistral local em vez do ChatGPT.

Integração na Aplicação (Laravel, Vue, etc.)

O Ollama expõe uma API REST muito simples. Se você já usou a API da OpenAI, vai notar que é quase idêntica.

Endpoint Principal: POST http://localhost:11434/api/generate
Exemplo de Payload:

{
  "model": "llama3",
  "prompt": "Por que o céu é azul?",
  "stream": false
}

Teste realizado usando `curl`

curl -s -X POST http://10.0.10.240:11434/api/generate -H "Accept: application/json" -H "Content-Type: application/json" -d '{ "model": "phi3.5", "prompt": "Por que o ceu é azul?", "stream": false }' | jq

GitHub Copilot for JetBrains: use Ollama without confusing local inference with local memory

Ahab — Wed, 12 Aug 2026 01:20:50 +0000

GitHub Copilot for JetBrains: use Ollama without confusing local inference with local memory

Quick answer

GitHub Copilot for JetBrains now supports Ollama as a local BYOK provider and can retain Copilot Memory across agent chat sessions. Treat them as two separate controls.

Ollama can keep model inference on your machine when the selected endpoint is loopback and the selected model is local. Copilot Memory stores repository facts and user preferences under GitHub's memory controls. Enabling a local model does not prove that memory, account, session sync, telemetry, MCP calls, or cloud-agent work also stays local.

Roll out in two lanes: first prove the local inference path with Memory disabled, then decide whether to enable Memory and audit what it stores.

Who this is for

This guide is for IntelliJ IDEA, Android Studio, PyCharm, WebStorm, and other JetBrains users who want Copilot's agent workflow with a locally served model, or who need to review the privacy boundary before enabling persistent context.

It is also for organization administrators. GitHub says local BYOK in IDEs can be disabled by organization or enterprise policy, while Copilot Memory has its own policy and user toggle. The existing Copilot managed-settings guide explains why client access and runtime controls must be tested separately.

What changed on August 11

GitHub's release adds four notable JetBrains capabilities: enterprise-managed settings, cross-session Copilot Memory, Ollama as a BYOK provider, and expanded Codex workflows. The release says Ollama provider configuration and model selection are available throughout the JetBrains experience.

GitHub's BYOK documentation distinguishes local BYOK from enterprise BYOK. Local keys and provider configuration are handled client-side, and a local model can remove the dependency on GitHub's Copilot model API. Enterprise BYOK is different: it is handled server-side and still requires a Copilot license and internet access.

Memory is a separate public-preview system. It can store repository-level facts such as build commands and architecture decisions, plus user-level preferences. Repository facts are citation-backed and revalidated against the current branch. Unused entries expire after 28 days, and users or administrators can review and delete them.

Boundary matrix

Surface	What the official material confirms	What you must verify
Ollama model inference	JetBrains supports Ollama as a local BYOK provider	Endpoint is `127.0.0.1`, local model is selected, and requests reach the local process
Copilot Memory	Repository facts and user preferences persist under GitHub controls	Policy, billing owner, stored entries, citations, deletion, and whether you need it
JetBrains plugin	Provider/model selection and agent chat are integrated	Exact plugin and IDE builds, selected provider after restart, and fallback behavior
Other agent paths	Release also covers MCP, Codex, terminals, and cloud agents	Network destinations, permissions, session sync, and every non-model tool path

The safe claim is narrow: the model request was served by local Ollama. Do not upgrade that claim to “Copilot is fully offline” without separate network and feature evidence.

A six-gate rollout

1. Record the exact client and policy state

Update to the latest compatible GitHub Copilot plugin, then record the plugin version, JetBrains product/build, account, active billing entity, organization policies, and current Memory state. Do not rely on “latest” as durable evidence.

2. Keep Ollama on loopback

Ollama binds to 127.0.0.1:11434 by default. Keep that default for a single-machine IDE setup. Do not set OLLAMA_HOST=0.0.0.0:11434, add a tunnel, or expose the port merely to make discovery easier.

If you want an explicitly local-only Ollama service, disable its cloud features:

{
  "disable_ollama_cloud": true
}

Then restart Ollama and confirm its logs report that cloud support is disabled. This controls Ollama; it does not disable GitHub-side Copilot features.

3. Prove the provider, not just the answer

In JetBrains, configure Ollama, choose one already-downloaded local model, start a fresh chat, and submit a harmless prompt with a unique marker. Capture:

the selected provider and model in the UI;
the local Ollama request log or process activity for that marker;
the absence of an automatic fallback to a GitHub-hosted model;
the result after restarting both the IDE and Ollama.

If the UI says Ollama but no local request appears, stop. A plausible answer is not routing evidence.

4. Test the failure path

Stop Ollama and repeat the harmless prompt. The acceptable result is a clear local-provider failure or an explicit, user-approved provider switch. Silent fallback fails the gate. Restart Ollama and confirm the same model works again.

For larger open models, check hardware before downloading. The local-model hardware checklist shows why “open” and “laptop-ready” are different claims.

5. Audit Memory separately

Keep Memory disabled during the routing test. If you later enable it, use a disposable repository with a harmless convention such as npm run verify:fixture. Interact as a user with write access, then review repository facts under the repository's Copilot Memory settings and user preferences in personal Copilot settings.

Record any fact, its citation, owner, and deletion result. Do not seed passwords, customer data, private URLs, or production architecture secrets. Memory creation is not guaranteed for every sentence; if no entry appears, record “not observed” instead of claiming it was stored.

6. Gate production use

Approve the rollout only if local routing, stopped-server behavior, restart persistence, policy enforcement, memory inspection, and deletion all match the intended boundary. Keep a one-step rollback: switch provider, disable Memory, or disable the local-BYOK policy.

Decision tree

Need only local model inference? Keep Memory off and validate Ollama routing first.
Need persistent repository context? Enable Memory only after ownership, review, retention, and deletion are acceptable.
Need a fully offline or air-gapped workflow? Disable unrelated cloud features and prove network behavior; the JetBrains release alone is not evidence.
Organization policy hides Ollama? Ask the administrator to review local BYOK policy; do not work around it with an exposed endpoint.
Local model lacks tool quality or context? Route only bounded chat tasks to it and keep agentic writes on a separately evaluated model.

Common mistakes

Calling the whole Copilot workflow local because the LLM runs in Ollama.
Enabling Memory and Ollama together, then being unable to attribute behavior.
Exposing port 11434 to the LAN or internet without a real access-control design.
Confusing local BYOK with enterprise BYOK, which uses a server-side path.
Trusting the model name shown in prose instead of observing the local request.
Testing persistence with secrets or sensitive repository facts.
Assuming a stored fact is permanent; unused Memory entries expire and facts are revalidated.

For other local-control boundaries, see the Qwen trusted-LAN checklist. For tools that can act on external data, use the connector security checklist.

Copyable acceptance record

date / owner:
jetbrains_product_build:
copilot_plugin_version:
account / billing_entity:
local_byok_policy: enabled | disabled
memory_policy / user_toggle:
ollama_bind: 127.0.0.1:11434 | other
ollama_cloud_disabled: yes | no
selected_provider / model:
local_request_observed: pass | fail
stopped_server_fails_closed: pass | fail
restart_preserves_selection: pass | fail
memory_fact_observed / citation:
memory_delete_verified: pass | fail | not tested
non_model_network_paths_reviewed:
rollback_action / owner / evidence:

FAQ

Does Ollama make GitHub Copilot for JetBrains fully offline?

It can localize the model-provider path. Other Copilot capabilities have separate account, memory, session, tool, and network behavior. Verify those independently before making an offline claim.

Is Copilot Memory stored inside Ollama?

No official source says that. GitHub documents Memory as repository facts and user preferences managed through GitHub Copilot settings. Treat it as a separate service and control plane.

Should I expose Ollama so JetBrains can find it?

Not for a same-machine setup. Ollama's documented default is loopback. Keep it there unless you have a deliberate authenticated network design.

Sources

GitHub Copilot for JetBrains release: https://github.blog/changelog/2026-08-11-copilot-memory-and-ollama-in-github-copilot-for-jetbrains
GitHub Copilot BYOK concepts: https://docs.github.com/en/copilot/concepts/models/bring-your-own-key
About GitHub Copilot Memory: https://docs.github.com/en/copilot/concepts/agents/copilot-memory
Manage personal Copilot Memory: https://docs.github.com/en/copilot/how-tos/use-copilot-agents/copilot-memory/manage-for-yourself
Manage organization and enterprise Memory: https://docs.github.com/en/copilot/how-tos/use-copilot-agents/copilot-memory/manage-as-administrator
Ollama network and local-only guidance: https://docs.ollama.com/faq
JetBrains local-LLM demand signal: https://github.com/microsoft/copilot-intellij-feedback/issues/593

I Asked Muse Glimmer to Pair Program With Me in OpenCode

Ahmed Nafies — Tue, 11 Aug 2026 22:04:13 +0000

I wanted to use Meta's Muse Glimmer as a local agentic engineering assistant inside OpenCode.

Not as a chatbot in a browser. I wanted an agent that could inspect a repository, understand a task, edit files, run tests, and explain what it did.

The command was refreshingly short:

ollama launch opencode --model muse-glimmer

The model, however, is a 30-billion-parameter coworker who arrives carrying an approximately 18 GB suitcase and expects a 128K-context office.

The onboarding meeting

First, I checked Ollama:

ollama --version

Then I started the local service if needed:

ollama serve

I verified that the local API was alive:

curl http://localhost:11434/api/version

Then I launched OpenCode with Muse Glimmer:

ollama launch opencode --model muse-glimmer

The first run downloads the model. This is a good opportunity to make coffee, reconsider your disk space, and remember that “local AI” does not mean “tiny AI.”

To inspect installed models:

ollama ls

On Apple Silicon, Ollama also lists an MLX variant:

ollama launch opencode --model muse-glimmer:30b-mlx

The standard model is listed at around 18 GB. That is the model's luggage, not the entire hotel bill. Leave memory for the context, OpenCode, the operating system, and the repository you are asking the agent to understand.

Why OpenCode changes the experiment

The interesting part is not just running Muse Glimmer locally. It is putting the model inside an engineering workflow.

With OpenCode, I can ask the agent to:

Inspect the repository.
Find the relevant files.
Explain the current behavior.
Propose a plan.
Make a focused change.
Run the project's tests.
Read failures.
Revise the patch.
Show the final diff and verification result.

That is the difference between asking a model for code and asking an agent to help complete an engineering task. The model can still be confidently wrong. The workflow gives me more chances to catch it before the wrongness becomes a pull request.

The first repository tour

I opened the repository in OpenCode and started with a read-only request:

You are working in a local software repository. Inspect the project structure and identify:

- The application entry point
- The test command
- The main source directories
- Configuration files that affect development
- Any obvious contribution or safety constraints

Do not modify files. Report the evidence and ask before making changes.

This is a better first task than “fix everything.” It tests whether Muse Glimmer can build context without immediately turning the repository into modern art.

The key signal is whether the response points to actual files and commands. A useful agent should report evidence, not invent a project architecture because the folder name sounded familiar.

The first engineering task

After the repository tour, I gave OpenCode a small, testable issue:

Find the bug in the relevant average function. Explain the edge case, propose the smallest fix, and add a regression test. Do not change unrelated files.

The current implementation is: average(values) = sum(values) / len(values) + 1

The extra + 1 is suspicious. It is the kind of bug that enters production wearing sunglasses and saying, “Nobody will notice.”

A good OpenCode session should locate the real implementation, find the project's existing test style, make the smallest patch, and run the relevant test. A locally generated patch is still a patch. It needs review, tests, and version control like any other change.

Planning before editing

I asked OpenCode to separate investigation from implementation:

Before editing, write a short implementation plan with:

- Files you expect to inspect
- The behavior you expect to change
- The test you will add or update
- The command you will run to verify the change

Wait for approval before modifying files.

This creates a useful checkpoint. It also prevents the classic coding-agent experience where you ask for one small fix and receive a surprise framework migration, a new abstraction layer, and a README poem.

OpenCode is the interface where this loop becomes practical: inspect the plan, approve the direction, watch the diff, and ask for verification in the same engineering session.

The test-and-repair loop

The useful workflow looks like this:

Inspect -> plan -> approve -> edit -> test -> inspect failure -> revise -> test again

I tested failure recovery with an intentionally broken test result:

The test runner returned:

ERROR: assertion failed in tests/test_average.py
Expected: 2.0
Received: 3.0

Explain the failure, identify the likely cause, and propose the smallest correction. Do not edit files yet.

Then I tested a repository constraint:

The repository check returned:

ERROR: the generated lockfile differs from the committed lockfile.

Do not regenerate dependencies. Explain what should be reviewed first.

The goal is not to make the agent retry everything until the terminal turns green. The goal is to see whether it can distinguish a fixable implementation error from a situation requiring human review.

“The model usually behaves” is not an engineering control. It is something you say immediately before opening an incident ticket.

Screenshot-driven debugging

Muse Glimmer supports image input, which is useful for UI bugs, screenshots, diagrams, and terminal output. From the local OpenCode workflow, I can give the agent a screenshot and ask for evidence-first analysis:

Inspect this screenshot as a debugging artifact. List only visible symptoms, separate observations from hypotheses, and suggest which source files or browser checks should be inspected next.

“The button is clipped” is an observation. “The CSS grid is broken because of a missing min-width” is a hypothesis. That distinction saves time and prevents a screenshot from becoming a license to rewrite the frontend.

Conclusion

Using Muse Glimmer with OpenCode through Ollama is straightforward to start:

ollama launch opencode --model muse-glimmer

The interesting work begins after OpenCode starts. Ask the agent to inspect, plan, edit, test, diagnose, and explain.

Muse Glimmer can be the clever local coworker. Ollama handles the model runtime. OpenCode handles the engineering conversation. You remain the person who decides whether the patch deserves to live.

Resources:

Does Ollama Include That New llama.cpp Feature?

Patrick Hughes — Tue, 11 Aug 2026 16:05:11 +0000

Does Ollama Include That New llama.cpp Feature?

A new llama.cpp release can look like an instant upgrade. That release note does not prove that my local application can use it.

Short answer: I trace the request from my application to the engine that runs the model. I pin that engine version, test the same workload, and keep the old recommendation until the new code reaches the active path.

Why is the release note not enough?

My sizing desk sends requests to the Ollama HTTP API. It does not call a standalone llama.cpp binary. Ollama includes its own llama.cpp code, so a new upstream release reaches my path only after Ollama includes the needed change.

The b10258 release is narrower than a new feature. It moves n_vocab from shared sampler data into the penalty sampler. That is internal sampler wiring, not proof that Ollama exposes a new capability. The upstream code can exist while my installed runtime still uses an older code base. A benchmark against the wrong binary answers the wrong question.

I treat runtime support as a separate gate from model fit. A model can fit in memory and still fail before its first useful token. I cover that first boundary in VRAM Fit Is Not Runtime Support.

How do I map the active local inference path?

I start at the application call. I write down the endpoint, process, and executable that receive the request. Then I find which component owns model loading and token generation.

For my test on August 6, 2026, the path ended at Ollama 0.31.2. No standalone llama-cli or llama-bench binary was on the host path. Testing a fresh llama.cpp build would have measured a different system from the sizing desk.

This check is simple, but it blocks a common error. The name in a release note is not the same thing as the code in production. The active executable and its version are the facts that matter.

What did my RTX 5090 smoke test prove?

I used the same llama3.1:8b request shape as an earlier baseline. The test fixed num_ctx at 2,048, num_predict at 128, and temperature at zero. Ollama reported full GPU placement during the run.

Three warm runs produced 173.13, 174.91, and 173.34 tokens per second. The spread was under 1%. The GPU used 7,262 MiB of 32,607 MiB, and Ollama showed a 5.0 GB resident model.

Those results proved that the active sizing path gave repeatable numbers in that session. They did not prove that llama.cpp b10258 was active. The installed Ollama build still owned the path, so I made no feature claim and changed no sizing recommendation.

Why did I not compare the new result with the old baseline?

The June run recorded 216.89 tokens per second on Ollama 0.22.1. The August warm runs were near 174 tokens per second. That looks like a drop of about 20%, but the host conditions did not match.

The August test ran while two package installs and the agent fleet used the same host. The first model load took 135 seconds on the busy disk, compared with 3.2 seconds in June. That test cannot isolate a runtime change.

I recorded the difference as an open question. A quiet-host rerun can test it later. This is why I bind each result to the driver, runtime, workload, and time in How I Keep LLM Results Valid After a Driver Update.

What evidence should change a recommendation?

I need four receipts. First, the target feature must exist upstream. Second, the active runtime must include it. Third, the model and setting must expose it. Fourth, the same workload must show a useful result without a quality loss.

A version string alone clears only one part of that chain. A fast test against a separate binary also falls short. I change the recommendation only when the application path and the measured path are the same.

This rule saves time. It keeps release news in a watch list until my software can use it. It also keeps a new speed result from reaching the sizing desk before the result has the right runtime and workload behind it.

Accompanying prompt

What the prompt does: This prompt checks whether a new local inference feature has reached the runtime that an application uses.

Copy/paste this prompt:

Role:
You are a local inference compatibility reviewer.

Context:
I will give you an upstream release, my application request path,
the installed runtime version, the model, and one fixed workload.

Task:
1. Map the request from the application to the inference engine.
2. Check whether the active engine includes the target feature.
3. Run the fixed workload through the application path.
4. Separate measured facts from open questions.

Output:
- A support result for each layer in the request path.
- The exact runtime and model versions.
- The test result and one recommendation.
- A list of claims that the evidence does not support.

Constraints:
- Do not test a different binary from the application path.
- Do not infer feature support from an upstream release alone.
- Keep the old recommendation when the active runtime lacks the feature.

Copy the block above.

Get measured local AI notes by email at The 5090 Reports.

Originally published on bmdpat.com. I run a one-person AI agent company and write about what actually works.

Want these in your inbox? Subscribe to the newsletter - no spam, unsubscribe anytime.

Nine ways to talk to a local model

the kilted dev — Tue, 11 Aug 2026 15:56:51 +0000

A nine-tool survey of local-model interfaces on one GPU: what worked, what silently failed, and
why what sits between you and the model matters more than which model you picked.

Nine pieces of software for talking to a local model went through this machine over about two
weeks. Same GPU, largely the same handful of models, one afternoon each.

The spread in outcomes was enormous, and almost none of it was about the model. The same weights
that scored full marks through one harness produced unparseable garbage through another, and hung
indefinitely through a third. What sits between you and the model turns out to matter more than
which model you picked, which is a boring conclusion until you notice it also means most model
comparisons are measuring something else.

Here is what each one was actually good at.

Tool	Good at	The number that mattered
Ollama	the default, broadly reliable	32.9 t/s gen on a 30B MoE
llama.cpp	fastest, once tuned	27.1 t/s vs Ollama's 23.4
LM Studio	document extraction to file	10 rows, exact match, no invented values
little-coder	small-model coding harness	8/8 in 2.1 min, then a version bump broke it
Unsloth Studio	fastest single-file result	unusable past a 4096-token ceiling
Goose	agent framework, right idea	30% GPU vs 98%, one setting
Claude Code as client	n/a	all three connection methods fail
Odysseus	the most polished interface	0/8, only one of three files actually saved
Continue.dev	autocomplete, nothing else does it	355MB free at 32K, no room for anything else

Ollama, the one everything else is measured against

The default answer, and it stayed the default. It runs a 30B mixture-of-experts model at 32.9
tokens/sec generation and 443 tokens/sec prefill on a 69/31 CPU/GPU split, which is the single
most useful capability on this box.

Two things about it are worth knowing before you trust it. Its desktop application stores a
context-length setting in a local database that overrides the environment variable you set,
which produced a memorable afternoon of a model ignoring configuration that was demonstrably
correct. And model aliases pin their own context, which is how a working alias silently acquired
a context it had never been tested at and started failing weeks later.

Neither is a defect exactly. Both are the kind of thing you only learn by being caught by them.

More: ollama.com

llama.cpp, faster if you're willing to tune it

For one specific job, a 36B mixture-of-experts model at 32K context, llama.cpp beats Ollama by
about 16% on generation and 34% on prefill. 27.1 tokens/sec against 23.4.

That margin isn't free. It came from three separate tuning discoveries: disabling memory-mapped
file loading, which llama.cpp itself warns is slow when combined with CPU tensor overrides and
which alone accounted for most of the gap; setting thread count to the CPU's fourteen physical
cores rather than the eight the script had; and tuning how many expert layers stay on the GPU.

Untuned, the same path runs at 13 tokens/sec, less than half. The default configuration of the
faster runtime is slower than the alternative, which is worth remembering whenever a benchmark
reports that one tool beats another.

llama.cpp's server also has the most complete tool-calling story of anything tested here. Given a
toy arithmetic tool, it emitted five valid parseable calls out of five. Given three near-identical
tools and six questions, it selected correctly six times out of six. That second number is the one
that matters, because choosing between similar tools is where tool-calling usually breaks.

It also has server-side built-in tools, off by default, with the server itself warning against
exposing them to untrusted environments (the available set includes shell execution). Only the
read-only subset was enabled here. Notably, the server does not auto-execute anything: it returns
the tool call and the client drives the loop. That's the right design, and it means the safety
boundary sits where you can see it.

More: github.com/ggml-org/llama.cpp

LM Studio, the graphical one, and the absolute-path rule

Good at exactly one thing this project needed: read some documents, extract structured data, write
the result to a file. Given real utility bills it produced ten rows across two months with every
usage quantity and cost matching the source exactly. No invented values.

Getting there took three attempts. The first two failed with filesystem permission errors even
after access was explicitly approved through its own dialog. The difference was the prompt: "the
files in your workspace" failed, and naming the absolute directory worked.

Whether the model requested bad relative paths or the permission scope failed to resolve them was
never determined. The rule is empirical and it's reliable: give it absolute paths.

More: lmstudio.ai

little-coder, the one that proved the whole thing was possible

A thin harness tuned for small models, giving them real file-writing tools rather than asking them
to print code into a chat window. It's what scored 8 out of 8 on this project's benchmark in 2.1
minutes, including the follow-up-fix-without-regression test that had defeated everything before
it.

Then a version bump broke it. With its default model it began emitting raw XML that nothing
parsed, exiting successfully, and writing no files, the worst failure shape available, because an
exit code of zero and no error output reads as success to anything automated.

The instructive part is the diagnosis, which was wrong the first time. The failure was recorded as
being caused by the version change breaking model-id resolution, because a warning about the model
id appeared next to the failure. A later controlled test held the model constant and varied only
id registration: the warning is benign, and the actual fault is the specific model. The same
harness works with a different one.

Adjacency isn't causation, and a warning that appears next to a failure is still just a warning
that appears next to a failure.

More: github.com/itayinbarr/little-coder

Unsloth Studio, fastest and unusable

The fastest single-file result of anything tested: ten seconds.

Its context window control is broken on Windows. The effective ceiling is 4096 tokens regardless
of what the interface is set to, which makes multi-file work impossible and makes any measurement
taken through it incomparable to anything else. A tool that silently runs at a fraction of the
context you configured is worse than one that refuses, because the number it shows you is a number
you will use.

It also gets flagged as malware by the antivirus on this machine, which is a false positive, and
which is its own small saga.

More: unsloth.ai

Goose, right idea, wrong backend

Goose works here, with one hard rule: use its Ollama provider, never its built-in llama.cpp.

The built-in path manages roughly 30% GPU and 25% CPU utilisation on Windows, a broken offload
that leaves the hardware idle while the model crawls. Through the Ollama provider the same machine
runs at 98% GPU.

Same tool, same model, same box. One configuration choice, and the difference is the entire
value of having a GPU.

More: github.com/block/goose

Claude Code itself, three methods, one root cause

The obvious idea, and the one people ask about most: point the agentic coding tool you already use
at a local model, and stop paying for tokens.

Three connection methods were tested. Setting the API base URL directly, which produces no output
without also setting an auth token to a dummy value. A proxy shim, which worked and ran at roughly
two and a half minutes per response on CPU offload. And the runtime's own native launch command,
which produced a model hallucinating unrelated tasks and emitting raw tool-call syntax as text.

All three fail, and they fail for the same reason: local models do not parse Claude Code's
system prompt format. That's not a configuration problem, and no amount of trying a fourth
connection method changes it. The finding was worth writing down precisely so nobody here burns
another afternoon on connection method five.

More: claude.com/product/claude-code

Odysseus, the most polished tool tested

The most polished thing tested, and the worst result.

It scored 0 out of 8 on the same benchmark. The run climbed from 0.74 to 10.19 tokens/sec, then
stalled dead at "4% complete" with a byte-identical transcript across a five-minute recheck.
Underneath, the backend had dropped the generation stream and was returning 404 to the frontend's
status polls, which the frontend kept making, forever, with no error surfaced, no retry, and no
timeout. A silent, indefinite hang.

Two things compounded it. Its interface exposes no control for disabling thinking mode, which this
project has needed since day one, so the model visibly re-planned and restarted its own draft
mid-stream. And checking its own document panel rather than trusting the chat transcript revealed
that only one of three files had actually been saved. The transcript showed content that was
never written anywhere.

A chat transcript is a record of what a model said, not what a system did, and those two things
diverge silently.

It's also structurally unable to do what this project needs: its tools are a fixed set of built-in
capabilities toggled per message, not a registry you can register an arbitrary function with.
There was no way to run the tool-calling probe at all. Not a bug, a different product.

More: github.com/odysseus-dev/odysseus

Continue.dev, the one that does something nothing else does

An editor extension, evaluated for autocomplete rather than chat, because autocomplete is the one
capability nothing else here provides.

A 4B model completes a fill-in-the-middle hole in 209 to 252 milliseconds at 127 tokens/sec,
comfortably inside the budget where ghost text feels helpful rather than laggy. A 9B does it in
around 328 milliseconds. A 14B coder model, the one nominally built for this, takes 2.5 seconds,
because it spills to CPU, and rambles past the hole into inventing further functions.

The measurement that mattered was not speed, though. With the autocomplete model warm at 32K
context, free video memory drops to about 355MB for the 9B and 1GB for the 4B. There is no room
for anything else. You can have interactive coding with autocomplete, or a background delegation
run. Not both.

More: continue.dev

The comparison we never ran

All of this was supposed to be settled by a formal benchmark. It was written into the handover
document that started this project, as open item one: the word search task run across five
entrants, Goose's built-in backend, Goose via Ollama, llama.cpp, Unsloth Studio and LM Studio,
against one baseline model, with context standardised at 89K on every runtime so the results would
be comparable.

The plan even predicted its own casualty:

Unsloth will fail this, document it.

It was closed without ever being run.

Not from lack of time. By the time it came up for scheduling, its actual question, which runtime
for which job, had already been answered by using them. Ollama for the coding harness. llama.cpp
for the 36B at 32K. LM Studio for graphical document extraction. Goose only via Ollama. Unsloth
ruled out on the context ceiling. The proxy shim never a peer to the others, because it's a
different category of thing entirely.

A generic five-way bake-off would have re-measured memory-spill behaviour already understood, at a
context size chosen to be fair rather than useful, and changed not one of those decisions. The
tools had already differentiated themselves on the only axis that mattered: what happened when
real work went through them.

One footnote, found while checking the above against the original plan rather than against our own
notes. The record written when the benchmark was closed describes it as having six entrants, and
lists a different set, merging Goose's two configurations into one, adding a runtime the plan had
not included, and adding a proxy shim the plan mentions only in its list of things already
rejected. The reasoning for closing it was sound and every per-use-case answer holds. The entrant
list was simply restated from memory rather than reread, in a note written the same week the plan
was still open.

A benchmark nobody ran is a low-stakes thing to miscount. It's the same move that makes a
miscounted one dangerous.

The rule that came out of it is narrow. Benchmark a use case, never a category. If a new runtime
appears, or a genuinely new job appears, measure that, and if the answer is already visible in
work you have done, the benchmark is a formality you are performing for the shape of it.

Nine tools. The differences that mattered were reliability, offload behaviour, honesty about what
was actually written to disk, and whether the thing exposed the one control the model needed. None
of that shows up in a table of tokens per second.

Originally published at thekilted.dev/nine-ways-to-talk-to-a-local-model.

I Benchmarked Local LLMs on the Laptop I Already Have

Konstantinas Mamonas — Tue, 11 Aug 2026 14:49:11 +0000

My last post was about companies discovering their AI bills and installing caps. A reasonable reaction to that post is the one the local-LLM crowd has been giving all year: stop renting tokens, the open models are good now, just run them on your own machine.

The benchmarks backing that advice have a hardware problem. The writeups that measure anything run on Mac Studios or 128GB M5 Max laptops, and the headline stories are worse. DeepSeek V4 Flash "runs on a MacBook" if the MacBook is a 128GB build that costs about $5,000. Kimi K3 has open weights that need eight H100s just to load. One guide this summer was literally titled "Open Weights You Can't Run."

My laptop is a 2021 M1 Pro with 16GB of RAM. It's almost five years old, and I suspect it's a lot closer to the median developer machine than anything in those benchmarks. So I benchmarked what it can actually do.

The Setup

I tested five models, all pulled through Ollama, all picked to fit in 16GB:

qwen3.5:4b and qwen3.5:9b, Alibaba's current small models, both of which think before answering by default
gemma4:e4b-it-qat and gemma4:12b-it-qat, Google's current small family, in the quantization-aware builds
qwen2.5:7b, a late-2024 model, as the baseline for whether a year of progress shows up at this size

The current models everyone actually talks about didn't make the list, and that's a finding on its own. glm-4.7-flash and qwen3-coder both sound small and both are 19GB downloads. Qwen3.8-27B, announced the week I ran this, needs about 17GB and its weights weren't even downloadable yet, they're promised for the week this post goes out. On 16GB of RAM the list of models you can actually run is much shorter than online discussions suggest.

The tasks are 21 problems of the kind that come up in a data engineer's week, in six categories: writing DuckDB SQL against a seeded database, fixing buggy Python files with a failing test suite, extracting JSON from messy text like Airflow logs and billing emails, predicting what tricky Python snippets print, answering quick questions I'd normally google, and finding facts planted in a 7,400-token runbook. Everything is graded by code, not by me reading the answers: generated SQL gets executed and compared row by row against a reference query, bug fixes have to pass pytest, extractions are checked field by field. Each model also has a temperature setting that controls how much randomness goes into its answers, I set it to zero and ran everything twice, so each model gets 42 runs.

What Passed

model	size	pass rate	total wall time
gemma4:12b-it-qat	7.2GB	38/42 (90%)	123 min
gemma4:e4b-it-qat	6.1GB	36/42 (86%)	18 min
qwen3.5:9b	6.6GB	34/42 (81%)	114 min
qwen3.5:4b	3.4GB	26/42 (62%)	125 min
qwen2.5:7b	4.7GB	22/42 (52%)	3 min

Two things surprised me here. The first is how much these models get right. Gemma 4's 12B answered 90% of the suite correctly, including 6/6 bug fixes that had to pass an actual test suite and 8/10 SQL questions graded by execution. The 2024 baseline managed 52% on the same tasks and went 0 for 6 on predicting what Python code prints, so the models did improve over the year, at least between those two.

The second surprise is the wall-time column. The old qwen2.5 ripped through the whole suite in 3 minutes. The models that beat it needed one to two hours for the same 42 runs, and almost all of that gap is reasoning tokens.

Accuracy against time for the whole suite. Only gemma4:e4b is both fast and accurate.

The Reasoning Tax

The current generation thinks before answering, and on a laptop that thinking is where most of the time goes.

My first run capped generation at 3,072 tokens per task, which is not much considering how many tokens something like Claude uses on a single request. The qwen3.5 models hit that cap in 48 out of 84 runs, spending the entire budget on reasoning and never producing an answer. Both scored around 40-57%. I doubled the budget to 8,192 and reran: both gained exactly ten passes. The ability was there, the models just needed thousands of extra tokens to reach the answer. Some runs blew through the doubled budget too. qwen3.5:9b spent eight minutes and 34,000 characters of thinking on one SQL question and still ran out of room.

The gemma models think too, usually less, but not always. On one run, gemma4:12b spent 14 minutes reasoning about which numeric chmod mode corresponds to rwxr-xr--, produced 19,000 characters of thinking, and answered 750. The answer is 754.

On the API this same behavior shows up as a line item, the reasoning tokens get billed like any others. Locally it shows up as time spent waiting.

Would You Stop Googling Things?

The speed question I actually cared about: for a quick factual lookup, does asking a local model beat a search engine?

For the non-thinking models the answer is yes, at least on latency. With the model already loaded, "what port does Postgres use" came back in 0.6 to 1.1 seconds from gemma4:e4b and qwen2.5. That's faster than I can type the query into a browser. The reasoning models don't compete here: qwen3.5:9b averaged 17 seconds per quick question and the 4B averaged 38, which is slower than any search engine.

Median time to a quick answer with the model already loaded. The slow ones spend that time reasoning.

The problem is what comes back at that speed. Three models gave three different wrong answers to the chmod question: 752, 744, 750. Two models suggested git reset HEAD~ or HEAD^ for "undo the last commit but keep changes staged", which does the opposite, plain git reset unstages. The wrong answers came back just as fast and just as confident as the right ones. A search result page gives you Stack Overflow votes and three competing answers to cross-check. A local model gives you one answer and no way to tell whether it's right.

The Long Document Test

Pasting a 7,400-token runbook and asking a question is where laptop hardware shows its actual limit. All five models found the planted facts, even the one buried near the very end, so ability isn't the problem. The problem is the wait: 32 seconds for the first question on gemma4:e4b, 84 on qwen3.5:4b, 148 on gemma4:12b. My M1 Pro processes an incoming prompt at 130 to 330 tokens per second, and that's why nobody runs coding agents against laptop models. A Claude Code-style harness sends tens of thousands of tokens of system prompt and context with every request. At laptop speeds that means waiting minutes before the answer even starts, on every request the agent makes.

One pleasant surprise: follow-up questions about the same document came back fast, 17 to 40 seconds, because Ollama keeps the already-processed document in memory. Chatting with one long document works locally, but anything that sends a fresh context on every request does not.

The Money

Here's the table I actually built this benchmark for. The whole suite, 210 runs, used 162,000 input tokens and 412,000 output tokens. Priced at current API rates, all of it together costs:

provider	whole benchmark
Claude Opus 5 ($5/$25 per 1M)	$11.11
GPT-5.6 Terra ($2/$12)	$5.27
DeepSeek V4 Flash ($0.14/$0.28)	$0.14
Muse contributor tier ($0.10/$0.20)	$0.10

The entire benchmark, every model, every rerun, is fourteen cents at DeepSeek's prices, and Meta will go lower if you let them train on your prompts. Against that, my laptop spent 6.4 hours producing the local answers. The electricity rounds to a few cents, so local is technically cheaper than Opus. That stops being true the moment my time is worth anything, and at DeepSeek's prices there is nothing to save at all. When Uber caps engineers at $1,500 a month, you cannot offload any meaningful part of that onto a machine that needs two hours to do fourteen cents of work.

So the real reason to run local models on a normal laptop isn't money. It's the situations where the API isn't an option: data that can't leave the machine, working offline, or the API being down. And I found one more reason I didn't expect. gemma4:e4b got every extraction task in the suite right, and running it costs nothing no matter how many times I do it. That feels different from using an API where every request adds to the bill.

What I'd Actually Keep

If one model earns a spot on this machine, it's gemma4:e4b-it-qat. It passed 86% of the suite at 30 tokens per second, answers quick questions in under a second, and finished the whole benchmark in 18 minutes while the 12B needed two hours for four percentage points more. That's enough to keep it as an offline fallback and a free JSON-extraction tool.

The rest can go. The 12B is too slow to wait for on this hardware, the reasoning models take longer than the answers are worth, and the 2024 baseline was a nice reminder of how much these small models improved in a year, 52% to 90% at roughly the same size.

So back to the advice from the top of the post. Running models locally works better than I expected on the laptop I already have. But it won't shrink an AI bill, and the bill was the reason to try this in the first place.

Agentic AI - Let Ollama Write and Run Your Python, A Local Code Generator

Austin Cunningham — Mon, 10 Aug 2026 13:42:23 +0000

Up until this point I have mostly used Ollama as an API that returned answers. I wanted a basic understanding of agentic AI. You give it a goal of creating code, point it at a folder under ~/repo, and let a model on my machine write and run Python until the job looks done.

No cloud API keys, no burning tokens. Just Ollama, a few Python files, and tool calling.

What we are building

You pass a task and a work directory. The agent talks to Ollama. When the model wants an action, it returns a tool call. Your code runs the matching Python function and sends the result back. Repeat until the model answers in plain text, or you hit a turn limit.

The model never opens files itself. It only requests actions. Your process executes them.

Prerequisites

Python 3.10+
Ollama installed and running
A tool-capable model (I use llama3.1:8b)
The ollama Python package

ollama serve
ollama pull llama3.1:8b
pip install ollama

Project layout

Four files keep the responsibilities clear:

File	Role
`main.py`	CLI, work-dir prompt / `--dir`, starts the loop
`agent.py`	Chat loop, tool execution, fake tool-call handling
`tools.py`	Read / write / run, plus a soft `~/repo` path limit
`config.py`	Model name, iteration cap, system prompt

Config first (`config.py`)

In config.py I keep the knobs in one place. Which model, how many turns, and a system prompt that pushes the model toward real tool calls (more on that later).

MODEL_NAME = "llama3.1:8b"
MAX_ITERATIONS = 15

SYSTEM_PROMPT = (
    "You are an autonomous local AI assistant with file tools. "
    "When creating or editing code, you MUST call write_local_file with the COMPLETE source in content. "
    "Never write empty files. Never create a placeholder file first. "
    "Always use relative filenames (for example 'calculator.py'). "
    "Do not invent absolute paths like /repo/.... "
    "Never describe or simulate tool calls in text, XML, JSON, or markdown — only use real tool calls. "
    "Keep calling tools until the task is actually done; then reply with a short plain-text summary."
)

NOTE: That prompt is not decoration. Smaller local models love to describe a tool call in chat instead of returning one in the API field. The system prompt is the first line of defence against this behaviour.

The tools (`tools.py`)

These are normal Python functions. Docstrings and type hints matter: the Ollama client turns them into tool schemas automatically.

I expose three actions the model can request.

`read_local_file`

Lets the model inspect what is already on disk — useful after a write, or when fixing a file it created earlier. Errors come back as a string so the model can see them in the next turn.

def read_local_file(filepath: str) -> str:
    """Reads and returns the contents of a local file.

    Args:
        filepath: The path to the file to read. Must be under ~/repo.
    """
    try:
        with open(_resolve_tool_path(filepath), "r") as f:
            return f.read()
    except Exception as e:
        return f"Error reading file: {str(e)}"

`write_local_file`

Creates or overwrites a file with the full source in content. I reject empty bodies here — early on the model wrote content: "", got a zero-byte file, “verified” it, and declared victory.

def write_local_file(filepath: str, content: str) -> str:
    """Writes or overwrites text content to a local file.

    Args:
        filepath: Target filename or path. Must be under ~/repo.
        content: The text content to write inside the file. Must not be empty.
    """
    try:
        if content is None or not str(content).strip():
            return (
                "Error writing file: content is empty. "
                "Call write_local_file again with the full file source in content."
            )
        path = _resolve_tool_path(filepath)
        path.parent.mkdir(parents=True, exist_ok=True)
        with open(path, "w") as f:
            f.write(content)
        return f"Successfully wrote file to {path} ({len(content)} bytes)"
    except Exception as e:
        return f"Error writing file: {str(e)}"

`run_python_script`

Runs the file with python3, captures stdout/stderr, and enforces a 10s timeout. That is how the agent “checks its work” after a write. Interactive scripts (input()) and GUI mainloop() calls do not verify cleanly here.

def run_python_script(filepath: str) -> str:
    """Executes a local Python file and returns STDOUT and STDERR.

    Args:
        filepath: The path to the Python file to run. Must be under ~/repo.
    """
    try:
        path = _resolve_tool_path(filepath)
        result = subprocess.run(
            ["python3", str(path)],
            capture_output=True,
            text=True,
            timeout=10,
            cwd=str(_work_dir),
        )
        return f"STDOUT:\n{result.stdout}\nSTDERR:\n{result.stderr}"
    except Exception as e:
        return f"Execution error: {str(e)}"

Registration

The agent looks tools up by name when Ollama returns a tool call:

REGISTERED_TOOLS = {
    "read_local_file": read_local_file,
    "write_local_file": write_local_file,
    "run_python_script": run_python_script,
}

Soft path limits under `~/repo`

For the demo I resolve tool paths under ~/repo and a chosen work directory. Relative names like hello.py land in that work dir. Absolute paths must stay under ~/repo. Models sometimes invent /repo/..., so I map that onto ~/repo/....

ALLOWED_ROOT = (Path.home() / "repo").resolve()
_work_dir = ALLOWED_ROOT

def _under_repo(path: Path) -> Path:
    resolved = path.expanduser().resolve()
    try:
        resolved.relative_to(ALLOWED_ROOT)
    except ValueError as exc:
        raise PermissionError(
            f"Path '{resolved}' is outside allowed root '{ALLOWED_ROOT}'."
        ) from exc
    return resolved

NOTE: This is a convenience guard on tool file paths, not a security sandbox. A script started with run_python_script can still touch the rest of the system. Fine for a personal demo, do not treat it as isolation. Hardening this takes the simplicity out of the demo.

The agent loop (`agent.py`)

This is the interesting bit: Thought → Action → Observation.

model reply from ollama.chat = thought / decision
running the tool/tool_calls = action
tool message (result) = observation

# ollama api call
response = ollama.chat(
    model=MODEL_NAME,
    messages=messages,
    tools=tools_list,  # the Python functions from REGISTERED_TOOLS
)

message = response["message"]
messages.append(message)

tool_calls = message.get("tool_calls")

What a real tool_calls look like

It is important that we get valid tool_calls back from the LLM's API, otherwise the agent will fail to function. When tool calling works, the useful data is not in content. It is in tool_calls array. Here name is the function from tools.py (for example write_local_file):

{
  "message": {
    "role": "assistant",
    "content": "",
    "tool_calls": [
      {
        "function": {
          "name": "write_local_file",
          "arguments": {
            "filepath": "hello.py",
            "content": "print('hi')\n"
          }
        }
      }
    ]
  }
}

Then I look up the name and run it:

for call in tool_calls:
    func_name = call["function"]["name"]
    func_args = call["function"]["arguments"]

    if func_name in REGISTERED_TOOLS:
        observation = REGISTERED_TOOLS[func_name](**func_args)
    else:
        observation = f"Error: Tool '{func_name}' is not registered."

    messages.append({
        "role": "tool",
        "name": func_name,
        "content": str(observation),
    })

That observation goes back into messages for the next Ollama turn. That is the whole agent idea in one paragraph.

What a fake tool_call looks like

Sometimes the model skips tool_calls and pastes something into content instead:

I'll create the file now.
<tool_call>
{"name": "write_local_file", "arguments": {"filepath": "hello.py", "content": "print('hi')"}}
</tool_call>

If you treat “no tool_calls” as “task complete,” you exit with nothing written. I hit that more than once.

So when there are no tool calls, I check whether the text looks like a fake one. If it does, I nudge (basically another prompt) instead of finishing:

CONTINUE_NUDGE = (
    "That response was not a valid tool call. "
    "Call write_local_file now with the full file contents. "
    "Do not write empty content. Do not describe the tool call in text."
)

if not tool_calls:
    content = message.get("content") or ""
    if _looks_like_fake_tool_call(content):
        messages.append({"role": "user", "content": CONTINUE_NUDGE})
        continue  # next iteration; yes, this burns a turn

    print(f"\n[Agent Completed Task]:\n{content}")
    return

A nudge is just another user message in the history: “that was not valid, try again with a real tool call.” It increments the iteration counter like any other turn.

If there are no tool calls and the text is a normal summary, we stop. That is the happy path.

Context pruning

Local models have limited context. I keep the system + original user message, then the last few turns:

def prune_context(messages: list, max_history: int = 6) -> list:
    if len(messages) <= max_history + 2:
        return messages
    return messages[:2] + messages[-max_history:]

Aggressive, but it keeps demos from falling over on long tool transcripts.

The CLI (`main.py`)

main.py is thin on purpose.

Optional task argument (default: build a factors calculator.py and verify it)
--dir for a folder under ~/repo, or an interactive prompt
Append a reminder that relative filenames resolve inside that work dir
Call run_agent_loop(task)

python main.py --dir demos/hello 'Create hello.py that prints Hello and run it to verify.'

Running it

The more detailed the prompt the more successful the outcome. Simple prompts like make me a calculator have no chance of success. Here is one for creating another type of calculator with a GUI that another LLM gave me.

cd ~/repo/python-agent
python main.py 'Single-file tkinter calculator with digit buttons 0-9, + - * / = C. Show expression and result. No matplotlib. Write calculator.py only. After writing, fix any syntax or import errors. Prefer a complete working file; do not rely on running mainloop in the test harness.'

You should see something like:

Files will be written under: /home/<user_name>/repo
Enter a directory under ~/repo for the program (relative path like 'demos/factorial', or absolute under ~/repo): calculator
Work directory: /home/<user_name>/repo/calculator
Starting agent with goal: Single-file tkinter calculator with digit buttons 0-9, + - * / = C. Show expression and result. No matplotlib. Write calculator.py only. After writing, fix any syntax or import errors. Prefer a complete working file; do not rely on running mainloop in the test harness.

[Iter 1] Executing Tool: write_local_file({'content': "import tkinter as tk\n...", 'filepath': 'calculator.py'})
  -> Successfully wrote file to ... (N bytes)

[Agent Completed Task]:
This is a complete working single-file tkinter calculator with digit buttons 0-9, + - * / = C. The expression and result are shown in the entry field at the top. The 'C' button clears the entry field.

This project only creates Python, because that is all run_python_script can execute. After the agent completes the task, run the app yourself:

cd ~/repo/calculator
python calculator.py

You get a somewhat working calculator:

I have run this a number of times and the outcome is always different. Sometimes it works and sometimes it does not.

My key takeaway from this is agentic AI is a coding pattern based on your tooling and feedback from an LLM.

Code lives at https://github.com/austincunningham/python-agent.

DEV Community: ollama

Moving Scheduled LLM Curation from Cloud APIs to Local Models

The decision point

Option A: Cloud APIs

Option B: Local models

The headless approval trap

Node pinning and image pulls

The memory that vanishes

Decision framework

My pick and why

When a 4B Local LLM Beats 26B on One Task

When a 4B Local LLM Beats 26B on One Task

Can a 4B local model beat a 26B model?

Why does task quality come before tokens per second?

What should a local model comparison hold fixed?

When would I still choose the 26B model?

What model-selection rule did I keep?

Accompanying prompt

Ollama v1 OpenAI Compat Drops Think Toggle

Building Your Own Private AI Server

Installing and Running Ollama Bare Metal: Building Your Own Private AI Server

Hardware requirements

The runtime tool: Ollama

Visual interface (Open WebUI)

The models (the "brain")

Private use cases

Choosing the Best Distro for Ollama

The best option: Fedora

The stability pick: Debian (Bookworm)

The enterprise alternative: openSUSE Tumbleweed

Comparison table for AI workloads

My recommendation depending on your profile

My choice: Debian (Stable)

Why I avoided Fedora and Tumbleweed for this server

Installing Ollama "Bare Metal"

Installation

Network configuration (important for external access)

Optimizing for 8GB of RAM (Swap)

Downloading strategic models

Where are the models stored? (Important)

The stress test (monitoring)

Managing models

Depending on your hardware and goals, you'll pick different models

Some models I managed to run on my server

Installing the Open WebUI Interface

Connecting with n8n and Other Applications

n8n integration

Integration into your application (Laravel, Vue, etc.)

Test performed using curl

Benchmarking Local AI: Building a llama.cpp vs. Ollama Comparison & Benchmarking App

Introduction

System Architecture & Overview

Interface & Local Tool Integration

Ollama: Frictionless Desktop & Daemon Workflows

llama.cpp: Granular Binary & Hardware Optimization

Benchmarking Inference: Streaming vs. Subprocess Parsing

Ollama Streaming Runner

llama.cpp Native Dispatcher Runner

Key Takeaways & Comparison Matrix

Conclusion

Links

Criando uma IA particular

Instalando e rodando Ollama Bare Metal

O Hardware Necessário (Requisitos)

A Ferramenta de Execução: Ollama

Interface Visual (Open WebUI)

Os Modelos (o Cérebro)

Casos de Uso Privados

Qual a melhor distro para o Ollama.

A Melhor: Fedora

A Escolha de Estabilidade: Debian (Bookworm)

A Alternativa Enterprise: openSUSE Tumbleweed

Tabela Comparativa para IA

Minha Recomendação dependendo do seu perfil:

A minha escolha: Debian (Stable)

Por que evitei Fedora e Tumbleweed para este Servidor?

Instalação do Ollame "Bare Metal"

Instalação

Configuração de Rede (Importante para utlização externa)

Otimização para 8GB de RAM (O Swap)

Test performed using `curl`

Teste realizado usando `curl`