DEV Community: Traven

What You Can Self-Host on Oracle Cloud Free Tier (2026)

Traven — Fri, 13 Nov 2026 05:39:14 +0000

Last updated: July 27, 2026

Status as of this writing: Oracle quietly cut the Always Free Ampere A1 allocation from 4 OCPUs / 24 GB RAM to 2 OCPUs / 12 GB RAM on June 15, 2026. No blog post, no email, an edited documentation page and nothing else. If you're reading an older guide (including some still-circulating 2026 posts) that budgets against 24 GB, check your own console before following its numbers. Everything below is sized for the current 2 OCPU / 12 GB reality, plus the two small AMD Micro instances that are unaffected by the change.

A free tier that quietly loses half its headroom is still worth building on. It means the sizing conversation matters more than it used to, and CPU, not RAM, ends up being the tighter constraint on two cores.

What Oracle gives you for free

The Always Free resources relevant to a self-hosting stack:

Ampere A1 Flex (ARM): 2 OCPUs and 12 GB RAM total, split across one to four instances as you choose. This is down from 4/24 as of June 2026.
AMD Micro instances: two of them, each 1/8 OCPU and 1 GB RAM. Small, but free on its own terms and separate from the Ampere allocation.
Block storage: 200 GB total, shared across your instances.
Outbound bandwidth: 10 TB per month.
Reclamation risk: Oracle's current published policy reclaims an Always Free compute instance if, over a 7-day period, its 95th-percentile CPU, network, and (for A1 shapes) memory utilization all stay under 20%. A stack that's serving real traffic rarely trips this; an instance you spun up once and forgot about is exactly what the policy targets.
Regional availability: Ampere A1 capacity is tightly constrained in popular regions. Frankfurt and Singapore tend to provision within minutes; US regions can sit at "out of host capacity" for hours or days. Budget for that when you're planning a migration deadline. What "2 OCPU" means on Ampere A1. On Oracle's x86 shapes, one OCPU is a full physical core with hyperthreading, billed as the equivalent of two vCPUs, so a workload can lean on simultaneous multithreading to squeeze more out of contended cores. Ampere A1 doesn't work that way: the Ampere Altra processor behind it has no hyperthreading, so one OCPU is exactly one physical core and one thread, nothing shared. Two OCPUs on this tier means two real cores and nothing more to borrow from. That's why CPU contention shows up immediately on this tier rather than being smoothed over by a hyperthreading scheduler the way it might be on an equivalent x86 box.

Two honest limitations before the stack: first, ARM64 isn't universal. Most actively maintained self-hosting images ship multi-arch builds, but some don't. Ghost's core image runs fine on ARM64; its optional ActivityPub and Traffic Analyzer add-ons are AMD64-only as of this writing, so check before assuming a feature works. Second, 2 shared OCPUs is a real ceiling once more than one service wants the CPU at the same time. RAM sizing matters, but CPU contention is what bites on this tier.

The stack: one Ampere instance, one AMD Micro

Put everything internet-facing on the single Ampere A1 instance, and use one AMD Micro purely as an external watchdog, something that checks whether the main box is reachable from outside its own failure domain. The second Micro is spare capacity for later.

Service	Job	Real RAM cost
Caddy	TLS termination, reverse proxy	Under 50 MB
Ghost + MySQL	Blog/CMS (this article's own platform)	Roughly 750 MB in production use
Vaultwarden	Password manager, Bitwarden-compatible	~10 MB idle
Syncthing	Peer-to-peer file sync, no central server needed	150–300 MB typical, spikes higher during large scans
Ollama (3B model, Q4)	Lightweight local AI for personal use	Roughly 3–4 GB
Uptime Kuma + Beszel agent	Uptime checks plus host/container metrics	150–180 MB combined
OS + Docker overhead	Everything else	Roughly 500 MB

That totals somewhere around 5–6 GB of the 12 GB available, leaving real headroom rather than a stack that fits with nothing to spare. The number that won't show up in a RAM table is CPU: an AI response generating tokens, a MySQL query, and a Syncthing hash scan happening at the same moment are competing for two cores, and that's where a personal Oracle box slows down, not memory pressure.

Running Vaultwarden already and want the full hardening pass? Our dedicated Vaultwarden guide covers backup strategy, admin panel lockdown, and the HTTPS requirements the official clients enforce. Take a look at our complete Vaultwarden self-hosting guide.

If you're enjoying what you've read so far, we recommend Oracle Free Tier ARM Cutoff: What to Do Before Aug 18 as your next read.

Why each piece is on this list

Ghost runs the blog you're reading right now, so it earns a place as a real-world example rather than a theoretical pick. The official minimum is 1 GB of RAM; in production with MySQL running alongside it, a well-known self-hosting write-up clocks its own long-running instance at 214 MB for Ghost and 538 MB for MySQL, comfortably under 1 GB combined. Budget for 2 GB if you expect real traffic or plan to turn on the newsletter feature, which adds Mailgun and additional background jobs.

Vaultwarden is the easiest sizing decision in this list. It's a Rust reimplementation of the Bitwarden server, and our own Vaultwarden instance (the one covered in our dedicated guide) idles around 10 MB of RAM serving eight active users, less than a browser tab. There's no meaningful tradeoff here: it's compatible with every official Bitwarden client, and the resource cost rounds to zero next to everything else on this instance.

Syncthing covers file synchronization without needing a central server the way Nextcloud does; it syncs directly between your devices and this instance acts as one more always-on peer. That's a deliberate tradeoff for this tier: Nextcloud gets you a web UI, calendar and contacts sync, and a proper multi-user file store, all of which cost meaningfully more RAM than 12 GB comfortably supports alongside everything else here. If you have RAM to spare on a bigger box later, our productivity-stack guide covers Nextcloud, Immich, Jellyfin, and Paperless-ngx together. On this tier, Syncthing does the one job (getting files onto more than one device) for a fraction of the cost.

Idle RAM use is modest, but it's not flat: Syncthing keeps file indexes in memory, and both very large individual files and folders with hundreds of thousands of small files can push memory use well above idle during an initial scan, in some reported cases into the gigabytes on a large enough folder tree. If you're syncing a large photo library or a folder with a huge file count, set maxConcurrentWrites to 1 and sync folders sequentially rather than in parallel; both are documented tuning options that trade sync speed for a lower peak. On a 12 GB instance already running an AI model, that peak matters more than it would on a dedicated NAS.

Ollama is the one service constrained by this tier's hardware and not only its RAM budget. There's no GPU on Oracle's free tier at all, so inference runs on the Ampere A1's ARM cores. A quantized 7B model is technically possible (public benchmarks put a Q4_K_M 7B model around 5–8 tokens per second on a 4-OCPU A1 instance), but at 2 OCPUs, a smaller 3B model at Q4 quantization is the more honest fit for anything resembling a responsive personal assistant, and it leaves the other two-thirds of your CPU budget for Ghost and Syncthing to use.

Uptime Kuma paired with Beszel is the monitoring choice that matters most on a RAM-constrained tier. A combined Prometheus, Grafana, and node_exporter stack, the setup in our own monitoring guide, runs 800 MB or more at idle before it's watched a single request; Uptime Kuma plus a Beszel agent covers external uptime checks and internal host/container metrics for roughly 150–180 MB combined. On a 24 GB instance, or a paid box with RAM to spare, Prometheus and Grafana's deeper PromQL querying and alerting rules are worth the extra memory. On 12 GB shared with an AI model, Uptime Kuma is the choice between having monitoring at all and not.

Elevated

Outgrown the free tier, or tired of Oracle reclaiming idle instances? Elevated runs managed VPS hosting from SudoSecurity, with real support behind it.

See Elevated VPS plans →

The Layout

Oracle free tier architecture · SVG Oracle Free Tier: Two-Instance Layout Ampere A1 Flex -- 2 OCPU / 12 GB RAM (Always Free) Caddy (TLS, reverse proxy) -- <50 MB Ghost + MySQL -- ~750 MB Vaultwarden -- ~50 MB Syncthing -- ~150-300 MB Ollama, 3B model Q4 -- ~3-4 GB Uptime Kuma + Beszel agent -- ~150-180 MB OS + Docker overhead -- ~500 MB Remaining headroom: roughly 6-7 GB of 12 GB Shared constraint: 2 OCPU total AI inference, MySQL queries, and Syncthing hashing compete for the same 2 cores -- CPU, not RAM, is the real ceiling on this tier AMD Micro -- 1/8 OCPU 1 GB RAM (Always Free) External uptime check Watches the Ampere box from outside its own failure domain Internet-facing CPU-heaviest service

Everything internet-facing sits behind Caddy on the Ampere instance. The AMD Micro doesn't run any of the stack; its only job is checking the Ampere box from a separate failure domain, so a problem with the main instance doesn't also take down the thing telling you about the problem.

A resource-bounded Compose file

services:
  caddy:
    image: caddy:latest
    restart: unless-stopped
    ports:
      - "443:443"
      - "80:80"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - caddy_data:/data
    networks:
      - stack_net

  ghost:
    image: ghost:5
    restart: unless-stopped
    environment:
      - database__client=mysql
      - database__connection__host=ghost-db
      - database__connection__user=ghost
      - database__connection__password=${GHOST_DB_PASSWORD}
      - database__connection__database=ghost
      - url=https://blog.yourdomain.com
    volumes:
      - ghost_content:/var/lib/ghost/content
    deploy:
      resources:
        limits:
          memory: 1g
    networks:
      - stack_net

  ghost-db:
    image: mysql:8
    restart: unless-stopped
    environment:
      - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
      - MYSQL_DATABASE=ghost
      - MYSQL_USER=ghost
      - MYSQL_PASSWORD=${GHOST_DB_PASSWORD}
    volumes:
      - ghost_db:/var/lib/mysql
    deploy:
      resources:
        limits:
          memory: 700m
    networks:
      - stack_net

  vaultwarden:
    image: vaultwarden/server:latest
    restart: unless-stopped
    environment:
      - ADMIN_TOKEN=${VAULTWARDEN_ADMIN_TOKEN}
      - SIGNUPS_ALLOWED=false
    volumes:
      - vaultwarden_data:/data
    deploy:
      resources:
        limits:
          memory: 128m
    networks:
      - stack_net

  syncthing:
    image: syncthing/syncthing:latest
    restart: unless-stopped
    environment:
      - STMAXCONCURRENTWRITES=1
    volumes:
      - syncthing_data:/var/syncthing
    deploy:
      resources:
        limits:
          memory: 768m
    networks:
      - stack_net

  ollama:
    image: ollama/ollama:latest
    restart: unless-stopped
    volumes:
      - ollama_data:/root/.ollama
    deploy:
      resources:
        limits:
          memory: 4g
    networks:
      - stack_net

  uptime-kuma:
    image: louislam/uptime-kuma:2
    restart: unless-stopped
    volumes:
      - uptime_kuma_data:/app/data
    deploy:
      resources:
        limits:
          memory: 150m
    networks:
      - stack_net

networks:
  stack_net:
    external: true

volumes:
  caddy_data:
  ghost_content:
  ghost_db:
  vaultwarden_data:
  syncthing_data:
  ollama_data:
  uptime_kuma_data:

Caddy stays the only exposed port on the instance. Everything else, Ghost, Vaultwarden, Syncthing's sync protocol, Ollama's API, and the Uptime Kuma dashboard, sits behind it on the internal Docker network, reachable by subdomain rather than by port number.

Still with us? We think you'll like How to Self-Host a Discord Bot: Building and Running Your Own Bot 24/7 as your next read.

Three Important Performance Fixes

The Compose file above will run. These three changes are what keep it running well once Syncthing hits a big scan or Ollama is mid-response while someone's reading the blog.

Add swap, even on a "fast enough" instance. Ampere A1 has no swap configured by default, and 12 GB with several services and an AI model loaded leaves less margin than it looks like on paper. A memory spike that would slow things down with swap available becomes an OOM-killed container without it, and Docker's OOM killer doesn't pick politely; it can take out MySQL as easily as the process that caused the spike. A modest swap file (4 GB is reasonable against 12 GB of RAM) plus a low swappiness value gives the kernel somewhere to put cold pages before it starts killing containers:

sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
sudo sysctl vm.swappiness=10
echo 'vm.swappiness=10' | sudo tee -a /etc/sysctl.conf

Swap on a boot volume backed by network-attached block storage is slower than swap on local NVMe, so this is a safety margin for spikes, not a substitute for the RAM budget in the table above. Don't plan to run the stack out of swap day to day.

Tune MySQL down from its defaults. Out of the box, MySQL 8's innodb_buffer_pool_size defaults to 128 MB, which is conservative enough to be safe here, but the rest of its default configuration still assumes more headroom than a 12 GB shared host has to give. A small my.cnf override keeps Ghost's database from creeping past the budget in the resource table:

[mysqld]
innodb_buffer_pool_size=256M
max_connections=50
table_open_cache=200
thread_cache_size=8

Mount this as /etc/mysql/conf.d/low-memory.cnf in the ghost-db service. It won't matter for a low-traffic personal blog, and it's the difference between MySQL sitting at 700 MB and MySQL slowly climbing toward whatever's left on the box during a traffic spike.

Cap Ollama's concurrency instead of leaving it to load-balance itself. Ollama's defaults assume it can decide how many models to keep loaded and how many requests to serve in parallel, which is a reasonable default on a dedicated GPU box and a bad one on a 2-core ARM instance sharing memory with everything else here. Setting explicit limits keeps one long generation from queuing up a second one and doubling memory use:

OLLAMA_NUM_PARALLEL=1
OLLAMA_MAX_LOADED_MODELS=1
OLLAMA_MAX_QUEUE=4

Add these as environment variables on the ollama service in the Compose file. OLLAMA_NUM_PARALLEL=1 means the second request waits instead of doubling memory use by running alongside the first; on a personal instance where you're the only user, that queueing is invisible in practice and the memory ceiling it buys is not.

Want the general self-hosting stack playbook, beyond the free-tier version? Our shared server-prep and reverse-proxy pattern applies whether you're on a free Ampere instance or a paid box. Check out the Mailcow Guide!

Lessons from running this

Check ARM64 support before you deploy, not after. Most of the images above have solid ARM builds. A few popular self-hosting tools still don't, or only support ARM in a beta tag, and finding that out mid-migration is worse than checking before the first docker compose up:

docker manifest inspect ghost:5 | grep -A2 '"architecture"'

Look for "architecture": "arm64" in the output alongside "amd64". If it's missing, the image is x86-only and won't run on the Ampere instance at all, or will run under slow QEMU emulation if Docker's configured to allow it, which defeats the point of a free ARM box.

Watch the reclamation window if traffic is thin. A personal blog with a handful of daily visitors, a password vault only you touch, and a sync client that mostly sits idle can plausibly dip under Oracle's 20% utilization thresholds across a full week. Real, if modest, usage from an actual project protects you here better than any artificial workaround, and running an artificial load generator specifically to dodge the reclamation policy sits in a gray area against Oracle's terms of service that isn't worth the risk for a free instance.

Pin your image tags once the stack is stable. latest is fine while you're getting things running; once Ghost, Vaultwarden, and Ollama are doing real work, an unannounced upstream change landing on a routine docker compose pull is a worse failure mode than a slightly stale version.

Back up the volumes that matter. Ghost's content volume and MySQL data, Vaultwarden's data directory, and Syncthing's config and index. Model weights for Ollama are re-downloadable and not worth backing up; Uptime Kuma's history is nice to keep but not critical if it's lost.

The Compose file, ready to adapt

The full stack above, Caddy, Ghost, Vaultwarden, Syncthing, Ollama, and Uptime Kuma, is a lot to retype by hand. Grab the complete docker-compose.yml and a matching .env template below, drop in your own domain and secrets, and you're running rather than transcribing.

Get the Compose file and.env template

Enter your email and we'll send the exact configuration from this article, including the resource limits, ready to paste onto a fresh Ampere A1 instance.

If you outgrow the 12 GB ceiling, that's a genuine milestone, not a failure of the free tier. It means the stack is doing enough real work that a small paid VPS or dedicated box starts paying for itself in time saved rather than money spent, and at that point the free tier has already done its job: it got you running for the cost of an afternoon instead of a monthly bill.

Your Code Doesn't Have to Leave the Building

Traven — Fri, 13 Nov 2026 05:39:12 +0000

Every developer I know has done the thing. Deadline pressure, a function that won't behave, and you paste it into a chat box because it's four in the afternoon and you want to go home.

I've done it. You've done it. And most of us have never read the retention policy of the service we did it to.

That's the actual situation. Not a hypothetical about corporate espionage — just a quiet, cumulative transfer of proprietary code out of the building, one frustrated paste at a time, into systems whose data-handling terms change by press release. You will find out what happened to it the same way you find out about anything: later, and from someone else.

The good news is that the alternative stopped being a compromise. In 2026 you can run a model on your own hardware that outscores GPT-4o on the coding benchmarks that still mean something. It costs an afternoon and about forty gigabytes of disk. Here's the whole stack, and enough of the internals that you can fix it yourself when it misbehaves.

Three Layers, and You Can Replace Any of Them

Before an install command, get the stack straight. It's three pieces and troubleshooting depends entirely on knowing which one broke.

The weights. A large binary — usually GGUF format — holding billions of numerical parameters. That's the model. A 7B at 4-bit quantization is roughly 4.6GB on disk; a 70B at the same quantization is about 43GB. Those numbers aren't rules of thumb, they're arithmetic, and the section on quantization below shows how to compute them for any model you're considering. These files are inert. They don't do anything.

The runtime. Software that loads those weights into memory and runs inference against them. We're using Ollama, which wraps llama.cpp behind a CLI and a REST API, handles GPU allocation, and manages model storage. It listens on localhost:11434. Think Docker: you pull a thing, you run it.

The interface. Anything that speaks HTTP to that port. curl. A Python script. A VS Code extension. A web UI. The runtime does not care.

Weights on disk. Runtime loads them and exposes an API. Interface talks to the API. Swap any layer without touching the others — and when something breaks, you now know which of three places to look instead of staring at the whole thing.

VRAM Is the Whole Question

Everything about local inference is a memory problem. The weights have to be somewhere fast, and how fast that somewhere is determines your tokens per second.

Discrete GPU: VRAM, and the model needs to fit entirely inside it. Spill into system RAM and you don't degrade gracefully — you fall off a cliff. How steep that cliff is comes out of the arithmetic below, and it is much worse than most guides admit.

Apple Silicon: unified memory. CPU and GPU share the same pool at hardware level, so all your system RAM is effectively VRAM. This is why a MacBook with 36GB competes with a desktop running a 3090, and why the Apple path scales past where consumer GPUs stop.

CPU only: system RAM. Works for 3B–7B models at conversational speed. Anything bigger gets painful — single-digit tokens per second, which is fine for a question you'll wait on and useless for autocomplete.

Hardware	Memory	Comfortable models	Rough speed
6–8GB VRAM	6–8GB	7B at Q4	20–40 tok/s
12–16GB VRAM	12–16GB	14B at Q4, 7B at Q8	30–60 tok/s
24GB VRAM (3090/4090)	24GB	32B at Q4, 14B at Q8	40–80 tok/s
Apple M-series, 36GB	36GB unified	32B comfortably	25–45 tok/s
Apple M-series, 64–96GB	64–96GB unified	70B at Q4	35–55 tok/s
CPU only, 32GB RAM	system RAM	7B at Q4, slowly	5–15 tok/s

Where Those Speeds Come From

Most guides hand you a table of tokens per second with no explanation, which makes it impossible to reason about hardware they didn't list. So here's the model.

For a dense transformer, generating one token requires reading every weight from memory once. That makes generation memory-bandwidth-bound, not compute-bound, and gives you a hard ceiling:

tokens/sec ceiling = memory bandwidth ÷ model size in memory

Published bandwidth figures, run against common model sizes at Q4:

                                    7B       14B       32B       70B
RTX 4090 (GDDR6X, 1008 GB/s)       214       112        50        25
RTX 3090 (GDDR6X,  936 GB/s)       199       104        47        23
M3 Ultra (unified, 800 GB/s)       170        89        40        20
M3 Max 40-core   (400 GB/s)         85        44        20        10
M3 Max 14-core   (300 GB/s)         64        33        15         8
DDR5-6000 dual channel (96 GB/s)    20        11         5         2

Those are ceilings, not measurements — real throughput lands somewhere under them depending on the runtime, the quantization, batch handling and how much of the bandwidth you actually get. But the ratios hold, and the ratios are what you're buying.

Check it against published benchmarks and the model behaves: a 70B at Q4 on an M3 Ultra has a ceiling of 20 tok/s, and reported real-world figures for 70–72B on high-end Apple Silicon sit around 9–14. Roughly half the ceiling, consistently, which is what you'd expect.

Now the important number. GDDR6X on a 4090 moves about 1,008 GB/s. Dual-channel DDR5 moves about 96. That's a 10.5× gap , which means any layer that doesn't fit in VRAM runs about ten times slower than the same layer would have in VRAM.

That is the cliff. Not a 30% penalty, not "somewhat slower" — an order of magnitude on every offloaded layer, which is why published tests of a 70B at Q4 on a single 24GB card report throughput collapsing into single digits. The model didn't get worse. Most of it is being read across a bus a tenth as fast.

It also explains the two things about this table that look strange at first glance. Apple Silicon competes with discrete GPUs despite lower peak bandwidth because the model fits — 400 GB/s that holds all the weights beats 1,008 GB/s that holds two thirds of them. And CPU-only inference isn't slow because CPUs are weak at math; it's slow because dual-channel DDR5 is a tenth the pipe.

Capacity first. Bandwidth second. In that order, always.

The practical floor for real work is an 8GB GPU running a 7B coding model. Twenty to forty tokens per second is faster than you read. That's enough.

If you're buying: a used RTX 3090 is still the value pick in 2026. Same 24GB as a 4090, runs the same models, costs meaningfully less. The 4090 is faster; it is not more capable. Above that, Apple Silicon with 64GB+ is the only consumer route to 70B-class models without stacking cards.

On AMD , be careful, because this is where the previous version of this guide contradicted itself. AMD acceleration on Linux via ROCm is genuinely competitive — a 7900 XTX trades blows with a 3090. On Windows, Ollama does not officially support AMD GPU acceleration; you're on WSL2 or CPU. And check Ollama's current ROCm version requirement before you install anything , because it has moved more than once and a guide written six months ago will send you to the wrong package.

Storage: 100GB of NVMe if you want a few models around. Capacity is the obvious reason; load time is the one people underestimate. Model load is a straight sequential read, so it's pure bandwidth:

                            4.7GB (7B)  20GB (32B)  43GB (70B)
NVMe Gen4 (~7 GB/s)               1 s         3 s         6 s
NVMe Gen3 (~3.5 GB/s)             1 s         6 s        12 s
SATA SSD (~550 MB/s)              9 s        37 s        79 s
7200rpm HDD (~150 MB/s)          32 s      2.3 min     4.8 min

A 32B off NVMe is a three-second pause. Off a spinning disk it's two and a half minutes , every cold start, every time the keep-alive window expires. That's not a minor inconvenience, it's the difference between a tool you reach for and one you avoid because you don't want to wait.

If you're enjoying what you've read so far, we recommend Stable Diffusion Local Setup: RTX 3090 vs Apple Silicon as your next read.

Installing the Runtime

Linux:

curl -fsSL https://ollama.com/install.sh | sh

That sets up a systemd service. Verify it's alive:

ollama --version
systemctl status ollama

macOS and Windows get installers from ollama.com/download. One macOS gotcha worth stating because it wastes an evening: if you install the app, do not also runollama serve. The app already runs the server. Both means a port conflict on 11434 and a confusing error.

Then the only verification that actually matters:

ollama run llama3.1:8b "say hello"
ollama ps

ollama ps shows you where the model actually landed:

NAME              ID              SIZE    PROCESSOR    UNTIL
llama3.1:8b       365c0bd3109b    6.7 GB  100% GPU     4 minutes from now

100% GPU is the goal. If you see 100% CPU on a machine with a supported card, stop and fix the driver before going further. The difference is roughly 35 tokens per second against 5, and every other decision you make downstream is meaningless until this line is right.

Which Models, and Why the Benchmark Everyone Quotes Is Useless

For coding, the Qwen2.5-Coder family is the answer on consumer hardware, and it isn't close.

ollama pull qwen2.5-coder:7b     # fits 8GB VRAM
ollama pull qwen2.5-coder:14b    # 16GB, noticeably better on hard tasks
ollama pull qwen2.5-coder:32b    # 24GB, where local stops apologising

Now, the numbers — and this is the part the previous version of this guide got wrong, so let's do it properly.

You will see HumanEval scores quoted everywhere. Ignore them. The benchmark is saturated: nearly every current model scores above 90%, which means it has stopped distinguishing between them. Qwen2.5-Coder-7B scores in the high 80s. The 32B is around 92%. GPT-4o is about 90%. Those numbers are all the same number wearing different hats, and a two-point gap on a saturated benchmark tells you nothing about whether the thing can refactor your service layer.

The benchmarks that still separate models are the ones that haven't been trained into oblivion:

LiveCodeBench — Qwen2.5-Coder-32B scores 37.2% , against GPT-4o's 29.2%. The 7B lands around 28.6%.
Aider (code repair, not generation) — the 32B scores 73.7 , roughly comparable to GPT-4o. Read that first line again. A 32B model you can run on a single 24GB card outscores GPT-4o on LiveCodeBench. Not "approaches." Outscores. That's the actual state of local coding models in 2026, and it's a much stronger claim than anything HumanEval will tell you.

For general chat and research, Llama 3.1 8B is the workhorse and 3.3 70B is the ceiling if you have the memory. For step-by-step debugging, DeepSeek R1 thinks out loud before answering — you'll see <think> tags in the output, which is the point, not a bug. On minimal hardware, Phi-4 Mini and Gemma 3 1B are surprisingly capable for their size.

Quantization: Fewer Bits, More Parameters

Weights are trained at FP16 or FP32. Quantization drops that precision to save memory, and the key insight is that inference doesn't need training precision. Going FP16 → 4-bit cuts the footprint roughly 75% for modest quality loss.

The naming is misleading, though, and it's worth knowing why. A "4-bit" k-quant is not 4 bits per weight — the K-quant formats store block scales and minimums alongside the weights, so the real figure is higher. Effective bits per weight, which is what actually determines file size:

Format	Bits/weight	vs F16
F16	16.00	—
Q8_0	8.50	47%
Q6_K	6.56	41%
Q5_K_M	5.67	35%
Q4_K_M	4.83	30%
Q3_K_M	3.91	24%

Which makes model size a one-line calculation:

size in GB = parameters (billions) x bits_per_weight / 8

Check it against reality — computed against what Ollama actually ships:

  qwen2.5-coder:7b     computed  4.6 GB | published  4.7 GB | delta 2.1%
  llama3.1:8b          computed  4.8 GB | published  4.9 GB | delta 1.1%
  qwen2.5-coder:32b    computed 19.8 GB | published 20.0 GB | delta 1.0%

Within two percent across the range. So you can size any model on any hardware before downloading a single byte, including ones nobody has written a guide about.

Note also that Q4_K_M lands at 70% smaller than F16, not the 75% everyone repeats — that gap is the k-quant metadata, and on a 32B model it's a couple of gigabytes you need to budget for.

The _K_M suffix means mixed quantization: more precision preserved in the layers that affect output quality most. It beats naive Q4 at nearly identical size, which is why Ollama pulls it by default.

Now the rule worth internalising: more parameters at lower precision beats fewer parameters at higher precision. Run the arithmetic on a 12GB budget:

  7B  @ Q8_0   =  8.1 GB
  14B @ Q4_K_M =  8.9 GB   <- more params, same budget

Nearly identical footprint, twice the parameters. Take the bigger model and the cheaper bits.

The Context Setting That Silently Eats Your Data

This is the single most expensive thing to not know, and it's why "the model forgot what we were talking about" is the most common complaint about local setups.

When your conversation exceeds the context window, Ollama does not error. It silently drops the oldest tokens and keeps going. The model answers confidently using half your input. Nothing in the output tells you this happened.

Worse, the default is smaller than people assume and it is no longer one number. Ollama now picks a default based on detected VRAM: roughly 4K under 24GiB, 32K for 24–48GiB, and 256K above that. So the same command on your workstation and your laptop gives you two very different context windows, and only one of them will quietly truncate the file you pasted.

There are three ways to change it, and they are not equivalent.

In the REPL, temporarily — gone when the model unloads:

/set parameter num_ctx 32768

For the whole server , via environment variable:

OLLAMA_CONTEXT_LENGTH=64000 ollama serve

Permanently, per model , with a Modelfile — the one that actually survives a reboot:

FROM qwen2.5-coder:14b
PARAMETER num_ctx 32768



ollama create qwen-coder-32k -f Modelfile

That rebuild doesn't re-download anything; it stacks a config layer on the existing weights. Note the underscore in num_ctx — a hyphenated num-ctx inside a Modelfile is silently invalid, which is a wonderful thing to discover three hours later.

And via the API, per request:

curl http://localhost:11434/api/generate -d '{
  "model": "qwen2.5-coder:14b",
  "prompt": "...",
  "options": { "num_ctx": 32768 }
}'

Context costs VRAM. A 32K window uses substantially more than 4K on the same model — budget for it, or you'll push layers out of GPU and wonder why everything got slow. For agent and coding-tool workloads, Ollama's own guidance is to set at least 64,000 tokens explicitly rather than trusting the auto-default.

If you take one thing from this article, take this: check your context length before you blame the model.

What Context Actually Costs in VRAM

"Larger context uses more VRAM" is where every guide stops. That's not good enough to plan with, because the number is computable and it's bigger than people expect.

The KV cache holds the key and value tensors for every token in the window, on every layer. Its size is:

bytes = 2 (K and V) x layers x kv_heads x head_dim x context x bytes_per_element

Qwen2.5-7B's published architecture: 28 layers, hidden size 3584, 28 query heads, 4 key-value heads. Head dimension is 3584 ÷ 28 = 128. At fp16, two bytes per element:

2 x 28 x 4 x 128 x 2 = 57,344 bytes per token

Fifty-seven kilobytes of VRAM per token of context. Which scales exactly how you'd fear:

   context     KV cache    total w/ 4.7GB Q4 weights
     4,096       0.22 GB                     4.92 GB
     8,192       0.44 GB                     5.14 GB
    16,384       0.88 GB                     5.58 GB
    32,768       1.75 GB                     6.45 GB
    65,536       3.50 GB                     8.20 GB
   131,072       7.00 GB                    11.70 GB

Read the right column against your card. A 7B at Q4 with a 4K window fits an 8GB GPU with room to spare. Push it to 64K and you need more than 8GB — the weights didn't change, the cache ate the difference. That's the actual mechanism behind "I raised my context and everything got slow." You didn't hit a software limit. You pushed layers out of VRAM and into system RAM, and now you're paying the spill penalty on every token.

There's a second thing hiding in that formula worth understanding, because it's why any of this is affordable at all. Qwen uses grouped-query attention: 28 query heads share just 4 key-value heads. The KV cache scales with kv_heads, not q_heads. Run the same arithmetic with 28 KV heads, as older architectures had:

without GQA (28 KV heads) at 32k: 12.25 GB   <- 7x larger

Seven times the cache. A 32K context on a 7B model would need over twelve gigabytes for the cache alone, before the weights. GQA is the single architectural change that made long context viable on consumer hardware, and it's invisible in every spec sheet you'll read.

Plug your own model's numbers into that formula before you set num_ctx. ollama show <model> gives you the architecture. Then you're sizing a context window on arithmetic instead of raising it until something breaks.

Halving the Cache Without Touching the Context

The KV cache doesn't have to be fp16. Ollama can quantize it, and this is the single biggest lever for running long context on a small card.

OLLAMA_KV_CACHE_TYPE=q8_0 ollama serve

Per Ollama's own FAQ, q8_0 uses about half the memory of f16 "with a very small loss in precision, this usually has no noticeable impact on the model's quality." q4_0 gets you to roughly a quarter with a real, measurable trade-off. Applied to the numbers above:

  context       f16      q8_0      q4_0
    8,192    0.44 GB   0.22 GB   0.11 GB
   32,768    1.75 GB   0.88 GB   0.44 GB
   65,536    3.50 GB   1.75 GB   0.88 GB
  131,072    7.00 GB   3.50 GB   1.75 GB

Now make it concrete. An 8GB card holding a 7B at Q4 has roughly 3GB left after weights and overhead:

    65,536 @ f16   =  3.50 GB  spills
    65,536 @ q8_0  =  1.75 GB  FITS
   131,072 @ f16   =  7.00 GB  spills
   131,072 @ q4_0  =  1.75 GB  FITS

A 64K context that does not fit at f16 fits comfortably at q8_0, on the same hardware, for a precision loss the Ollama docs describe as unnoticeable. That's a genuinely free upgrade and almost nobody sets it.

Three caveats, and the third one will cost you an afternoon if you don't know it.

It requires flash attention to be active. Older guides tell you to set OLLAMA_FLASH_ATTENTION=1; on current builds it's on by default for supported hardware and the variable now acts as a three-state override.

Flash attention does not shrink the cache. This gets repeated wrong constantly. It computes attention in blocks that fit in GPU SRAM, which makes the operation faster and lowers peak transient memory. The K and V vectors are still stored in full. It is a speed optimisation that happens to be a prerequisite for the memory optimisation.

On unsupported architectures it silently falls back to f16. Not a warning. Not an error. Ollama's own issue tracker documents developers setting q8_0, calculating their VRAM budget against half-size cache, and then hitting unexpected OOMs because the server quietly kept using f16. So verify rather than assume:

grep "KV self size" ~/.ollama/logs/server.log

That line reports the actual cache size and type the runtime allocated. If it says f16 after you asked for q8_0, your architecture isn't on the allowlist and every capacity calculation you just did is wrong.

One more detail worth knowing if you push to q4_0: the K cache is measurably more sensitive to quantization than the V cache. Ollama exposes a single global setting, but the underlying llama.cpp allows them to be set separately, and the quality-per-byte sweet spot in community testing has been q8_0 keys with q4_0 values. If you're running llama.cpp directly, that's the configuration to try before accepting q4_0 across the board.

Still with us? We think you'll like The Complete Self-Hosted AI Stack for 2026 as your next read.

A Web Interface, in One Command

Terminal is fine for smoke tests. For real use you want history, file uploads, and model switching. Open WebUI, entirely local:

docker run -d \
  -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -v open-webui:/app/backend/data \
  --name open-webui \
  --restart always \
  ghcr.io/open-webui/open-webui:main

The flag that people drop and then can't debug is --add-host=host.docker.internal:host-gateway. Without it the container has no route to Ollama on the host, and the failure looks like Open WebUI simply not finding any models.

It lands at http://localhost:3000 and asks you to create an admin account — stored locally, no registration, nothing leaves. If auto-detection fails, point it at http://host.docker.internal:11434 on macOS and Windows, or http://172.17.0.1:11434 on Linux.

Wiring It Into VS Code

This is where local stops being a privacy argument and starts being a tool you actually use. The Continue extension gives you inline completion and chat against your local runtime.

Configure two models, not one, and configure them differently:

Autocomplete fires on every keystroke. Put the 7B there. Latency is the entire product.
Chat is something you deliberately invoked and are already waiting on. Put the 14B or 32B there. Continue has changed its configuration schema more than once — check its current docs for the exact file and format rather than copying a config from any guide, including this one. What matters conceptually is that each entry points at provider: ollama and apiBase: http://localhost:11434, and that the autocomplete model is the small fast one.

The feature that justifies the whole exercise is codebase context. Asking "find everywhere we don't handle the error from the database call" means indexing your repository. With a cloud model that means shipping your repository to somebody else — which is exactly the thing most employers prohibit and most developers do anyway. Locally it's just a disk read.

A review loop that works:

Make changes on a branch.
In chat: @diff Summarize these changes and flag any bugs or security issues.
Read the response as a first-pass reviewer, not a verdict. Nothing leaves. The diff stays on the machine that made it.

When It Breaks

Five failures cover almost everything, and each one has a diagnostic that takes ten seconds.

ollama ps says 100% CPU on a machine with a GPU.

The runtime never found your card. Check the driver first, then the logs:

nvidia-smi                       # NVIDIA: driver present and version?
rocm-smi                         # AMD on Linux: ROCm present?
journalctl -u ollama -n 50       # what did detection actually say?

The Ollama log records the detection attempt and the specific reason it failed. Usual causes: driver older than Ollama's minimum, missing ROCm runtime on AMD Linux, or a card that isn't on the supported list. Fix this before anything else — you are otherwise tuning a system running seven times slower than it should.

Output is garbled, repetitive, or incoherent.

Check ollama ps for a split load. If the processor column says something like 60% GPU / 40% CPU, the model didn't fit and part of it is in system RAM. Pull a smaller quantization, pull a smaller model, or reduce context so the whole thing fits. Coherent-but-slow is a spill; incoherent usually means something worse — try re-pulling the model in case the download corrupted.

The model "forgot" the beginning of the conversation.

Context truncation, silently. Covered above — check your num_ctx against what you actually fed it. This is the single most common complaint about local setups and it is almost never a model quality problem.

Open WebUI can't see any models.

Test the runtime directly first, from the host:

curl http://localhost:11434/api/tags

JSON list of models means Ollama is fine and the problem is Docker networking — confirm --add-host=host.docker.internal:host-gateway is present, and on Linux try pointing the UI at http://172.17.0.1:11434. A connection error means Ollama itself isn't running and the UI is innocent.

Everything is slow despite a working GPU.

Something else is holding VRAM. Watch it live:

nvidia-smi --loop=1

Another model still loaded, a browser doing GPU compositing, a game you forgot about. Unload what you're not using:

ollama stop <model>

Models stay resident for a keep-alive window after use, which is a feature until you're running two and wondering why the second one crawls.

What to Check Before You Ask Anyone for Help

In order, because this sequence answers most questions before they get asked:

ollama ps                                    # is it on the GPU at all?
ollama show <model>                          # architecture, context, quantization
grep "KV self size" ~/.ollama/logs/server.log  # what cache did it actually allocate?
journalctl -u ollama -n 50                   # what did it say when it loaded?

Four commands. Between them they tell you where the model is running, how big its window is, what the cache cost, and what the runtime thought about all of it. Almost every "why is my local LLM bad" question is answered somewhere in that output.

When the Cloud Is Still the Right Call

I'd rather be useful than partisan, so: local is not always the answer.

Frontier reasoning still favours the cloud. Decompiling obfuscated binaries, architecting a distributed system, chasing a subtle race condition across a large codebase — the biggest hosted models remain better at the genuinely hard end. Local has closed most of the gap on everyday work and not all of it on the hard stuff.

Wanting a 70B without the memory for one is not a plan. Running a 70B at Q4 on an 8GB card means spilling to system RAM and living at single-digit tokens per second. You will hate it and stop using it.

Serving customers is a different problem. A model on your workstation is a tool. A model serving production traffic is inference infrastructure — batching, queueing, autoscaling, GPU scheduling. Don't confuse the two because the ollama run command looked easy.

And you'll lag the frontier by days to weeks. New models reach hosted APIs before they reach the Ollama library.

For everything else — daily coding, review, research, documentation, the endless small questions you'd never open a ticket for — local wins on cost, on latency, on being there at 3AM when the hosted API is having an incident.

And on the one thing you can't buy back. Every other cost here is recoverable: hardware depreciates, models get better, a bad quantization choice is a re-pull. Code that left the building did that permanently, and no amount of hardware brings it home.

That's the argument. Everything above is just the implementation.