I Gave My Coding Agent A Free Tier

Dustin VanKrimpen — Sun, 12 Jul 2026 16:33:42 +0000

Note: There is a TL;DR section near the bottom for anyone who just wants the setup notes without the story.

The world of AI coding is moving at lightspeed in 2026. Every week I learn about some cool new tooling that people are using to build software with AI assistance. It's an exciting time to be a software engineer, but an uncertain one as well. To satisfy my own curiosity (and to ensure my skills and knowledge stay relevant), I've been investing a lot of my personal time in experimenting with AI development. This is outside of my day-job, so I use my own personal licenses. The deeper I go, the more I run into usage limits.

I like using the latest frontier class models for the hard stuff. I do not love spending their rate limits on "rename this variable" and "write the obvious boilerplate." So I decided to host a large language model (LLM) on my own hardware, free and off of rate limits. As an aside, I also like the idea of my data being kept local, and not shipped off to some major AI corporation on every prompt. But the main motivation is to soak up the easy work with the local model and leave the frontier credits for the work that actually needs a frontier model.

And that's what this post is about. The first part of a series where I will explore what it takes to host your own LLM, make it securely available from anywhere, evaluate its usefulness, and explore the cool things you can do with your own self-hosted AI model. Today's post is about step #1: setting up hardware to host an LLM.

Sourcing hardware during the RAMpocalypse

2026 is a brutal market for PC components. There's a massive AI-driven shortage of DRAM and GDDR which has made prices skyrocket. As of this post, a 32 GB DDR4 kit that used to cost about $50-$70 is sitting at roughly $200. Rather than get fleeced in a seller's market, I decided to just use hardware that I had lying around.

Enter my gaming PC. I built it back in 2020 during the pandemic restrictions. Back then I had all the free time in the world for gaming. It's mostly been collecting dust since my first son was born in 2022. The perfect candidate for my AI server.

The Specs

Originally the gaming PC had a GTX 1080 carried over from an even earlier build. Around 3 years ago I impulse bought a RX 6750 XT 12GB on a good sale and swapped it in. The RX 6750 XT is a fine mid-level card for 1080p and some 1440p gaming today. However, it is generally not a card that you'll see recommended for running LLMs locally.

That said, it's not a bad card for it either (with some caveats I will get to later). For inference, the most important factor is VRAM, because that's what determines what size of model you can hold. The 12 GB in my RX 6750 XT is enough to hold a real coding model, with extra room for context. I didn't set out to choose AMD over NVIDIA here. It's just what I had.

The rest of the box is nothing special. The CPU is a genuinely dated Ryzen 5 2600, but that's ok because it doesn't need to do much during GPU inference. It has 16 GB of DDR4 RAM. A bit tight since I also want to run orchestration processes on this machine, but it's fine given the sheer redonkulousness of RAM prices in 2026.

I did a fresh install of Ubuntu 26.04 since it's free, lean, fits the tooling, and I plan on mostly using it headless via SSH anyway.

Full spec table for the curious:

Part	What I Have	Notes
Case	Cooler Master NR200 (SFF, 18L)	Fits triple-slot GPUs up to 330mm / 156mm; SFX PSU only
PSU	Cooler Master V850 SFX Gold (850W, full-modular)	Upgrade ready
Board	ASUS ROG Strix B450-I (Mini-ITX, AM4, DDR4, PCIe 3.0)	AM4 tops out at Ryzen 5000, which would probably serve my use case just fine for a while
CPU	Ryzen 5 2600 (Zen+, 6c/12t, 2018)	Adequate for now
GPU	Radeon RX 6750 XT 12GB (RDNA2, gfx1031)	Fine for now, but I'm keeping an eye out for deals on something like an RX 7900 XTX or RTX 3090 to upgrade
RAM	16 GB DDR4	Will upgrade when RAM prices come down
Storage	1 x 500 GB NVMe & 1 x 4 TB NVMe	Lucky to have bought before the price increases. More than enough for my use

The unsupported card problem

Here is where the caveat of my GPU comes into play. AMD's compute stack is called ROCm. This is their answer to NVIDIA's CUDA. There is a hardcoded list of supported GPU targets for ROCm, and your card has to be on it. My RX 6750 XT reports its chip target as gfx1031, which is not supported. The very similar chip in the 6800 and 6900 series cards is gfx1030, and that one is supported. gfx1031, the 6700 series, is not. Same architecture, one digit off, left off the list.

Note:
ROCm's official supported-GPU list changes between releases, and gfx1031 support has been a moving target. Best to check for yourself since things may have changed since the time of this posting.

The usual workaround

There is a very simple workaround that a lot of people have reported success with on Linux. It basically tricks ROCm into thinking your card is supported. You install the GPU drivers and ROCm runtime as usual, add your user account to the video and render groups, and then spoof the GFX version to 10.3.0 like so:

export HSA_OVERRIDE_GFX_VERSION=10.3.0

Vulkan: the alternative

Vulkan is a graphics and compute API that these cards support natively with no ROCm needed. I am using Ollama to serve models, which runs inference on the Vulkan backend no problem. The main downside is it's inference-only, meaning you can't fine-tune or train models. Since I'm only trying to serve an existing model for now, it's a non-issue for me.

Note: It's not that Vulkan is inherently incapable of training/tuning models. It's just that the typical training stack targets CUDA/ROCm.

Comparison: ROCm workaround vs. Vulkan (RX 6750 XT / gfx1031)

Factor	ROCm (via `HSA_OVERRIDE_GFX_VERSION=10.3.0`)	Vulkan
Setup	Requires the override env var to spoof GPU target	Natively supported (no workaround)
Reliability	Fragile: can fail to init (known "Tensile host" error); kernel/ROCm updates can silently break it	Stable and low-maintenance; part of Mesa, survives kernel updates, set-and-forget
Legitimacy	Unsanctioned spoof that's untested by AMD on this chip	Uses the card as itself
Inference speed	Historically faster, but the gap has narrowed. Recent benchmarks range from ROCm ahead ~10–20% (mostly on prompt processing) to Vulkan matching or beating it	Very competitive on RDNA2 specifically. The backend was first developed on an RDNA2 card, so token-generation speed is often on par with ROCm
Fine-tuning / training	Supported. ROCm is the ML-native path (PyTorch-ROCm, vLLM, training libs)	Unsupported. Inference-only; cannot fine-tune or run PyTorch training
Tooling	Richer (`rocm-smi` for monitoring/management)	Lighter (`radeontop`)
Footprint	Large stack to install and keep working	Small, often already present

Verdict

For my use case (mostly-headless inference box), Vulkan wins. And I'm not giving up much for it. The old assumption that ROCm is meaningfully faster has largely eroded, and on RDNA2 in particular (the generation the Vulkan backend was first tuned on) the two are close enough that speed isn't the deciding factor.

The goal of this project is to make AI-assisted workflows easier for me. I want to spend my time building cool things, not debugging why the latest update broke my hacky workaround. Vulkan just works, and keeps working across driver updates, which is what I want for an always-on server.

Setup

I started with a clean slate. A fresh install of Ubuntu 26.04 LTS Desktop. I went with desktop because I do have a monitor attached (set up at a desk in my basement), and will occasionally work from it directly using IDEs and such. But most of the time it will be connected to and used headless via my MacBook Air.

Step 1: The Vulkan graphics stack

Update the system and install the Vulkan userspace drivers and a couple of
diagnostic tools:

sudo apt update && sudo apt full-upgrade -y
sudo apt install -y mesa-vulkan-drivers vulkan-tools radeontop

Confirm the card is visible to Vulkan:

vulkaninfo | grep -i deviceName
# should print "AMD Radeon RX 6750 XT" (mine shows RADV NAVI22)

radeontop is the tool we'll use later to prove the GPU is actually doing the
math, so it's worth installing now.

Step 2: Install Ollama and pull a model

Ollama is the easiest way to serve a local model. Install it with one line:

curl -fsSL https://ollama.com/install.sh | sh

On this hardware Ollama installs as a systemd service. During the first run, it
resolves the Vulkan backend, drops the unsupported ROCm device, and
loads onto the discrete RX 6750 XT with ~11.6 GiB of usable VRAM.

Note: When Ollama starts, it will show a warning that it dropped the ROCm device. This is expected on a gfx1031 card. It realized that the GPU doesn't support ROCm and fell back to Vulkan. That's what you want.

Picking a model for 12GB of VRAM

Now that Ollama is set up, it's time to choose a model for it to serve. This is where the VRAM budgeting gets real.

My VRAM limitation rules out the current flagships: qwen3-coder:30b (~19GB), Codestral 22B (~13GB), and anything 24B+ dense. What's left is still a solid field of 7–16B coders.

Model	Params	~Q4 size	Context	Trade-off
qwen2.5-coder:14b (chosen)	14.8B dense	~9GB	32K	Best dense code quality in the tier; native tool-calling + fill-in-the-middle. Non-reasoning, so it's fast.
qwen2.5-coder:7b	7B dense	~4.7GB	32K	Same family, more headroom/speed, lower ceiling.
qwen3:14b	14.8B dense	~9GB	32K+	Strong generalist that codes well, but not code-specialized.
deepseek-coder-v2:16b	16B MoE (2.4B active)	~8.9GB	160K	Fastest option (MoE) and by far the largest context. Weaker at multi-step tool use.
deepseek-r1-distill-qwen:14b	14B dense (reasoning)	~9GB	32K	Thinks step-by-step. Great for hard debugging, slower for everyday generation.

I went with qwen2.5-coder:14b because this box is intended to be the worker tier of a two-tier setup: easy tasks run locally, hard ones escalate to externally hosted models. I wanted the fastest, most reliable, tool-capable code specialist that fits. If I need a higher-reasoning model, I will use an externally hosted 'frontier-class' model.

The closest competitor for my use was deepseek-coder-v2:16b. I may yet try it out, and benchmark it just to see how it compares. We'll see.

Pulling

With the model chosen, pulling it is one line:

ollama pull qwen2.5-coder:14b-instruct-q4_K_M

Quick smoke test:

ollama run qwen2.5-coder:14b-instruct-q4_K_M "write a python function that reverses a string"

If it answers, the model runs. However, that does not prove it's using the
GPU. That part is next.

Step 3: Prove it's actually on the GPU (not silently on the CPU)

It's possible for Ollama to discover your GPU, load the weights into VRAM, and still run the compute on CPU.
The logs will look normal, VRAM will show occupied, and the tokens will trickle while your CPU melts.

To verify the GPU compute, I used radeontop. Open it in one terminal like so:

radeontop

Then, in a second terminal, fire off a prompt long enough to sustain load for several seconds:

ollama run qwen2.5-coder:14b-instruct-q4_K_M \
  "write a detailed, heavily commented python implementation of a red-black tree with insert, delete, and search"

A tmux tip: split the terminal (Ctrl-b %), run radeontop on the left and
your ollama run on the right, and you can literally watch the GPU spike as it
"thinks." It's oddly satisfying.

Now watch radeontop while tokens are streaming. What you're looking for:

Metric	Idle (between generations)	Active (mid-generation)
Graphics pipe	~1–2%	jumps high (~93% on mine)
Shader Clock	~0.4%	climbs toward max (~90%)
VRAM	~10.3 / 12.2 GB	same (model stays resident)

The signal is the transition: idle → active → idle, synced to token output.

Note: Watch Graphics pipe and Shader Clock, not VRAM. The weights can be in VRAM while the math runs on the CPU. You have to watch during streaming. The compute drops to zero as soon as generation ends.

On the VRAM numbers: Ollama reports usable VRAM in GiB (it logged ~11.6 GiB available), while radeontop reads out in GB using its own accounting. The two don't line up exactly.

On my box, Graphics pipe hit ~93% and Shader Clock ~90% during generation, then fell to 0% and ~0.11% after. That confirms GPU inference
over Vulkan. If your compute stays flat while tokens are streaming, that indicates CPU fallback, so you'll need to do some debugging.

Step 4: The context-window decision (8K, on purpose)

Ollama picks a default context length for you, but I wanted to set it explicitly
because context costs VRAM. A bigger context window means a bigger KV cache
sitting in memory alongside the weights. On a 12 GB card, that budget is tight, so
I capped the default served context at 8K, which gives ~10.3 GB resident
with headroom to spare (even though Qwen2.5-Coder's native context is 32K).

I set it with a systemd drop-in, so I'm not editing Ollama's packaged unit file:

sudo mkdir -p /etc/systemd/system/ollama.service.d
sudo tee /etc/systemd/system/ollama.service.d/override.conf >/dev/null <<'EOF'
[Service]
Environment="OLLAMA_CONTEXT_LENGTH=8192"
EOF

sudo systemctl daemon-reload
sudo systemctl restart ollama

Verify it took:

systemctl show ollama -p Environment --no-pager   # echoes OLLAMA_CONTEXT_LENGTH=8192

8K is a fine default for most coding prompts. If a specific task needs
more, you don't have to change the service. You can override context per
request:

curl http://localhost:11434/api/generate -d '{
  "model": "qwen2.5-coder:14b-instruct-q4_K_M",
  "prompt": "...",
  "stream": false,
  "options": { "num_ctx": 16384, "temperature": 0.2 }
}'

Raise num_ctx for the one long-context call that needs it. Don't leave it high
globally or you'll pay the VRAM tax on every request.

A couple of other handy per-request knobs:

temperature 0.1–0.3 for deterministic code, higher for brainstorming
keep_alive pass -1 to pin the model resident when you're about to hammer it (by default it unloads after 5 idle minutes and the next request eats a few-second reload).

Ollama also exposes an OpenAI-compatible API at http://localhost:11434/v1.
Any tool compatible with "OpenAI API, custom base
URL" can point at it locally with a dummy key. The server is stateless. It takes
the full message history on every call, so your client holds the conversation.

Step 5: zram, and a packaging gotcha

This box has 16 GB of RAM, and the DDR4 shortage of 2026 makes adding more
expensive. The model itself lives in VRAM, so system RAM isn't the hot path,
but the OS plus whatever I stack on top can spike. I'm planning on setting up some
orchestration and automation for unattended workflows, so zram is cheap insurance.

If you don't know, zram is a compressed swap device that lives in RAM, giving you meaningfully more
usable memory at a small CPU cost.

On Ubuntu 26.04 the right tool is systemd-zram-generator (not the older
zram-tools / zram-config packages, which use different config paths and will
just confuse you):

sudo apt install -y systemd-zram-generator

sudo tee /etc/systemd/zram-generator.conf >/dev/null <<'EOF'
[zram0]
zram-size = min(ram, 8192)
compression-algorithm = zstd
EOF

sudo tee /etc/sysctl.d/99-zram.conf >/dev/null <<'EOF'
vm.swappiness = 180
EOF
sudo sysctl --system >/dev/null

zram-size = min(ram, 8192) is evaluated in MiB, so on a
16 GB machine that's an 8 GB compressed device.

vm.swappiness = 180 looks crazy if you remember the old "100 is the max" rule,
but kernels since 5.8 cap at 200. Since zram swap is RAM-fast, biasing the kernel
toward evicting pages into compressed RAM is cheap and frees space for page cache.
150 is a good option too. The difference is noise.

The zram gotcha

Here's what confused me. The package ships a vendor default that activates a
4 GB / lzo-rle device at install time, before my config file existed. A
plain systemctl start on an already-active zram device does nothing. It keeps
the install-time parameters and silently ignores your config.

Symptom: you set up 8 GB / zstd, run zramctl, and see 4G / lzo-rle
instead. Your config appears to do nothing.

Fix: use restart, not start. It tears the device down and rebuilds it
from your /etc config:

sudo systemctl daemon-reload
sudo systemctl restart systemd-zram-setup@zram0.service

Verify the end state:

zramctl            # want: /dev/zram0  zstd  8G  [SWAP]
swapon --show      # zram0 at priority 100; disk swapfile as the low-priority tail
cat /proc/sys/vm/swappiness   # 180

One more caveat: zram is not an OOM guarantee. A hard enough runaway process
blows past RAM + zram and still hits the OOM killer. The on-disk swapfile
(priority -1) is the tail that catches overflow; zram (priority 100) fills first.
For a box that's going to run unattended workloads, the layering is worth having.

The end result

I ended up with a machine that:

runs Qwen2.5-Coder 14B on a 12 GB AMD GPU over Vulkan (no ROCm, no override hacks), with GPU compute verified, not assumed
serves it on both the native Ollama API and an OpenAI-compatible endpoint, so any local tool or editor plugin can point at it
caps context at a VRAM-sane 8K by default with per-request overrides for the rare long task
has zram as a RAM-pressure cushion for whatever I stack on top

And the whole thing runs on hardware I already owned, for the cost of electricity.

TL;DR

The cliffnotes on setting up your own self-hosted LLM server:

Install an LTS Linux (Ubuntu 24.04/26.04) with a recent kernel for your GPU.
Check your GPU's compute support. AMD: is your gfx target on ROCm's list? If not (e.g. RX 6700-series gfx1031), plan on Vulkan: sudo apt install mesa-vulkan-drivers vulkan-tools radeontop.
Install Ollama: curl -fsSL https://ollama.com/install.sh | sh.
Pull a model that fits your VRAM. 12 GB → a 14B at Q4_K_M (~10 GB). Don't try to cram a 30B into 12 GB.
Verify GPU compute with radeontop during generation. Watch Graphics pipe / Shader Clock, not VRAM. Discovery ≠ compute.
Set the context ceiling with a systemd drop-in (OLLAMA_CONTEXT_LENGTH=8192) to keep the KV cache inside your VRAM budget; override per request with num_ctx when needed.
Set up zram with systemd-zram-generator, and remember to restart (not start) the setup service so your config isn't shadowed by the vendor default.

Next steps

Now I have my very own local-tier coding model, running for free (minus the cost of electricity) on a GPU that I had lying around in a computer that lives in my basement.

What I am building towards is a coding agent that can do the easy work locally for free, allowing me to save my frontier model usage limits for harder work. Eventually, I would like to have a fully automated workflow that I can leave unattended (with guardrails in place) while I am doing other things. This box is the first step in reaching that.

There are a few more tasks that need to be done, with more posts to come documenting my process and results.

Reaching it from anywhere, privately

This post covers setting up a local AI server that only serves on localhost. Next I will cover getting to it from my laptop or my phone, from anywhere, over an encrypted mesh network with nothing exposed to the public internet. That's the next post.

Is it actually any good? The uncomfortable question.

Can I really trust a quantized 14B running on a consumer card with real work? I put it through the standard coding benchmarks, and the results were good, but surprising, and also alarming in a way that turned out to be the benchmark's fault, not the model's. I am working on a full write-up, and may even try out some other models for comparison.

The agent loop itself

The whole point of doing all this. I'm still experimenting with the many tools available for orchestrating multiple models, and running feedback loops so they can keep running even when I am not attending to them. Maybe I'll even end up writing some scripts of my own to do this. There are a few side projects that I have in the works to test everything out with, so expect future write-ups on what I discover.

Final notes

For now, I am happy to have my own free model running on a mid-level card that I didn't have to pay 2026 prices for. You don't need the latest, most expensive hardware to run a real and useful model at home. All you need is something with enough VRAM. It doesn't even have to be on AMD's supported list.

Go and check out what your idle GPU could be doing instead.

DEV Community: Dustin VanKrimpen