DEV Community: Dustin VanKrimpen

Using My Local Coding Model From Anywhere

Dustin VanKrimpen — Sun, 26 Jul 2026 13:43:25 +0000

In the last post I got a coding model (qwen2.5-coder:14b) running on a repurposed GPU (Radeon RX 6750 XT 12GB) that AMD's compute stack officially won't support. Which is great, except for one thing: the box it runs on sits in a corner in my basement. A local model you can only use while physically sitting at the machine it runs on is just inconvenient.

The whole point of a local tier is that it is there when I am. At my laptop on the couch, on my phone waiting in line somewhere, from a coffee shop. This post is the connective tissue, detailing how I reach that headless box from anywhere without punching a single hole in my router.

Why the obvious answer is the wrong one

Two things I wanted, and they pull against each other:

Reach the machine, and its model endpoint, from any of my devices, wherever I am.
Expose nothing to the public internet.

The naive way to do (1) is port forwarding: tell your router to send traffic on some port straight to the box. It works, and it flatly violates (2). And it is especially reckless here, because the model endpoint has no authentication. Ollama does not ask for a password. Anyone who can reach it can use it. Forwarding that to the open internet means giving your GPU to the first scanner that wanders by.

So port forwarding is out. What I wanted was a way for my own devices to reach each other as if they were on the same local network, no matter what physical network each one is actually on, without anything being visible to anyone else. Keeping everything private, with nothing exposed, is the whole point.

Tailscale: a private network that follows your devices around

The tool I settled on is Tailscale. It is a mesh VPN built on WireGuard. It's not the kind of VPN that hides your traffic from your ISP. Instead, it's the kind that connects your own devices to each other. You install it on each machine, they all join one private network (your "tailnet"), and every device gets a stable address in the 100.x range that is reachable from any other device on the tailnet, wherever they physically are.

The clever part is how the connection gets made. A coordination server helps your devices find each other and then gets out of the way. The actual traffic is a direct, end-to-end encrypted, peer-to-peer link. There is no port forwarding, and nothing is exposed to the public internet. The same mechanism carries everything: SSH on port 22, the model endpoint on :11434, and whatever else I run, all over the tailnet. Nothing public.

A few practical notes:

The free Personal tier is plenty. Unlimited devices of your own, up to 6 users. A handful of personal machines is nowhere near any limit, and it costs nothing.
Install it on everything, one account. The GPU box, the laptop, the phone, all on the same tailnet.
MagicDNS gives your devices names instead of making you memorize 100.x addresses... in theory. See the gotchas.

Note: For those who would rather not trust Tailscale's coordination server at all, there is an open-source, self-hostable replacement for it called Headscale. I plan on trying it out at some point and documenting my verdict here in the future, so stay tuned!

Two gotchas

This is the part worth documenting, because these are the things that will make someone bounce off Tailscale thinking it's broken when it isn't.

The Mac app can get stuck, and the fix is the CLI build. The standalone Tailscale Mac app leans on a macOS system extension, and that extension can wedge itself: no login prompt, the VPN toggle throws errors, and reinstalling does nothing. It is a known issue, more likely if you ever had an old App Store install or you're on a newer macOS. This happened to me, unfortunately. The fix that worked was installing the Homebrew CLI build instead (brew install tailscale), which uses the utun interface and sidesteps the system extension entirely. The tradeoff is you lose the menu-bar GUI and drive it from the terminal. Really no problem at all for me. It connects reliably, which is the part that matters.

MagicDNS short names are flaky on that CLI build. Once you are on the Homebrew CLI client, MagicDNS does not wire into the Mac's resolver as cleanly as the GUI app does. The name you set up may just not resolve. The fix is to use the full MagicDNS name or the raw 100.x IP. Slightly annoying, entirely livable once you know it. It's worth noting the MagicDNS names resolved fine on the Android client.

SSH, and making it survive real life

With the tailnet up, SSH into the box is just ssh you@100.x.x.x from any of your devices. A few things make it pleasant instead of fiddly:

Key auth, no passwords. Standard ssh-copy-id. If you import your keys from GitHub during the Ubuntu install, this is basically already done.
An SSH config alias so you can type ssh yourMachine and never think about IPs again. In ~/.ssh/config, point Host yourMachine at the raw 100.x address. This has a nice side effect: because you hardcoded the IP, the alias works even when MagicDNS is being flaky. It routes around the gotcha above entirely.
tmux for anything long-running. An SSH session dies when your laptop sleeps or your WiFi drops, and anything running directly in that session dies with it. tmux runs the session on the box, decoupled from your connection, so you start something, detach, close the lid, and reattach later from a different device exactly where you left off. This is load-bearing once you are kicking off unattended work.
mosh for the phone. Plain SSH drops when your phone roams between WiFi and cellular. mosh survives that, so a mobile session does not die every time you walk out of the building. mosh plus tmux is the mobile combo.

Exposing the model endpoint (only to the tailnet)

Here is the piece specific to the model. By default Ollama binds to 127.0.0.1, meaning it only listens to the box itself. To let other devices reach it, you tell it to listen on all interfaces with a systemd drop-in that sets OLLAMA_HOST=0.0.0.0.

That sounds alarming, and it would be, except for what actually holds the boundary. The bind is broad, but the firewall is the fence. ufw is set to default-deny incoming, with a single allow rule scoped to the tailnet interface (ufw allow in on tailscale0). So the endpoint listens broadly, but it's only reachable over the tailnet. Nothing on the physical LAN or the public internet can touch it. This matters enormously precisely because the endpoint is unauthenticated: the firewall scoping is the only thing standing between "my private model" and "a GPU for anyone on the internet." It is not optional.

The one command that proves the whole chain

There are three moving parts stacked here: the broad bind, the ufw rule, and Tailscale itself. When something doesn't connect, it is tempting to guess which layer is at fault. Don't guess. Prove it from a real client with one request:

curl http://100.x.x.x:11434/api/tags

Run that from the laptop, not from the box. If it returns JSON listing your models, the entire chain is proven end to end. The bind is listening, the firewall is allowing tailnet traffic, and Tailscale is carrying it. If it hangs or refuses, you've narrowed it to the network path rather than the model. This is the fastest "is it actually working" check, and I run it before wiring up anything else.

Actually using it from the laptop

Now the payoff. Anything that speaks to Ollama can be pointed at the box over the tailnet. The ollama CLI and most Ollama-aware tools respect an OLLAMA_HOST environment variable, so on the laptop:

export OLLAMA_HOST=http://100.x.x.x:11434

and the CLI is now a remote control for the GPU box. Nothing runs locally on the laptop; it is purely a client. For tools that only speak the OpenAI API shape, Ollama also exposes an OpenAI-compatible endpoint at the /v1 path, which you point the tool at with any dummy API key.

The reason this works so cleanly is a small architectural point worth internalizing: the model is stateless per request, and "workspace context" is just files you put into the request. So you can gather context on the machine that has your files (the laptop) and send only the model call to the remote GPU. The code and the compute do not have to live in the same place.

In a future post, I will dive deeper into the tooling that you can connect to your local LLM for development. Currently, I've been experimenting with using JetBrains Gateway and Cline.

The honest tradeoff: plan a way back in

As configured, SSH into this box rides only the tailnet firewall rule. So if Tailscale fails to come up on boot, the machine is unreachable over the tailnet, and right now that is the main door.

The fix is cheap and does not compromise the posture: add a second ufw rule that allows SSH from your local network, scoped to your subnet (ufw allow from 192.168.x.0/24 to any port 22). That is still not public. Your LAN is not the internet, and you are not touching the router or forwarding any ports. You are just widening the way in from "tailnet only" to "tailnet, or the same physical network." When Tailscale is down and you are home, you can still walk over to your own network and get in.

The LAN rule only saves you when you are physically on the LAN. If the tunnel dies while you are out, you are still waiting until you get home to console in or reboot. But for a headless box sitting in my house, "I can always reach it from my own network" covers nearly every lockout worth worrying about, and it is a rule that only needs to be set once. The thing to actually avoid is having neither path, tailnet-only with no LAN fallback, on a box you have made a dependency.

Where this leaves the project

So now the model from post 1 is not stuck in a corner. I can reach it from my laptop or my phone, from anywhere, and point tools at it as if it were local, with nothing exposed to the internet and one curl to prove the whole path. That is the local tier.

The next question is the one that decides whether any of this was worth it: is the model actually good enough to trust with real software engineering work? A quantized 14B on a consumer card is capable, but capable enough to hand real tasks to and route around a frontier model? The benchmarking of it will be my next post. And it has a twist, because the first numbers I got were badly wrong for reasons that had nothing to do with the model.

The bigger goal underneath all of it is the same: a coding agent that does the cheap work locally, for free and off any rate limit, and only spends frontier-model credits when the problem actually earns one. A free tier LLM reachable from anywhere is most of the setup for that. Now I need to know how much of my real work it can actually carry.

That is all for today. Go make the machine in your closet reachable from anywhere.

I Gave My Coding Agent A Free Tier

Dustin VanKrimpen — Sun, 12 Jul 2026 16:33:42 +0000

Note: There is a TL;DR section near the bottom for anyone who just wants the setup notes without the story.

I like using the latest frontier class models for the hard stuff. I do not love spending their rate limits on "rename this variable" and "write the obvious boilerplate." So I decided to host a large language model (LLM) on my own hardware, free and off of rate limits: soak up the easy work locally, save the frontier credits for the work that needs them, and keep my data off some AI corporation's servers while I'm at it. This post is step #1: setting up the hardware.

Sourcing hardware during the RAMpocalypse

2026 is a brutal market for PC components. There's a massive AI-driven shortage of DRAM and GDDR which has made prices skyrocket. As of this post, a 32 GB DDR4 kit that used to cost about $50-$70 is sitting at roughly $200. Rather than get fleeced in a seller's market, I decided to just use hardware that I had lying around.

Enter my gaming PC. I built it back in 2020 during the pandemic restrictions. Back then I had all the free time in the world for gaming. It's mostly been collecting dust since my first son was born in 2022. The perfect candidate for my AI server.

The Specs

Originally the gaming PC had a GTX 1080 carried over from an even earlier build. Around 3 years ago I impulse bought a RX 6750 XT 12GB on a good sale and swapped it in. The RX 6750 XT is a fine mid-level card for 1080p and some 1440p gaming today. However, it is generally not a card that you'll see recommended for running LLMs locally.

That said, it's not a bad card for it either (with some caveats I will get to later). For inference, the most important factor is VRAM, because that's what determines what size of model you can hold. The 12 GB in my RX 6750 XT is enough to hold a real coding model, with extra room for context. I didn't set out to choose AMD over NVIDIA here. It's just what I had.

The rest of the box is nothing special. The CPU is a genuinely dated Ryzen 5 2600, but that's ok because it doesn't need to do much during GPU inference. It has 16 GB of DDR4 RAM. A bit tight since I also want to run orchestration processes on this machine, but it's fine given the sheer redonkulousness of RAM prices in 2026.

I did a fresh install of Ubuntu 26.04 since it's free, lean, fits the tooling, and I plan on mostly using it headless via SSH anyway.

Full spec table for the curious:

Part	What I Have	Notes
CPU	Ryzen 5 2600 (Zen+, 6c/12t, 2018)	Adequate for now
GPU	Radeon RX 6750 XT 12GB (RDNA2, gfx1031)	Fine for now, but I'm keeping an eye out for deals on something like an RX 7900 XTX or RTX 3090 to upgrade
RAM	16 GB DDR4	Will upgrade when RAM prices come down

The unsupported card problem

Here is where the caveat of my GPU comes into play. AMD's compute stack is called ROCm. This is their answer to NVIDIA's CUDA. There is a hardcoded list of supported GPU targets for ROCm, and your card has to be on it. My RX 6750 XT reports its chip target as gfx1031, which is not supported. The very similar chip in the 6800 and 6900 series cards is gfx1030, and that one is supported. gfx1031, the 6700 series, is not. Same architecture, one digit off, left off the list.

Note:
ROCm's official supported-GPU list changes between releases, and gfx1031 support has been a moving target. Best to check for yourself since things may have changed since the time of this posting.

The usual workaround

There is a very simple workaround that a lot of people have reported success with on Linux. It basically tricks ROCm into thinking your card is supported. You install the GPU drivers and ROCm runtime as usual, add your user account to the video and render groups, and then spoof the GFX version to 10.3.0 like so:

export HSA_OVERRIDE_GFX_VERSION=10.3.0

Vulkan: the alternative

Vulkan is a graphics and compute API that these cards support natively with no ROCm needed. I am using Ollama to serve models, which runs inference on the Vulkan backend no problem. The main downside is it's inference-only, meaning you can't fine-tune or train models. Since I'm only trying to serve an existing model for now, it's a non-issue for me.

Note: It's not that Vulkan is inherently incapable of training/tuning models. It's just that the typical training stack targets CUDA/ROCm.

Comparison: ROCm workaround vs. Vulkan (RX 6750 XT / gfx1031)

Factor	ROCm (via `HSA_OVERRIDE_GFX_VERSION=10.3.0`)	Vulkan
Setup	Needs the override env var to spoof the GPU target	Natively supported, no workaround
Reliability	Fragile: can fail to init; updates can silently break it	Stable; part of Mesa, survives kernel updates
Fine-tuning / training	Supported; the ML-native path (PyTorch-ROCm, vLLM)	Unsupported; inference only
Tooling	Richer (`rocm-smi` for monitoring)	Lighter (`radeontop`)

Verdict

For my use case (a mostly-headless inference box), Vulkan wins, and I'm not giving up much for it. The old assumption that ROCm is meaningfully faster has largely eroded: recent benchmarks put ROCm anywhere from ~10–20% ahead (mostly on prompt processing) to level with or behind Vulkan. On RDNA2 in particular, the generation the Vulkan backend was first tuned on, the two are close enough that speed isn't the deciding factor.

The goal of this project is to make AI-assisted workflows easier for me. I want to spend my time building cool things, not debugging why the latest update broke my hacky workaround. Vulkan just works, and keeps working across driver updates, which is what I want for an always-on server.

Setup

I started with a clean slate. A fresh install of Ubuntu 26.04 LTS Desktop. I went with desktop because I do have a monitor attached (set up at a desk in my basement), and will occasionally work from it directly using IDEs and such. But most of the time it will be connected to and used headless via my MacBook Air.

Step 1: The Vulkan graphics stack

Update the system and install the Vulkan userspace drivers and a couple of
diagnostic tools:

sudo apt update && sudo apt full-upgrade -y
sudo apt install -y mesa-vulkan-drivers vulkan-tools radeontop

Confirm the card is visible to Vulkan:

vulkaninfo | grep -i deviceName
# should print "AMD Radeon RX 6750 XT" (mine shows RADV NAVI22)

radeontop is the tool we'll use later to prove the GPU is actually doing the
math, so it's worth installing now.

Step 2: Install Ollama and pull a model

Ollama is the easiest way to serve a local model. Install it with one line:

curl -fsSL https://ollama.com/install.sh | sh

On this hardware Ollama installs as a systemd service. During the first run, it
resolves the Vulkan backend, drops the unsupported ROCm device, and
loads onto the discrete RX 6750 XT with ~11.6 GiB of usable VRAM.

Note: When Ollama starts, it will show a warning that it dropped the ROCm device. This is expected on a gfx1031 card. It realized that the GPU doesn't support ROCm and fell back to Vulkan. That's what you want.

Picking a model for 12GB of VRAM

Now that Ollama is set up, it's time to choose a model for it to serve. This is where the VRAM budgeting gets real.

My VRAM limitation rules out the current flagships: qwen3-coder:30b (~19GB), Codestral 22B (~13GB), and anything 24B+ dense. What's left is still a solid field of 7–16B coders.

Model	Params	~Q4 size	Context	Trade-off
qwen2.5-coder:14b (chosen)	14.8B dense	~9GB	32K	Best dense code quality in the tier; native tool-calling + fill-in-the-middle. Non-reasoning, so it's fast.
qwen2.5-coder:7b	7B dense	~4.7GB	32K	Same family, more headroom/speed, lower ceiling.
qwen3:14b	14.8B dense	~9GB	32K+	Strong generalist that codes well, but not code-specialized.
deepseek-coder-v2:16b	16B MoE (2.4B active)	~8.9GB	160K	Fastest option (MoE) and by far the largest context. Weaker at multi-step tool use.
deepseek-r1-distill-qwen:14b	14B dense (reasoning)	~9GB	32K	Thinks step-by-step. Great for hard debugging, slower for everyday generation.

I picked it because this box is intended to be the worker tier of a two-tier setup: easy tasks run locally, hard ones escalate to externally hosted models. I wanted the fastest, most reliable, tool-capable code specialist that fits. If I need a higher-reasoning model, I reach for an externally hosted 'frontier-class' one.

Pulling

With the model chosen, pulling it is one line:

ollama pull qwen2.5-coder:14b-instruct-q4_K_M

Quick smoke test:

ollama run qwen2.5-coder:14b-instruct-q4_K_M "write a python function that reverses a string"

If it answers, the model runs. However, that does not prove it's using the
GPU. That part is next.

Step 3: Prove it's actually on the GPU (not silently on the CPU)

It's possible for Ollama to discover your GPU, load the weights into VRAM, and still run the compute on CPU.
The logs will look normal, VRAM will show occupied, and the tokens will trickle while your CPU melts.

To verify the GPU compute, I used radeontop. Open it in one terminal like so:

radeontop

Then, in a second terminal, fire off a prompt long enough to sustain load for several seconds:

ollama run qwen2.5-coder:14b-instruct-q4_K_M \
  "write a detailed, heavily commented python implementation of a red-black tree with insert, delete, and search"

A tmux tip: split the terminal (Ctrl-b %), run radeontop on the left and
your ollama run on the right, and you can literally watch the GPU spike as it
"thinks." It's oddly satisfying.

Now watch radeontop while tokens are streaming. What you're looking for:

Metric	Idle (between generations)	Active (mid-generation)
Graphics pipe	~1–2%	jumps high (~93% on mine)
Shader Clock	~0.4%	climbs toward max (~90%)
VRAM	~10.3 / 12.2 GB	same (model stays resident)

The signal is the transition: idle → active → idle, synced to token output.

Note: Watch Graphics pipe and Shader Clock, not VRAM. The weights can be in VRAM while the math runs on the CPU. You have to watch during streaming. The compute drops to zero as soon as generation ends.

On the VRAM numbers: Ollama reports usable VRAM in GiB (it logged ~11.6 GiB available), while radeontop reads out in GB using its own accounting. The two don't line up exactly.

On my box, Graphics pipe hit ~93% and Shader Clock ~90% during generation, then fell to 0% and ~0.11% after. That confirms GPU inference
over Vulkan. If your compute stays flat while tokens are streaming, that indicates CPU fallback, so you'll need to do some debugging.

GPU compute verified.

Step 4: The context-window decision (8K, on purpose)

Ollama picks a default context length for you, but I wanted to set it explicitly
because context costs VRAM. A bigger context window means a bigger KV cache
sitting in memory alongside the weights. On a 12 GB card, that budget is tight, so
I capped the default served context at 8K, which gives ~10.3 GB resident
with headroom to spare (even though Qwen2.5-Coder's native context is 32K).

I set it with a systemd drop-in, so I'm not editing Ollama's packaged unit file:

sudo mkdir -p /etc/systemd/system/ollama.service.d
sudo tee /etc/systemd/system/ollama.service.d/override.conf >/dev/null <<'EOF'
[Service]
Environment="OLLAMA_CONTEXT_LENGTH=8192"
EOF

sudo systemctl daemon-reload
sudo systemctl restart ollama

Verify it took:

systemctl show ollama -p Environment --no-pager   # echoes OLLAMA_CONTEXT_LENGTH=8192

8K is a fine default for most coding prompts. If a specific task needs
more, you don't have to change the service. You can override context per
request:

curl http://localhost:11434/api/generate -d '{
  "model": "qwen2.5-coder:14b-instruct-q4_K_M",
  "prompt": "...",
  "stream": false,
  "options": { "num_ctx": 16384, "temperature": 0.2 }
}'

Raise num_ctx for the one long-context call that needs it. Don't leave it high
globally or you'll pay the VRAM tax on every request.

A couple of other handy per-request knobs:

temperature 0.1–0.3 for deterministic code, higher for brainstorming
keep_alive pass -1 to pin the model resident when you're about to hammer it (by default it unloads after 5 idle minutes and the next request eats a few-second reload).

Ollama also exposes an OpenAI-compatible API at http://localhost:11434/v1.
Any tool compatible with "OpenAI API, custom base
URL" can point at it locally with a dummy key. The server is stateless. It takes
the full message history on every call, so your client holds the conversation.

I also set up zram as a compressed-RAM cushion for the orchestration I plan to stack on top, since 16 GB is tight.

TL;DR

The cliffnotes on setting up your own self-hosted LLM server:

Install an LTS Linux (Ubuntu 24.04/26.04) with a recent kernel for your GPU.
Check your GPU's compute support. AMD: is your gfx target on ROCm's list? If not (e.g. RX 6700-series gfx1031), plan on Vulkan: sudo apt install mesa-vulkan-drivers vulkan-tools radeontop.
Install Ollama: curl -fsSL https://ollama.com/install.sh | sh.
Pull a model that fits your VRAM. 12 GB → a 14B at Q4_K_M (~10 GB). Don't try to cram a 30B into 12 GB.
Verify GPU compute with radeontop during generation. Watch Graphics pipe / Shader Clock, not VRAM. Discovery ≠ compute.
Set the context ceiling with a systemd drop-in (OLLAMA_CONTEXT_LENGTH=8192) to keep the KV cache inside your VRAM budget; override per request with num_ctx when needed.

Next steps

This box is the worker tier of a two-tier setup: the easy work runs here, the hard work escalates to a frontier model. Reaching it from anywhere without exposing anything to the internet is the next post. Whether a quantized 14B is actually good enough to trust is the one after.

Final notes

For now, I am happy to have my own free model running on a mid-level card that I didn't have to pay 2026 prices for. You don't need the latest, most expensive hardware to run a real and useful model at home. All you need is something with enough VRAM. It doesn't even have to be on AMD's supported list.

Go and check out what your idle GPU could be doing instead.