Self-Hosting a Terminal AI Coding Assistant (Ollama + Tailscale + Qwen Code)

A few weeks ago our team of about ten engineers decided we wanted the "AI pair programmer in the terminal" experience without paying per-seat API costs for every developer, and without sending our codebase to a third party for every single query.

The answer we landed on: run an open model (Qwen) on a single shared GPU box we already owned, and have everyone's laptop talk to it over a private network. No cloud inference bill, no code leaving our own hardware, one box serving the whole team.

This post is the "why we built it, what broke, what we learned" writeup.
The step-by-step setup instructions are at the bottom if you want to replicate it.

Why we did it this way

We had a GPU workstation sitting mostly idle. Cloud-hosted coding assistants are billed per user or per token, which adds up fast for a team our size, and some of our repos aren't things we want routed through someone else's API regardless of their data policies. Self-hosting the model on hardware we control solved both problems at once - the only recurring cost is electricity.

The architecture is simple on purpose:

Laptop (Qwen Code CLI) → Tailscale mesh → GPU box (Ollama, serving Qwen)

• One GPU box runs Ollama serving a Qwen model.
• Tailscale creates a private mesh network so every laptop can reach the box securely, without opening anything to the public internet or dealing with VPN configs.
• Qwen Code runs locally on each developer's laptop, pointed at the shared box via an OpenAI-compatible endpoint. It only ever sees the developer's own local repo - the GPU box never touches anyone's files, it just does the "thinking."

Each person still works in their own repo, with their own git credentials, on their own machine. The shared box is purely inference compute - one model loaded, serving all ten of us, handling up to a handful of requests in parallel.

What broke, and what we learned

Everyone hammering the model at once causes silent queuing, not errors.
Ollama on that box will serve about four requests in parallel, but once you've got all ten of us firing off prompts around the same time, some of them simply queue and wait rather than failing loudly. The first time this happened, a couple of people assumed something was broken and started restarting things, when really the fix was just "wait a few seconds." The lesson: document the expected behavior up front so people don't chase a phantom bug. We now tell new teammates explicitly - if it's slow, someone else is probably using it, just wait it out.

Swapping models evicts everyone else. Ollama keeps one model loaded in GPU memory at a time. Someone loading a second model for a quick experiment silently kicks the shared model out of memory, and the next request from anyone else on the team pays a 30-60 second reload penalty (or worse, hits a model that no longer matches what their tooling expects). We made "one model, for everyone, don't swap it without warning the team" an explicit rule after this happened more than once.

Reboots need a defined recovery order. GPU boxes get restarted - for updates, power blips, whatever. The services (networking, the inference server, the firewall) need to come up in a sane order and take a minute or two to settle before the box is actually ready to answer requests. Rather than have people ad hoc SSH in and poke around every time, we wrote down a short, repeatable checklist: wait for boot, run a one-line connectivity test, and if that fails, SSH in and check the three services in a fixed order. That turned a "is it just me?" Slack thread into a 90-second, no-thinking fix.

A one-line health check is worth more than any dashboard. The single most useful thing we did was standardize on one curl command that anyone can run from their laptop to answer "is the shared box reachable and healthy right now?" It's the first troubleshooting step for basically every problem, and it means most issues get diagnosed without ever involving whoever manages the box.

The setup, if you want to do this yourself

Below is the cleaned-up version of the internal guide we hand new teammates.
Replace the placeholder values with your own.

What you need

• A machine with a GPU that can stay on and reachable (this is your "inference box")
• Tailscale (or any private mesh VPN) connecting that box to every developer's laptop
• Ollama running on the inference box, serving your model of choice
• Qwen Code installed on each developer's laptop

One-time laptop setup (Windows example)

• Install Tailscale from tailscale.com/download, log in with your team's shared account via the system tray icon.

• Verify you can reach the inference box:
  

   curl http://<your-tailscale-ip>:11434/v1/models

You should get back JSON listing the available model(s). If it hangs, check that Tailscale shows "Connected" and reconnect if not.

• Install Node.js LTS from nodejs.org if you don't already have node -v reporting v20 or higher.

• Install Qwen Code:
  

   npm install -g @qwen-code/qwen-code

• Point it at your shared box by setting three environment variables:

   [System.Environment]::SetEnvironmentVariable("OPENAI_API_KEY", "ollama", "User")
   [System.Environment]::SetEnvironmentVariable("OPENAI_BASE_URL", "http://<your-tailscale-ip>:11434/v1", "User")
   [System.Environment]::SetEnvironmentVariable("OPENAI_MODEL", "<your-model-name>", "User")

Close and reopen PowerShell, then confirm with echo $env:OPENAI_BASE_URL.

• Use it:

   cd C:\path\to\your-repo
   qwen

Ask it things like "explain this codebase" or "find the bug in auth.py."
Git works exactly as normal - commit and push with your own credentials.

Day-to-day (nothing to redo after a laptop restart)

Tailscale, the environment variables, and the CLI install all persist across reboots. Just cd into a repo and run qwen.

If it can't connect, the one-line health check above is your first move.
If that returns JSON, the problem is local - restart your terminal. If it hangs, the shared box itself may be down.

After the inference box restarts

1. Give it a minute or two to fully boot all services.
2. Run the same health check from any laptop.
3. If it responds, you're good - the very first query will be slow (~30-60 seconds) while the model loads into GPU memory, then it's fast again for everyone.
4. If it doesn't respond, SSH in and check services in order: the mesh network, the inference server, and the firewall - starting whichever one isn't running.

The rules that keep it running smoothly for a shared team

• One model, agreed by the whole team. Loading a second model evicts the first and stalls everyone.
• Expect queuing under load. The box serves a limited number of requests in parallel - simultaneous use from the whole team means some requests wait a bit, that's normal, not a failure.
• Don't change server settings solo. Whoever manages the box tunes things like the firewall and server config for the whole team; changes should be communicated first.

If you're on a team weighing "pay for hosted per-seat AI coding tools" against "stand up something on hardware we already have," this has worked well enough for us that I'd recommend giving it a try - just go in expecting to write down your own version of the "what to do when it breaks" checklist early, rather than after the third confused Slack thread.