Running a local LLM is easy.
Running multiple specialized models for coding, reasoning and AI agents on a 12GB GPU is where things get interesting.
My local AI machine is fairly ordinary:
- NVIDIA RTX 3060 — 12GB VRAM
- AMD Ryzen 7 5800X
- 64GB RAM
- Ubuntu Linux
Modern 7B–12B models run surprisingly well on this hardware.
But my workflow doesn’t need just one model.
I might want one model for coding, another for reasoning, one verified for tool calling, one for embedding and a last one for reranking while another I’m experimenting with.
My GPU can run them.
It just can’t keep all of them loaded at once.
This Became a Scheduling Problem
Initially I was thinking about inference. Which model? Which quantization? GGUF or SafeTensors? llama.cpp or vLLM?
But the bigger problem turned out to be scheduling: with only 12GB, deciding which model gets the GPU right now, and getting out of the way of the one that's leaving.
Which model? Which quantization? GGUF or SafeTensors? llama.cpp or vLLM?
But eventually the bigger problem became:
flowchart LR
A["Request<br/>requested model"] --> B{"Requested model<br/>already loaded?"}
B -->|Yes| G["Serve request"]
B -->|No| C["Drain<br/>current model"]
C --> D["Unload<br/>current model"]
D --> E["Free<br/>GPU / VRAM"]
E --> F["Load<br/>requested model"]
F --> GI wanted the request to determine which model occupies the GPU.
And I didn’t want every application to understand my inference infrastructure.
Instead of:
Coding model → :8081
Reasoning model → :8082
vLLM model → :8000
Other model → :8083
I wanted:
flowchart TB
OC["opencode"]
AG["AI Agents"]
SDK["Python / Node SDKs"]
APP["Applications"]
OC --> API
AG --> API
SDK --> API
APP --> API
API["One OpenAI-Compatible API<br/>:9090"]
API --> SW["GGUF Switchboard<br/>Model Scheduler"]
SW --> LC["llama.cpp"]
SW --> VL["vLLM"]
LC --> GG["GGUF Models"]
VL --> ST["SafeTensors Models"]When a request comes in for a model that isn't resident, Switchboard drains the current model — it stops accepting new requests and lets in-flight ones finish — then unloads it, waits for VRAM to actually free, and loads the requested model before serving. On my 3060 a swap between 7–12B GGUF models costs a few seconds of cold-load latency, so the scheduler batches consecutive requests for the same model and only pays that cost on an actual switch.
The clients should be boring.
The infrastructure should handle the complexity.
Why Not Ollama or LM Studio?
Both are excellent tools.
Ollama makes downloading and running local models incredibly easy.
LM Studio is excellent for discovering, configuring and experimenting with them.
My requirement was slightly different.
I wanted to treat multiple local models as shared infrastructure, especially on machines where VRAM is constrained.
I also didn’t want to choose one inference ecosystem.
On my 3060, I often prefer:
GGUF → llama.cpp
On larger GPU systems, I may want:
SafeTensors → vLLM
The application shouldn’t care.
It should ask for a model and let the infrastructure determine how to serve it.
So I Built GGUF Switchboard
That experiment became GGUF Switchboard.
It’s an open-source model scheduler that provides one OpenAI-compatible endpoint while managing local model lifecycles underneath it.
Some of the problems I’m trying to solve:
- hardware-aware model discovery
- GGUF + SafeTensors support
- llama.cpp + vLLM backends
- request-driven model switching
- drain/unload/load lifecycle
- GPU resource management
- tool-call conformance testing for AI agents
For example:
ggs models search "qwen"
Instead of just searching for Qwen models, Switchboard considers the hardware it’s running on so I can answer the question I actually care about:
What makes sense on this machine?
The Goal
The simplest way I can describe the project is:
Ollama and LM Studio make running a local model easy. GGUF Switchboard is trying to make operating a collection of local models boring.
For constrained GPUs, that’s exactly what I wanted.
Try It
GGUF Switchboard is open source: GitHub — GGUF Switchboard
Demos, screenshots, architecture and documentation: GGUF Switchboard Product Page
For the deeper technical write-up: Read the full article on my blog
I’ve also published a shorter version on Medium: Running Local AI Coding Models on an RTX 3060 12GB
I’m Looking for Real-World Configurations
If you’re running local LLMs on an 8–16GB GPU, I’d like to know what actually works for you day-to-day:
- GPU
- model + quantization
- llama.cpp / Ollama / LM Studio / vLLM
- coding agent
- tool-calling reliability
If GGUF Switchboard looks useful, try it and star the repo.
More importantly, open an issue if your hardware/model combination doesn’t work.
Real-world configurations are exactly what I want to use to improve the project.
Top comments (2)
The biggest headache with dynamic swapping on a 12GB card is context thrashing when an agent alternates between a small tool router and an 8B coder. If you rely on mmap with llama.cpp server, warm swaps are fast, but memory fragmentation builds up if the previous runner process does not cleanly release before the next subprocess claims VRAM. Adding a hard SIGTERM fallback with a two-second drain window solved most of the zombie allocation hangs in my setup.
@reidmarlow
Short answer: Yes, it handles this.
that's basically what the force-kill path does; default drain is Ns, configurable in
config.tomlswitch_drain_timeout_secs = 120Default is 120 seconds. Set it lower for your 12GB card — something like 5 or 10 is reasonable if your requests are short-lived. The example in config.example.toml:25 shows the same 120s default.
When you switch between models (like going from a small tool router to an 8B coder), gguf-switchboard:
This prevents the "zombie memory" problem where the old model is technically gone but still hogging GPU memory.
One thing to know: It watches your system's overall memory, not your GPU's memory directly. So it's good at catching big problems, but it won't detect subtle GPU memory fragmentation.
For a 12GB card setup, this should work well. If you want faster switches, you can shorten the wait time in your config file.