DEV Community

Anton “Admiral” Fishman
Anton “Admiral” Fishman

Posted on

Local LLM generation on a consumer GPU: AirColibri streams Qwen3.5-9B into modest VRAM

I run language models at home on a mid-range NVIDIA card. A 9B-class hybrid model still asks for more memory than a lot of cards have if you load full-precision (or even comfortably quantized) weights all at once. Off-the-shelf servers are excellent when the model fits. I wanted a path that still works when it does not: measure free VRAM, keep as many layers resident as that budget allows, and stream the rest.

The result is a stack for local LLM generation on a consumer GPU. AirColibri is a Python inference engine for a single NVIDIA card. I wanted to run a hybrid Qwen3.5 model locally when the card does not have enough VRAM to keep every layer resident, and to serve it on the LAN through an OpenAI-compatible API with a small web chat.

This article is about how it is put together, what I traded away, and how to try it. The repository is a new public release (mid-November 2026). AirColibri is 1.0.0; the project NOTICE still describes it as an experimental integration, and I treat the quality and speed numbers below as measurements from my own machine, not as a promise about yours.

Layer streaming instead of a bigger card

AirColibri is aimed at Qwen3.5 hybrid models (Gated DeltaNet + attention) on one NVIDIA GPU, with an OpenAI-compatible server for the local network. It combines ideas from AirLLM (layer streaming, selective weight reads) and Colibrì (tiered placement, native MTP speculation) with its own W4A16 weights (Q4, group size 64) and NVRTC CUDA kernels. It is an independent project under Apache-2.0; those names do not imply endorsement.

The design choice I care about most is that standard Hugging Face shards are read directly (model-0000X-of-0000N.safetensors plus model.safetensors.index.json). There is no converter and no second full-size copy of the model on disk. On first start, Q4 weights are built once into data/q4-g64-cache/. The recommended model is Qwen/Qwen3.5-9B; the first download is about 19 GB. You can also point --model at a local folder, or drop a supported folder into models/. Unsupported layouts (the README mentions OLMoE) are skipped. Colibrì per-layer containers still work.

VRAM planning is measured, not profiled. At start the engine looks at free VRAM. --vram-mode normal (the default) uses almost all of it, but keeps a reserve for the OS, the KV / DeltaNet state of --max-context tokens, and the prefill workspace, so a long prompt is less likely to OOM. --vram-mode turbo keeps only a minimal reserve. If the whole model fits, every layer stays resident. If it does not, the remaining budget is split into equal slots that rotate roles — computing, cleaning, receiving the next layer's weights — so the next layer is already on the GPU when the current one finishes, and as many layers as fit stay resident. Prompts are prefilled in chunks (--prefill-chunk, default 512 tokens), and only the last position is projected onto the 248k-token vocabulary.

The LAN server binds web chat and the API on port 8787, with a UDP beacon on 8788:

  • POST /v1/chat/completions — OpenAI-compatible, JSON or stream: true SSE (reasoning_content for <think>)
  • POST /v1/chat/stop — stops the running generation immediately
  • GET /v1/models, GET /api/info — model list and node status
  • GET / — the built-in web chat

Only one generation runs at a time. A new request, or a client that drops its connection (including during prefill), cancels the current one and frees VRAM before the next starts. --no-preempt queues instead. Bodies larger than --max-request-bytes (default 8 MiB) get HTTP 413. The server is open on the LAN by default; --require-auth turns on a Bearer API key. Logs go to logs/server.log (rotating, 10 MB × 5).

Tradeoffs, including numbers I actually measured. Q4 is lossy. Against the original weights, Q4 (W4A16, group 64) logits had a cosine similarity of about 0.987–0.993 in my checks. "Bit-exact" applies only to the fused kernels versus the reference Q4 computation, not versus BF16. MTP speculation does not change greedy output: with greedy decoding the text is identical with and without MTP (tests/test_engine_gpu.py covers this).

On an RTX 3060 Ti 8 GB, Qwen3.5-9B, all layers resident, 400-token answers, one answer at a time, I measured about 31 tok/s without MTP, and 30–37 tok/s with MTP, depending on how many drafts were accepted (43–71 % in those runs; on new text MTP may give no gain). That is not a throughput number for several parallel requests, and I am not quoting one. --fast-vocab is experimental and off by default; it can drop rare tokens on long outputs.

The runtime I document is narrower than "any Python box": NVIDIA GPU with CUDA, Python 3.12, PyTorch 2.6 with CUDA 12.4, Windows x64 or Linux. I tested on the 3060 Ti 8 GB; the README states that cards with around 4 GB stream part of the layers.

Install and run. There is no PyPI package. On Windows, aircolibri.cmd picks .venv automatically; on any OS you can run python aircolibri.py with the same subcommands. The README's virtualenv paths are Windows-style; on Linux, use .venv/bin/python in place of .venv/Scripts/python.

git clone https://github.com/DIIIUA/air-colibri.git
cd air-colibri
python -m venv .venv
.venv/Scripts/python -m pip install torch==2.6.0 --index-url https://download.pytorch.org/whl/cu124
.venv/Scripts/python -m pip install -r requirements.txt

# Windows: aircolibri.cmd
# Any OS:  python aircolibri.py …
aircolibri doctor --model Qwen/Qwen3.5-9B   # downloads once (~19 GB) and checks GPU, cache, LAN addresses
aircolibri ask "Who are you?"
aircolibri chat                             # Ctrl+C stops the current answer
aircolibri serve                            # web chat + OpenAI API on :8787
aircolibri discover                         # find servers on the LAN
Enter fullscreen mode Exit fullscreen mode

CPU tests: python -m pytest tests -q. The GPU engine tests and the model-dependent Q4 tests run only when a CUDA GPU and a local model are present.

If you already use an OpenAI-compatible client, point it at http://<lan-ip>:8787. If the server is reachable without --require-auth, you should treat the LAN as trusted.

What is still rough

AirColibri 1.0.0 is a public release of a deliberately experimental stack: Q4 quantization, layer streaming, and a single-flight server. It will not match a fully resident, less lossy runtime on a card that can hold the model comfortably. It is CUDA-only. The first model download is large. I have not published numbers for parallel request throughput, and I have not claimed results on GPUs I have not measured.

I also will not invent a head-to-head against llama.cpp, vLLM, or Ollama. If your model already fits in one of those, use them.

What I would like to hear back

Quality of Q4 Qwen3.5-9B on real tasks, not just logit cosine; behaviour on cards around 4 GB, where layers have to stream; MTP accept rates on your prompts; and whether POST /v1/chat/completions and /v1/chat/stop behave correctly in the clients you use.

Issues and pull requests are welcome. Please do not open a PR that includes API keys, hostnames, or device identifiers.

Links

I am Anton (DIIIUA). If you try this on your hardware, I am more interested in a failed setup than in a compliment.

Top comments (0)