Can a 27B Coding Model Be Useful on a 16GB GPU?

arczhi — Thu, 03 Sep 2026 02:49:53 +0000

Repository with scripts and benchmark notes:

https://github.com/arczhi/5060ti-qwen3.8-27b

I wanted to answer a practical question:

Can a local coding agent actually finish work on a 16GB GPU, or will it spend most of its time reading code and never produce a reliable patch?

The test machine was:

AMD Ryzen 9 9950X, 16 cores / 32 threads
NVIDIA RTX 5060 Ti with 16,311 MiB VRAM
32GB DDR5-6000 system RAM
NVIDIA driver 595.71.05
CUDA 13.2
Linux x86_64
Docker and llama.cpp CUDA server

The model was Qwen3.8-27B GGUF using UD-IQ4_XS quantization.

The tested configuration used an 80K context window, MTP-1, Q4_0 KV cache, Flash Attention, one parallel slot, disabled reasoning, and four FFN layers offloaded to the CPU.

The agent test was more important than the short benchmark. I gave the agent a design document and asked it to implement a feature in a medium-sized Go repository.

It took about 14m20s to:

Read the design document
Modify seven files
Add five tests
Complete scoped build, vet, formatting, and diff checks

Two failing checks were confirmed to be pre-existing baseline failures.

Across 14 server requests, the measured results were:

Prefill: 22.18-326.36 tok/s
Decode: 11.06-12.95 tok/s
Generation time: 7.89-184.60 seconds per request
MTP acceptance: 84.9%-100%

The short smoke test was much faster, reaching 86.54 tok/s prefill and 17.77 tok/s decode. However, the longest request still took 184.6 seconds.

I also compared CPU FFN offload:

No CPU FFN offload: 14m20s, 11.06-12.95 tok/s decode
Four CPU FFN layers: 14m04s, 19.40 tok/s weighted decode
Eight CPU FFN layers: stopped after 25m30s, 18.36 tok/s weighted decode

In this real coding-agent run, FFN4 was the best balance. It reduced GPU pressure without adding as much CPU/GPU transfer overhead as the larger offload configuration.

The main lesson is that tokens per second are only part of the story. For coding agents, I care more about:

Whether the task was completed
Tests passed
Total wall-clock time
Number of agent turns
Context compactions
Final repository state

This benchmark comes from one machine and one runtime configuration. The goal is reproducibility and useful comparison, not a universal performance claim.

DEV Community: arczhi

Can a 27B Coding Model Be Useful on a 16GB GPU?