A specialist fork of llama.cpp is making AMD's decade-old GFX906 cards usable for local language-model inference, which matters because those cards sell used for roughly $200 with 16 gigabytes of memory each. The project ships hand-written kernels under its own directory tree, built on llama.cpp build 7924 and tested on the Instinct MI50 and MI60. Upstream llama.cpp contributors are now discussing porting the work back into the mainline, and their testing has already found a tuning parameter in the fork that was leaving significant speed on the table.
Key facts
- Targets AMD GFX906 hardware, specifically the MI50 and MI60, based on llama.cpp build 7924.
- Ships custom kernels for flash-attention, RoPE, and matrix multiplication organized around Q4_0, Q4_1, and Q8_0 quantization formats.
- Active release trail across at least seven tagged releases, plus overclocking and power-scaling scripts.
- Primary source: the llama.cpp-gfx906 repository, with upstream discussion at llama.cpp discussion 23881.
The economics are the whole reason this exists. Current eBay listings put MI50 16GB cards roughly in the $198 to $230 range and Radeon VII 16GB cards roughly $198 to $320, with at least one Radeon VII sold at $179. These are datacenter and prosumer cards from a generation AMD's current software stack has largely moved past. They have real memory bandwidth and real capacity, and because LLM inference is memory-bound rather than compute-bound for single-user generation, memory bandwidth is most of what you need. What they lack is anyone maintaining kernels for them.
The fork's author frames it in exactly those terms, describing the first release as optimizing llama.cpp for "our beloved cheap and slow videocards." That is the honest version of what this is: not a claim that MI50s are secretly good, but that a card with 16 gigabytes at $200 is worth writing code for even if the code has to be written by hand. A GPU kernel is essentially a recipe for how to split a mathematical operation across thousands of tiny parallel workers, and the right split depends heavily on the chip's specific memory layout and instruction set. Modern kernels are tuned for modern architectures. Run them on an older design and the work still happens, just badly, with workers idling and memory accessed in patterns the hardware handles poorly. Retuning is unglamorous, high-skill labor with no commercial payoff, which is why it usually does not happen.
The upstream discussion is the most interesting artifact here, because it shows the retuning is not finished. A contributor investigating a port back to mainline llama.cpp reported that the fork's thread-group setting of 2 was actively hurting prompt-processing speed, and that raising it produced major gains. A later reply complicated that: raising it helped one quantization format and regressed another on a Radeon VII. That is what real hardware tuning looks like, and it is a reminder that a single number in a kernel configuration can be worth a large fraction of your throughput on the specific chip in front of you. This is quantization meeting hardware reality, where the format you chose changes which kernel configuration is optimal.
The broader trend this sits inside is the steady expansion of what hardware can run useful models locally. Ground Truth has covered two-bit models running on every major llama.cpp backend, a 753-billion-parameter model running on a single workstation GPU, and FastFlowLM bringing inference to AMD NPUs. The direction is consistent: the floor for running capable models on hardware you own keeps dropping, and much of the work is being done by individuals rather than vendors.
The honest caveat is about what is and is not verifiable from the repository itself. The README documents which custom kernel files exist and the release notes describe the optimization categories, but the specific throughput numbers circulating in summaries of this project are not exposed on the accessible repository pages, and neither are the more granular claims about particular instructions or forced precision modes. There is also no blanket statement in the README that only certain model shapes are supported, despite that claim appearing in secondary write-ups. What can be said with confidence: the fork is real, actively released, organized around specific quantization paths, has drawn serious upstream engagement, and is aimed squarely at making $200 cards do useful work. If you own GFX906 hardware, that is enough to justify trying it. If you are shopping based on someone's quoted tokens per second, get the number yourself.
Originally published on Ground Truth, where every claim is checked against the primary source.
Top comments (0)