Cliff K

Posted on Oct 31 Edited on Nov 3 AI-assisted

I let my coding agent tune my local LLMs: 3x faster prompts on an 8 GB AMD GPU

#llamacpp #localllm #performance #opensource

Update (Nov 3, v0.2.0): the project is now called local-llm-speedtune. It was tune-my-llm, then llamacpp-speedtune, and old links redirect. It now picks the best inference engine first, then the backend, then the flags. It asks all its questions up front, gives you a finish time, runs while you're away, and saves each model's best config. It also has a 2-minute check, export to Ollama and community results. Details are in the skill section below.

I run a 35B MoE model (Qwen3.6-35B-A3B) as the main model for OpenCode, on a very ordinary PC: Ryzen 7 5700X, 32 GB DDR4, and a Radeon RX 6600 XT with 8 GB of VRAM. It worked, but long sessions felt sluggish, and every guide I read said the same thing: on AMD, just use llama.cpp with Vulkan.

So I asked my coding agent to verify that, and to find the best config for every model I have. It took a day of experiments. The headline result, measured at the point where OpenCode compacts its context:

Qwen3.6-35B-A3B at 49K tokens	Backend	Key flags	Prompt reading	Generation
Before	Vulkan	`--n-cpu-moe 33`, `-ub 512`, q8_0 cache	172 t/s	19.2 t/s
After	HIP (ROCm)	`--n-cpu-moe 34`, `-ub 2048`, f16 cache, `-t 8`	526–562 t/s	24–25 t/s

Re-reading a full 49.5K context went from ~4.8 minutes to ~95 seconds. On six real OpenCode tasks the tuned config passed 6/6 (the old one 5/6), and tasks with long prompts finished about 2× faster.

Then I packaged the whole process as an open-source agent skill, local-llm-speedtune, so your agent can do the same on your hardware.

Why "tok/s on an empty prompt" is the wrong benchmark for agents

A coding agent isn't a chatbot. Every turn, OpenCode resends ~12K tokens of system prompt and tool definitions plus the whole conversation. With a 64K context it auto-compacts at roughly:

usable = ctx − max(0.15 · ctx, 16000)   →   ~49.5K at 64K

So the model spends its life between ~12K and ~49.5K tokens deep. A config that wins at depth 0 can lose badly at 48K, and prompt-reading speed at depth matters as much as generation speed. Measurements were taken at the depths the agent actually works at (16K–48K), not just at 0, then confirmed in a real llama-server at the full 64K.

(One side effect: at 32K context OpenCode reserves 16K, which leaves a 32K model only ~4.7K tokens of working room after the system prompt. Context size choices look very different once you do this maths.)

What I learned (some of it surprised me)

1. "Vulkan is best on AMD" was only half true. With a fair comparison (same llama.cpp commit, flash attention on):

Qwen3.6 MoE was 47–55% faster at reading and 22–35% faster at generating on HIP/ROCm.
Dense 7B/3B models generated up to 28% faster on Vulkan.
Gemma-4 12B only fit on Vulkan at all. HIP wanted 1.1 GB more compute buffer, and ROCm can't spill to RAM.

The fix: build llama.cpp with both backends (-DGGML_HIP=ON -DGGML_VULKAN=ON). --list-devices then shows ROCm0 and Vulkan0, and in router mode each preset picks its own:

[qwen3.6-35b-a3b]
device = ROCm0
n-cpu-moe = 34
ubatch-size = 2048
cache-type-k = f16
cache-type-v = f16

[gemma-4-12b]
device = Vulkan0
ubatch-size = 1024
cache-type-k = q8_0
cache-type-v = q8_0
no-mmproj-offload = true

One server, one port, and every model on its best backend. Switching models in OpenCode switches the whole config.

2. -ub 2048 nearly doubled prompt speed for the MoE with experts in RAM (328 → 599 t/s at 16K). Dense models fully on the GPU barely cared.

3. q8_0 KV cache isn't free. On HIP, generation was 10% slower than with f16 at 16K and 23% slower at 45K (20.0 vs 26.0 t/s). Use f16 wherever it fits, and treat q8_0 as the floor (not q4) for tool calling.

4. SMT threads hurt. -t 16 on an 8-core/16-thread CPU halved MoE generation (27.9 → 12.7 t/s). Anything from 4 to 12 threads performed the same.

5. llama-bench doesn't load vision encoders. Gemma looked fine in benchmarks, but behind the real server, with its mmproj on the GPU, it spilled past 8 GB: 121 t/s reading at 28K. With --no-mmproj-offload it read at 212 t/s, and an image still takes only ~11 s. Always validate in the real server.

6. The draft model made it slower. Gemma's own assistant draft (--spec-type draft-mtp) halved generation speed (26 → 12–16 t/s, ~45% acceptance). N-gram speculation did nothing measurable either. Measure before you enable speculation.

7. OpenCode's title requests evict your conversation. Every few turns OpenCode sends a tiny "name this conversation" prompt, which pushes the long conversation out of a single-slot server's KV cache. llama-server's RAM prompt cache (--cache-ram, on by default) restores it: disabling it cost 7% and 46K extra tokens on a replayed 49-turn session. Leave it on.

8. No flag fixes a model that can't drive the agent. On the same six tasks: Gemma-4 12B passed 5/6, Qwen2.5-7B passed 2/6, and Qwen2.5-Coder-3B passed 0/6, without making a single tool call. Check this before spending hours tuning a model.

Testing agents headless is full of traps

Running OpenCode non-interactively in a benchmark harness broke in creative ways:

opencode run waits forever on a non-TTY stdin → pass an empty stdin.
OpenCode takes its working directory from $PWD, not the process cwd, so my first runs edited files in the benchmark folder → set PWD explicitly.
OpenCode identifies projects by git root commit, so copies of one repo collided → fresh git init per run.
pkill -f <pattern> matched (and killed) the agent's own shell → kill by PID.

These are all fixed in the scripts and documented in the repo's gotchas.md.

The skill: let your agent do this for you

local-llm-speedtune is an Agent Skill plus scripts. It picks the best engine, then the best backend, then the best flags for your hardware.

llama.cpp is the default engine. It runs on CUDA, HIP/ROCm, Vulkan, Metal or CPU, and it can split a model between GPU and RAM. The skill tries another engine only where the hardware gives it a real chance: ExLlama on NVIDIA when the model fits in VRAM, MLX on Apple Silicon, ik_llama.cpp for MoE models with experts in RAM, OpenVINO on Intel. A challenger has to win by measurement: at least 10% faster, at the same bits per weight, passing the same agent tasks. The agent may research what's current for your hardware online, but that research only proposes candidates. After the Vulkan lesson above, only measurements on your machine decide.

How a tune runs

Inventory (~2 min, read-only): GPU, VRAM, cores, RAM speed and channels, the real PCIe link (AMD cards report a misleading x16), your models and whether they fit, the engines you already have, and your current settings.
One round of questions, with what it found filled in:
- what you use the models for
- which models to tune
- context size
- just you, or several users at once (if several won't fit, it says so in numbers)
- what matters most
- how long you can be away
- what it may do while you're away: build backends, install engines, download up to N GB, stop your model server, keep the computer awake

You can reply "all defaults" to most of them.

A plan with a finish time, e.g. "3 models, about 1 h 40 min, done around 21:40". Then you can leave.
The unattended run: your current config is measured first, then engines, backends, flags and agent tasks. It resumes where it stopped if interrupted, never uses sudo, and never changes your live config while you're away.
Results:
- before/after for every model
- each model's best config saved as a profile
- one endpoint for all of them (router presets, or llama-swap if models ended up on different engines)

You get a notification when it's done. The new config is applied only when you're back and say yes.

Start with the 2-minute check

Before spending any GPU time, find out whether tuning is worth it. The check reads your current settings and hardware and points out likely problems. It needs no config, runs no benchmarks and changes nothing:

git clone https://github.com/cliffkoome/local-llm-speedtune && cd local-llm-speedtune/scripts
./quick_check.py --router http://127.0.0.1:8080      # a llama-server router
./quick_check.py --preset ~/models/presets.ini       # or a presets file

== qwen3.6-35b-a3b  (qwen35moe, MoE 8/256 experts, 40 layers, 20.6 GB)
   [big   ] Small batch size with experts in RAM
            fix: ubatch-size = 2048 and batch-size = 2048, if VRAM allows

I ran it on my original, untuned presets file. It flagged the problems the tune later fixed: batch size 512 with experts in RAM, GPU layers left to --fit, and every model on the same backend. On my tuned router it says all four models look tuned.

Three modes

	Check	Quick tune	Comprehensive tune
Time	~2 min, GPU stays free	25–45 min per model	1.5–3 h per model (run it overnight)
Does	Reads current flags, rule-of-thumb findings, changes nothing	Uses what's installed; key sweeps at your agent's max context; fit check; 2 agent tasks	Everything: challenger engines, every backend, all sweeps at 3–4 depths, speculative decoding, session replay, 6 agent tasks

One prompt to install and run it:

Install the agent skill from https://github.com/cliffkoome/local-llm-speedtune (clone it into your skills folder, or into
~/local-llm-speedtune and read its SKILL.md if you don't support skills). Then use it to tune my local models.
Check my hardware first, then ask me all your questions in one go and tell me when it will be done.
After that, run it unattended to the end. Don't change my live server or agent config until I'm back and say yes.

Run the tune from an agent that isn't using the models being tuned. A tune needs the whole GPU, so it unloads your live server's models. If your agent runs on one of them (OpenCode on local Qwen, in my case), the tune would stop the agent itself. Use an agent on a cloud model, or run the scripts yourself. The skill checks for this and stops rather than unloading its own model.

Using Ollama?

Ollama runs llama.cpp underneath but exposes fewer settings. The 2-minute check also reads Ollama's settings (--ollama). If you're open to switching, the tune measures your Ollama setup as the baseline. If you want to stay on Ollama, to_ollama.py turns the winning config into a Modelfile and service settings (num_ctx, num_gpu, OLLAMA_KV_CACHE_TYPE, …). It also tells you what Ollama can't do, such as moving only the experts of an MoE model to RAM, or choosing a backend per model.

Share your numbers

There's a community results table and a short share-your-results form. After a tune, the skill offers to fill the form in for you. It never posts anything itself.

Honest limits: the full workflow has only been run end to end on Linux with an AMD GPU and llama.cpp. The NVIDIA, Apple and Intel paths and the challenger engines follow each project's docs but haven't been run on real hardware yet. If you try them, results or an issue would help a lot, especially one where a challenger beats llama.cpp. The agent tasks currently support OpenCode, and adding another agent is one adapter function. Each agentic task was run once, so treat differences under ~20% on short tasks as noise.

DEV Community