DEV Community

Cover image for Running a 76 GB MoE model with 262K context on a 16 GB RX 6900 XT
Amuro Ray
Amuro Ray

Posted on

Running a 76 GB MoE model with 262K context on a 16 GB RX 6900 XT

A 76 GB mixture-of-experts model should not run on a 16 GB graphics card. Strata, a new inference engine, can run one on a 2020 Radeon RX 6900 XT (RDNA2) with the full 262,144-token context, and I measured 49 to 66 tokens/s of generation. I only tested speed: this is what I measured, which settings mattered, and why running several requests at once gained almost nothing.

Measurements, scripts and raw results are on GitHub: local-llm-benchmarks, in the strata folder.

The short version

  • Decode speed holds up with context. 65.7 tokens/s on a short prompt, 54.9 at 32K tokens, 48.8 at 256K.
  • Reading a long prompt is the cost. A fresh 256K prompt takes 5.8 minutes before the first token (about 740 tokens/s). A follow-up that keeps the same start took 0.6 s when it added 35 tokens.
  • The token predictor was the biggest single gain. Turning it off cut decode from 56.2 to 37.8 tokens/s on short prompts (a 49% gain with it on) and from 48.5 to 35.1 at 256K.
  • Fitting more of the model on the GPU beat keeping the whole cache there. Leaving the older part of the cache in system RAM freed room for about 70% more experts on the GPU and gave about 19% faster decode at long context.
  • Several requests at once do not help. Total decode went from about 58 tokens/s with one request to at most 69 with four or eight (my guess at why is below).

What I ran

  • Machine: Radeon RX 6900 XT (16 GB), Ryzen 9 7900X, 64 GB DDR5, Windows 11.
  • Model: Qwen3.8-Flash-Next, a mixture-of-experts model that uses 10 of 512 experts per token. I used a community build (SC117's GSQ-RCO "abliterated" variant) at the IQ3_XXS quantization tier, 76 GB across two GGUF files.
  • Engine: Strata, versions 0.1.55 and 0.1.56. It keeps the experts in system RAM and moves the ones it needs onto the GPU, and it can keep the older part of the KV cache in RAM too.
  • Not a llama.cpp run. Everything in this post uses Strata.

Strata's release notes say its RDNA2 package was built but not run on a real RX 6000 card. Everything below ran without errors on mine, but I would still call this card experimental for this engine.

The configuration I used for the main runs:

setting what it does
262,144-token context the full window the model supports
--kv int8 same prefill speed as fp16 and faster decode at long context, in half the memory
--kv-resident 32768 keeps the most recent 32K of cache on the GPU and the rest in RAM, leaving room for experts
--vram-reserve-mib 1400 leaves VRAM for Windows and the monitors
STRATA_HIP_PROMPT_F16=1 about 55% faster prefill on this card (392 to 610 tokens/s at 32K)
--mtp ... --spec 4 --spec-min-p 0.5 the token predictor: it proposes up to 4 tokens and the main model checks them
--adapt-async 1 expert swaps run between check windows instead of blocking them

Speed against context length

Version 0.1.56 with the configuration above, 512 generated tokens per run:

prompt length prefill decode time to first token
short (about 80 tokens) n/a 65.7 tokens/s 2.1 s
32K tokens 685 tokens/s 54.9 tokens/s 47 s
128K tokens 750 tokens/s 51.1 tokens/s 170 s
256K tokens 740 tokens/s 48.8 tokens/s 5.8 minutes

Decode drops about 26% from a short prompt to 256K, which is much gentler than the prompt-reading cost. A recall test also passed at every size from 8K to 256K: I buried three facts in the text and the model found all three each time. That is a sanity check that the long context works, not a quality benchmark.

The cache matters for anything you ask twice. A follow-up that keeps the first 256K tokens identical took 0.6 s when it added 35 new tokens. If the text near the top changes, even a timestamp, the whole prompt is read again.

What mattered, and what did not

These runs used version 0.1.55, with 32K and 256K prompts, medians of 2 or 3 runs up to 64K and single runs above:

setup decode, short decode, 32K decode, 256K first token, 256K
int8 cache, older part in RAM (main config) 56.2 49.4 48.5 6.2 min
fp16 cache, older part in RAM 54.9 48.2 43.3 6.2 min
int8 cache, all in VRAM 52.3 41.5 40.6 10.0 min
fp16 cache, all in VRAM 42.3 38.8 33.8 44.9 min
int8, in RAM, predictor off 37.8 35.5 35.1 8.0 min

Decode in tokens/s.

  • Keep the cache small and the experts on the GPU. Putting the whole cache in VRAM left room for only 2,406 experts, against 4,138 when most of it lived in RAM. That cost about 16% of decode speed at 32K and 256K, and prefill at 32K fell from 706 to 434 tokens/s. With an fp16 cache all in VRAM it was far worse: 97 tokens/s of prefill, which makes a 256K prompt take 45 minutes.
  • int8 cache over fp16. Prefill was the same, and decode at 256K was 12% faster (48.5 vs 43.3 tokens/s) with half the cache memory.
  • The token predictor. Off, decode fell to 37.8 tokens/s on short prompts, a 49% gap, and prefill was lower as well. The engine still runs a simpler drafter in that case, so this measures the predictor, not speculation as a whole. I did not test why prefill dropped.
  • Async expert swaps (0.1.56). On version 0.1.56 these took short-answer decode from 60 to 61 up to 66 to 68 tokens/s, and 32K from 46 to 48 up to 54. The results are not bit-identical from run to run.

Things I tried that gave nothing, all on 0.1.56:

change decode, short decode, 32K
main config (async on) 65.9 to 68.2 54.0 to 54.2
--spec 5 61.5 to 65.9 49.3 to 52.4
a second drafter (DFlash2) at 0.5 confidence 65.8 48.5
same, at 0.35 or 0.7 67.7 or 65.0 50.2 or 47.8
--kv k8v4 61.2 49.6
--kv-resident 20480 60.8 47.2
--lookup-chain 2 59.5 49.2

The second drafter ran fine on this card, and it was level with the main config, not better.

Parallel streams: almost nothing

Same machine and configuration, 16K context, 512 generated tokens per request, and about 4K-token prompts for the prefill numbers. Two runs per row. Speeds are tokens per second summed over all requests.

slots decode, total decode, per request prefill, total last request's first token
1 56 to 60 68 to 74 539 to 542 7 s
2 54 to 55 30 to 34 489 to 495 16 s
2, with batched predictor 60 to 64 33 to 40 515 to 519 15 s
4 66 to 68 17 to 22 486 33 s
4, with batched predictor 64 to 69 17 to 23 506 to 508 31 s
8, with batched predictor 62 to 64 8 to 11 510 to 515 62 s

Total decode barely moves: the best I saw was about 69 tokens/s, roughly 1.2x one request. Each request gets slower in proportion, and prompts are read one at a time, so the last of eight users waits about a minute for a first token. One request at a time is the best setup here.

In my earlier post, a smaller mixture-of-experts model on llama.cpp scaled to about 2.5x with several streams (post). That is a different engine, a different model and a model that fits in VRAM, so this is not a like-for-like comparison. My reading of the difference is that here every token picks 10 of 512 experts and different requests rarely pick the same ones, so batching saves little of the work of moving expert weights. I have not tested that explanation.

Limits

  • Memory is tight. Free RAM fell to 3.3 GB at 256K on 0.1.55 (6.1 GB on 0.1.56). Strata warns that the engine can end at the limit with no message, and my 4 GB Windows page file leaves little room. If the server vanishes during a long prompt, check memory first.
  • Part of the model is read from the SSD. About 4 GiB of experts come from disk, because the system's commit limit caps the RAM copy.
  • One machine, one model, one engine. Prompts up to 64K are medians of 2 or 3 runs and larger ones are single runs. The 8-slot row was run only with the batched predictor.
  • The model is a modified community build. I only measured speed. I did not compare its quality with the original.

All scripts, raw results and commands: github.com/amurocje-cpu/local-llm-benchmarks (the strata folder).

Top comments (0)