<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Amuro Ray</title>
    <description>The latest articles on DEV Community by Amuro Ray (@pastgengamer).</description>
    <link>https://dev.to/pastgengamer</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4249178%2F57ab8f9e-859f-48b8-8288-fd36c3da772e.png</url>
      <title>DEV Community: Amuro Ray</title>
      <link>https://dev.to/pastgengamer</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/pastgengamer"/>
    <language>en</language>
    <item>
      <title>Running a 76 GB MoE model with 262K context on a 16 GB RX 6900 XT</title>
      <dc:creator>Amuro Ray</dc:creator>
      <pubDate>Tue, 17 Nov 2026 10:34:52 +0000</pubDate>
      <link>https://dev.to/pastgengamer/running-a-76-gb-moe-model-with-262k-context-on-a-16-gb-rx-6900-xt-4gc8</link>
      <guid>https://dev.to/pastgengamer/running-a-76-gb-moe-model-with-262k-context-on-a-16-gb-rx-6900-xt-4gc8</guid>
      <description>&lt;p&gt;&lt;em&gt;A 76 GB mixture-of-experts model should not run on a 16 GB graphics card. Strata, a new inference engine, can run one on a 2020 Radeon RX 6900 XT (RDNA2) with the full 262,144-token context, and I measured 49 to 66 tokens/s of generation. I only tested speed: this is what I measured, which settings mattered, and why running several requests at once gained almost nothing.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Measurements, scripts and raw results are on GitHub: &lt;a href="https://github.com/amurocje-cpu/local-llm-benchmarks" rel="noopener noreferrer"&gt;local-llm-benchmarks&lt;/a&gt;, in the &lt;code&gt;strata&lt;/code&gt; folder.&lt;/p&gt;

&lt;h2&gt;
  
  
  The short version
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Decode speed holds up with context.&lt;/strong&gt; 65.7 tokens/s on a short prompt, 54.9 at 32K tokens, 48.8 at 256K.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Reading a long prompt is the cost.&lt;/strong&gt; A fresh 256K prompt takes 5.8 minutes before the first token (about 740 tokens/s). A follow-up that keeps the same start took 0.6 s when it added 35 tokens.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The token predictor was the biggest single gain.&lt;/strong&gt; Turning it off cut decode from 56.2 to 37.8 tokens/s on short prompts (a 49% gain with it on) and from 48.5 to 35.1 at 256K.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Fitting more of the model on the GPU beat keeping the whole cache there.&lt;/strong&gt; Leaving the older part of the cache in system RAM freed room for about 70% more experts on the GPU and gave about 19% faster decode at long context.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Several requests at once do not help.&lt;/strong&gt; Total decode went from about 58 tokens/s with one request to at most 69 with four or eight (my guess at why is below).&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What I ran
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Machine:&lt;/strong&gt; Radeon RX 6900 XT (16 GB), Ryzen 9 7900X, 64 GB DDR5, Windows 11.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Model:&lt;/strong&gt; Qwen3.8-Flash-Next, a mixture-of-experts model that uses 10 of 512 experts per token. I used a community build (SC117's GSQ-RCO "abliterated" variant) at the IQ3_XXS quantization tier, 76 GB across two GGUF files.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Engine:&lt;/strong&gt; Strata, versions 0.1.55 and 0.1.56. It keeps the experts in system RAM and moves the ones it needs onto the GPU, and it can keep the older part of the KV cache in RAM too.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Not a llama.cpp run.&lt;/strong&gt; Everything in this post uses Strata.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Strata's release notes say its RDNA2 package was built but not run on a real RX 6000 card. Everything below ran without errors on mine, but I would still call this card experimental for this engine.&lt;/p&gt;

&lt;p&gt;The configuration I used for the main runs:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;setting&lt;/th&gt;
&lt;th&gt;what it does&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;262,144-token context&lt;/td&gt;
&lt;td&gt;the full window the model supports&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;--kv int8&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;same prefill speed as fp16 and faster decode at long context, in half the memory&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;--kv-resident 32768&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;keeps the most recent 32K of cache on the GPU and the rest in RAM, leaving room for experts&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;--vram-reserve-mib 1400&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;leaves VRAM for Windows and the monitors&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;STRATA_HIP_PROMPT_F16=1&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;about 55% faster prefill on this card (392 to 610 tokens/s at 32K)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;--mtp ... --spec 4 --spec-min-p 0.5&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;the token predictor: it proposes up to 4 tokens and the main model checks them&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;--adapt-async 1&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;expert swaps run between check windows instead of blocking them&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  Speed against context length
&lt;/h2&gt;

&lt;p&gt;Version 0.1.56 with the configuration above, 512 generated tokens per run:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;prompt length&lt;/th&gt;
&lt;th&gt;prefill&lt;/th&gt;
&lt;th&gt;decode&lt;/th&gt;
&lt;th&gt;time to first token&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;short (about 80 tokens)&lt;/td&gt;
&lt;td&gt;n/a&lt;/td&gt;
&lt;td&gt;65.7 tokens/s&lt;/td&gt;
&lt;td&gt;2.1 s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;32K tokens&lt;/td&gt;
&lt;td&gt;685 tokens/s&lt;/td&gt;
&lt;td&gt;54.9 tokens/s&lt;/td&gt;
&lt;td&gt;47 s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;128K tokens&lt;/td&gt;
&lt;td&gt;750 tokens/s&lt;/td&gt;
&lt;td&gt;51.1 tokens/s&lt;/td&gt;
&lt;td&gt;170 s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;256K tokens&lt;/td&gt;
&lt;td&gt;740 tokens/s&lt;/td&gt;
&lt;td&gt;48.8 tokens/s&lt;/td&gt;
&lt;td&gt;5.8 minutes&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Decode drops about 26% from a short prompt to 256K, which is much gentler than the prompt-reading cost. A recall test also passed at every size from 8K to 256K: I buried three facts in the text and the model found all three each time. That is a sanity check that the long context works, not a quality benchmark.&lt;/p&gt;

&lt;p&gt;The cache matters for anything you ask twice. A follow-up that keeps the first 256K tokens identical took 0.6 s when it added 35 new tokens. If the text near the top changes, even a timestamp, the whole prompt is read again.&lt;/p&gt;

&lt;h2&gt;
  
  
  What mattered, and what did not
&lt;/h2&gt;

&lt;p&gt;These runs used version 0.1.55, with 32K and 256K prompts, medians of 2 or 3 runs up to 64K and single runs above:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;setup&lt;/th&gt;
&lt;th&gt;decode, short&lt;/th&gt;
&lt;th&gt;decode, 32K&lt;/th&gt;
&lt;th&gt;decode, 256K&lt;/th&gt;
&lt;th&gt;first token, 256K&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;int8 cache, older part in RAM (main config)&lt;/td&gt;
&lt;td&gt;56.2&lt;/td&gt;
&lt;td&gt;49.4&lt;/td&gt;
&lt;td&gt;48.5&lt;/td&gt;
&lt;td&gt;6.2 min&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;fp16 cache, older part in RAM&lt;/td&gt;
&lt;td&gt;54.9&lt;/td&gt;
&lt;td&gt;48.2&lt;/td&gt;
&lt;td&gt;43.3&lt;/td&gt;
&lt;td&gt;6.2 min&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;int8 cache, all in VRAM&lt;/td&gt;
&lt;td&gt;52.3&lt;/td&gt;
&lt;td&gt;41.5&lt;/td&gt;
&lt;td&gt;40.6&lt;/td&gt;
&lt;td&gt;10.0 min&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;fp16 cache, all in VRAM&lt;/td&gt;
&lt;td&gt;42.3&lt;/td&gt;
&lt;td&gt;38.8&lt;/td&gt;
&lt;td&gt;33.8&lt;/td&gt;
&lt;td&gt;44.9 min&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;int8, in RAM, predictor off&lt;/td&gt;
&lt;td&gt;37.8&lt;/td&gt;
&lt;td&gt;35.5&lt;/td&gt;
&lt;td&gt;35.1&lt;/td&gt;
&lt;td&gt;8.0 min&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Decode in tokens/s.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Keep the cache small and the experts on the GPU.&lt;/strong&gt; Putting the whole cache in VRAM left room for only 2,406 experts, against 4,138 when most of it lived in RAM. That cost about 16% of decode speed at 32K and 256K, and prefill at 32K fell from 706 to 434 tokens/s. With an fp16 cache all in VRAM it was far worse: 97 tokens/s of prefill, which makes a 256K prompt take 45 minutes.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;int8 cache over fp16.&lt;/strong&gt; Prefill was the same, and decode at 256K was 12% faster (48.5 vs 43.3 tokens/s) with half the cache memory.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The token predictor.&lt;/strong&gt; Off, decode fell to 37.8 tokens/s on short prompts, a 49% gap, and prefill was lower as well. The engine still runs a simpler drafter in that case, so this measures the predictor, not speculation as a whole. I did not test why prefill dropped.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Async expert swaps (0.1.56).&lt;/strong&gt; On version 0.1.56 these took short-answer decode from 60 to 61 up to 66 to 68 tokens/s, and 32K from 46 to 48 up to 54. The results are not bit-identical from run to run.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Things I tried that gave nothing, all on 0.1.56:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;change&lt;/th&gt;
&lt;th&gt;decode, short&lt;/th&gt;
&lt;th&gt;decode, 32K&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;main config (async on)&lt;/td&gt;
&lt;td&gt;65.9 to 68.2&lt;/td&gt;
&lt;td&gt;54.0 to 54.2&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;--spec 5&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;61.5 to 65.9&lt;/td&gt;
&lt;td&gt;49.3 to 52.4&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;a second drafter (DFlash2) at 0.5 confidence&lt;/td&gt;
&lt;td&gt;65.8&lt;/td&gt;
&lt;td&gt;48.5&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;same, at 0.35 or 0.7&lt;/td&gt;
&lt;td&gt;67.7 or 65.0&lt;/td&gt;
&lt;td&gt;50.2 or 47.8&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;--kv k8v4&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;61.2&lt;/td&gt;
&lt;td&gt;49.6&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;--kv-resident 20480&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;60.8&lt;/td&gt;
&lt;td&gt;47.2&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;--lookup-chain 2&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;59.5&lt;/td&gt;
&lt;td&gt;49.2&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The second drafter ran fine on this card, and it was level with the main config, not better.&lt;/p&gt;

&lt;h2&gt;
  
  
  Parallel streams: almost nothing
&lt;/h2&gt;

&lt;p&gt;Same machine and configuration, 16K context, 512 generated tokens per request, and about 4K-token prompts for the prefill numbers. Two runs per row. Speeds are tokens per second summed over all requests.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;slots&lt;/th&gt;
&lt;th&gt;decode, total&lt;/th&gt;
&lt;th&gt;decode, per request&lt;/th&gt;
&lt;th&gt;prefill, total&lt;/th&gt;
&lt;th&gt;last request's first token&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;56 to 60&lt;/td&gt;
&lt;td&gt;68 to 74&lt;/td&gt;
&lt;td&gt;539 to 542&lt;/td&gt;
&lt;td&gt;7 s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;54 to 55&lt;/td&gt;
&lt;td&gt;30 to 34&lt;/td&gt;
&lt;td&gt;489 to 495&lt;/td&gt;
&lt;td&gt;16 s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2, with batched predictor&lt;/td&gt;
&lt;td&gt;60 to 64&lt;/td&gt;
&lt;td&gt;33 to 40&lt;/td&gt;
&lt;td&gt;515 to 519&lt;/td&gt;
&lt;td&gt;15 s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;66 to 68&lt;/td&gt;
&lt;td&gt;17 to 22&lt;/td&gt;
&lt;td&gt;486&lt;/td&gt;
&lt;td&gt;33 s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4, with batched predictor&lt;/td&gt;
&lt;td&gt;64 to 69&lt;/td&gt;
&lt;td&gt;17 to 23&lt;/td&gt;
&lt;td&gt;506 to 508&lt;/td&gt;
&lt;td&gt;31 s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;8, with batched predictor&lt;/td&gt;
&lt;td&gt;62 to 64&lt;/td&gt;
&lt;td&gt;8 to 11&lt;/td&gt;
&lt;td&gt;510 to 515&lt;/td&gt;
&lt;td&gt;62 s&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Total decode barely moves: the best I saw was about 69 tokens/s, roughly 1.2x one request. Each request gets slower in proportion, and prompts are read one at a time, so the last of eight users waits about a minute for a first token. One request at a time is the best setup here.&lt;/p&gt;

&lt;p&gt;In my earlier post, a smaller mixture-of-experts model on llama.cpp scaled to about 2.5x with several streams (&lt;a href="https://dev.to/pastgengamer/how-much-does-running-multiple-streams-raise-aggregate-tokenss-on-one-gpu-31jn"&gt;post&lt;/a&gt;). That is a different engine, a different model and a model that fits in VRAM, so this is not a like-for-like comparison. My reading of the difference is that here every token picks 10 of 512 experts and different requests rarely pick the same ones, so batching saves little of the work of moving expert weights. I have not tested that explanation.&lt;/p&gt;

&lt;h2&gt;
  
  
  Limits
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Memory is tight.&lt;/strong&gt; Free RAM fell to 3.3 GB at 256K on 0.1.55 (6.1 GB on 0.1.56). Strata warns that the engine can end at the limit with no message, and my 4 GB Windows page file leaves little room. If the server vanishes during a long prompt, check memory first.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Part of the model is read from the SSD.&lt;/strong&gt; About 4 GiB of experts come from disk, because the system's commit limit caps the RAM copy.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One machine, one model, one engine.&lt;/strong&gt; Prompts up to 64K are medians of 2 or 3 runs and larger ones are single runs. The 8-slot row was run only with the batched predictor.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The model is a modified community build.&lt;/strong&gt; I only measured speed. I did not compare its quality with the original.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;All scripts, raw results and commands: &lt;a href="https://github.com/amurocje-cpu/local-llm-benchmarks" rel="noopener noreferrer"&gt;github.com/amurocje-cpu/local-llm-benchmarks&lt;/a&gt; (the &lt;code&gt;strata&lt;/code&gt; folder).&lt;/p&gt;

</description>
      <category>llm</category>
      <category>amd</category>
      <category>performance</category>
      <category>gpu</category>
    </item>
    <item>
      <title>How much does running multiple streams raise aggregate tokens/s on one GPU?</title>
      <dc:creator>Amuro Ray</dc:creator>
      <pubDate>Tue, 17 Nov 2026 09:15:18 +0000</pubDate>
      <link>https://dev.to/pastgengamer/how-much-does-running-multiple-streams-raise-aggregate-tokenss-on-one-gpu-31jn</link>
      <guid>https://dev.to/pastgengamer/how-much-does-running-multiple-streams-raise-aggregate-tokenss-on-one-gpu-31jn</guid>
      <description>&lt;p&gt;&lt;em&gt;If several people share one local model, how much does total throughput go up as you add streams, and what does each person give up? I tested a dense 27B and a mixture-of-experts 35B-A3B on a single 16 GB Radeon RX 6900 XT with 1 to 8 parallel streams and context from empty to 16k tokens. It is a 2020 RDNA2 card running through Vulkan, so treat these as conservative, entry-level numbers: a baseline that newer hardware and newer kernels should have room to improve on. This includes the part where my first results were wrong.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Raw data, scripts and the full write-up are on GitHub: &lt;a href="https://github.com/amurocje-cpu/local-llm-benchmarks" rel="noopener noreferrer"&gt;local-llm-benchmarks&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  The short version
&lt;/h2&gt;

&lt;p&gt;Four users on the MoE model get about &lt;strong&gt;240 tokens/s in total, around 60 each&lt;/strong&gt;, on a 2020 card. Here is how total and per-stream speed change with the number of streams (empty context, tokens/s):&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;streams&lt;/th&gt;
&lt;th&gt;dense total&lt;/th&gt;
&lt;th&gt;dense per stream&lt;/th&gt;
&lt;th&gt;MoE total&lt;/th&gt;
&lt;th&gt;MoE per stream&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;31.9&lt;/td&gt;
&lt;td&gt;31.9&lt;/td&gt;
&lt;td&gt;108.2&lt;/td&gt;
&lt;td&gt;108.2&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;53.9&lt;/td&gt;
&lt;td&gt;26.9&lt;/td&gt;
&lt;td&gt;171.9&lt;/td&gt;
&lt;td&gt;86.0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;62.9&lt;/td&gt;
&lt;td&gt;15.7&lt;/td&gt;
&lt;td&gt;239.8&lt;/td&gt;
&lt;td&gt;59.9&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;td&gt;68.4&lt;/td&gt;
&lt;td&gt;8.5&lt;/td&gt;
&lt;td&gt;264.6&lt;/td&gt;
&lt;td&gt;33.1&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;ul&gt;
&lt;li&gt;Run more streams at once and &lt;strong&gt;total&lt;/strong&gt; throughput goes up, but nowhere near linearly. Dense topped out around &lt;strong&gt;2.3x&lt;/strong&gt; one stream; the MoE around &lt;strong&gt;2.5x&lt;/strong&gt;. Most of the gain arrives by 3 to 4 streams.&lt;/li&gt;
&lt;li&gt;The MoE model generates &lt;strong&gt;3.4x faster than dense for one user and 3.8x faster for four&lt;/strong&gt;. Eight users on the MoE each still get roughly the speed one user gets on the dense model (32 vs 31 tokens/s).&lt;/li&gt;
&lt;li&gt;Long context is cheap for generation but &lt;strong&gt;expensive for the first token&lt;/strong&gt;, and there is a &lt;strong&gt;cliff&lt;/strong&gt; when the GPU runs out of memory.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The setup
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;GPU:&lt;/strong&gt; one Radeon RX 6900 XT (16 GB, RDNA2, released in 2020), Windows 11, Ryzen 9 7900X, 64 GB DDR5.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Software:&lt;/strong&gt; llama.cpp &lt;code&gt;llama-server&lt;/code&gt; with N slots, Vulkan backend (build 11624).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Models:&lt;/strong&gt; Qwen3.6-27B (dense) and Qwen3.6-35B-A3B (MoE, about 3B parameters active per token), both the same &lt;code&gt;IQ3_XXS&lt;/code&gt; quantization so they fit in VRAM (11.2 and 12.3 GB).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Method:&lt;/strong&gt; for each combination of 1 to 8 simultaneous streams and 0 to 16k tokens of context per stream, I started a fresh server and measured how fast all streams generate while they are all running.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This is the card I have. It is also the kind of hardware plenty of people already own: lots of VRAM, not the newest silicon, and not what most inference work is tuned on first. If it handles a small group of users, a newer card should be at least as comfortable, though I have not tested that.&lt;/p&gt;

&lt;h2&gt;
  
  
  Parallel streams: more total, less each
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4gaaktdxi5l51hhb8mvh.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4gaaktdxi5l51hhb8mvh.png" alt="Throughput vs streams" width="800" height="563"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;For the dense model a single stream runs at 32 tokens/s. Two streams give 54 total (1.7x), four give 63, and the best I saw was 73 at 7 streams. Each stream gets slower as you add people (16 tokens/s each at four, 9 at eight), and past about four streams you are mostly trading speed per user for a small total gain.&lt;/p&gt;

&lt;p&gt;The MoE starts at 108 tokens/s for one stream and reaches about 270 total at 7 streams. That is a better multiple than dense, and it means eight users each get about 32 tokens/s, which is already comfortably faster than reading speed.&lt;/p&gt;

&lt;p&gt;Why is scaling so far from linear? In theory decoding is limited by reading the weights once per step, so a bigger batch should be almost free. I did not isolate the cause, but in a quick comparison an older ROCm/HIP build of llama.cpp scaled the dense model further on this same card (97 vs 68 tokens/s at 8 streams). That points at software kernels more than at the hardware ceiling, and it is why I read these Vulkan numbers as a conservative baseline. It is one comparison with a different llama.cpp version, so I would not draw more from it than that.&lt;/p&gt;

&lt;h2&gt;
  
  
  Context: cheap to generate with, costly to start
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fub7inkrf486nhcm2b6wv.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fub7inkrf486nhcm2b6wv.png" alt="Prefill" width="800" height="563"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;With everything in memory, depth costs only a little: a single stream loses about 10% (dense) or 12% (MoE) of its speed at 16k tokens of context compared with an empty one.&lt;/p&gt;

&lt;p&gt;The painful part is &lt;strong&gt;prefill&lt;/strong&gt;, reading the prompt. Dense processes about 326 tokens/s and the MoE about 1280 at 1k, and both slow down as prompts get longer. At 16k the dense model needs 2.2 minutes for one prompt and the MoE 42 seconds. Prefill speed barely changes with the number of streams, so four users with 4k prompts each means the last one waits 66 s (dense) or 18 s (MoE) for a first token.&lt;/p&gt;

&lt;h2&gt;
  
  
  The cliff
&lt;/h2&gt;

&lt;p&gt;Some of my cells fell off a cliff (hollow dots in the chart): for example dense with two streams at 16k context, or the MoE with six streams at 4k each, where throughput roughly halves. I measured GPU memory during these runs: in every slow case the GPU started using shared (system) memory, and in every normal case it did not. Adding up weights, KV cache, and the per-slot recurrent state these models carry, the slow cases all sit just past roughly 13 GiB of allocations (13.1 for dense, 12.9 for the MoE) on a 16 GiB card where Windows already holds about 1.7 GiB.&lt;/p&gt;

&lt;p&gt;That is a correlation with a plausible mechanism, not a proof, but the practical lesson is clear: &lt;strong&gt;if you plan for N users, budget VRAM for N slots (KV plus state), and leave headroom.&lt;/strong&gt; A smaller quant, quantized KV cache or fewer slots would likely move the cliff. I did not test those.&lt;/p&gt;

&lt;h2&gt;
  
  
  I got it wrong the first time
&lt;/h2&gt;

&lt;p&gt;My first sweep sized each server differently per row and reused servers across cells. It looked like parallelism behaved erratically (weird dips at three and six streams), and deep-context numbers were 20 to 40% too low. A repeat run of one cell on a smaller server gave 221 tokens/s versus 129 in the original sweep. I redid everything with a fresh, size-matched server per cell, repeated a few cells (they agreed within 1%), and those dips disappeared. If you benchmark parallel serving, hold the KV allocation constant, or match it to the cell, and re-run something to see your noise floor.&lt;/p&gt;

&lt;h2&gt;
  
  
  Caveats
&lt;/h2&gt;

&lt;p&gt;One machine, one quant, two models. Streams all start at once with different prompts (real traffic is messier). 128 generated tokens per stream. Single run per cell. I only sanity-checked that both models answer simple prompts correctly; this is a speed test, not a quality comparison. I only tested this one card, so how much newer hardware improves on these numbers is my expectation, not a measurement.&lt;/p&gt;

&lt;p&gt;Full numbers, method and raw data: &lt;a href="https://github.com/amurocje-cpu/local-llm-benchmarks/blob/main/results/WRITEUP.md" rel="noopener noreferrer"&gt;WRITEUP.md&lt;/a&gt;. All scripts, CSVs and notes: &lt;a href="https://github.com/amurocje-cpu/local-llm-benchmarks" rel="noopener noreferrer"&gt;github.com/amurocje-cpu/local-llm-benchmarks&lt;/a&gt;&lt;/p&gt;

</description>
      <category>llm</category>
      <category>performance</category>
      <category>llamacpp</category>
      <category>gpu</category>
    </item>
  </channel>
</rss>
