<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Hyperbolic</title>
    <description>The latest articles on DEV Community by Hyperbolic (@elcances).</description>
    <link>https://dev.to/elcances</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4041529%2F6136a961-0000-4d14-a881-ec82f7ffea8d.webp</url>
      <title>DEV Community: Hyperbolic</title>
      <link>https://dev.to/elcances</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/elcances"/>
    <language>en</language>
    <item>
      <title>qwen3.5-4b on a 16 GB RTX 5060 Ti: 823 tokens per second for 16 users</title>
      <dc:creator>Hyperbolic</dc:creator>
      <pubDate>Mon, 16 Nov 2026 14:15:00 +0000</pubDate>
      <link>https://dev.to/elcances/qwen35-4b-on-a-16-gb-rtx-5060-ti-823-tokens-per-second-for-16-users-6i</link>
      <guid>https://dev.to/elcances/qwen35-4b-on-a-16-gb-rtx-5060-ti-823-tokens-per-second-for-16-users-6i</guid>
      <description>&lt;p&gt;qwen3.5-4b is a 4-billion-parameter language model in the small (&amp;lt;8B) class. On the test machine it generates text &lt;strong&gt;21 times faster than you can read&lt;/strong&gt;, even with a 100-page book in the prompt. Models at that size can be interesting to run on mobile or at the edge.&lt;/p&gt;

&lt;h2&gt;
  
  
  Straight to the Results
&lt;/h2&gt;

&lt;p&gt;qwen3.5-4b is quick on an NVIDIA GeForce RTX 5060 Ti when prompts are short. Long prompts change that: the first word takes much longer to arrive, and generation slows down.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Speed: &lt;strong&gt;114 tokens per second&lt;/strong&gt; on a short prompt, dropping to &lt;strong&gt;34 tokens per second&lt;/strong&gt; at the full 260k-token context.&lt;/li&gt;
&lt;li&gt;Waiting: &lt;strong&gt;101 ms&lt;/strong&gt; to the first word on a short prompt, &lt;strong&gt;6.5 s&lt;/strong&gt; on a 32k-token prompt.&lt;/li&gt;
&lt;li&gt;Electricity: &lt;strong&gt;0.63 kWh per 1M generated tokens&lt;/strong&gt;, roughly &lt;strong&gt;€0.16&lt;/strong&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The measurements below come from llama.cpp installed with its standard settings.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;512 Context&lt;/th&gt;
&lt;th&gt;32k Context&lt;/th&gt;
&lt;th&gt;Max Context (260k)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Prompt Processing (tok/s)&lt;/td&gt;
&lt;td&gt;5050&lt;/td&gt;
&lt;td&gt;5031&lt;/td&gt;
&lt;td&gt;1946&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;TTFT&lt;/td&gt;
&lt;td&gt;101 ms&lt;/td&gt;
&lt;td&gt;6.5 s&lt;/td&gt;
&lt;td&gt;134.5 s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Generation (tok/s)&lt;/td&gt;
&lt;td&gt;114&lt;/td&gt;
&lt;td&gt;88&lt;/td&gt;
&lt;td&gt;34&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  Which Program Should You Run It With?
&lt;/h2&gt;

&lt;p&gt;The model ran under llama.cpp, Ollama and vLLM, each with its default settings.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;llama.cpp&lt;/th&gt;
&lt;th&gt;Ollama&lt;/th&gt;
&lt;th&gt;vLLM&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Writing speed, short prompt (tok/s)&lt;/td&gt;
&lt;td&gt;114&lt;/td&gt;
&lt;td&gt;114&lt;/td&gt;
&lt;td&gt;107&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Wait for first word, short prompt&lt;/td&gt;
&lt;td&gt;101 ms&lt;/td&gt;
&lt;td&gt;165 ms&lt;/td&gt;
&lt;td&gt;101 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Wait for first word, 32k prompt&lt;/td&gt;
&lt;td&gt;6.5 s&lt;/td&gt;
&lt;td&gt;7.3 s&lt;/td&gt;
&lt;td&gt;7.5 s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Longest context (tokens)&lt;/td&gt;
&lt;td&gt;262k&lt;/td&gt;
&lt;td&gt;262k&lt;/td&gt;
&lt;td&gt;256k&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Start-up time&lt;/td&gt;
&lt;td&gt;1.03 s&lt;/td&gt;
&lt;td&gt;4.7 s&lt;/td&gt;
&lt;td&gt;48 s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Electricity (kWh per 1M tokens)&lt;/td&gt;
&lt;td&gt;0.63&lt;/td&gt;
&lt;td&gt;0.65&lt;/td&gt;
&lt;td&gt;0.52&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;For one person on one machine, llama.cpp is the straightforward choice. It ties Ollama on short-prompt speed, matches vLLM for the quickest first word, and starts in &lt;strong&gt;1.03 s&lt;/strong&gt; against Ollama's 4.7 s and vLLM's &lt;strong&gt;48 s&lt;/strong&gt;. Ollama runs at the same speed but takes longer to start and to answer.&lt;/p&gt;

&lt;p&gt;vLLM earns its place on a shared machine. It uses the least electricity at &lt;strong&gt;0.52 kWh per 1M tokens&lt;/strong&gt;, and its writing speed stays close to the other two on short prompts. The wait for the first word on a 32k prompt is 7.5 s, compared with 6.5 s for llama.cpp, a small gap next to its heavier setup and long start-up.&lt;/p&gt;

&lt;h2&gt;
  
  
  Writing Speed as the Conversation Grows
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fm97ml41u9baz45z2ed84.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fm97ml41u9baz45z2ed84.png" alt="Writing speed vs context size" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Decode speed stays close to the short-prompt figure until the context grows. llama.cpp drops from &lt;strong&gt;114 tokens per second&lt;/strong&gt; at 512 tokens to 88 at 32k, giving up about a quarter of its pace, and Ollama matches it at both sizes. At the top, llama.cpp reaches &lt;strong&gt;34 tokens per second&lt;/strong&gt; and Ollama 33, which makes the two interchangeable for raw throughput.&lt;/p&gt;

&lt;p&gt;Past 32k, all three slow sharply. llama.cpp ends up about &lt;strong&gt;3.4x&lt;/strong&gt; slower than at 512 tokens as the KV cache starts to eat into the card's memory. vLLM begins slightly behind at 107 tokens per second but holds up better; at the largest context it can manage, it still runs at &lt;strong&gt;36 tokens per second&lt;/strong&gt;, about 6% ahead of llama.cpp and 7% ahead of Ollama. It has no figure at the full 262k size because it cannot hold that much context.&lt;/p&gt;

&lt;p&gt;The wait for the first word at full context, about &lt;strong&gt;135 s&lt;/strong&gt;, is mostly the time needed to process the 262k-token prompt at &lt;strong&gt;1946 tokens per second&lt;/strong&gt;. The card spends 12 GB of its 16 GB on the KV cache, so the delay comes from reading the prompt rather than from slow generation.&lt;/p&gt;

&lt;h2&gt;
  
  
  When Many People Use It at Once
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fh4fqerqqw8rqs13pgy88.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fh4fqerqqw8rqs13pgy88.png" alt="Total tokens produced per second vs simultaneous users" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;A single vLLM server can serve &lt;strong&gt;16 people at once&lt;/strong&gt;. Together they get &lt;strong&gt;823 tokens per second&lt;/strong&gt;, compared with 103 tokens per second for one user alone, and each person still sees the first word after about 1210 ms. One budget computer can back a whole team or a small product, not just one user.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Test Bench
&lt;/h2&gt;

&lt;p&gt;Every model in this series ran on the same machine, with the same &lt;strong&gt;16 GB&lt;/strong&gt; GeForce RTX 5060 Ti, so the results can be compared directly. Any difference between models comes from the model itself rather than from a change in hardware.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Component&lt;/th&gt;
&lt;th&gt;Details&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;CPU&lt;/td&gt;
&lt;td&gt;Intel Core Ultra 7 265K&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Motherboard&lt;/td&gt;
&lt;td&gt;ASUS ROG MAXIMUS Z890 APEX Hovedkort - Intel Z890&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;RAM&lt;/td&gt;
&lt;td&gt;Kingston FURY Beast DDR5-5600 - 64GB (32 Gb)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GPU&lt;/td&gt;
&lt;td&gt;MSI GeForce RTX 5060 Ti (16 Gb GDDR7)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;System&lt;/td&gt;
&lt;td&gt;Linux 7.0.0-38-generic&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;llama.cpp version&lt;/td&gt;
&lt;td&gt;llama.cpp@b11960&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Ollama version&lt;/td&gt;
&lt;td&gt;&lt;a href="mailto:ollama@v0.40.13"&gt;ollama@v0.40.13&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;vLLM version&lt;/td&gt;
&lt;td&gt;&lt;a href="mailto:vllm@0.27.1"&gt;vllm@0.27.1&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Model compression (llama.cpp)&lt;/td&gt;
&lt;td&gt;gguf-Q4_K_M&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Model compression (Ollama)&lt;/td&gt;
&lt;td&gt;gguf-Q4_K_M&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Model compression (vLLM)&lt;/td&gt;
&lt;td&gt;awq-AWQ-4bit&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;For a single user on an RTX 5060 Ti, llama.cpp is the straightforward pick. It starts in about a second, matches Ollama on speed, and needs no server setup.&lt;/p&gt;

&lt;p&gt;When several people share the same card, vLLM is the better fit. Its slower start-up matters little on a machine that stays running.&lt;/p&gt;

&lt;p&gt;The practical limit is context length. Near the model's ceiling, the KV cache takes 12 GB of the card's 16 GB, and a full-length prompt takes &lt;strong&gt;135 s&lt;/strong&gt; to process before the first word arrives. Short prompts are where this setup feels fast; feeding it whole documents is a different story.&lt;/p&gt;

&lt;h2&gt;
  
  
  Further reading
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Every number, chart and caveat:&lt;/strong&gt; &lt;a href="https://bench.auralis-lab.com/qwen3-5-4b-on-a-16-gb-rtx-5060-ti-260k-token-context-in-12-gb/" rel="noopener noreferrer"&gt;the complete analysis&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>benchmark</category>
      <category>devops</category>
    </item>
  </channel>
</rss>
