<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Gosha</title>
    <description>The latest articles on DEV Community by Gosha (@gpunz).</description>
    <link>https://dev.to/gpunz</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4196496%2F53ab88cc-7117-4542-b41c-fb46cda92333.png</url>
      <title>DEV Community: Gosha</title>
      <link>https://dev.to/gpunz</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/gpunz"/>
    <language>en</language>
    <item>
      <title>A coding agent on your own model: OpenCode, Qwen Code and more</title>
      <dc:creator>Gosha</dc:creator>
      <pubDate>Mon, 16 Nov 2026 16:42:55 +0000</pubDate>
      <link>https://dev.to/gpunz/a-coding-agent-on-your-own-model-opencode-qwen-code-and-more-2l34</link>
      <guid>https://dev.to/gpunz/a-coding-agent-on-your-own-model-opencode-qwen-code-and-more-2l34</guid>
      <description>&lt;p&gt;A coding agent on your own model needs a GPU with 24 GB or more, a context of at least 64 thousand tokens and a model that can call tools. Qwen 3.8 27B at a 64K context takes 20.7 GB by our calculation, and Qwen3 Coder 30B-A3B takes 24.3 GB. Below: which agents work with a model that is not theirs, how much memory a long context costs, and two ways to start.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;20.7 GB&lt;/strong&gt; — Qwen 3.8 27B, Q4_K_M, 64K context, 16-bit cache&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;24.3 GB&lt;/strong&gt; — Qwen3 Coder 30B-A3B, same conditions&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;64K&lt;/strong&gt; — the least context the Ollama documentation advises for agents&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;~6.5 min&lt;/strong&gt; — from the click to the first answer in our 2026-11-11 run (a single run)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;em&gt;Memory is computed by the &lt;a href="https://gpu.nz/en/tools/vram" rel="noopener noreferrer"&gt;calculator&lt;/a&gt;, data as of 2026-10-19. The run was ours; the Ollama documentation was read on 2026-11-16.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What is a coding agent, and why run it on your own model?
&lt;/h2&gt;

&lt;p&gt;A coding agent is a program on your computer. It reads your project files, runs commands and tests, and edits code. A language model does the thinking, and the agent reaches it over the network: either someone's paid service or an open model on your own rented GPU.&lt;/p&gt;

&lt;p&gt;The names are easy to mix up. Qwen Code is a terminal agent, while Qwen3 Coder is a family of models for code. An agent works with any model that speaks the right protocol and can call tools.&lt;/p&gt;

&lt;h2&gt;
  
  
  Which agents can work with a model of your own?
&lt;/h2&gt;

&lt;p&gt;Most agents talk to a model over an OpenAI-compatible protocol, which is exactly what our Ollama and vLLM templates serve. The exception is Claude Code, which uses Anthropic's format. The last column says what we tested ourselves and what we only read in the documentation.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Agent&lt;/th&gt;
&lt;th&gt;What your model needs and where to set it&lt;/th&gt;
&lt;th&gt;What was checked&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;OpenCode&lt;/td&gt;
&lt;td&gt;The OpenAI Chat Completions protocol. In opencode.json: a provider with the @ai-sdk/openai-compatible package, a baseURL and a list of models&lt;/td&gt;
&lt;td&gt;Documentation, 2026-11-16. Our run on 2026-11-11: OpenCode 1.19.5 and Qwen 3.8 27B in Ollama on an RTX 4090, tool calls worked (a single run)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen Code&lt;/td&gt;
&lt;td&gt;An OpenAI-compatible address. In ~/.qwen/settings.json: a modelProviders block with baseUrl and envKey; the server must support tool calls&lt;/td&gt;
&lt;td&gt;Documentation, 2026-11-16. We have not run it&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Kimi Code CLI&lt;/td&gt;
&lt;td&gt;A provider with type = "openai" in ~/.kimi-code/config.toml: base_url, api_key, and max_context_size on the model&lt;/td&gt;
&lt;td&gt;Documentation, 2026-11-16. We have not run it&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Claude Code&lt;/td&gt;
&lt;td&gt;The Anthropic Messages format; the address is set by the ANTHROPIC_BASE_URL variable&lt;/td&gt;
&lt;td&gt;Anthropic says it does not support running Claude Code on non-Claude models through gateways; Ollama describes such a setup in its own documentation. We have not tested it&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://gpu.nz/en/code" rel="noopener noreferrer"&gt;gpu.nz Code&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Nothing: the app rents the server, starts the model and connects the agent itself&lt;/td&gt;
&lt;td&gt;Our run on 2026-11-11, the same as OpenCode, which the app is built on&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  Which open model should you take for an agent?
&lt;/h2&gt;

&lt;p&gt;Start with Qwen 3.8 27B: it is the only model we tested with the agent, and it fits a 24 GB card. The others below are picked for their size and purpose; we have not compared their quality, so try them on your own tasks.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;&lt;a href="https://gpu.nz/en/models/qwen3.8-27b" rel="noopener noreferrer"&gt;Qwen 3.8 27B&lt;/a&gt;&lt;/strong&gt; — a dense model with 27.78B parameters and a context of up to 256K. The gpu.nz Code app offers it out of the box.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;a href="https://gpu.nz/en/models/qwen3-coder-30b-a3b" rel="noopener noreferrer"&gt;Qwen3 Coder 30B-A3B&lt;/a&gt;&lt;/strong&gt; — a Mixture-of-Experts model for code: 30.53B parameters, about 3.3B of them working on each token. Context up to 256K.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;a href="https://gpu.nz/en/models/devstral-small-2-24b" rel="noopener noreferrer"&gt;Devstral Small 2 24B&lt;/a&gt;&lt;/strong&gt; — dense, 24.01B parameters, context up to 384K. Its context cache is noticeably heavier than its neighbours'.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;a href="https://gpu.nz/en/models/glm-4.7-flash" rel="noopener noreferrer"&gt;GLM-4.7 Flash&lt;/a&gt;&lt;/strong&gt; — 31.22B parameters, about 3B active, context up to 198K.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;a href="https://gpu.nz/en/models/gpt-oss-20b" rel="noopener noreferrer"&gt;GPT-OSS 20B&lt;/a&gt;&lt;/strong&gt; — 20.91B parameters in its native MXFP4, the lightest on the list: a 16 GB card holds even a 64K context.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Larger models come down to the number of cards. Qwen3 Coder 480B-A35B at Q4_K_M with a 64K context takes 286.7 GB: at least 4 × 80 GB. Kimi K2 from Moonshot AI needs at least 8 × 80 GB. The Kimi Code CLI agent is not tied to it: you can point it at any model from the list above.&lt;/p&gt;

&lt;p&gt;The key requirement is tool calling. An agent does not just answer questions: it asks the model to “read a file” or “run a command”. A model that cannot do that cannot work with your files. The Qwen Code documentation says outright that the server must have tool calling on, and OpenCode has a tip for Ollama: if tool calls fail, raise num_ctx to 16–32 thousand tokens.&lt;/p&gt;

&lt;h2&gt;
  
  
  How much VRAM does an agent need with a long context?
&lt;/h2&gt;

&lt;p&gt;An agent sends the model not a question but everything it has gathered: instructions, pieces of files, command output. Contexts of tens of thousands of tokens are normal, and the context cache grows with every token. The Ollama documentation advises at least 64 thousand tokens for agents and coding tools.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;At a 64K context the models need 13.9 GB to 24.3 GB: a 24 GB card holds some of them only with the 8-bit cache&lt;/strong&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;GB&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Qwen 3.8 27B&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;20.7&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3 Coder 30B-A3B&lt;/td&gt;
&lt;td&gt;24.3&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Devstral Small 2 24B&lt;/td&gt;
&lt;td&gt;24.3&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GLM-4.7 Flash&lt;/td&gt;
&lt;td&gt;21.4&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GPT-OSS 20B&lt;/td&gt;
&lt;td&gt;13.9&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;em&gt;One request, Q4_K_M (GPT-OSS 20B in its native MXFP4), 64K tokens, cache in FP16. Data as of 2026-10-19.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The full table: memory at 32K, 64K and 128K tokens with the usual 16-bit cache and with an 8-bit one. It matters most for models with a heavy cache. Devstral Small 2 at 128K with a 16-bit cache wants a 48 GB card, and with an 8-bit one 32 GB.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model&lt;/th&gt;
&lt;th&gt;32K&lt;/th&gt;
&lt;th&gt;64K&lt;/th&gt;
&lt;th&gt;128K&lt;/th&gt;
&lt;th&gt;64K, 8-bit cache&lt;/th&gt;
&lt;th&gt;128K, 8-bit cache&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://gpu.nz/en/models/qwen3.8-27b" rel="noopener noreferrer"&gt;Qwen 3.8 27B&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;18.7 GB, 24 GB card&lt;/td&gt;
&lt;td&gt;20.7 GB, 24 GB card&lt;/td&gt;
&lt;td&gt;24.7 GB, 32 GB card&lt;/td&gt;
&lt;td&gt;18.8 GB, 24 GB card&lt;/td&gt;
&lt;td&gt;20.9 GB, 24 GB card&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://gpu.nz/en/models/qwen3-coder-30b-a3b" rel="noopener noreferrer"&gt;Qwen3 Coder 30B-A3B&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;21.3 GB, 24 GB card&lt;/td&gt;
&lt;td&gt;24.3 GB, 32 GB card&lt;/td&gt;
&lt;td&gt;30.3 GB, 32 GB card&lt;/td&gt;
&lt;td&gt;21.5 GB, 24 GB card&lt;/td&gt;
&lt;td&gt;24.7 GB, 32 GB card&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://gpu.nz/en/models/devstral-small-2-24b" rel="noopener noreferrer"&gt;Devstral Small 2 24B&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;19.3 GB, 24 GB card&lt;/td&gt;
&lt;td&gt;24.3 GB, 32 GB card&lt;/td&gt;
&lt;td&gt;34.3 GB, 48 GB card&lt;/td&gt;
&lt;td&gt;19.7 GB, 24 GB card&lt;/td&gt;
&lt;td&gt;25.0 GB, 32 GB card&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://gpu.nz/en/models/glm-4.7-flash" rel="noopener noreferrer"&gt;GLM-4.7 Flash&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;19.8 GB, 24 GB card&lt;/td&gt;
&lt;td&gt;21.4 GB, 24 GB card&lt;/td&gt;
&lt;td&gt;24.7 GB, 32 GB card&lt;/td&gt;
&lt;td&gt;19.9 GB, 24 GB card&lt;/td&gt;
&lt;td&gt;21.6 GB, 24 GB card&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://gpu.nz/en/models/gpt-oss-20b" rel="noopener noreferrer"&gt;GPT-OSS 20B&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;13.2 GB, 16 GB card&lt;/td&gt;
&lt;td&gt;13.9 GB, 16 GB card&lt;/td&gt;
&lt;td&gt;15.4 GB, 16 GB card&lt;/td&gt;
&lt;td&gt;13.2 GB, 16 GB card&lt;/td&gt;
&lt;td&gt;14.0 GB, 16 GB card&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;An 8-bit cache takes about half the memory of a 16-bit one with a very small loss in precision, according to the Ollama documentation. At 64K it moves Qwen3 Coder 30B-A3B from a 32 GB card to a 24 GB one (24.3 GB against 21.5 GB). Devstral Small 2 moves from a 32 GB card to a 24 GB one. In Ollama these are three server variables, and the 8-bit cache works only with flash attention:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;OLLAMA_FLASH_ATTENTION&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;1
&lt;span class="nv"&gt;OLLAMA_KV_CACHE_TYPE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;q8_0
&lt;span class="nv"&gt;OLLAMA_CONTEXT_LENGTH&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;65536
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The calculation is a lower bound. For Qwen 3.8 27B at 64K the calculator gives 20.7 GB and a 24 GB card. Yet in our 2026-11-11 run on an RTX 4090 with the default cache part of the model spilled to the CPU and the speed dropped by about a third. With an 8-bit cache (18.8 GB by calculation) the model stayed on the GPU. Check with ollama ps: the PROCESSOR column should read “100% GPU”.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Set the context yourself: Ollama cuts a long prompt silently.&lt;/strong&gt; In our 2026-11-11 run Ollama cut a prompt longer than the context window without an error: the answer came back with HTTP 200. The agent cannot tell that part of its request was lost, and the server keeps billing by the minute. The Ollama documentation says the default window depends on memory: 4 thousand tokens below 24 GiB, 32 thousand at 24–48 GiB. The &lt;a href="https://gpu.nz/en/templates/ollama" rel="noopener noreferrer"&gt;Open WebUI + Ollama&lt;/a&gt; template starts with 8 thousand, so set the window for an agent yourself: OLLAMA_CONTEXT_LENGTH=65536 in Ollama or --max-model-len in vLLM.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;To check your own model, quantisation and context length, use the calculator:&lt;/p&gt;

&lt;h2&gt;
  
  
  What does an hour cost, and how is it different from a subscription?
&lt;/h2&gt;

&lt;p&gt;There is no per-token price and no request limit here: you pay for the minutes the server is on. Subscriptions to coding services work differently: a fixed monthly fee with limits of their own. Which is cheaper depends on how many hours a month your agent really works. Multiply those hours by the hourly price on the cards below, and stop the server whenever the agent is idle.&lt;/p&gt;

&lt;h2&gt;
  
  
  How do you point an agent at your own address?
&lt;/h2&gt;

&lt;p&gt;You need two things: an address ending in /v1 and a token. In the Ollama templates the address is https://-11434.run.gpu.nz/v1, in vLLM https://-8000.run.gpu.nz/v1, and the token is on the server page (more in the &lt;a href="https://gpu.nz/en/guides/ollama-vllm-api" rel="noopener noreferrer"&gt;Ollama and vLLM guide&lt;/a&gt;). The formats below were checked against the documentation on 2026-11-16. OpenCode now has two branches, V1 and V2, and V2 reads a V1 config as it is, so one file suits both.&lt;/p&gt;

&lt;p&gt;OpenCode: opencode.json in the project folder or ~/.config/opencode/opencode.json. The token comes from the GPUNZ_TOKEN variable.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"$schema"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"https://opencode.ai/config.json"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"model"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"gpunz/qwen3.8:27b"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"provider"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nl"&gt;"gpunz"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="nl"&gt;"npm"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"@ai-sdk/openai-compatible"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nl"&gt;"name"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"gpu.nz"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nl"&gt;"options"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nl"&gt;"baseURL"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"https://&amp;lt;id&amp;gt;-11434.run.gpu.nz/v1"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="nl"&gt;"apiKey"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"{env:GPUNZ_TOKEN}"&lt;/span&gt;
      &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nl"&gt;"models"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nl"&gt;"qwen3.8:27b"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
          &lt;span class="nl"&gt;"name"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Qwen 3.8 27B"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
          &lt;span class="nl"&gt;"limit"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"context"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;65536&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"output"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;8192&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;
      &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Qwen Code: ~/.qwen/settings.json. Set GPUNZ_TOKEN in your environment; the settings name it in envKey.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"modelProviders"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nl"&gt;"openai"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
      &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nl"&gt;"id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"qwen3.8:27b"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="nl"&gt;"name"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Qwen 3.8 27B (gpu.nz)"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="nl"&gt;"envKey"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"GPUNZ_TOKEN"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="nl"&gt;"baseUrl"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"https://&amp;lt;id&amp;gt;-11434.run.gpu.nz/v1"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="nl"&gt;"generationConfig"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"contextWindowSize"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;65536&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
      &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;]&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"security"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"auth"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"selectedType"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"openai"&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"model"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"name"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"qwen3.8:27b"&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Kimi Code CLI: ~/.kimi-code/config.toml. By default it guesses a model's capabilities from its name, so for your own model it is better to list them.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight toml"&gt;&lt;code&gt;&lt;span class="na"&gt;default_model&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"gpunz"&lt;/span&gt;

&lt;span class="k"&gt;[providers.gpunz]&lt;/span&gt;
&lt;span class="na"&gt;type&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"openai"&lt;/span&gt;
&lt;span class="na"&gt;base_url&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"https://&amp;lt;id&amp;gt;-11434.run.gpu.nz/v1"&lt;/span&gt;
&lt;span class="na"&gt;api_key&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"&amp;lt;token&amp;gt;"&lt;/span&gt;

&lt;span class="k"&gt;[models."gpunz"]&lt;/span&gt;
&lt;span class="na"&gt;provider&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"gpunz"&lt;/span&gt;
&lt;span class="na"&gt;model&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"qwen3.8:27b"&lt;/span&gt;
&lt;span class="na"&gt;max_context_size&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;65536&lt;/span&gt;
&lt;span class="na"&gt;capabilities&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"tool_use"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Route 1: gpu.nz Code, with no setup
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://gpu.nz/en/code" rel="noopener noreferrer"&gt;gpu.nz Code&lt;/a&gt; is our free open-source app (MIT): a coding agent that rents a server itself, starts an open model on it and connects to it. It is built on OpenCode and is not affiliated with its team. It runs on Windows, macOS and Linux.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Install the app and sign in.&lt;/strong&gt; Download the build for your system from the &lt;a href="https://gpu.nz/en/code" rel="noopener noreferrer"&gt;gpu.nz Code&lt;/a&gt; page and sign in with your gpu.nz account. Your balance needs funds: the server is paid from it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Pick a model.&lt;/strong&gt; Qwen 3.8 27B is the one tested with the agent. For your own, enter an Ollama name or a link to a GGUF on Hugging Face: the app works out the GPU and shows the hourly price before launch. It picks the server itself.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Wait for the launch.&lt;/strong&gt; The first launch usually takes 5–14 minutes while the server downloads the model. In our 2026-11-11 run it was about six and a half minutes from the click to the first answer.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Open your project folder and describe the task.&lt;/strong&gt; The agent runs on your computer and the project is not uploaded: only requests to the model go to your server. The agent asks before destructive commands.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Choose the idle shutdown.&lt;/strong&gt; The server is deleted after 15, 30 or 60 minutes of idling and when you quit the app. If you turned the shutdown off, delete the server yourself. Details are in the &lt;a href="https://gpu.nz/en/docs/code" rel="noopener noreferrer"&gt;documentation&lt;/a&gt;.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Route 2: your own server and any agent
&lt;/h2&gt;

&lt;p&gt;Want another agent or another model? Rent a server with a template and connect the agent with the address from the section above.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Rent a server with a 24 GB card or more.&lt;/strong&gt; For Qwen 3.8 27B take the &lt;a href="https://gpu.nz/en/templates/chat-qwen38" rel="noopener noreferrer"&gt;Qwen 3.8&lt;/a&gt; template: its context window grows with the server's memory, 64K tokens on 24 GB and 128K on 32 GB, and the cache is already 8-bit. For another model take &lt;a href="https://gpu.nz/en/templates/ollama" rel="noopener noreferrer"&gt;Ollama&lt;/a&gt; or &lt;a href="https://gpu.nz/en/templates/vllm" rel="noopener noreferrer"&gt;vLLM&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Copy the address and the token.&lt;/strong&gt; On the server page the «Ollama API» button opens the model list; paste the address up to /v1 and the token into the agent's settings.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Set the context and check it.&lt;/strong&gt; In the Ollama template set the variables from the memory section. Over &lt;a href="https://gpu.nz/en/guides/ssh-gpu-server" rel="noopener noreferrer"&gt;SSH&lt;/a&gt; run ollama ps: the CONTEXT column should show your value, and PROCESSOR should say “100% GPU”.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Paste the settings into the agent.&lt;/strong&gt; The OpenCode, Qwen Code or Kimi Code CLI config from the section above; the model name is the one in the «Ollama API» list.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Stop the server when you are done.&lt;/strong&gt; A stopped server costs nothing for the GPU, only its disk.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;On vLLM the start looks like this. The tool-calling flags are what the agent needs, and the parser name depends on the model: take it from the vLLM documentation on tool calling.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;vllm serve &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;model&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; --port 8000 --api-key &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;token&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; --max-model-len 65536 --enable-auto-tool-choice --tool-call-parser &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;parser&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  What can go wrong
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Part of the model spilled to the CPU. Answers slow down by about a third, and ollama ps shows a CPU share. Shrink the cache to 8 bits, cut the context or take a bigger card.&lt;/li&gt;
&lt;li&gt;The model does not call tools. The agent answers in text and does not touch files. In vLLM turn on --enable-auto-tool-choice and a parser for your model. In OpenCode V2 models discovered on vLLM start with tools disabled, so describe the model in the config yourself.&lt;/li&gt;
&lt;li&gt;The context window is smaller than the agent thinks. The agent's setting (limit.context, contextWindowSize, max_context_size) must hold the same number as the server.&lt;/li&gt;
&lt;li&gt;A long first launch. While the server downloads the model it is already billed by the minute: those 5–14 minutes.&lt;/li&gt;
&lt;li&gt;A forgotten server. The agent is silent and the server keeps running. Turn on the idle shutdown or stop the server by hand.&lt;/li&gt;
&lt;li&gt;The tools change fast. Our run used OpenCode 1.19.5, and on 2026-11-16 the project's repository was already at version 2.3.0. Check the settings against the documentation for your version.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  How we counted
&lt;/h2&gt;

&lt;p&gt;Memory comes from the &lt;a href="https://gpu.nz/en/tools/vram" rel="noopener noreferrer"&gt;gpu.nz calculator&lt;/a&gt;: weights are real file sizes, the context cache follows each model's config.json, and the 8-bit cache is counted as 8.5 bits per value. Data as of 2026-10-19. One request, Q4_K_M, GPT-OSS 20B in its native MXFP4. 1 GB here is 1024³ bytes, as on GPUs. More on quantisation and the cache is in the &lt;a href="https://gpu.nz/en/guides/gguf-quantization-explained" rel="noopener noreferrer"&gt;GGUF article&lt;/a&gt;, and on LLM memory in the &lt;a href="https://gpu.nz/en/guides/llm-vram-table" rel="noopener noreferrer"&gt;table for 35 models&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Our run was on 2026-11-11, one launch of each variant, and we do not generalise it: OpenCode 1.19.5 with Qwen 3.8 27B in Ollama through an OpenAI-compatible address on a rented RTX 4090. Tool calls worked; with a 64K context on a 24 GB card the cache had to be 8-bit for the model to stay on the GPU.&lt;/p&gt;

&lt;p&gt;The agents' settings and what we say about them were checked on 2026-11-16 against the official documentation: opencode.ai/docs (V1 and V2), Qwen Code (qwenlm.github.io/qwen-code-docs), Kimi Code CLI (moonshotai.github.io/kimi-code), Claude Code (code.claude.com/docs, the gateways page), Ollama (docs.ollama.com: context-length, faq, anthropic-compatibility) and vLLM (docs.vllm.ai, tool calling). We have not run Qwen Code, Kimi Code CLI or Claude Code on our servers, and we have not compared model quality. There are no prices in the text: they are on the cards.&lt;/p&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;How do I connect OpenCode to my own model in Ollama?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;In opencode.json add a provider with the @ai-sdk/openai-compatible package, the server address ending in /v1 as options.baseURL, and the model name under models. If the API is behind a token, put it in options.apiKey. Set the Ollama context to at least 16–32 thousand tokens, and for an agent 64K is better.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Which model suits OpenCode and Qwen Code on a 24 GB card?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Qwen 3.8 27B: at a 64K context it takes 20.7 GB by calculation and 18.8 GB with an 8-bit cache. Qwen3 Coder 30B-A3B and Devstral Small 2 fit 24 GB at 64K only with an 8-bit cache: 21.5 GB and 19.7 GB.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How much VRAM does a coding agent need at a 64K context?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;From 13.9 GB for GPT-OSS 20B to 24.3 GB for Qwen3 Coder 30B-A3B and Devstral Small 2 with the usual 16-bit cache. With an 8-bit cache the same two take 21.5 GB and 19.7 GB.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Can I run Claude Code on my own open model?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Claude Code uses the Anthropic Messages format, and the address is set by the ANTHROPIC_BASE_URL variable. Anthropic says it does not support running Claude Code on non-Claude models through gateways, while Ollama describes such a setup in its documentation. We have not tested it and do not promise that it works.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What is Qwen Code, and how is it different from Qwen3 Coder?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Qwen Code is a terminal agent under the Apache-2.0 licence, while Qwen3 Coder is a family of models for code. You can point the agent at your own OpenAI-compatible address and run the model on a rented GPU.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How do I start a coding agent without setting up a server?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Install the free gpu.nz Code app: it rents the server, starts the model and connects the agent itself. The first launch takes 5–14 minutes, the server is billed by the minute and deleted after the idle time you choose: 15, 30 or 60 minutes.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>opensource</category>
      <category>programming</category>
    </item>
    <item>
      <title>Which Qwen model to run: versions, sizes and VRAM for 11 models</title>
      <dc:creator>Gosha</dc:creator>
      <pubDate>Wed, 21 Oct 2026 03:26:07 +0000</pubDate>
      <link>https://dev.to/gpunz/which-qwen-model-to-run-versions-sizes-and-vram-for-11-models-3466</link>
      <guid>https://dev.to/gpunz/which-qwen-model-to-run-versions-sizes-and-vram-for-11-models-3466</guid>
      <description>&lt;p&gt;The Qwen model you can run depends on your GPU memory. At Q4_K_M with an 8K context, Qwen3.5 9B needs 6.8 GB, Qwen3.8 27B 17.2 GB, Qwen3 Next 80B-A3B 46.5 GB and Qwen3 Coder 480B-A35B 273.1 GB. These are single-request figures for the 11 family models. Below: what separates the generations, which model fits which card and how to run it.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;6.8 GB&lt;/strong&gt; — Qwen3.5 9B, Q4_K_M, 8K context&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;17.2 GB&lt;/strong&gt; — Qwen3.8 27B, Q4_K_M, 8K context&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;46.5 GB&lt;/strong&gt; — Qwen3 Next 80B-A3B, Q4_K_M, 8K context&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;256K&lt;/strong&gt; — context of Qwen3.8, Qwen3.5, Next and Coder, tokens; Qwen3 has 40K&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;em&gt;Computed by the &lt;a href="https://gpu.nz/en/tools/vram" rel="noopener noreferrer"&gt;VRAM calculator&lt;/a&gt;, data as of 2026-10-19.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Which Qwen models are there?
&lt;/h2&gt;

&lt;p&gt;Qwen is Alibaba's family of open models. The names are easy to mix up: versions, sizes and suffixes all run together. Five lines matter for a GPU in our data.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Qwen3 — dense models at 8B, 14B and 32B plus two Mixture-of-Experts (MoE) models, 30B-A3B and 235B-A22B. They work with text and think before answering by default.&lt;/li&gt;
&lt;li&gt;Qwen3 Coder — programming models, 30B-A3B and 480B-A35B. The authors built them for agentic coding, and they have no thinking mode.&lt;/li&gt;
&lt;li&gt;Qwen3 Next 80B-A3B — a MoE with 81.3B parameters, 3B of them working on each token. Its attention is hybrid, more on that below.&lt;/li&gt;
&lt;li&gt;Qwen3.5 — the 4B and 9B versions are in the table. They take images along with text.&lt;/li&gt;
&lt;li&gt;Qwen3.8 27B — the newest in the table. Its authors call it the next generation after Qwen3.5 and Qwen3.6, with gains in code, professional work and agents. It takes images and video.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;“A3B” and “A22B” mean 3B and 22B active parameters. In a dense model every parameter works on every token.&lt;/p&gt;

&lt;p&gt;Part of the family is not in the calculator yet: Qwen3.5 has 27b, 35b and 122b sizes in the Ollama library, Qwen3.8 has Flash-Next and 2.4T-A95B on Hugging Face, and Qwen3.6 is not in the data. We do not count these.&lt;/p&gt;

&lt;h2&gt;
  
  
  How much VRAM does each Qwen model need?
&lt;/h2&gt;

&lt;p&gt;At Q4_K_M and an 8K context, from 4.1 GB (Qwen3.5 4B) to 273.1 GB (Qwen3 Coder 480B-A35B). Quantisation compresses the weights: the Q4_K_M file of Qwen3.8 27B is 3.3 times smaller than the original BF16, with little loss in answer quality. The total covers the weights, the context cache (memory for the text the model has already read) and the CUDA runtime.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Qwen up to 32B fits a 24 GB card at Q4_K_M, Next 80B needs 48 GB&lt;/strong&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;GB&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Qwen3.5 4B&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;4.1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Qwen3.5 9B&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;6.8&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3 8B&lt;/td&gt;
&lt;td&gt;6.8&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3 14B&lt;/td&gt;
&lt;td&gt;10.7&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Qwen3.8 27B&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;17.2&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3 30B-A3B (MoE)&lt;/td&gt;
&lt;td&gt;19.1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3 Coder 30B-A3B (MoE)&lt;/td&gt;
&lt;td&gt;19.1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3 32B&lt;/td&gt;
&lt;td&gt;21.4&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;strong&gt;Qwen3 Next 80B-A3B&lt;/strong&gt; (MoE)&lt;/td&gt;
&lt;td&gt;46.5&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;em&gt;8K context, one request, FP16 cache. In bold: models with hybrid attention. Weights are the GGUF files on Hugging Face; data as of 2026-10-19.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The whole family in one table: parameters, file size, memory at two context lengths, the smallest card and the context limit set in each model's config.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model&lt;/th&gt;
&lt;th&gt;Parameters&lt;/th&gt;
&lt;th&gt;Q4_K_M weights&lt;/th&gt;
&lt;th&gt;Memory at 8K / 32K&lt;/th&gt;
&lt;th&gt;Smallest card&lt;/th&gt;
&lt;th&gt;Context limit&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://gpu.nz/en/models/qwen3.5-4b" rel="noopener noreferrer"&gt;Qwen3.5 4B&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;4.7B&lt;/td&gt;
&lt;td&gt;2.6 GB&lt;/td&gt;
&lt;td&gt;4.1 / 4.8 GB&lt;/td&gt;
&lt;td&gt;8 GB&lt;/td&gt;
&lt;td&gt;256K&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://gpu.nz/en/models/qwen3.5-9b" rel="noopener noreferrer"&gt;Qwen3.5 9B&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;9.7B&lt;/td&gt;
&lt;td&gt;5.3 GB&lt;/td&gt;
&lt;td&gt;6.8 / 7.6 GB&lt;/td&gt;
&lt;td&gt;8 GB&lt;/td&gt;
&lt;td&gt;256K&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://gpu.nz/en/models/qwen3-8b" rel="noopener noreferrer"&gt;Qwen3 8B&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;8.2B&lt;/td&gt;
&lt;td&gt;4.7 GB&lt;/td&gt;
&lt;td&gt;6.8 / 10.2 GB&lt;/td&gt;
&lt;td&gt;8 GB&lt;/td&gt;
&lt;td&gt;40K&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://gpu.nz/en/models/qwen3-14b" rel="noopener noreferrer"&gt;Qwen3 14B&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;14.8B&lt;/td&gt;
&lt;td&gt;8.4 GB&lt;/td&gt;
&lt;td&gt;10.7 / 14.4 GB&lt;/td&gt;
&lt;td&gt;12 GB&lt;/td&gt;
&lt;td&gt;40K&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://gpu.nz/en/models/qwen3.8-27b" rel="noopener noreferrer"&gt;Qwen3.8 27B&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;27.8B&lt;/td&gt;
&lt;td&gt;15.3 GB&lt;/td&gt;
&lt;td&gt;17.2 / 18.7 GB&lt;/td&gt;
&lt;td&gt;24 GB&lt;/td&gt;
&lt;td&gt;256K&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://gpu.nz/en/models/qwen3-30b-a3b" rel="noopener noreferrer"&gt;Qwen3 30B-A3B&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;30.5B (3.3B active)&lt;/td&gt;
&lt;td&gt;17.3 GB&lt;/td&gt;
&lt;td&gt;19.1 / 21.3 GB&lt;/td&gt;
&lt;td&gt;24 GB&lt;/td&gt;
&lt;td&gt;40K&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://gpu.nz/en/models/qwen3-coder-30b-a3b" rel="noopener noreferrer"&gt;Qwen3 Coder 30B-A3B&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;30.5B (3.3B active)&lt;/td&gt;
&lt;td&gt;17.3 GB&lt;/td&gt;
&lt;td&gt;19.1 / 21.3 GB&lt;/td&gt;
&lt;td&gt;24 GB&lt;/td&gt;
&lt;td&gt;256K&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://gpu.nz/en/models/qwen3-32b" rel="noopener noreferrer"&gt;Qwen3 32B&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;32.8B&lt;/td&gt;
&lt;td&gt;18.4 GB&lt;/td&gt;
&lt;td&gt;21.4 / 27.4 GB&lt;/td&gt;
&lt;td&gt;24 GB&lt;/td&gt;
&lt;td&gt;40K&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://gpu.nz/en/models/qwen3-next-80b-a3b" rel="noopener noreferrer"&gt;Qwen3 Next 80B-A3B&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;81.3B (3B active)&lt;/td&gt;
&lt;td&gt;45.2 GB&lt;/td&gt;
&lt;td&gt;46.5 / 47.0 GB&lt;/td&gt;
&lt;td&gt;48 GB, tight&lt;/td&gt;
&lt;td&gt;256K&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://gpu.nz/en/models/qwen3-235b-a22b" rel="noopener noreferrer"&gt;Qwen3 235B-A22B&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;235B (22B active)&lt;/td&gt;
&lt;td&gt;132.4 GB&lt;/td&gt;
&lt;td&gt;134.9 / 139.3 GB&lt;/td&gt;
&lt;td&gt;141 GB, or 2 × 80 GB, tight&lt;/td&gt;
&lt;td&gt;40K&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://gpu.nz/en/models/qwen3-coder-480b-a35b" rel="noopener noreferrer"&gt;Qwen3 Coder 480B-A35B&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;480B (35B active)&lt;/td&gt;
&lt;td&gt;270.1 GB&lt;/td&gt;
&lt;td&gt;273.1 / 278.9 GB&lt;/td&gt;
&lt;td&gt;288 GB, or 4 × 80 GB, tight&lt;/td&gt;
&lt;td&gt;256K&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;A3B does not mean “3 GB”.&lt;/strong&gt; Qwen3 30B-A3B uses 3.3B parameters per token, but all 30.5B load into memory: 17.3 GB for the weights alone. MoE buys answer speed, not memory. The model will not fit a card smaller than 24 GB, and renting a weak card just burns minutes.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  How do Qwen3.5 and Qwen3.8 differ from Qwen3?
&lt;/h2&gt;

&lt;p&gt;The biggest difference shows in memory: the new models have a much smaller context cache. In Qwen3 every layer keeps the whole conversation. In Qwen3.8 27B and Qwen3.5 9B (according to the model cards) three layers in four are linear attention, Gated DeltaNet, with a tiny state. Only every fourth layer keeps a cache. That is 64 KiB per token against 256 KiB for Qwen3 32B, 4 times less.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;From 8K to 32K context Qwen3 32B takes 6.0 GB more, Qwen3.8 27B only 1.5 GB&lt;/strong&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;GB&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3 32B&lt;/td&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3 14B&lt;/td&gt;
&lt;td&gt;3.8&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3 8B&lt;/td&gt;
&lt;td&gt;3.4&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3 30B-A3B&lt;/td&gt;
&lt;td&gt;2.3&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3 Coder 30B-A3B&lt;/td&gt;
&lt;td&gt;2.3&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Qwen3.8 27B&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;1.5&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Qwen3.5 4B&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;0.8&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Qwen3.5 9B&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;0.8&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Qwen3 Next 80B-A3B&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;0.6&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;em&gt;Extra memory at 32K tokens over 8K, Q4_K_M, one request. In bold: hybrid attention.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Qwen3 Next is built like Qwen3.8, and Qwen3 Coder like plain Qwen3. The gap widens with long contexts. Qwen3.8 27B at 128K needs 24.7 GB and stays on a 32 GB card; at the full 256K it takes 32.7 GB and a 48 GB card. Qwen3 Coder 30B-A3B at 128K needs 30.3 GB for 17.3 GB of weights: its cache is the ordinary kind.&lt;/p&gt;

&lt;p&gt;The new models take 256K tokens of context, about 400 pages of text. Qwen3 is limited to 40K in our data. The Qwen3 cards say YaRN stretches it to 128K, with a warning that quality on short texts can drop. The calculator does not model that mode.&lt;/p&gt;

&lt;p&gt;We have not measured answer quality in Russian or any other language. The authors claim support for more than 100 languages in Qwen3 and 201 in Qwen3.5. Try the model on your own texts: an hour on a rented card is enough.&lt;/p&gt;

&lt;h2&gt;
  
  
  Which Qwen model fits your GPU?
&lt;/h2&gt;

&lt;p&gt;The table shows which models each next memory step adds at 8K, 32K and 128K context (Q4_K_M, one request). A dash means nothing new.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Card memory&lt;/th&gt;
&lt;th&gt;New at 8K context&lt;/th&gt;
&lt;th&gt;New at 32K&lt;/th&gt;
&lt;th&gt;New at 128K&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;8 GB&lt;/td&gt;
&lt;td&gt;Qwen3.5 4B, Qwen3.5 9B, Qwen3 8B&lt;/td&gt;
&lt;td&gt;Qwen3.5 4B, Qwen3.5 9B&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;12 GB&lt;/td&gt;
&lt;td&gt;Qwen3 14B&lt;/td&gt;
&lt;td&gt;Qwen3 8B&lt;/td&gt;
&lt;td&gt;Qwen3.5 4B, Qwen3.5 9B&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;16 GB&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;Qwen3 14B&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;24 GB&lt;/td&gt;
&lt;td&gt;Qwen3.8 27B, Qwen3 30B-A3B, Qwen3 Coder 30B-A3B, Qwen3 32B&lt;/td&gt;
&lt;td&gt;Qwen3.8 27B, Qwen3 30B-A3B, Qwen3 Coder 30B-A3B&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;32 GB&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;Qwen3 32B&lt;/td&gt;
&lt;td&gt;Qwen3.8 27B, Qwen3 Coder 30B-A3B&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;48 GB&lt;/td&gt;
&lt;td&gt;Qwen3 Next 80B-A3B&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;80 GB&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;Qwen3 Next 80B-A3B&lt;/td&gt;
&lt;td&gt;Qwen3 Next 80B-A3B&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;ul&gt;
&lt;li&gt;8 GB — Qwen3.5 9B: 6.8 GB at 8K and 7.6 GB at 32K. Qwen3 8B fits at 8K too, but for 32K it needs a 12 GB card.&lt;/li&gt;
&lt;li&gt;12 GB and 16 GB — Qwen3 14B: 10.7 GB at 8K. A 32K context (14.4 GB) needs a 16 GB card.&lt;/li&gt;
&lt;li&gt;24 GB — the widest step: Qwen3.8 27B (17.2 GB), Qwen3 30B-A3B and Coder 30B-A3B (19.1 GB), Qwen3 32B (21.4 GB). All but Qwen3 32B hold a 32K context.&lt;/li&gt;
&lt;li&gt;32 GB — Qwen3 32B with a 32K context (27.4 GB) and Qwen3.8 27B with 128K (24.7 GB).&lt;/li&gt;
&lt;li&gt;48 GB — Qwen3 Next 80B-A3B: 46.5 GB, almost the whole card. A 32K context no longer fits and needs an 80 GB card.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Within one line, more parameters usually mean better answers. We have no quality comparison across lines: take two or three models that fit and run them on your own tasks.&lt;/p&gt;

&lt;h2&gt;
  
  
  Which Qwen for code, chat and images?
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Code in an editor, agents — Qwen3 Coder. The 30B-A3B needs 19.1 GB and runs on a 24 GB card; the 480B-A35B needs several cards.&lt;/li&gt;
&lt;li&gt;Chat, writing, analysis — Qwen3.8 27B (24 GB), or Qwen3.5 9B on a smaller card. Both think by default.&lt;/li&gt;
&lt;li&gt;Images as input — Qwen3.5 and Qwen3.8: attach a photo or a screenshot. Qwen3 and Coder take text only.&lt;/li&gt;
&lt;li&gt;An API for several people — MoE models such as 30B-A3B and Next: fewer parameters work on each token than in a dense model of the same size. vLLM suits such an API.&lt;/li&gt;
&lt;li&gt;Image generation — a different family, Qwen-Image. Setup and use are in &lt;a href="https://gpu.nz/en/guides/qwen-image-edit-comfyui" rel="noopener noreferrer"&gt;Qwen-Image-Edit in ComfyUI&lt;/a&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  How many GPUs do Qwen3 235B and Coder 480B need?
&lt;/h2&gt;

&lt;p&gt;The big ones do not fit an ordinary card. Qwen3 235B-A22B at Q4_K_M needs 134.9 GB: almost the whole 141 GB card. Qwen3 Coder 480B-A35B needs 273.1 GB: one 288 GB card or several smaller ones.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model&lt;/th&gt;
&lt;th&gt;Q4_K_M weights&lt;/th&gt;
&lt;th&gt;Memory at 8K&lt;/th&gt;
&lt;th&gt;One card&lt;/th&gt;
&lt;th&gt;80 GB cards&lt;/th&gt;
&lt;th&gt;141 GB cards&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://gpu.nz/en/models/qwen3-next-80b-a3b" rel="noopener noreferrer"&gt;Qwen3 Next 80B-A3B&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;45.2 GB&lt;/td&gt;
&lt;td&gt;46.5 GB&lt;/td&gt;
&lt;td&gt;48 GB&lt;/td&gt;
&lt;td&gt;80 GB&lt;/td&gt;
&lt;td&gt;141 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://gpu.nz/en/models/qwen3-235b-a22b" rel="noopener noreferrer"&gt;Qwen3 235B-A22B&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;132.4 GB&lt;/td&gt;
&lt;td&gt;134.9 GB&lt;/td&gt;
&lt;td&gt;141 GB&lt;/td&gt;
&lt;td&gt;2 × 80 GB&lt;/td&gt;
&lt;td&gt;141 GB, tight&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://gpu.nz/en/models/qwen3-coder-480b-a35b" rel="noopener noreferrer"&gt;Qwen3 Coder 480B-A35B&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;270.1 GB&lt;/td&gt;
&lt;td&gt;273.1 GB&lt;/td&gt;
&lt;td&gt;288 GB&lt;/td&gt;
&lt;td&gt;4 × 80 GB&lt;/td&gt;
&lt;td&gt;2 × 141 GB, tight&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The model is split across cards: the calculation halves the weights and the cache, and every card keeps its own runtime memory.&lt;/p&gt;

&lt;h2&gt;
  
  
  How to run it
&lt;/h2&gt;

&lt;p&gt;With a card that fits, Ollama is the shortest path; the tags match the models above:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;ollama run qwen3.5:9b      &lt;span class="c"&gt;# 8 GB card&lt;/span&gt;
ollama run qwen3:14b       &lt;span class="c"&gt;# 12-16 GB&lt;/span&gt;
ollama run qwen3.8:27b     &lt;span class="c"&gt;# 24 GB&lt;/span&gt;
ollama run qwen3-coder:30b &lt;span class="c"&gt;# 24 GB&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ollama tags are separate builds, so their sizes differ a little from the Q4_K_M GGUF figures here; if the fit is tight, take the next card size.&lt;/p&gt;

&lt;p&gt;For the big models and for an API with several users, use vLLM. This is how the model card runs Qwen3 Next 80B-A3B: four cards, a 256K context, vLLM 0.10.2 or newer.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;vllm serve Qwen/Qwen3-Next-80B-A3B-Instruct &lt;span class="nt"&gt;--port&lt;/span&gt; 8000 &lt;span class="nt"&gt;--tensor-parallel-size&lt;/span&gt; 4 &lt;span class="nt"&gt;--max-model-len&lt;/span&gt; 262144
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;--tensor-parallel-size&lt;/code&gt; sets the number of cards and &lt;code&gt;--max-model-len&lt;/code&gt; the context length.&lt;/p&gt;

&lt;p&gt;No card that big? You can rent one by the minute on &lt;a href="https://gpu.nz/en/gpu" rel="noopener noreferrer"&gt;gpu.nz&lt;/a&gt; (my service): Qwen3.8 27B has a ready &lt;a href="https://gpu.nz/en/templates/chat-qwen38" rel="noopener noreferrer"&gt;template&lt;/a&gt; with a chat and an API on a 24 GB card, and the rest run through the &lt;a href="https://gpu.nz/en/templates/ollama" rel="noopener noreferrer"&gt;Open WebUI + Ollama&lt;/a&gt; template.&lt;/p&gt;

&lt;h2&gt;
  
  
  Check your own setup
&lt;/h2&gt;

&lt;p&gt;Pick the model, quantisation and context length in the free &lt;a href="https://gpu.nz/en/tools/vram" rel="noopener noreferrer"&gt;VRAM calculator&lt;/a&gt;: it shows the memory breakdown and which GPU sizes hold it. Every model above also has a page with the full quantisation table.&lt;/p&gt;

&lt;h2&gt;
  
  
  What can go wrong
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Thinking uses up context. Qwen3, Qwen3.5 and Qwen3.8 think first by default, and long reasoning grows the cache. In Qwen3 and Qwen3.5 you can switch it off with enable_thinking=False. Coder has no thinking.&lt;/li&gt;
&lt;li&gt;Parallel requests multiply the cache. The figures above are for one request. In vLLM every concurrent request keeps its own cache, so an API for several people needs headroom.&lt;/li&gt;
&lt;li&gt;A tight fit. Above 92% of the card's memory the model starts, but a long conversation will not fit. The table marks Qwen3 Next 80B-A3B, Qwen3 235B-A22B and Qwen3 Coder 480B-A35B this way.&lt;/li&gt;
&lt;li&gt;Licences differ. The cards of Qwen3.8 27B, Qwen3.5 9B, Qwen3 32B and 235B, Next and Coder say Apache-2.0, while Qwen3.8 Flash-Next says qwen-community-1.0. Open the card of your model before a commercial launch.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  How we counted
&lt;/h2&gt;

&lt;p&gt;Weights are the real file sizes: GGUF from Hugging Face, BF16 from Qwen's original repositories. The context cache is computed from each model's config.json, counting the linear layers of the hybrid models. The runtime is about 0.75 GB plus the logits buffer. A card counts as enough when the model takes at most 97.5% of its memory. 1 GB here is 1024³ bytes, as on GPUs. Data as of 2026-10-19; the method is in the &lt;a href="https://gpu.nz/en/tools/vram" rel="noopener noreferrer"&gt;calculator&lt;/a&gt;, and one table for 35 models is in &lt;a href="https://gpu.nz/en/guides/llm-vram-table" rel="noopener noreferrer"&gt;How much VRAM do open LLMs really need?&lt;/a&gt;. Ollama tags by memory step are in the &lt;a href="https://gpu.nz/en/guides/best-ollama-models" rel="noopener noreferrer"&gt;Ollama model roundup&lt;/a&gt;, and a neighbouring read is &lt;a href="https://gpu.nz/en/guides/gemma-4-vram" rel="noopener noreferrer"&gt;Gemma 4: which version fits&lt;/a&gt;.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Disclosure: I run &lt;a href="https://gpu.nz/en" rel="noopener noreferrer"&gt;gpu.nz&lt;/a&gt;, a per-minute GPU rental; the memory figures come from the model files and configs and do not depend on that. The original, with a live calculator, is at &lt;a href="https://gpu.nz/en/guides/which-qwen-model" rel="noopener noreferrer"&gt;gpu.nz/en/guides/which-qwen-model&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>llm</category>
      <category>ai</category>
      <category>machinelearning</category>
      <category>gpu</category>
    </item>
    <item>
      <title>Ollama vs LM Studio vs llama.cpp vs vLLM: which to pick, and how much VRAM each needs</title>
      <dc:creator>Gosha</dc:creator>
      <pubDate>Wed, 21 Oct 2026 03:25:39 +0000</pubDate>
      <link>https://dev.to/gpunz/ollama-vs-lm-studio-vs-llamacpp-vs-vllm-which-to-pick-and-how-much-vram-each-needs-3647</link>
      <guid>https://dev.to/gpunz/ollama-vs-lm-studio-vs-llamacpp-vs-vllm-which-to-pick-and-how-much-vram-each-needs-3647</guid>
      <description>&lt;p&gt;At home, for yourself, pick LM Studio or Ollama; for an API on a server, pick Ollama for one user and vLLM for many. The choice changes the memory too: Qwen3.8 27B takes 17.2 GB as GGUF Q4_K_M and 52.8 GB as the original BF16 weights that vLLM runs. Below: a feature table checked against the docs on 2026-10-20, the memory for each format and what goes wrong.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;17.2 GB&lt;/strong&gt; — Qwen3.8 27B, GGUF Q4_K_M, 8K context: Ollama, LM Studio, llama.cpp&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;30.6 GB&lt;/strong&gt; — the same model at FP8, 8K context: vLLM&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;52.8 GB&lt;/strong&gt; — original BF16 weights, 8K context: vLLM&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;40.2 GB&lt;/strong&gt; — FP8 with 16 requests at once, 8K each&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;em&gt;Computed by the &lt;a href="https://gpu.nz/en/tools/vram" rel="noopener noreferrer"&gt;VRAM calculator&lt;/a&gt;, data as of 2026-10-19.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  How do Ollama, LM Studio, llama.cpp and vLLM differ?
&lt;/h2&gt;

&lt;p&gt;All four run open models, but they split the work differently: LM Studio and Ollama are ready-made programs, llama.cpp is an engine with many flags, and vLLM is a server for an API under load.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Ollama is a program that downloads a model and runs it with one command, ollama run. It has a REST API and an OpenAI-compatible /v1 address.&lt;/li&gt;
&lt;li&gt;LM Studio is a desktop app: a chat, a model catalog and a local-server switch. For servers there is a windowless mode (llmster) and the lms command-line tool. It runs models on the llama.cpp engine, and on Macs with Apple Silicon on MLX as well.&lt;/li&gt;
&lt;li&gt;llama.cpp is an open C/C++ engine under the MIT licence. It also provides llama-server, with an OpenAI-compatible API and a web interface.&lt;/li&gt;
&lt;li&gt;vLLM is a library and server that serves models through an API. It was built for simultaneous requests and has no web chat.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;What they can do in practice is in the table. Its rows come from the projects' READMEs and documentation, checked on 2026-10-20.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Feature&lt;/th&gt;
&lt;th&gt;Ollama&lt;/th&gt;
&lt;th&gt;LM Studio&lt;/th&gt;
&lt;th&gt;llama.cpp&lt;/th&gt;
&lt;th&gt;vLLM&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Interface&lt;/td&gt;
&lt;td&gt;Terminal and API, an app on Windows and macOS; Open WebUI adds a web chat&lt;/td&gt;
&lt;td&gt;A window with a chat, a model catalog and a server switch&lt;/td&gt;
&lt;td&gt;Terminal and API; llama-server has a web interface&lt;/td&gt;
&lt;td&gt;API only, no ready web chat&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;API&lt;/td&gt;
&lt;td&gt;Its own REST API and /v1 in the OpenAI format (a subset), port 11434&lt;/td&gt;
&lt;td&gt;OpenAI-compatible /v1 and an Anthropic-compatible one; port 1234 in the examples&lt;/td&gt;
&lt;td&gt;OpenAI-compatible chat completions, responses and embeddings, plus Anthropic Messages; --api-key&lt;/td&gt;
&lt;td&gt;OpenAI-compatible (chat, completions, responses, embeddings) and Anthropic Messages; started with vllm serve&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Several users&lt;/td&gt;
&lt;td&gt;By default 1 request per model, the rest queue; set with OLLAMA_NUM_PARALLEL&lt;/td&gt;
&lt;td&gt;Continuous batching in the llama.cpp engine, 4 requests at once by default&lt;/td&gt;
&lt;td&gt;Parallel slots (-np) and continuous batching, on by default&lt;/td&gt;
&lt;td&gt;Continuous batching and PagedAttention: built for throughput&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Weight formats&lt;/td&gt;
&lt;td&gt;GGUF, and safetensors on import; the Ollama library and hf.co/…&lt;/td&gt;
&lt;td&gt;GGUF; MLX on Macs with Apple Silicon&lt;/td&gt;
&lt;td&gt;GGUF, 1.5- to 8-bit quantisation&lt;/td&gt;
&lt;td&gt;Hugging Face weights: FP8, INT8, INT4, GPTQ, AWQ. GGUF is “highly experimental”&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Systems&lt;/td&gt;
&lt;td&gt;Windows 10 22H2+, macOS, Linux, Docker; NVIDIA, AMD, Apple Metal, Vulkan&lt;/td&gt;
&lt;td&gt;Windows (x64, ARM), Linux (x64, ARM64), macOS on Apple Silicon only&lt;/td&gt;
&lt;td&gt;Windows, macOS, Linux, Docker; CUDA, HIP, Metal, Vulkan, SYCL, CPU&lt;/td&gt;
&lt;td&gt;Linux; Windows only through WSL; NVIDIA from compute capability 7.5, AMD, Intel&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Licence&lt;/td&gt;
&lt;td&gt;MIT&lt;/td&gt;
&lt;td&gt;Free at home and at work (since July 2025)&lt;/td&gt;
&lt;td&gt;MIT&lt;/td&gt;
&lt;td&gt;Apache 2.0&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Sources: docs.ollama.com and the Ollama README, lmstudio.ai/docs, the ggml-org/llama.cpp repository on GitHub, docs.vllm.ai. We did not measure generation speed and do not compare it: it depends on the card, the model and the quantisation.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where to download Ollama, LM Studio, llama.cpp and vLLM
&lt;/h2&gt;

&lt;p&gt;Ollama, llama.cpp and vLLM install with a terminal command; LM Studio is an ordinary app from lmstudio.ai. The commands come from the projects' documentation.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Ollama: Linux and macOS&lt;/span&gt;
curl &lt;span class="nt"&gt;-fsSL&lt;/span&gt; https://ollama.com/install.sh | sh

&lt;span class="c"&gt;# Ollama: Windows, PowerShell&lt;/span&gt;
irm https://ollama.com/install.ps1 | iex

&lt;span class="c"&gt;# llama.cpp: Linux and macOS&lt;/span&gt;
curl &lt;span class="nt"&gt;-LsSf&lt;/span&gt; https://llama.app/install.sh | sh

&lt;span class="c"&gt;# vLLM: Linux with an NVIDIA card&lt;/span&gt;
uv pip &lt;span class="nb"&gt;install &lt;/span&gt;vllm &lt;span class="nt"&gt;--torch-backend&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;auto
vllm download-kernels
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;LM Studio runs on Windows, Linux and Macs with an Apple Silicon chip; it does not run on Intel Macs. llama.cpp has ready builds on the Releases page of the ggml-org/llama.cpp repository, and a Docker image. vLLM needs Linux and, for NVIDIA cards, compute capability 7.5 or higher: an RTX 20-series card or newer.&lt;/p&gt;

&lt;h2&gt;
  
  
  How much memory does one model take in each format?
&lt;/h2&gt;

&lt;p&gt;The same Qwen3.8 27B takes from 17.2 GB to 52.8 GB, and the difference is in the weight format, not the program. Ollama, LM Studio and llama.cpp work with quantised GGUF files: Q4_K_M stores about 4.8 bits per parameter instead of 16. vLLM takes Hugging Face weights: the original BF16, or compressed FP8, AWQ or GPTQ.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Qwen3.8 27B: 17.2 GB at Q4_K_M and 52.8 GB at BF16 — a 24 GB card and an 80 GB one&lt;/strong&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;GB&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;strong&gt;Q4_K_M&lt;/strong&gt; (GGUF: Ollama, LM Studio, llama.cpp)&lt;/td&gt;
&lt;td&gt;17.2&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Q8_0 (GGUF)&lt;/td&gt;
&lt;td&gt;28.9&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;strong&gt;FP8&lt;/strong&gt; (vLLM)&lt;/td&gt;
&lt;td&gt;30.6&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BF16 (original weights, vLLM)&lt;/td&gt;
&lt;td&gt;52.8&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;em&gt;8K context, one request, FP16 cache. Weights are file sizes; data as of 2026-10-19.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The Q4_K_M weights are 3.3 times lighter than BF16 (15.3 GB against 50.9 GB). Q4_K_M fits a 24 GB card. Q8_0 and FP8 need a 32 GB card, and FP8 almost fills it. BF16 needs an 80 GB card.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Qwen3.8 27B at BF16 does not fit a 48 GB card.&lt;/strong&gt; The BF16 weights are 50.9 GB, and 52.8 GB with the cache and buffers. They will not start on a 48 GB card: you need an 80 GB card or the FP8 build (30.6 GB). The first vLLM start takes 10–30 minutes or more, and the card is billed the whole time. Work out the memory before you rent.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;&lt;a href="https://gpu.nz/en/models/gpt-oss-20b" rel="noopener noreferrer"&gt;GPT-OSS 20B&lt;/a&gt; looks different: it was released in MXFP4 from the start, and its weights are 11.3 GB. With an 8K context it needs 12.6 GB, a 16 GB card. Expanding the weights to BF16 gives about 40.3 GB (estimated from the parameter count) and a 48 GB card. Before you pick a program, check the format the model was released in: the memory depends on it.&lt;/p&gt;

&lt;h2&gt;
  
  
  What changes when many requests arrive?
&lt;/h2&gt;

&lt;p&gt;All four can serve several requests at once, but every request has its own context cache, and it takes GPU memory. The Ollama documentation says so directly: memory grows as OLLAMA_NUM_PARALLEL × OLLAMA_CONTEXT_LENGTH.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Qwen3.8 27B at FP8: one request 30.6 GB, 16 at once at 8K 40.2 GB, at 32K 64.2 GB&lt;/strong&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;GB&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;strong&gt;1 request&lt;/strong&gt; (8K)&lt;/td&gt;
&lt;td&gt;30.6&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;strong&gt;4 requests&lt;/strong&gt; (8K)&lt;/td&gt;
&lt;td&gt;32.5&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;strong&gt;16 requests&lt;/strong&gt; (8K)&lt;/td&gt;
&lt;td&gt;40.2&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;16 requests, 32K (32K window each)&lt;/td&gt;
&lt;td&gt;64.2&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;em&gt;FP8 weights, FP16 cache. Every request fills its window, so this is an upper estimate. Data as of 2026-10-19.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;For one request with an 8K window, the Qwen3.8 27B cache takes 0.64 GB: the model has a hybrid architecture, so it is modest. For 16 requests that is 10.3 GB, and at a 32K window 34.3 GB. One request fits a 32 GB card, four need 48 GB, and sixteen at 32K need 80 GB.&lt;/p&gt;

&lt;p&gt;The number of simultaneous requests is set differently in each tool. In Ollama it is the OLLAMA_NUM_PARALLEL variable, in llama.cpp the -np flag (the number of slots), in LM Studio the Max Concurrent Predictions setting. In vLLM it is limited by the memory left after the weights.&lt;/p&gt;

&lt;h2&gt;
  
  
  What to pick: a PC, a server or maximum control
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;One person, a home computer. LM Studio if you want a window with a chat and a model catalog. Ollama if you prefer the command line and need an API for your own programs. On Windows and Mac these two are the easiest.&lt;/li&gt;
&lt;li&gt;No GPU of your own, or the model does not fit. Rent a server with Ollama: the template gives a browser chat and an API. It is enough for you and a small team.&lt;/li&gt;
&lt;li&gt;An API for many users or a service. vLLM: continuous batching, Hugging Face weights, FP8 and AWQ. It needs Linux; on Windows it runs only through WSL.&lt;/li&gt;
&lt;li&gt;You need to squeeze a model into limited memory. llama.cpp: fine control over the cache, layers and experts, described below.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;llama.cpp helps when a model barely fits. It can store the K and V cache compressed: the flags -ctk q8_0 -ctv q8_0 make it about half the size. For Qwen3.8 27B at Q4_K_M with a 128K context that is 20.9 GB instead of 24.7 GB, so a 24 GB card is enough instead of 32 GB. Ollama has the same trick (OLLAMA_KV_CACHE_TYPE), but with one cache type for all models. Its documentation calls the precision loss at q8_0 very small.&lt;/p&gt;

&lt;p&gt;With MoE models, llama.cpp can keep the experts in system RAM (the --cpu-moe and --n-cpu-moe flags), and a model that does not fit in GPU memory runs in a hybrid CPU and GPU mode. In that mode answers are slower than with the whole model in GPU memory.&lt;/p&gt;

&lt;h2&gt;
  
  
  Check your own model
&lt;/h2&gt;

&lt;p&gt;Pick a model, a format and a context in the free &lt;a href="https://gpu.nz/en/tools/vram" rel="noopener noreferrer"&gt;VRAM calculator&lt;/a&gt;: it shows the memory breakdown and which GPU sizes hold it. For vLLM look at the FP8 and BF16 options, for Ollama, LM Studio and llama.cpp at Q4_K_M and Q8_0. The rule of thumb is in the &lt;a href="https://gpu.nz/en/guides/vram-for-llm" rel="noopener noreferrer"&gt;VRAM guide for LLMs&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  What can go wrong
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Parallel requests multiply the cache. The first tables are computed for one request. For an API with several people, add memory for the cache of every simultaneous request, especially with a long context.&lt;/li&gt;
&lt;li&gt;GGUF in vLLM is not the main path. The vLLM documentation calls GGUF support highly experimental and requires a separate plugin. For vLLM take FP8, AWQ or the original weights.&lt;/li&gt;
&lt;li&gt;vLLM takes almost the whole card. The --gpu-memory-utilization parameter sets the share, so other jobs on the same card usually cannot run.&lt;/li&gt;
&lt;li&gt;The vLLM key does not guard every address. By the documentation, --api-key is checked only on the /v1, /v2 and /inference paths. If you install vLLM yourself, put the port behind a proxy.&lt;/li&gt;
&lt;li&gt;The model did not fit in GPU memory. Ollama and llama.cpp put some layers in system RAM and still start, but answers become noticeably slower. In Ollama the share shows in the Processor column of ollama ps.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  How we counted
&lt;/h2&gt;

&lt;p&gt;Weights are real file sizes: GGUF from Hugging Face repositories, FP8 and BF16 from the models' original repositories. The BF16 size of GPT-OSS 20B is computed from its parameter count. The context cache is computed from the model's config.json; for several requests it is multiplied by their number, each request filling its whole window. The runtime is about 0.75 GB plus the logits buffer. 1 GB here is 1024³ bytes, as on GPUs. Calculator data as of 2026-10-19; the method is in the &lt;a href="https://gpu.nz/en/tools/vram" rel="noopener noreferrer"&gt;calculator&lt;/a&gt;, and one table for 35 models is in &lt;a href="https://gpu.nz/en/guides/llm-vram-table" rel="noopener noreferrer"&gt;How much VRAM do open LLMs really need?&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;The tools' features were checked against their official documentation on 2026-10-20: the README and docs.ollama.com, lmstudio.ai/docs, the llama.cpp README and server documentation, and docs.vllm.ai. Releases are frequent, so open the project page before you install.&lt;/p&gt;

&lt;h2&gt;
  
  
  No GPU that big?
&lt;/h2&gt;

&lt;p&gt;If the model you want does not fit your own card, the quickest test is a rented GPU for an hour. On gpu.nz (my service, billed by the minute) Ollama and vLLM have ready templates: &lt;a href="https://gpu.nz/en/templates/ollama" rel="noopener noreferrer"&gt;Open WebUI + Ollama&lt;/a&gt; from an 8 GB card and &lt;a href="https://gpu.nz/en/templates/vllm" rel="noopener noreferrer"&gt;vLLM&lt;/a&gt; from 24 GB, both with an OpenAI-compatible &lt;code&gt;/v1&lt;/code&gt; API. The walkthrough with request examples and common mistakes is in &lt;a href="https://gpu.nz/en/guides/ollama-vllm-api" rel="noopener noreferrer"&gt;Ollama and vLLM on a rented GPU&lt;/a&gt;. LM Studio and llama.cpp you install yourself over &lt;a href="https://gpu.nz/en/guides/ssh-gpu-server" rel="noopener noreferrer"&gt;SSH&lt;/a&gt;.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Disclosure: I run &lt;a href="https://gpu.nz/en" rel="noopener noreferrer"&gt;gpu.nz&lt;/a&gt;, a per-minute GPU rental; the feature table and the memory figures do not depend on that. The original, with a live calculator, is at &lt;a href="https://gpu.nz/en/guides/ollama-lm-studio-llama-cpp-vllm" rel="noopener noreferrer"&gt;gpu.nz/en/guides/ollama-lm-studio-llama-cpp-vllm&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>llm</category>
      <category>ollama</category>
      <category>ai</category>
      <category>machinelearning</category>
    </item>
    <item>
      <title>How much VRAM do open LLMs really need? Real GGUF sizes for 35 models (October 2026)</title>
      <dc:creator>Gosha</dc:creator>
      <pubDate>Tue, 20 Oct 2026 15:55:43 +0000</pubDate>
      <link>https://dev.to/gpunz/how-much-vram-do-open-llms-really-need-real-gguf-sizes-for-35-models-october-2026-28n2</link>
      <guid>https://dev.to/gpunz/how-much-vram-do-open-llms-really-need-real-gguf-sizes-for-35-models-october-2026-28n2</guid>
      <description>&lt;p&gt;"Parameters × bits ÷ 8" is the rule everyone quotes for how much GPU memory a model needs. It gets you in the right neighbourhood and then lets you down exactly where it matters: on the card that is 2 GB too small, or on the 32k-token chat that suddenly no longer fits.&lt;/p&gt;

&lt;p&gt;So I stopped estimating and pulled the numbers from the source for 35 current open models:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Weights&lt;/strong&gt;: the real size of the published file — GGUF builds by unsloth, ggml-org, bartowski and the model authors (Q4_K_M unless the model ships quantised natively, like GPT-OSS in MXFP4), read from the Hugging Face API.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Context (KV) cache&lt;/strong&gt;: computed from each model's own &lt;code&gt;config.json&lt;/code&gt; — number of layers, KV heads and head size — including the architectures where the naive formula is wrong (more on that below).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Runtime&lt;/strong&gt;: CUDA context and buffers, about 0.75 GiB plus the logits buffer of a 512-token batch (it adds up for 256k-token vocabularies).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;All figures are GiB (1024³ bytes), the unit GPU memory is sold in, so they compare directly with "24 GB". Hugging Face lists decimal gigabytes, which is why its file sizes look ~7% larger.&lt;/p&gt;

&lt;h2&gt;
  
  
  The table
&lt;/h2&gt;

&lt;p&gt;One GPU, one request, FP16 cache. "Smallest single card" is the smallest common card size where the total fits in 97.5% of the memory (the driver and CUDA take the rest).&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model&lt;/th&gt;
&lt;th&gt;Params&lt;/th&gt;
&lt;th&gt;Weights (format, GiB)&lt;/th&gt;
&lt;th&gt;VRAM @ 8k ctx&lt;/th&gt;
&lt;th&gt;VRAM @ 32k ctx&lt;/th&gt;
&lt;th&gt;Smallest single card (8k)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3.8 27B&lt;/td&gt;
&lt;td&gt;27.78B&lt;/td&gt;
&lt;td&gt;Q4_K_M 15.3&lt;/td&gt;
&lt;td&gt;17.2&lt;/td&gt;
&lt;td&gt;18.7&lt;/td&gt;
&lt;td&gt;24 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3.5 9B&lt;/td&gt;
&lt;td&gt;9.65B&lt;/td&gt;
&lt;td&gt;Q4_K_M 5.3&lt;/td&gt;
&lt;td&gt;6.8&lt;/td&gt;
&lt;td&gt;7.6&lt;/td&gt;
&lt;td&gt;8 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3.5 4B&lt;/td&gt;
&lt;td&gt;4.66B&lt;/td&gt;
&lt;td&gt;Q4_K_M 2.6&lt;/td&gt;
&lt;td&gt;4.1&lt;/td&gt;
&lt;td&gt;4.8&lt;/td&gt;
&lt;td&gt;8 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3 8B&lt;/td&gt;
&lt;td&gt;8.19B&lt;/td&gt;
&lt;td&gt;Q4_K_M 4.7&lt;/td&gt;
&lt;td&gt;6.8&lt;/td&gt;
&lt;td&gt;10.2&lt;/td&gt;
&lt;td&gt;8 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3 14B&lt;/td&gt;
&lt;td&gt;14.77B&lt;/td&gt;
&lt;td&gt;Q4_K_M 8.4&lt;/td&gt;
&lt;td&gt;10.7&lt;/td&gt;
&lt;td&gt;14.4&lt;/td&gt;
&lt;td&gt;12 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3 32B&lt;/td&gt;
&lt;td&gt;32.76B&lt;/td&gt;
&lt;td&gt;Q4_K_M 18.4&lt;/td&gt;
&lt;td&gt;21.4&lt;/td&gt;
&lt;td&gt;27.4&lt;/td&gt;
&lt;td&gt;24 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3 30B-A3B&lt;/td&gt;
&lt;td&gt;30.53B (3.3B active)&lt;/td&gt;
&lt;td&gt;Q4_K_M 17.3&lt;/td&gt;
&lt;td&gt;19.1&lt;/td&gt;
&lt;td&gt;21.3&lt;/td&gt;
&lt;td&gt;24 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3 Coder 30B-A3B&lt;/td&gt;
&lt;td&gt;30.53B (3.3B active)&lt;/td&gt;
&lt;td&gt;Q4_K_M 17.3&lt;/td&gt;
&lt;td&gt;19.1&lt;/td&gt;
&lt;td&gt;21.3&lt;/td&gt;
&lt;td&gt;24 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3 Next 80B-A3B&lt;/td&gt;
&lt;td&gt;81.32B (3B active)&lt;/td&gt;
&lt;td&gt;Q4_K_M 45.2&lt;/td&gt;
&lt;td&gt;46.5&lt;/td&gt;
&lt;td&gt;47.0&lt;/td&gt;
&lt;td&gt;48 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3 235B-A22B&lt;/td&gt;
&lt;td&gt;235.09B (22B active)&lt;/td&gt;
&lt;td&gt;Q4_K_M 132.4&lt;/td&gt;
&lt;td&gt;134.9&lt;/td&gt;
&lt;td&gt;139.3&lt;/td&gt;
&lt;td&gt;141 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3 Coder 480B-A35B&lt;/td&gt;
&lt;td&gt;480.15B (35B active)&lt;/td&gt;
&lt;td&gt;Q4_K_M 270.1&lt;/td&gt;
&lt;td&gt;273.1&lt;/td&gt;
&lt;td&gt;278.9&lt;/td&gt;
&lt;td&gt;288 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Gemma 4 12B&lt;/td&gt;
&lt;td&gt;11.96B&lt;/td&gt;
&lt;td&gt;Q4_K_M 6.6&lt;/td&gt;
&lt;td&gt;8.3&lt;/td&gt;
&lt;td&gt;8.7&lt;/td&gt;
&lt;td&gt;12 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Gemma 4 26B-A4B&lt;/td&gt;
&lt;td&gt;25.81B (4B active)&lt;/td&gt;
&lt;td&gt;Q4_K_M 15.8&lt;/td&gt;
&lt;td&gt;17.4&lt;/td&gt;
&lt;td&gt;17.9&lt;/td&gt;
&lt;td&gt;24 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Gemma 4 31B&lt;/td&gt;
&lt;td&gt;31.27B&lt;/td&gt;
&lt;td&gt;Q4_K_M 17.1&lt;/td&gt;
&lt;td&gt;19.7&lt;/td&gt;
&lt;td&gt;21.6&lt;/td&gt;
&lt;td&gt;24 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Gemma 3 27B&lt;/td&gt;
&lt;td&gt;27.43B&lt;/td&gt;
&lt;td&gt;Q4_K_M 15.4&lt;/td&gt;
&lt;td&gt;17.7&lt;/td&gt;
&lt;td&gt;19.6&lt;/td&gt;
&lt;td&gt;24 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GPT-OSS 20B&lt;/td&gt;
&lt;td&gt;20.91B (3.6B active)&lt;/td&gt;
&lt;td&gt;MXFP4 11.3&lt;/td&gt;
&lt;td&gt;12.6&lt;/td&gt;
&lt;td&gt;13.2&lt;/td&gt;
&lt;td&gt;16 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GPT-OSS 120B&lt;/td&gt;
&lt;td&gt;116.83B (5.1B active)&lt;/td&gt;
&lt;td&gt;MXFP4 59.0&lt;/td&gt;
&lt;td&gt;60.5&lt;/td&gt;
&lt;td&gt;61.3&lt;/td&gt;
&lt;td&gt;80 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DeepSeek R1 671B&lt;/td&gt;
&lt;td&gt;684.53B (37B active)&lt;/td&gt;
&lt;td&gt;Q4_K_M 377.1&lt;/td&gt;
&lt;td&gt;378.7&lt;/td&gt;
&lt;td&gt;380.3&lt;/td&gt;
&lt;td&gt;multi-GPU&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DeepSeek V3.2&lt;/td&gt;
&lt;td&gt;685.36B (37B active)&lt;/td&gt;
&lt;td&gt;Q4_K_M 377.6&lt;/td&gt;
&lt;td&gt;379.1&lt;/td&gt;
&lt;td&gt;380.7&lt;/td&gt;
&lt;td&gt;multi-GPU&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DeepSeek V4 Flash&lt;/td&gt;
&lt;td&gt;290.94B&lt;/td&gt;
&lt;td&gt;UD-Q4_K_XL 144.4&lt;/td&gt;
&lt;td&gt;145.5&lt;/td&gt;
&lt;td&gt;145.8&lt;/td&gt;
&lt;td&gt;180 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DeepSeek R1 Distill Qwen 32B&lt;/td&gt;
&lt;td&gt;32.76B&lt;/td&gt;
&lt;td&gt;Q4_K_M 18.5&lt;/td&gt;
&lt;td&gt;21.5&lt;/td&gt;
&lt;td&gt;27.5&lt;/td&gt;
&lt;td&gt;24 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DeepSeek R1 0528 Qwen3 8B&lt;/td&gt;
&lt;td&gt;8.19B&lt;/td&gt;
&lt;td&gt;Q4_K_M 4.7&lt;/td&gt;
&lt;td&gt;6.8&lt;/td&gt;
&lt;td&gt;10.2&lt;/td&gt;
&lt;td&gt;8 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Llama 3.1 8B&lt;/td&gt;
&lt;td&gt;8.03B&lt;/td&gt;
&lt;td&gt;Q4_K_M 4.6&lt;/td&gt;
&lt;td&gt;6.6&lt;/td&gt;
&lt;td&gt;9.6&lt;/td&gt;
&lt;td&gt;8 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Llama 3.3 70B&lt;/td&gt;
&lt;td&gt;70.55B&lt;/td&gt;
&lt;td&gt;Q4_K_M 39.6&lt;/td&gt;
&lt;td&gt;43.1&lt;/td&gt;
&lt;td&gt;50.6&lt;/td&gt;
&lt;td&gt;48 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Mistral Small 3.2 24B&lt;/td&gt;
&lt;td&gt;24.01B&lt;/td&gt;
&lt;td&gt;Q4_K_M 13.3&lt;/td&gt;
&lt;td&gt;15.6&lt;/td&gt;
&lt;td&gt;19.3&lt;/td&gt;
&lt;td&gt;16 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Devstral Small 2 24B&lt;/td&gt;
&lt;td&gt;24.01B&lt;/td&gt;
&lt;td&gt;Q4_K_M 13.3&lt;/td&gt;
&lt;td&gt;15.6&lt;/td&gt;
&lt;td&gt;19.3&lt;/td&gt;
&lt;td&gt;16 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Ministral 3 14B&lt;/td&gt;
&lt;td&gt;13.95B&lt;/td&gt;
&lt;td&gt;Q4_K_M 7.7&lt;/td&gt;
&lt;td&gt;9.9&lt;/td&gt;
&lt;td&gt;13.7&lt;/td&gt;
&lt;td&gt;12 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Mistral Nemo 12B&lt;/td&gt;
&lt;td&gt;12.25B&lt;/td&gt;
&lt;td&gt;Q4_K_M 7.0&lt;/td&gt;
&lt;td&gt;9.2&lt;/td&gt;
&lt;td&gt;13.0&lt;/td&gt;
&lt;td&gt;12 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GLM-4.7 Flash&lt;/td&gt;
&lt;td&gt;31.22B (3B active)&lt;/td&gt;
&lt;td&gt;Q4_K_M 17.1&lt;/td&gt;
&lt;td&gt;18.5&lt;/td&gt;
&lt;td&gt;19.8&lt;/td&gt;
&lt;td&gt;24 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GLM-4.5 Air&lt;/td&gt;
&lt;td&gt;110.47B (12B active)&lt;/td&gt;
&lt;td&gt;Q4_K_M 68.0&lt;/td&gt;
&lt;td&gt;70.4&lt;/td&gt;
&lt;td&gt;74.8&lt;/td&gt;
&lt;td&gt;80 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GLM-5.3 Flash&lt;/td&gt;
&lt;td&gt;321.32B&lt;/td&gt;
&lt;td&gt;UD-Q4_K_XL 186.0&lt;/td&gt;
&lt;td&gt;187.3&lt;/td&gt;
&lt;td&gt;187.6&lt;/td&gt;
&lt;td&gt;288 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Kimi K2 1T&lt;/td&gt;
&lt;td&gt;1026.47B (32B active)&lt;/td&gt;
&lt;td&gt;Q4_K_M 578.6&lt;/td&gt;
&lt;td&gt;580.2&lt;/td&gt;
&lt;td&gt;581.8&lt;/td&gt;
&lt;td&gt;multi-GPU&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Hermes 4.3 36B&lt;/td&gt;
&lt;td&gt;36.15B&lt;/td&gt;
&lt;td&gt;Q4_K_M 20.3&lt;/td&gt;
&lt;td&gt;23.3&lt;/td&gt;
&lt;td&gt;29.3&lt;/td&gt;
&lt;td&gt;24 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Phi-4 14B&lt;/td&gt;
&lt;td&gt;14.66B&lt;/td&gt;
&lt;td&gt;Q4_K_M 8.3&lt;/td&gt;
&lt;td&gt;10.8&lt;/td&gt;
&lt;td&gt;12.3&lt;/td&gt;
&lt;td&gt;12 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Hunyuan Hy3&lt;/td&gt;
&lt;td&gt;298.79B&lt;/td&gt;
&lt;td&gt;FP8 279.3&lt;/td&gt;
&lt;td&gt;282.8&lt;/td&gt;
&lt;td&gt;290.3&lt;/td&gt;
&lt;td&gt;multi-GPU&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  What surprised me
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Hybrid and sliding-window attention change the picture.&lt;/strong&gt; Qwen3 32B and Qwen3.8 27B are about the same size on disk, but going from an 8k to a 32k context costs Qwen3 32B &lt;strong&gt;+6 GiB&lt;/strong&gt; and Qwen3.8 27B &lt;strong&gt;+1.5 GiB&lt;/strong&gt;. Qwen3.8 keeps a KV cache on only 16 of its 64 layers; the other 48 are linear-attention layers with a fixed state per sequence. Gemma 4 does something similar with a 1024-token sliding window on 5 of every 6 layers. If you plan for long contexts, the architecture matters as much as the parameter count.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;MoE models need all of their weights in memory.&lt;/strong&gt; Qwen3 30B-A3B uses 3.3B parameters per token, but all 30.5B have to be loaded: it needs the same 24 GB card as a dense 27B model. It just runs much faster on it. (llama.cpp can keep experts in system RAM with &lt;code&gt;--n-cpu-moe&lt;/code&gt;, at a speed cost.)&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;GPT-OSS is the best fit for its size class.&lt;/strong&gt; GPT-OSS 20B ships in MXFP4 and fits a 16 GB card with room for context; GPT-OSS 120B fits a single 80 GB H100.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;70B still means 48 GB.&lt;/strong&gt; Llama 3.3 70B in Q4_K_M is 43 GiB at 8k context: a 48 GB card, or two 24 GB cards with tensor parallelism. At 32k it no longer fits in 48 GB without a quantised KV cache.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;DeepSeek's MLA cache is tiny.&lt;/strong&gt; DeepSeek V3.2 and R1 cache one 576-wide latent per token and layer instead of full keys and values: about 70 KB per token for the whole model. The weights are the problem there, not the context.&lt;/p&gt;

&lt;h2&gt;
  
  
  The formula, if you want to check a model yourself
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;weights   = file size of the quantisation you run (or params × bits ÷ 8)
kv_cache  = tokens × layers_with_full_attention × 2 × kv_heads × head_dim × 2 bytes
          + min(tokens, window) × sliding_layers × 2 × kv_heads × head_dim × 2 bytes
runtime   ≈ 0.75 GiB + 512 × vocab × 4 bytes
total     = weights + kv_cache × parallel_requests + runtime
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A q8_0 KV cache (&lt;code&gt;-ctk q8_0 -ctv q8_0&lt;/code&gt; in llama.cpp, &lt;code&gt;--kv-cache-dtype fp8&lt;/code&gt; in vLLM) roughly halves the second line. vLLM preallocates most of the card for its cache by default (&lt;code&gt;--gpu-memory-utilization&lt;/code&gt;), so &lt;code&gt;nvidia-smi&lt;/code&gt; will always look full there; the numbers above are what the model needs, not what vLLM grabs.&lt;/p&gt;

&lt;h2&gt;
  
  
  Calculator
&lt;/h2&gt;

&lt;p&gt;I turned this into a free calculator: pick a model (35 LLMs plus FLUX, Qwen-Image, Wan 2.2, HunyuanVideo and other image and video models), a quantisation, the context length and the number of parallel requests, and it shows the memory breakdown and the GPUs that fit: &lt;strong&gt;&lt;a href="https://gpu.nz/en/tools/vram" rel="noopener noreferrer"&gt;gpu.nz/en/tools/vram&lt;/a&gt;&lt;/strong&gt;. Every model also has a page with the full quantisation table: &lt;a href="https://gpu.nz/en/models" rel="noopener noreferrer"&gt;gpu.nz/en/models&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Disclosure: I run gpu.nz, a per-minute GPU rental, so the calculator also lists hourly prices of the cards that fit. The numbers themselves don't depend on that — if you spot a model where they look wrong, tell me and I'll check it against the files.&lt;/p&gt;

</description>
      <category>llm</category>
      <category>ai</category>
      <category>machinelearning</category>
      <category>gpu</category>
    </item>
  </channel>
</rss>
