<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Sparkle Qwen27b</title>
    <description>The latest articles on DEV Community by Sparkle Qwen27b (@csunx233).</description>
    <link>https://dev.to/csunx233</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4253811%2Fc42db226-8b6a-435d-a44c-32fdedf10fa5.png</url>
      <title>DEV Community: Sparkle Qwen27b</title>
      <link>https://dev.to/csunx233</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/csunx233"/>
    <language>en</language>
    <item>
      <title>NVFP4 Won't Save Your VRAM: Quantization Tradeoffs on a 12GB Laptop GPU</title>
      <dc:creator>Sparkle Qwen27b</dc:creator>
      <pubDate>Thu, 19 Nov 2026 12:32:59 +0000</pubDate>
      <link>https://dev.to/csunx233/nvfp4-wont-save-your-vram-quantization-tradeoffs-on-a-12gb-laptop-gpu-56ij</link>
      <guid>https://dev.to/csunx233/nvfp4-wont-save-your-vram-quantization-tradeoffs-on-a-12gb-laptop-gpu-56ij</guid>
      <description>

&lt;p&gt;Running a 27B model on a 12GB laptop GPU teaches you to question every assumption about memory — including the most obvious one: that a lower-precision format always saves VRAM.&lt;/p&gt;

&lt;p&gt;I run Qwen3 27B as a local agent on an RTX 5070 Ti Laptop (12GB VRAM, i7-14650HX, 32GB RAM, Windows), using NInfer + KVMem tiered KV + MTP3. The repo is Sparkle-Qwen27b-Kvmem-ninfer: &lt;a href="https://github.com/CSUNX233/Sparkle-Qwen27b-Kvmem-ninfer" rel="noopener noreferrer"&gt;https://github.com/CSUNX233/Sparkle-Qwen27b-Kvmem-ninfer&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;I switched the daily quantization profile from NVFP4 to RK4V4, and the VRAM numbers surprised me. Here is what I measured, and what it means if you are sizing a local setup.&lt;/p&gt;

&lt;h2&gt;
  
  
  The two profiles
&lt;/h2&gt;

&lt;p&gt;Same model, same machine, same workload shape:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Model: Qwen3.8-27B GSQ-RCO IQ3_XXS + MTP3&lt;/li&gt;
&lt;li&gt;KV budget: DeviceKV 36864 / Retrieve 3840 / HostKV 7168 MiB, xhigh&lt;/li&gt;
&lt;li&gt;Previous daily profile: NVFP4 — whole-GPU peak 11,168 MiB&lt;/li&gt;
&lt;li&gt;Current daily profile: RK4V4 + GDN FP32 — whole-GPU peak 11,220–11,236 MiB&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So switching to NVFP4 saved roughly 50–70 MiB at the peak. That sounds like a win for NVFP4, until you look closer: at the same pool size, RK4V4 actually uses about 20 MiB &lt;em&gt;less&lt;/em&gt;. The difference between the two peaks is the pool configuration, not the weight format.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why the weights are not the deciding factor
&lt;/h2&gt;

&lt;p&gt;Once a 27B model is already aggressively quantized, the weights are a modest slice of the 12GB budget. The KV cache is the part that scales with context, and the pool is sized to use almost everything available — deliberately. Peak VRAM here is dominated by the DeviceKV pool, not by whether the weights are NVFP4 or RK4V4.&lt;/p&gt;

&lt;p&gt;That is why "switch to a cheaper quantization" is bad first advice when you run out of memory on this kind of setup. The lever that actually moves VRAM is the pool size: if you OOM, drop the pool to 38,912 / 38,400 first. Swapping quantization to chase a rescue will not save you.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the quality check said
&lt;/h2&gt;

&lt;p&gt;The reason RK4V4 is the default anyway is quality, not VRAM. On a 39-question short-context check:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;RK4V4 + GDN32: 35/39 correct, mean total variation vs BF16 0.0262, 0 prediction flips&lt;/li&gt;
&lt;li&gt;NVFP4: 34/39 correct, 1 prediction flip&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Tool/JSON regression was 20/22 for both — same score, same failures. Long-context results held on the RK4V4 profile: a 110K task finished 12/12, a 218K task 26/26, and a 189,954-token real-content multi-turn run went 24/24. Peak VRAM stayed in the 11,220–11,236 MiB band.&lt;/p&gt;

&lt;p&gt;One question and one flip is a small sample, and I would not overclaim from it. But combined with the VRAM finding, the conclusion is clear enough for my setup: NVFP4 costs a little quality and buys essentially no VRAM. The cheapest quantization on paper is not always the cheapest once the KV budget is included.&lt;/p&gt;

&lt;h2&gt;
  
  
  Honest limits
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;These are measurements from one machine, not a unified benchmark. Different binaries, ports and fixtures across runs.&lt;/li&gt;
&lt;li&gt;256K is the logical context capacity (262,144 tokens). Full 256K is not validated; compaction is around 184K, and middle history is retrieved by page selection.&lt;/li&gt;
&lt;li&gt;The prebuilt engine is Windows x64 / sm120a only. RTX 30/40 need an architecture-matched build.&lt;/li&gt;
&lt;li&gt;The 232K random-order stress test was unstable across repeats and is not a release claim.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you run local models on consumer GPUs, I would like to hear your experience: where does your memory actually go — weights, KV cache, or something else entirely?&lt;/p&gt;

&lt;h2&gt;
  
  
  Repo: &lt;a href="https://github.com/CSUNX233/Sparkle-Qwen27b-Kvmem-ninfer" rel="noopener noreferrer"&gt;https://github.com/CSUNX233/Sparkle-Qwen27b-Kvmem-ninfer&lt;/a&gt;
&lt;/h2&gt;

</description>
      <category>llm</category>
      <category>localai</category>
      <category>gpu</category>
      <category>opensource</category>
    </item>
    <item>
      <title>How I Run a 27B Local Agent on a 12GB Laptop GPU: KVMem, Tiered KV, and Honest Long-Context Limits</title>
      <dc:creator>Sparkle Qwen27b</dc:creator>
      <pubDate>Thu, 19 Nov 2026 12:21:22 +0000</pubDate>
      <link>https://dev.to/csunx233/how-i-run-a-27b-local-agent-on-a-12gb-laptop-gpu-kvmem-tiered-kv-and-honest-long-context-limits-1d28</link>
      <guid>https://dev.to/csunx233/how-i-run-a-27b-local-agent-on-a-12gb-laptop-gpu-kvmem-tiered-kv-and-honest-long-context-limits-1d28</guid>
      <description>&lt;p&gt;Most local-LLM advice stops at the weights. Pick a smaller quantized model, fit it in VRAM, done. That works for chat. It breaks the moment you want a real agent: long files, long tool histories, multi-turn work. The weights are only half the memory problem. The KV cache is the other half, and it grows with context.&lt;/p&gt;

&lt;p&gt;Sparkle-Qwen27b-Kvmem-ninfer is my attempt to solve the second half on ordinary hardware. It runs Qwen3 27B as a local coding, files and web agent on an RTX 5070 Ti Laptop with 12GB of VRAM. The stack is NInfer inference, KVMem tiered KV, MTP3 speculative decoding, and a local agent workspace called dsh, with an optional Tauri desktop UI. The code is Apache-2.0.&lt;/p&gt;

&lt;p&gt;This article explains how the memory budget works, what KVMem actually does, what I measured, and where it still falls short. I am not going to claim this beats cloud models, or that 256K context means 256K of attention on every step. It does not.&lt;/p&gt;

&lt;h2&gt;
  
  
  The budget: 12GB is smaller than it looks
&lt;/h2&gt;

&lt;p&gt;The validated machine is an RTX 5070 Ti Laptop with 12GB VRAM, an i7-14650HX, and 32GB of system RAM, running Windows. The default model is Qwen3.8-27B GSQ-RCO IQ3_XXS with MTP3, using RK4V4 plus GDN FP32, with DeviceKV 36864, Retrieve 3840, and HostKV 7168 MiB.&lt;/p&gt;

&lt;p&gt;The daily peak is 11,220 to 11,236 MiB of whole-GPU memory. That leaves about 12 MiB of headroom. This is not a comfortable margin. It is deliberate: the pool is sized to use almost everything available, and the first advice when you run out of memory is to shrink the pool to 38,912 or 38,400, not to switch quantization and hope.&lt;/p&gt;

&lt;p&gt;Quantization choice matters here in a surprising way. The previous daily profile used NVFP4 and peaked at 11,168 MiB. The current RK4V4 profile peaks slightly higher, but it is the default anyway, because on a 39-question short-context check it scored 35 out of 39 correct with a mean total variation of 0.0262 against BF16 and zero prediction flips, while NVFP4 scored 34 out of 39 with one flip. NVFP4 is also not a VRAM-saving option here: at the same pool, RK4V4 already uses about 20 MiB less. The cheapest quantization on paper is not always the cheapest once the KV budget is included.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why the KV cache is the real problem
&lt;/h2&gt;

&lt;p&gt;Transformer inference stores a key and a value for every token in the context. That cache grows linearly with context length, and for a 27B model it grows fast. Hold the whole cache on the GPU and a long agent session exhausts 12GB long before the weights do. Evict aggressively and you lose the history the agent needs: the file it edited three turns ago, the error from the last tool call, the plan it agreed to at the start.&lt;/p&gt;

&lt;p&gt;The usual answers are to shorten the context, summarize early, or restart the session. All three work, and all three make the agent dumber at exactly the moment a task gets interesting. KVMem takes a different approach: do not keep all of the cache hot. Keep some of it, retrieve the rest.&lt;/p&gt;

&lt;h2&gt;
  
  
  How KVMem works
&lt;/h2&gt;

&lt;p&gt;KVMem divides the KV cache into tiers. The GPU holds hot pages: sink pages at the start, recent pages, and retrieved pages that a query scores highly. Cold history lives in host RAM — 7 GiB of Host KV in the default profile — and is restored to the GPU on demand.&lt;/p&gt;

&lt;p&gt;A request flows like this: the agent sends a request to the cache guard, which keeps the sent prefix stable and reduces repeated prefill; the engine scores the query against stored pages, selects a small set, and restores only those pages. The model then decodes with sink, recent, and retrieved pages in GPU memory. It does not reread all history on every step.&lt;/p&gt;

&lt;p&gt;Two details from v1.1.0 show why retrieval quality matters. First, evidence can span two KV pages. If the selector keeps only the second half, the model sees a fragment. The fix uses overlap-retrieval slots to add the predecessor page: at most eight pages per selection, replacing only lower-ranked members of the recent range, with no increase in total selected pages. Second, filling is allowed only after the known query end has been read and the full query has been scored. That change removed a regression that had appeared at 218K in an earlier candidate.&lt;/p&gt;

&lt;p&gt;MTP3 sits on top of this. It uses three draft tokens for speculative decoding, which reduces decode steps. Daily sampling stays unchanged. It does not change what the model knows; it changes how fast it produces tokens it was already going to produce.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I measured, and what I did not
&lt;/h2&gt;

&lt;p&gt;All measurements are from the maintainer's machine, with different binaries, ports and fixtures across runs. They are not a unified benchmark. I report them as fixtures, not leaderboard scores.&lt;/p&gt;

&lt;p&gt;A 110K task with 12 sequential pins finished 12 out of 12, producing 733 output tokens in 275.9 seconds at a peak of 11,232 MiB. The previous daily build had exhausted its 4,096 output budget without a final answer on the same fixture. A 218K task finished 26 out of 26, with normalized JSON matching the correct baseline. A 159K task finished 16 out of 16. A 232K sequential-pin task finished 24 out of 24, but sequential codes are easy to extrapolate, so I treat that as a weaker signal than the real-content test.&lt;/p&gt;

&lt;p&gt;The real-content test is the one I trust most. It used 189,954 tokens of real content across multiple turns, with 16 code facts and 8 earlier conversation facts to recall. It passed 24 out of 24. That is a better proxy for agent work than needle tests, because it requires the model to use retrieved history to answer questions about code it saw earlier in the same session.&lt;/p&gt;

&lt;p&gt;Short-context fidelity was checked on 39 questions. The default RK4V4 plus GDN32 scored 35 out of 39 with zero flips. Tool and JSON regression was 20 out of 22, the same score and failures as v1.0.0. A 66K input with 3,668 output tokens completed without running out of memory.&lt;/p&gt;

&lt;p&gt;Now the limits, stated plainly. The logical context capacity is 262,144 tokens, which is where the 256K number comes from. Full 256K is not validated. The dsh compaction threshold is around 184K, and long-history middle sections are retrieved by page selection. A step does not read all history. The 232K random-order stress test was unstable across repeats, so it is not a release claim. Anyone who tells you a tiered cache gives you full attention over 256K on a 12GB card is selling something.&lt;/p&gt;

&lt;p&gt;There are practical limits too. The prebuilt engine covers Windows x64 with sm120a only. RTX 30 and 40 series cards need an architecture-matched build. Search needs the internet, because dsh search is not an offline index. Private requests, sessions and logs stay local. Version 1.2.0 fixes a CPU vision chunk-boundary scatter error and adds bounded recovery from truncated summaries, but a known multi-image plus compaction failure mode remains: if it hits, start a new session.&lt;/p&gt;

&lt;h2&gt;
  
  
  The agent layer, and how to try it
&lt;/h2&gt;

&lt;p&gt;The engine alone would be a fast inference server. The dsh agent workspace is what makes it usable for work. It provides edit validation before execution, a task ledger that reminds the model of remaining actions, tool-result fidelity, and a deterministic state card after significant tool results. The optional Pmod desktop, built with Tauri 2 and Vue 3.5, adds chat, model configuration, VRAM monitoring, and an OpenAI-compatible gateway. The engine API listens on &lt;a href="http://127.0.0.1:1234/v1" rel="noopener noreferrer"&gt;http://127.0.0.1:1234/v1&lt;/a&gt;, the cache guard on port 1236, and the dsh UI on port 3080.&lt;/p&gt;

&lt;p&gt;You need Node.js 24 or later, the VC++ x64 runtime, a CUDA 13-compatible NVIDIA driver, and a v3 .ninfer model containing MTP. Install with: git clone &lt;a href="https://github.com/CSUNX233/Sparkle-Qwen27b-Kvmem-ninfer.git" rel="noopener noreferrer"&gt;https://github.com/CSUNX233/Sparkle-Qwen27b-Kvmem-ninfer.git&lt;/a&gt;, then run setup.ps1 with your model path, then start.cmd. The first start downloads checksum-pinned dependencies, loads weights, and captures CUDA graphs. The private dsh login URL is written to runtime/logs/dsh.stdout.log. Do not publish it. git pull alone does not replace the executable; rerun setup.ps1 after updating source.&lt;/p&gt;

&lt;p&gt;If you try it, I would like to hear where it breaks for you. Does retrieval hold up on your long tasks, or does the model lose the middle of the history first? Does the pool size need to change for your GPU? Those are the questions that will decide what v1.3.0 focuses on.&lt;/p&gt;

&lt;p&gt;Repo: &lt;a href="https://github.com/CSUNX233/Sparkle-Qwen27b-Kvmem-ninfer" rel="noopener noreferrer"&gt;https://github.com/CSUNX233/Sparkle-Qwen27b-Kvmem-ninfer&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Official community group coming soon — invite will be posted in the repo.&lt;/p&gt;

</description>
      <category>llm</category>
      <category>localai</category>
      <category>gpu</category>
      <category>opensource</category>
    </item>
  </channel>
</rss>
