Most local-LLM advice stops at the weights. Pick a smaller quantized model, fit it in VRAM, done. That works for chat. It breaks the moment you want a real agent: long files, long tool histories, multi-turn work. The weights are only half the memory problem. The KV cache is the other half, and it grows with context.
Sparkle-Qwen27b-Kvmem-ninfer is my attempt to solve the second half on ordinary hardware. It runs Qwen3 27B as a local coding, files and web agent on an RTX 5070 Ti Laptop with 12GB of VRAM. The stack is NInfer inference, KVMem tiered KV, MTP3 speculative decoding, and a local agent workspace called dsh, with an optional Tauri desktop UI. The code is Apache-2.0.
This article explains how the memory budget works, what KVMem actually does, what I measured, and where it still falls short. I am not going to claim this beats cloud models, or that 256K context means 256K of attention on every step. It does not.
The budget: 12GB is smaller than it looks
The validated machine is an RTX 5070 Ti Laptop with 12GB VRAM, an i7-14650HX, and 32GB of system RAM, running Windows. The default model is Qwen3.8-27B GSQ-RCO IQ3_XXS with MTP3, using RK4V4 plus GDN FP32, with DeviceKV 36864, Retrieve 3840, and HostKV 7168 MiB.
The daily peak is 11,220 to 11,236 MiB of whole-GPU memory. That leaves about 12 MiB of headroom. This is not a comfortable margin. It is deliberate: the pool is sized to use almost everything available, and the first advice when you run out of memory is to shrink the pool to 38,912 or 38,400, not to switch quantization and hope.
Quantization choice matters here in a surprising way. The previous daily profile used NVFP4 and peaked at 11,168 MiB. The current RK4V4 profile peaks slightly higher, but it is the default anyway, because on a 39-question short-context check it scored 35 out of 39 correct with a mean total variation of 0.0262 against BF16 and zero prediction flips, while NVFP4 scored 34 out of 39 with one flip. NVFP4 is also not a VRAM-saving option here: at the same pool, RK4V4 already uses about 20 MiB less. The cheapest quantization on paper is not always the cheapest once the KV budget is included.
Why the KV cache is the real problem
Transformer inference stores a key and a value for every token in the context. That cache grows linearly with context length, and for a 27B model it grows fast. Hold the whole cache on the GPU and a long agent session exhausts 12GB long before the weights do. Evict aggressively and you lose the history the agent needs: the file it edited three turns ago, the error from the last tool call, the plan it agreed to at the start.
The usual answers are to shorten the context, summarize early, or restart the session. All three work, and all three make the agent dumber at exactly the moment a task gets interesting. KVMem takes a different approach: do not keep all of the cache hot. Keep some of it, retrieve the rest.
How KVMem works
KVMem divides the KV cache into tiers. The GPU holds hot pages: sink pages at the start, recent pages, and retrieved pages that a query scores highly. Cold history lives in host RAM — 7 GiB of Host KV in the default profile — and is restored to the GPU on demand.
A request flows like this: the agent sends a request to the cache guard, which keeps the sent prefix stable and reduces repeated prefill; the engine scores the query against stored pages, selects a small set, and restores only those pages. The model then decodes with sink, recent, and retrieved pages in GPU memory. It does not reread all history on every step.
Two details from v1.1.0 show why retrieval quality matters. First, evidence can span two KV pages. If the selector keeps only the second half, the model sees a fragment. The fix uses overlap-retrieval slots to add the predecessor page: at most eight pages per selection, replacing only lower-ranked members of the recent range, with no increase in total selected pages. Second, filling is allowed only after the known query end has been read and the full query has been scored. That change removed a regression that had appeared at 218K in an earlier candidate.
MTP3 sits on top of this. It uses three draft tokens for speculative decoding, which reduces decode steps. Daily sampling stays unchanged. It does not change what the model knows; it changes how fast it produces tokens it was already going to produce.
What I measured, and what I did not
All measurements are from the maintainer's machine, with different binaries, ports and fixtures across runs. They are not a unified benchmark. I report them as fixtures, not leaderboard scores.
A 110K task with 12 sequential pins finished 12 out of 12, producing 733 output tokens in 275.9 seconds at a peak of 11,232 MiB. The previous daily build had exhausted its 4,096 output budget without a final answer on the same fixture. A 218K task finished 26 out of 26, with normalized JSON matching the correct baseline. A 159K task finished 16 out of 16. A 232K sequential-pin task finished 24 out of 24, but sequential codes are easy to extrapolate, so I treat that as a weaker signal than the real-content test.
The real-content test is the one I trust most. It used 189,954 tokens of real content across multiple turns, with 16 code facts and 8 earlier conversation facts to recall. It passed 24 out of 24. That is a better proxy for agent work than needle tests, because it requires the model to use retrieved history to answer questions about code it saw earlier in the same session.
Short-context fidelity was checked on 39 questions. The default RK4V4 plus GDN32 scored 35 out of 39 with zero flips. Tool and JSON regression was 20 out of 22, the same score and failures as v1.0.0. A 66K input with 3,668 output tokens completed without running out of memory.
Now the limits, stated plainly. The logical context capacity is 262,144 tokens, which is where the 256K number comes from. Full 256K is not validated. The dsh compaction threshold is around 184K, and long-history middle sections are retrieved by page selection. A step does not read all history. The 232K random-order stress test was unstable across repeats, so it is not a release claim. Anyone who tells you a tiered cache gives you full attention over 256K on a 12GB card is selling something.
There are practical limits too. The prebuilt engine covers Windows x64 with sm120a only. RTX 30 and 40 series cards need an architecture-matched build. Search needs the internet, because dsh search is not an offline index. Private requests, sessions and logs stay local. Version 1.2.0 fixes a CPU vision chunk-boundary scatter error and adds bounded recovery from truncated summaries, but a known multi-image plus compaction failure mode remains: if it hits, start a new session.
The agent layer, and how to try it
The engine alone would be a fast inference server. The dsh agent workspace is what makes it usable for work. It provides edit validation before execution, a task ledger that reminds the model of remaining actions, tool-result fidelity, and a deterministic state card after significant tool results. The optional Pmod desktop, built with Tauri 2 and Vue 3.5, adds chat, model configuration, VRAM monitoring, and an OpenAI-compatible gateway. The engine API listens on http://127.0.0.1:1234/v1, the cache guard on port 1236, and the dsh UI on port 3080.
You need Node.js 24 or later, the VC++ x64 runtime, a CUDA 13-compatible NVIDIA driver, and a v3 .ninfer model containing MTP. Install with: git clone https://github.com/CSUNX233/Sparkle-Qwen27b-Kvmem-ninfer.git, then run setup.ps1 with your model path, then start.cmd. The first start downloads checksum-pinned dependencies, loads weights, and captures CUDA graphs. The private dsh login URL is written to runtime/logs/dsh.stdout.log. Do not publish it. git pull alone does not replace the executable; rerun setup.ps1 after updating source.
If you try it, I would like to hear where it breaks for you. Does retrieval hold up on your long tasks, or does the model lose the middle of the history first? Does the pool size need to change for your GPU? Those are the questions that will decide what v1.3.0 focuses on.
Repo: https://github.com/CSUNX233/Sparkle-Qwen27b-Kvmem-ninfer
Official community group coming soon — invite will be posted in the repo.
Top comments (0)