DEV Community

Cover image for I Cut Dedicated GPU Memory to 4GB, and a Local LLM Still Ran at 90% Speed
maruhana
maruhana

Posted on Edited on Originally published at zenn.dev

I Cut Dedicated GPU Memory to 4GB, and a Local LLM Still Ran at 90% Speed

On a Surface Laptop Ultra (RTX Spark, unified memory), I measured local LLM speed and Windows' free memory with the GPU's dedicated memory set to 4, 64, 82 and 96GB. Speed barely moved; memory left for Windows did.

This is an English adaptation of an article I originally published in Japanese on Zenn: 専用 GPU メモリーを 4GB にしても、ローカル LLM は 9 割の速さで動いた. The measurements and conclusions are mine; I used an AI tool to help translate and restructure the text, and I checked the numbers against the original.

Short version: on this laptop, setting the GPU's dedicated memory to the minimum (4GB) still left a local LLM at about 90% of its speed, while the memory left for Windows changed from 122.5GB to 30.5GB across the settings.

Update (2026-11-16): After publishing, I also generated video inside WSL2 with dedicated GPU memory at 4GB. All 6 videos finished, but Wan 2.2 took about 1.9× as long as at 64GB, while the other video model (MiniMax H3) took about the same. See "Update: video generation in WSL2 at 4GB" below.

A 71-second video summary (narration is synthetic, English captions available):

TL;DR

  • This laptop's CPU and GPU share one pool of physical memory. A setting decides how much is dedicated to the GPU, from 4GB to 96GB. The rest is main memory for Windows and apps.
  • With dedicated GPU memory at the minimum 4GB, gpt-oss-120b (63.4GB) still generated 58.8 tokens/s, about 92% of the 63.9–64.2 at 64GB and up. Most of the model sat outside the dedicated part, in memory the GPU borrows from main memory.
  • Memory left for Windows went from 122.5GB (4GB setting) to 30.5GB (96GB setting). At 82GB and 96GB, with the model loaded and a browser open, committed memory reached 99% of the commit limit.
  • Moving MoE experts to the CPU (-ncmoe) was slower than leaving everything on the GPU, even at 4GB: about 2.1–2.6× slower for gpt-oss-120b and 1.8× for the other model.
  • I lowered my everyday setting from 82GB to 64GB.
  • One laptop, two models, one run per setting. On a PC with a separate graphics card (its own VRAM), I'd expect very different results.

Setup

PC Surface Laptop Ultra (Windows 11 Pro, NVIDIA RTX Spark N1X, 128GB unified memory)
How the dedicated amount was changed NVIDIA App, "Dedicated GPU memory (GB)", with a reboot each time. Options ran from 4GB (minimum) to 96GB (maximum)
Settings tested 4GB (minimum), 64GB (about the size of gpt-oss-120b's 63.4GB file), 82GB (my setting until then), 96GB (maximum)
Driver 621.24
llama.cpp b12110 (CUDA), the same build as my 8-model benchmark
Models gpt-oss-120b (MXFP4, 63.4GB), Nemotron 3.5 Bolt (74B total, about 9B active per token, 2-bit, about 22GB)
Stopped during the test WSL, ComfyUI and other LLMs. Only at 82GB, an Ollama instance with no model loaded was running

What I call each part of memory

There is one pool of physical memory, split in two by the setting:

Name What it is
Dedicated GPU memory The part reserved for the GPU. Set in the NVIDIA App. This is what nvidia-smi reports
Main memory The rest. Used by Windows and apps. This is what Task Manager shows as "Memory"
Shared GPU memory The part of main memory the GPU borrows when it needs more

Dedicated plus main came to 126.5GB at every setting.

How I measured

At each setting, in the same order:

  1. 3 minutes idle, logging free memory and more every 5 seconds.
  2. llama-bench, average of 3 runs:
    • pp512 (prompt processing): how fast the model reads a 512-token prompt before answering.
    • tg128 (generation): how fast it writes 128 tokens. Closest to how fast text appears in a chat.
  3. A 5-minute load with gpt-oss-120b loaded in llama-server: 10 browser tabs, plus a script grabbing main memory 4GB at a time up to 16GB. Still logging every 5 seconds.

I logged Windows performance counters (Available MBytes, Committed Bytes, Commit Limit, page file usage, disk writes) and dedicated GPU memory use from nvidia-smi. Memory sizes from the counters are converted at 1GB = 10^9 bytes.

# Everything assigned to the GPU (the main runs)
llama-bench -m <model>.gguf -ngl 999 -fa 1 -p 512 -n 128 -r 3

# Some MoE experts on the CPU side (for comparison)
llama-bench -m <model>.gguf -ngl 999 -ncmoe 16 -fa 1 -p 512 -n 128 -r 3
Enter fullscreen mode Exit fullscreen mode

Result 1: about 90% of the speed at 4GB

All layers assigned to the GPU (-ngl 999). Tokens per second, average of 3 runs.

Dedicated GPU memory gpt-oss-120b pp512 gpt-oss-120b tg128 Bolt pp512 Bolt tg128
4GB (min) 1,950 58.8 1,163 39.9
64GB 2,136 63.9 1,186 43.5
82GB 2,028 64.2 1,122 41.4
96GB (max) 2,127 64.0 1,204 43.6

At 4GB, generation was about 90% of the speed at 64GB and up: 58.8 ÷ 63.9 = 92% for gpt-oss-120b, and 39.9 ÷ 43.6–41.4 = 92–96% for Bolt.

The standard deviation across the 3 runs was within 4% of the mean for every generation number. gpt-oss-120b generated 58.8 ± 0.5 at 4GB and 63.9 ± 0.5 at 64GB, so the ~10% gap is larger than the run-to-run spread; the differences between 64, 82 and 96GB (63.9–64.2) are within it. The 3 runs were repeats within one launch; each setting was measured once after a reboot. gpt-oss-120b's prompt processing at 82GB had a larger spread (2,028 ± 214, about 11%), so its slightly lower value is within that range. Bolt's 41.4 at 82GB was a separate run from the 42.7 in my 8-model benchmark.

At 4GB, nvidia-smi showed at most 3,808MiB of dedicated GPU memory in use while gpt-oss-120b was fully assigned to the GPU. Most of the 63.4GB model sat outside the dedicated part, in shared GPU memory. It still ran at about 90% speed.

From 64GB up, speed barely changed. At 64GB, dedicated use peaked at 61,586MiB: the model fit inside.

Result 2: memory left for Windows changes a lot

Values with gpt-oss-120b loaded (all on the GPU) under the 5-minute load. All columns except the first are main-memory values. Committed memory is the total that apps have been promised; the limit is main memory plus the page file.

Dedicated GPU memory Main memory (Windows can use) Free when idle Free under load (min) Commit under load (max) / limit Page file usage (max)
4GB (min) 122.5GB ~112GB 47.1GB 102.6 / 238.9GB 0.6%
64GB 62.5GB ~50.8GB 45.8GB 103.1 / 134.2GB 0.8%
82GB 44.5GB ~33.8GB 26.9GB 108.0 / 108.8GB 1.6%
96GB (max) 30.5GB ~19.4GB 13.3GB 101.8 / 102.9GB 0.3%

At 82GB and 96GB, committed memory under load reached 99% of the limit. At 96GB the page file grew and the limit went from 96.6GB to 102.9GB. In both cases, a little more load would have hit the limit.

Generation speed, on the other hand, barely dropped under this load. I sent the same prompt (400 tokens, temperature 0) to llama-server once before and once during the load:

Dedicated GPU memory Before load Under load
4GB (min) 57.4 55.9
64GB 62.1 60.2
82GB 61.6 60.3
96GB (max) 61.6 60.4

The drop was about 3% or less everywhere, and page file usage stayed under 2%. Close to the limit, but not swapping to disk.

At 4GB, 47.1GB of main memory was still free under load, about the same as at 64GB, with a commit limit of 238.9GB.

Result 3: keeping everything on the GPU beat moving experts to the CPU

When dedicated GPU memory is small, one option is to keep some MoE experts on the CPU side (-ncmoe). The number after -ncmoe (8, 16, 24, 36, 99) is how many layers put their experts on the CPU side. gpt-oss-120b has 36 layers, so -ncmoe 36 is all of them. For Bolt I used 99, more than its layer count, to put all of them on the CPU side.

Dedicated GPU memory Setting pp512 tg128
4GB (min) gpt-oss-120b, all on GPU 1,950 58.8
4GB (min) gpt-oss-120b -ncmoe 24 1,139 27.4
4GB (min) gpt-oss-120b -ncmoe 36 (all layers) 943 22.3
4GB (min) Bolt, all on GPU 1,163 39.9
4GB (min) Bolt -ncmoe 99 (all layers) 796 22.4
64GB gpt-oss-120b, all on GPU 2,136 63.9
64GB gpt-oss-120b -ncmoe 8 1,535 45.3
64GB gpt-oss-120b -ncmoe 16 1,331 36.0

In every case, the more layers on the CPU side, the slower it got. Even at 4GB, leaving everything assigned to the GPU and letting shared GPU memory take the rest was about 2.1–2.6× faster for gpt-oss-120b and about 1.8× faster for Bolt.

-ncmoe seems to come up often as a way to keep speed when GPU memory runs short. I think that advice is for PCs where VRAM and main memory are separate parts. On this PC, at least with these two models, the result went the other way.

Why shared GPU memory stays fast (partly a guess)

On this PC the CPU and GPU use the same physical memory. NVIDIA's porting guide page on unified memory says the carve-out reserved for the GPU is reported by Windows as dedicated GPU memory and is physically backed by system DRAM, not separate VRAM. So "dedicated" and "shared" are two ways of allocating the same parts.

From here on it's my guess: if the GPU reads from the same parts either way, a model in shared GPU memory doesn't pay for a trip through a separate part. I don't know why 4GB was about 10% slower, and I didn't look inside the driver.

This also half-answers something I left open in the 8-model benchmark: an 86.9GB model ran fully assigned to the GPU while nvidia-smi showed only about 82GB in use. Since a model mostly outside dedicated memory ran fine at 4GB, the part that didn't fit in 82GB was most likely in shared GPU memory. I didn't observe that placement directly, though.

What I changed: 82GB → 64GB

After this, I lowered my everyday setting from 82GB to 64GB:

  • gpt-oss-120b (63.4GB) fits inside dedicated memory, at the same speed as 82GB and 96GB.
  • Free main memory under load is 45.8GB, about 19GB more than at 82GB (26.9GB).
  • The commit limit (134.2GB) has room.

I didn't go down to 4GB because there's a use I haven't measured: video generation in ComfyUI inside WSL2. WSL2 sees the GPU through the Windows driver, so it may not behave the same. Image editing didn't slow down when it spilled a little past dedicated memory (see below), but I haven't generated video at 4GB. (After publishing, I measured it at 4GB; see the update section below.)

Extra: prefill with longer prompts

At the 64GB setting, I ran the same Qwen3.6-35B-A3B file as in the 8-model benchmark (UD-Q4_K_M, 22.1GB) with longer prompts: llama-bench -ngl 999 -fa 1 -p 512,4096,8192 -n 128 -r 3, llama.cpp b12110. No other LLM was on the GPU.

Test Tokens/s (mean ± std. dev. of 3 runs) Time per run
pp512 2,831 ± 40 0.18 s
pp4096 2,820 ± 35 1.45 s
pp8192 2,735 ± 39 3.00 s
tg128 74.9 ± 1.8 1.71 s

At 8,192 tokens it still read at about 97% of the 512-token speed. Dedicated GPU memory use peaked at 23,099MiB.

In the 8-model benchmark (82GB setting), the same file gave pp512 3,092, pp4096 2,834 and tg128 75.7. pp4096 and tg128 are about the same, but pp512 is about 8% lower this time. These were different days and settings, and I haven't separated the cause.

Extra: changing WSL2's memory limit for video generation

I also wanted to know whether raising WSL2's memory limit (memory in .wslconfig) speeds up video generation. With dedicated GPU memory at 64GB, I set the limit to 24GB, 48GB and 40GB (in that order) and generated Wan 2.2 videos (720×1056, 16fps, 49 frames) in ComfyUI inside WSL2. The model files total about 30GB.

Each time I shut WSL2 down and started it again, then made: ① one video right after startup, ② one more with the same workflow, ③ one with a different workflow (different start image and prompt). Times run from the start of the measuring script to the end of writing the video. Only the limit changed; the environment was the same all three times (torch 2.14.1+cu132, ComfyUI v0.45.0, Wan 2.2 i2v 14B fp8 with a 4-step LoRA). Memory values use 1GB = 10^9 bytes as above.

WSL2 limit 24GB 48GB 40GB
① right after startup 145 s 143 s 143 s
② same workflow again 127 s 129 s 128 s
③ different workflow 129 s 129 s 128 s
Free main memory (min) 28.4GB 5.9GB 12.4GB
Commit (max) 88.8GB 111.2GB 106.1GB
Page file usage (max) 0.8% 2.0% 1.5%
Dedicated GPU memory used (max) 44.4GB 44.4GB 44.3GB

Times differed by 2 seconds or less: a higher limit wasn't faster. What changed was free main memory, which fell to 5.9GB at 48GB.

In the 40GB run I also looked inside WSL2. After the three videos, processes used about 4.3GB and the file cache about 35GB. WSL2's swap wasn't used. Dedicated GPU memory use was about 44GB in all three runs, so the model weights were on the dedicated GPU side, outside WSL2's limit.

My guess is that a higher limit reduces free main memory because WSL2 holds the model files as cache and takes that from main memory. Disk reads for the three videos were about 30GB in every run. For this video on this PC, re-reading from disk seems to have been fast enough even with a smaller cache.

When I started these runs, my WSL2 setting was 40GB. The day before, at 24GB, switching workflows made one step take 53 seconds while the model was re-read from disk, and I suspected 24GB was too small. At the time dedicated GPU memory was 82GB, and 40GB would have left only about 4.5GB of main memory, so I didn't raise it. After lowering dedicated memory to 64GB, I raised it to 40GB. I didn't measure 40GB's speed before raising it. Now that I have, 24GB gave the same time per video. The alternating test below was just as fast down to 16GB, so I now use 16GB.

Alternating between two models

A higher limit might let WSL2 keep both models' files cached, so switching would be faster. I generated 6 videos alternating between two video models, A→B→A→B→A→B. I measured at 24GB and 40GB, then also at 8GB, 1GB and 16GB to see how low the limit could go. A was the same Wan 2.2 setup (model files about 38.5GB); B was MiniMax H3 video (640×352, 56 frames, about 40.5GB). The two share no files. Dedicated GPU memory stayed at 64GB, same environment as above.

Order Model 1GB 8GB 16GB 24GB 40GB
1 A (Wan 2.2) failed 155 s 145 s 144 s 143 s
2 B (MiniMax H3) failed 50 s 32 s 30 s 30 s
3 A failed 158 s 143 s 140 s 141 s
4 B failed 48 s 30 s 27 s 27 s
5 A failed 157 s 141 s 139 s 140 s
6 B failed 47 s 28 s 30 s 28 s
Disk reads (6 videos total) — 519.8GB 229.0GB 229.1GB 226.7GB
Free main memory (min) — 45.7GB 36.8GB 28.1GB 12.1GB
Dedicated GPU memory used (max) — 10,112MiB 43,538MiB 62,285MiB 61,424MiB

At 16GB, 24GB and 40GB, alternating gave almost the same times. At every limit from 16GB up, each switch re-read about as much as that model's files from disk (about 33–43GB per video). The two models total about 79GB, so even at 40GB they didn't both fit in the cache, and I think the previous model was evicted each time.

Generating A back to back with the same workflow took 127–129 s. Alternating, A took 139–145 s, about 10–15 s slower for the re-read. At 40GB free main memory dropped to 12.1GB, but page file usage stayed at 1.2% or less and WSL2's swap was not in use at the end of the run.

At 8GB all 6 videos finished, but compared with 40GB, A was about 15 s (about 10%) slower and B about 20 s (about 70%) slower. Disk reads for the 6 videos were 519.8GB, about 2.3× the runs at 16GB and up; each B video read about 101–105GB, more than twice its model files. Dedicated GPU memory peaked at only 10,112MiB, versus 43,538–62,285MiB at 16GB and up. My guess is that the model couldn't stay on the GPU side and was read and dropped over and over (I haven't confirmed how). WSL2's 2GB swap was also used up.

At 1GB, WSL2 was already using swap right after it started. About 13 s into loading the first model, the Linux kernel's OOM killer stopped ComfyUI's Python process. WSL2 then shut itself down, and all 6 videos failed.

With these two models, any limit from 16GB to 40GB gave the same speed when alternating. 8GB was slower, and 1GB didn't work. I didn't measure where it starts slowing down (between 8GB and 16GB). A limit big enough for both (80GB or more) might remove the re-reads, but with dedicated GPU memory at 64GB this PC has only 62.5GB of main memory, so I can't test that.

Image editing that spilled past dedicated memory

With the same settings (dedicated 64GB, WSL2 limit 40GB), I also have a log of 19 image edits in ComfyUI inside WSL2, with an LLM (llama-server, about 33GB) running at the same time.

  • Dedicated GPU memory use peaked at 64,258MiB, and about 40% of the samples during the 19 edits sat at around 64,000MiB. I think the rest went to shared GPU memory.
  • Time per image (from an agent requesting the edit to getting the result) was 124.3–132.7 s (median 126.3 s). On the morning of the same day, at 82GB, it was 126.8–138.5 s (median 130.5 s), so 64GB wasn't slower. Inside ComfyUI, one image took 109.8–118.7 s, of which the 6 sampling steps took 73–80 s. (ComfyUI's own timings weren't kept for the 82GB morning.)
  • Free main memory fell to 6.9GB at the lowest. Even so, SSD activity peaked at 18%, page file usage stayed at 4.7% or less, WSL2's swap didn't grow, and nothing stalled. (SSD and page file logging started partway through the 19 images.)

This was image editing, not video, and dedicated memory wasn't cut to 4GB. Still, it's a record that running out of dedicated memory from inside WSL2 didn't slow things down here.

Update: video generation in WSL2 at 4GB

When I published, I hadn't generated video with dedicated GPU memory at 4GB. So I repeated the 16GB column of the alternating test above, changing only dedicated GPU memory to 4GB. The WSL2 limit (16GB), both models, the workflows, the seeds, and the ComfyUI and model versions were the same. One run of 6 videos, A→B→A→B→A→B.

Order Model Dedicated 4GB Dedicated 64GB (16GB column above)
1 A (Wan 2.2) 276 s 145 s
2 B (MiniMax H3) 36 s 32 s
3 A 270 s 143 s
4 B 31 s 30 s
5 A 266 s 141 s
6 B 33 s 28 s
Disk reads (6 videos) 227.6GB 229.0GB
Peak shared GPU memory use 47.0GB not recorded
Peak dedicated GPU memory use 3,983MiB 43,538MiB

At 4GB all 6 videos finished. Nothing was killed for lack of memory, the SSD never pinned, and WSL2 didn't swap. What didn't fit in the 4GB went to Windows' shared GPU memory (peak 47.0GB).

Speed split by model, though. A (Wan 2.2) averaged 271 s per video, about 1.9× the 143 s at 64GB. B (MiniMax H3) averaged 33 s, about the same as the 30 s at 64GB. The LLM kept about 90% of its speed at 4GB, but in this video test one model ran at roughly half speed.

I haven't confirmed why only A slowed down. A is 720×1056 with 49 frames and B is 640×352 with 56 frames, so A's intermediate data is larger; my guess is that more of it had to go back and forth through shared GPU memory. Disk reads were about the same at 4GB and 64GB (the model was re-read at every switch either way), so the slowdown didn't come from re-reading files.

Free main memory bottomed out at 48.5GB (36.8GB at 64GB). That's because at 4GB the memory Windows can use is itself larger (122.5GB), not because anything became more efficient.

What I didn't measure

  • One run per setting. llama-bench averages 3 runs, but each setting was measured once.
  • Video generation between 4GB and 64GB. The 4GB update was one run, with WSL2 at 16GB only. I didn't test 8GB or 16GB, larger resolutions, or running an LLM at the same time.
  • A WSL2 limit big enough to cache both models, or anything between 8GB and 16GB. The alternating test covered 1, 8, 16, 24 and 40GB, one run each. At 16GB and up, every switch re-read the model. I didn't find where it starts slowing down, and didn't test other model pairs.
  • Heavier load or longer runs. The load was 16GB plus 10 browser tabs, for 5 minutes.
  • Why shared GPU memory stays fast. That it's the same physical memory is from NVIDIA's guide; why the speed holds is my guess, and I didn't look inside the driver.
  • Other models and other PCs. Two models, one laptop. On a PC where VRAM is a separate part from main memory, I'd expect very different results.

Takeaways

  • On this PC, with dedicated GPU memory at the minimum 4GB, an LLM fully assigned to the GPU ran at about 90% of its speed at 64GB and up.
  • Memory left for Windows changed from 122.5GB to 30.5GB. At 82GB and 96GB, running the LLM and a browser together pushed committed memory to 99% of the limit.
  • Leaving everything on the GPU and letting shared GPU memory take the rest was faster than moving MoE layers to the CPU (-ncmoe).
  • Raising WSL2's limit from 24GB to 48GB didn't change the time per video; it only reduced free main memory. Alternating between two models, 16GB, 24GB and 40GB gave almost the same times; 8GB was slower and 1GB didn't work.
  • At 4GB dedicated GPU memory, all 6 WSL2 videos still finished, but Wan 2.2 took about 1.9× as long as at 64GB (MiniMax H3 took about the same).
  • I set my everyday dedicated GPU memory to 64GB. On this PC, unless I plan to use a bigger model, not over-allocating dedicated GPU memory leaves more room in main memory.

Update history

  • 2026-11-16 — Added "Update: video generation in WSL2 at 4GB" and a note at the top. At publication I said I hadn't measured video at 4GB; I measured it and added the results, and updated the 82GB → 64GB section, What I didn't measure, and Takeaways to match.

About this series

This is part of my series on the Surface Laptop Ultra (Windows on Arm with an NVIDIA GPU). Related articles:

The original of this article: 専用 GPU メモリーを 4GB にしても、ローカル LLM は 9 割の速さで動いた.

Top comments (0)