Soup CLI Lets You Fine-Tune an 8B LLM on a 4 GB Laptop GPU

#soup #llm #finetuning #opensource

Laptop GPU" style="width:100%;height:auto;" />

Soup CLI, an open-source Python tool licensed under Apache-2.0, lets you fine-tune an 8B-parameterLLM on a 4 GB laptop GPU using a technique called layer streaming. The frozen base model stays in system RAM while decoder layers stream into GPU memory one at a time, achieving 119.6 tokens per second on an RTX 3050 Laptop with 3.32 GB peak VRAM usage.

Layer streaming was introduced as a practical solution to the memory bottleneck that has long prevented researchers and developers from fine-tuning large language models on consumer hardware. Soup CLI, available on GitHub at github.com/MakazhanAlpamys/Soup, implements this approach with a focus on correctness, enabling users to run preference training algorithms like DPO, ORPO, SimPO, and KTO without loading the entire model into GPU memory.

The tool uses NF4 (four-bit normalfloat) quantization to reduce model size while preserving accuracy. In benchmarks, Llama-3.1-8B-Instruct with NF4 quantization achieved 119.6 tok/s on an RTX 3050 Laptop 4 GB, peaking at just 3.32 GB of VRAM. The same configuration on an 8xH100 cluster reproduced the 3.32 GB peak at a median 113.00 tok/s, and the streamed run produced bit-exact results compared to a resident (fully loaded) run.

Key Takeaways

Soup CLI is an open-source CLI for LLM fine-tuning and post-training, licensed Apache-2.0
Layer streaming keeps the frozen base model in system RAM, streaming decoder layers into GPU one at a time
Llama-3.1-8B-Instruct + NF4 on RTX 3050 Laptop 4 GB: 119.6 tok/s, 3.32 GB peak VRAM
8xH100 reproduction: 113.00 tok/s median, same 3.32 GB peak, bit-exact vs resident run
v0.73.0 (Aug 5-9, 2026) fixed a silent wrong-gradient bug on NF4 layers above ~165 MiB
Preference training (DPO/ORPO/SimPO/KTO) supported at 0.914x SFT peak, avoiding +730 MB overhead
Install: pip install "soup-cli[train]" and run soup train with a soup.yaml config

How Layer Streaming Works

Traditional fine-tuning loads the entire model into GPU memory, which for an 8B model in standard precision requires roughly 16 GB of VRAM. Soup CLI takes a different approach. The base model remains frozen in system RAM, and only one decoder layer at a time streams into the GPU during forward and backward passes. This means a 4 GB laptop GPU can handle models that would normally require 24 GB or more.

The technique relies on careful memory management. As each layer processes, its activations and gradients are computed, and then the layer is swapped back to RAM before the next one loads. The system tracks memory usage precisely, ensuring the peak VRAM stays well within the GPU's capacity. For Llama-3.1-8B-Instruct with NF4 quantization, the peak sits at 3.32 GB on a 4 GB card, leaving headroom for training overhead.

Correctness is maintained through a verification protocol. Every release of Soup CLI compares the streamed run against a resident run and requires exact logits match. This means the output of the streaming approach is identical to loading the full model, just with a different memory access pattern.

Benchmark Results and Performance

The benchmark results demonstrate that layer streaming does not come with a steep performance penalty. On an RTX 3050 Laptop GPU with 4 GB of VRAM, Llama-3.1-8B-Instruct quantized to NF4 achieved 119.6 tokens per second. The same configuration on an 8xH100 cluster produced a median of 113.00 tok/s at the same 3.32 GB peak VRAM.

The bit-exact comparison is the critical detail. Streaming is not an approximation method. The logits produced by the streamed run match the resident run exactly, which means fine-tuning results are reproducible regardless of whether you use a laptop GPU or a datacenter cluster.

Preference training algorithms like DPO, ORPO, SimPO, and KTO are supported with the streamed base and adapters turned off. Benchmarks show these run at 0.914x the peak SFT speed while avoiding the +730 MB overhead that would come from loading a second model instance.

v0.73.0: Fixing Silent Wrong Gradients

Version 0.73.0, validated between August 5 and 9, 2026 on a borrowed H100 cluster, addressed a silent wrong-gradient bug that affected NF4 models with layers above approximately 165 MiB per layer. This was a correctness issue where the streamed run produced gradients that did not match the resident run, potentially corrupting fine-tuning without any visible error.

The validation caught the bug through the standard correctness protocol. Every release includes a comparison between streamed and resident runs, requiring exact logits match. When the bug was found, the four backends that previously failed to run at all were also fixed. Additional improvements in v0.73.0 include vLLM now using the model's chat template, support for training.seed, and the ability to run full fine-tuning by setting lora.r to 0.

Installation and Usage

Soup CLI installs via pip with the training extras:

pip install "soup-cli[train]"

The workflow is straightforward. Create a soup.yaml configuration file that specifies your model, dataset, and training parameters, then run the training command:

soup train

The configuration supports multiple backends and training modes. The project is licensed under Apache-2.0, with the source code hosted at github.com/MakazhanAlpamys/Soup and documentation at trysoup.dev.

Who This Is For

Soup CLI targets developers and researchers who want to fine-tune large models but do not have access to multi-GPU setups or cloud compute. The layer streaming approach makes it possible to run experiments on consumer hardware, which reduces cost and increases iteration speed for prototyping.

The preference training support is particularly useful for alignment work. DPO, ORPO, SimPO, and KTO are the standard algorithms for shaping model behavior, and having them run efficiently on a 4 GB GPU removes a common bottleneck.

Conclusion

Soup CLI demonstrates that layer streaming is production-ready for fine-tuning workflows. The v0.73.0 fix for silent wrong gradients is a sign of a project that takes correctness seriously, and the benchmark results show that performance remains strong even on modest hardware. For teams working with 8B-class models on laptop GPUs, it provides a path to fine-tuning that was previously only available on datacenter equipment.

Frequently Asked Questions

Q: What is layer streaming?

A: Layer streaming is a memory optimization technique where the frozen base model stays in system RAM and decoder layers are streamed into GPU memory one at a time during forward and backward passes. This allows fine-tuning models much larger than the GPU's VRAM capacity.

Q: Does Soup CLI lose accuracy compared to full GPU loading?

A: No. Soup CLI requires exact logits match between streamed and resident runs for every release. The v0.73.0 validation confirmed bit-exact results against the full-model baseline.

Q: What hardware do I need to run Soup CLI?

A: A GPU with at least 4 GB of VRAM is sufficient for 8B models with NF4 quantization. The benchmarks tested on an RTX 3050 Laptop 4 GB and achieved 119.6 tok/s.

Q: Can I use Soup CLI for preference training?

A: Yes. Soup CLI supports DPO, ORPO, SimPO, and KTO with the streamed base. These run at 0.914x SFT peak speed while avoiding the +730 MB overhead of a second model instance.

Q: How do I install Soup CLI?

A: Run pip install "soup-cli[train]", create a soup.yaml configuration file, and start training with soup train. The project is Apache-2.0 licensed.