DEV Community

Paramveer Singh
Paramveer Singh

Posted on Originally published at mlblueprints.substack.com

Swizzle Runtime v0.1.0: How We Reached 209 tok/s and 43ms TTFT on a 4GB Laptop GPU in Pure Rust

Swizzle Benchmark Comparison

Autonomous coding agents, local developer tools, and edge robotics are bottlenecked by the traditional Python/PyTorch serving stack. On local workstations or 4 GB laptop GPUs, gigabytes of runtime overhead add steep cold-start latency and memory pressure.

We open-sourced Swizzle Runtime (swizzle-rt v0.1.0) under Apache-2.0: a single static Rust binary with custom x86-64 AVX2, NVIDIA CUDA (sm_75 to sm_120), and cross-vendor Vulkan compute kernels.

Swizzle Runtime vs llama.cpp Side-by-Side Terminal Demo

Core Architecture

  1. Bitwise XOR Shared-Memory GEMM Tiling (GemmTile): Eliminates GPU SRAM bank conflicts and cuts shared-memory footprint by 9.1%.
  2. RAD-1 Self-Speculative Decoding: Reuses the target model's own quantized weights to verify candidate branches in a single batched pass with 100% bit-identical greedy parity.
  3. Warm Start & Zero Wait: Two-tier KV/recurrent prefix cache plus double-buffered CUDA graph overlap.
  4. Ed25519 + BLAKE3 Execution Receipts: Every run can emit a cryptographic receipt (swizzle-rt verify-receipt).

Verified Benchmarks vs. llama.cpp

Measured back-to-back on an AMD Ryzen 7 170 (8C/16T) + NVIDIA GeForce RTX 3050 Laptop GPU (4 GB VRAM, sm_86, CUDA 12.8), 5 repetitions with 5s cooldown, verified bit-identical greedy parity:

Model Device Test Baseline (tok/s) Swizzle Runtime (tok/s) Delta Swizzle Latency
Qwen3-0.6B-Q8_0.gguf CUDA pp512 11,550.31 11,804.23 +2.2% TTFT 43.37 ms
Qwen3-0.6B-Q8_0.gguf CUDA tg128 196.84 209.70 +6.5% ITL p50: 4.84 ms
Qwen3-0.6B-Q8_0.gguf CUDA edit256 737.47 759.75 +3.0% edit_identical: yes
Qwen3-0.6B-Q8_0.gguf CPU edit256 65.80 90.61 +37.7% edit_identical: yes
LFM3-1.7B-Q4_K_M.gguf CUDA tg128 134.82 144.26 +7.0% ITL p50: 7.04 ms
LFM3-1.7B-Q4_K_M.gguf CPU tg128 16.15 18.06 +11.8% ITL p50: 55.85 ms

Try It in One Command

Linux x86_64 (CUDA 12.8):

curl -fsSL https://raw.githubusercontent.com/Swizzle-AI/swizzle-runtime/main/install.sh | sh
Enter fullscreen mode Exit fullscreen mode

Windows x86_64 (PowerShell):

irm https://raw.githubusercontent.com/Swizzle-AI/swizzle-runtime/main/install.ps1 | iex
Enter fullscreen mode Exit fullscreen mode

Docker (GHCR):

docker run --rm -p 8080:8080 -v "$PWD/models:/models:ro" ghcr.io/swizzle-ai/swizzle-runtime:latest /models/model.gguf
Enter fullscreen mode Exit fullscreen mode

Top comments (5)

Collapse
 
max_quimby profile image
Max Quimby •

The framing I appreciate most here is naming cold-start and runtime overhead as the actual bottleneck for local agents, not just raw tok/s. On a laptop GPU, a Python/PyTorch stack can lose more wall-clock to import + warmup than it ever makes back on throughput, and for an agent that spawns short-lived inference calls that overhead dominates. A single static binary sidesteps the whole dependency-resolution tax.

The Ed25519 + BLAKE3 execution receipts are the part I didn't expect and find genuinely interesting — a cryptographic record that a specific run produced a specific output is exactly what agent auditing needs, and almost nobody ships it at the runtime layer.

My one honest question: the benchmarks are all 0.6B–1.7B models, which is where a 4GB card lives, but also where the Python overhead is proportionally largest. Does the GEMM tiling / self-speculative decoding advantage hold up on 7B+ quantized models where you're bandwidth-bound rather than overhead-bound? That's the regime where most coding agents actually run, and it'd be a stronger headline if the delta survives there.

Collapse
 
maxbuildsdev profile image
Max Kaya •

The +6.5% on tg128 is the honest number here and it is worth underlining why: at 0.6B Q8 you are nowhere near the 4 GB limit, so this is a memory-bandwidth and launch-overhead comparison, not a VRAM-pressure one. The 4 GB framing holds for LFM3-1.7B-Q4 too (~1 GB resident), so the interesting case — a model where the KV cache actually forces layer offload — is not in the table yet. The +37.7% CPU edit256 row is the one that says something structural, since that's where the GEMM tiling and thread scheduling matter more than the GPU graph overlap.

On RAD-1: self-speculative decoding with bit-identical greedy parity is the right constraint, but the speedup lives or dies on acceptance rate, and small models draft their own continuations very well while 7B+ quantised ones do not. Did you log acceptance length per step, and does the double-buffered CUDA graph still overlap when the batch is a single sequence with a cold prefix? The warm-start numbers would look very different from a cold pp512 on the same card.

Collapse
 
qianchuan profile image
qianchuan meng (qianchuan) •

Impressive numbers — and a good reminder that most of us are nowhere near GPU-bound ceilings.

For contrast from the no-GPU end: I run a 35B-A3B MoE (4-bit, CPU-only) on a mini PC with 16GB single-channel DDR4 — 4.27 tok/s decode, purely bandwidth-bound (25.6GB/s shared across everything). Every token streams expert weights through that pipe, so the "209 tok/s vs 4 tok/s" gap isn't really about model quality — it's bytes-per-token × available bandwidth.

Makes me curious how Swizzle behaves in a fully bandwidth-starved config (no GPU at all, experts streamed from SSD). That's the tier where most "can my laptop run it?" questions actually live.

Collapse
 
suquant profile image
George Kutsurua •

The table shows a 37.7% gain on CPU edit256 and only 3.0% on CUDA edit256. Does RAD-1 self-speculative decoding explain that gap, or is it something else?

Collapse
 
mininglamp profile image
Mininglamp •

Naming cold-start as the real local-agent bottleneck is the right framing — raw tokens/s benchmarks mostly measure the wrong thing. 43ms TTFT on 4GB VRAM is genuinely impressive.