Swizzle Runtime v0.1.0: How We Reached 209 tok/s and 43ms TTFT on a 4GB Laptop GPU in Pure Rust

Paramveer Singh — Sat, 14 Nov 2026 14:55:59 +0000

Autonomous coding agents, local developer tools, and edge robotics are bottlenecked by the traditional Python/PyTorch serving stack. On local workstations or 4 GB laptop GPUs, gigabytes of runtime overhead add steep cold-start latency and memory pressure.

We open-sourced Swizzle Runtime (swizzle-rt v0.1.0) under Apache-2.0: a single static Rust binary with custom x86-64 AVX2, NVIDIA CUDA (sm_75 to sm_120), and cross-vendor Vulkan compute kernels.

Core Architecture

Bitwise XOR Shared-Memory GEMM Tiling (GemmTile): Eliminates GPU SRAM bank conflicts and cuts shared-memory footprint by 9.1%.
RAD-1 Self-Speculative Decoding: Reuses the target model's own quantized weights to verify candidate branches in a single batched pass with 100% bit-identical greedy parity.
Warm Start & Zero Wait: Two-tier KV/recurrent prefix cache plus double-buffered CUDA graph overlap.
Ed25519 + BLAKE3 Execution Receipts: Every run can emit a cryptographic receipt (swizzle-rt verify-receipt).

Verified Benchmarks vs. `llama.cpp`

Measured back-to-back on an AMD Ryzen 7 170 (8C/16T) + NVIDIA GeForce RTX 3050 Laptop GPU (4 GB VRAM, sm_86, CUDA 12.8), 5 repetitions with 5s cooldown, verified bit-identical greedy parity:

Model	Device	Test	Baseline (tok/s)	Swizzle Runtime (tok/s)	Delta	Swizzle Latency
`Qwen3-0.6B-Q8_0.gguf`	CUDA	`pp512`	11,550.31	11,804.23	+2.2%	TTFT 43.37 ms
`Qwen3-0.6B-Q8_0.gguf`	CUDA	`tg128`	196.84	209.70	+6.5%	ITL p50: 4.84 ms
`Qwen3-0.6B-Q8_0.gguf`	CUDA	`edit256`	737.47	759.75	+3.0%	`edit_identical: yes`
`Qwen3-0.6B-Q8_0.gguf`	CPU	`edit256`	65.80	90.61	+37.7%	`edit_identical: yes`
`LFM3-1.7B-Q4_K_M.gguf`	CUDA	`tg128`	134.82	144.26	+7.0%	ITL p50: 7.04 ms
`LFM3-1.7B-Q4_K_M.gguf`	CPU	`tg128`	16.15	18.06	+11.8%	ITL p50: 55.85 ms

Try It in One Command

Linux x86_64 (CUDA 12.8):

curl -fsSL https://raw.githubusercontent.com/Swizzle-AI/swizzle-runtime/main/install.sh | sh

Windows x86_64 (PowerShell):

irm https://raw.githubusercontent.com/Swizzle-AI/swizzle-runtime/main/install.ps1 | iex

Docker (GHCR):

docker run --rm -p 8080:8080 -v "$PWD/models:/models:ro" ghcr.io/swizzle-ai/swizzle-runtime:latest /models/model.gguf

GitHub Repo: https://github.com/Swizzle-AI/swizzle-runtime
Submit your GPU numbers to the Hardware Leaderboard: https://github.com/Swizzle-AI/swizzle-runtime/issues/5
Original Substack deep-dive: https://mlblueprints.substack.com/p/swizzle-runtime-v010-how-we-reached

DEV Community: Paramveer Singh

Swizzle Runtime v0.1.0: How We Reached 209 tok/s and 43ms TTFT on a 4GB Laptop GPU in Pure Rust

Core Architecture

Verified Benchmarks vs. llama.cpp

Try It in One Command

Verified Benchmarks vs. `llama.cpp`