Autonomous coding agents, local developer tools, and edge robotics are bottlenecked by the traditional Python/PyTorch serving stack. On local workstations or 4 GB laptop GPUs, gigabytes of runtime overhead add steep cold-start latency and memory pressure.
We open-sourced Swizzle Runtime (swizzle-rt v0.1.0) under Apache-2.0: a single static Rust binary with custom x86-64 AVX2, NVIDIA CUDA (sm_75 to sm_120), and cross-vendor Vulkan compute kernels.
Core Architecture
-
Bitwise XOR Shared-Memory GEMM Tiling (
GemmTile): Eliminates GPU SRAM bank conflicts and cuts shared-memory footprint by 9.1%. - RAD-1 Self-Speculative Decoding: Reuses the target model's own quantized weights to verify candidate branches in a single batched pass with 100% bit-identical greedy parity.
- Warm Start & Zero Wait: Two-tier KV/recurrent prefix cache plus double-buffered CUDA graph overlap.
-
Ed25519 + BLAKE3 Execution Receipts: Every run can emit a cryptographic receipt (
swizzle-rt verify-receipt).
Verified Benchmarks vs. llama.cpp
Measured back-to-back on an AMD Ryzen 7 170 (8C/16T) + NVIDIA GeForce RTX 3050 Laptop GPU (4 GB VRAM, sm_86, CUDA 12.8), 5 repetitions with 5s cooldown, verified bit-identical greedy parity:
| Model | Device | Test | Baseline (tok/s) | Swizzle Runtime (tok/s) | Delta | Swizzle Latency |
|---|---|---|---|---|---|---|
Qwen3-0.6B-Q8_0.gguf |
CUDA | pp512 |
11,550.31 | 11,804.23 | +2.2% | TTFT 43.37 ms |
Qwen3-0.6B-Q8_0.gguf |
CUDA | tg128 |
196.84 | 209.70 | +6.5% | ITL p50: 4.84 ms |
Qwen3-0.6B-Q8_0.gguf |
CUDA | edit256 |
737.47 | 759.75 | +3.0% | edit_identical: yes |
Qwen3-0.6B-Q8_0.gguf |
CPU | edit256 |
65.80 | 90.61 | +37.7% | edit_identical: yes |
LFM3-1.7B-Q4_K_M.gguf |
CUDA | tg128 |
134.82 | 144.26 | +7.0% | ITL p50: 7.04 ms |
LFM3-1.7B-Q4_K_M.gguf |
CPU | tg128 |
16.15 | 18.06 | +11.8% | ITL p50: 55.85 ms |
Try It in One Command
Linux x86_64 (CUDA 12.8):
curl -fsSL https://raw.githubusercontent.com/Swizzle-AI/swizzle-runtime/main/install.sh | sh
Windows x86_64 (PowerShell):
irm https://raw.githubusercontent.com/Swizzle-AI/swizzle-runtime/main/install.ps1 | iex
Docker (GHCR):
docker run --rm -p 8080:8080 -v "$PWD/models:/models:ro" ghcr.io/swizzle-ai/swizzle-runtime:latest /models/model.gguf
- GitHub Repo: https://github.com/Swizzle-AI/swizzle-runtime
- Submit your GPU numbers to the Hardware Leaderboard: https://github.com/Swizzle-AI/swizzle-runtime/issues/5
- Original Substack deep-dive: https://mlblueprints.substack.com/p/swizzle-runtime-v010-how-we-reached


Top comments (5)
The framing I appreciate most here is naming cold-start and runtime overhead as the actual bottleneck for local agents, not just raw tok/s. On a laptop GPU, a Python/PyTorch stack can lose more wall-clock to import + warmup than it ever makes back on throughput, and for an agent that spawns short-lived inference calls that overhead dominates. A single static binary sidesteps the whole dependency-resolution tax.
The Ed25519 + BLAKE3 execution receipts are the part I didn't expect and find genuinely interesting — a cryptographic record that a specific run produced a specific output is exactly what agent auditing needs, and almost nobody ships it at the runtime layer.
My one honest question: the benchmarks are all 0.6B–1.7B models, which is where a 4GB card lives, but also where the Python overhead is proportionally largest. Does the GEMM tiling / self-speculative decoding advantage hold up on 7B+ quantized models where you're bandwidth-bound rather than overhead-bound? That's the regime where most coding agents actually run, and it'd be a stronger headline if the delta survives there.
The +6.5% on tg128 is the honest number here and it is worth underlining why: at 0.6B Q8 you are nowhere near the 4 GB limit, so this is a memory-bandwidth and launch-overhead comparison, not a VRAM-pressure one. The 4 GB framing holds for LFM3-1.7B-Q4 too (~1 GB resident), so the interesting case — a model where the KV cache actually forces layer offload — is not in the table yet. The +37.7% CPU edit256 row is the one that says something structural, since that's where the GEMM tiling and thread scheduling matter more than the GPU graph overlap.
On RAD-1: self-speculative decoding with bit-identical greedy parity is the right constraint, but the speedup lives or dies on acceptance rate, and small models draft their own continuations very well while 7B+ quantised ones do not. Did you log acceptance length per step, and does the double-buffered CUDA graph still overlap when the batch is a single sequence with a cold prefix? The warm-start numbers would look very different from a cold pp512 on the same card.
Impressive numbers — and a good reminder that most of us are nowhere near GPU-bound ceilings.
For contrast from the no-GPU end: I run a 35B-A3B MoE (4-bit, CPU-only) on a mini PC with 16GB single-channel DDR4 — 4.27 tok/s decode, purely bandwidth-bound (25.6GB/s shared across everything). Every token streams expert weights through that pipe, so the "209 tok/s vs 4 tok/s" gap isn't really about model quality — it's bytes-per-token × available bandwidth.
Makes me curious how Swizzle behaves in a fully bandwidth-starved config (no GPU at all, experts streamed from SSD). That's the tier where most "can my laptop run it?" questions actually live.
The table shows a 37.7% gain on CPU
edit256and only 3.0% on CUDAedit256. DoesRAD-1self-speculative decoding explain that gap, or is it something else?Naming cold-start as the real local-agent bottleneck is the right framing — raw tokens/s benchmarks mostly measure the wrong thing. 43ms TTFT on 4GB VRAM is genuinely impressive.