The latest articles on DEV Community by IDKLETSCODESOMETHIN (@idkletscodesomethin). https://dev.to/idkletscodesomethin https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4239108%2F384e9178-7047-4991-85e6-aa55073adde5.png https://dev.to/idkletscodesomethin en IDKLETSCODESOMETHIN Wed, 11 Nov 2026 13:00:12 +0000 https://dev.to/idkletscodesomethin/i-built-a-bare-metal-deep-learning-engine-in-c20-and-rust-velograd-v100-5ek3 https://dev.to/idkletscodesomethin/i-built-a-bare-metal-deep-learning-engine-in-c20-and-rust-velograd-v100-5ek3 <p>Hey developers!</p> <p>My name is Ayushman Singh Rathore, and over the past several months I have been building VeloGrad (v1.0.0), a native-first machine learning and autoregressive inference runtime written from scratch in modern C++20, Rust, and CUDA.</p> <p>GitHub Repository: <a href="https://github.com/IDKLETSCODESOMETHIN/VeloGrad" rel="noopener noreferrer">https://github.com/IDKLETSCODESOMETHIN/VeloGrad</a></p> <p>Why I Built It: PyTorch is great for research, but running modern long-context LLMs in production usually requires duct-taping multiple heavy frameworks together (vLLM, DeepSpeed, TensorRT-LLM). This carries Python GIL stalls, massive multi-gigabyte dependencies, and severe memory fragmentation.</p> <p>I wanted to explore how far we could push runtime efficiency by designing an engine from bare-metal primitives:</p> <p>Static Arena Allocator: Instead of dynamic heap allocations during inference, memory is managed via a contiguous arena aligned to 64-byte hardware cache lines (alignas(64)), eliminating GC stalls and latency jitter.</p> <p>Virtual Paged Attention: We break the KV-cache into discrete physical pages (similar to OS virtual memory paging), reducing KV-cache VRAM usage by ~60% on long contexts without memory fragmentation.</p> <p>Universal 1-Command Converter (velograd-convert): A zero-dependency binary parser that translates between Hugging Face Safetensors, PyTorch checkpoints, and GGUF v3 (for Ollama / llama.cpp) with no external packages.</p> <p>PyTorch Drop-in API: Familiar .forward(), .backward(), and autograd syntax with array exchange protocols for NumPy, JAX, and DLPack.</p> <p>Try it on local hardware in 30 seconds: You can clone the repository, install it locally, and run our universal test suite in under a second:</p> <p>git clone <a href="https://github.com/IDKLETSCODESOMETHIN/VeloGrad.git" rel="noopener noreferrer">https://github.com/IDKLETSCODESOMETHIN/VeloGrad.git</a><br> cd VeloGrad pip install -e . python tests/test_velograd_universal_compat.py</p> <p>The project is completely open source under AGPLv3: <a href="https://github.com/IDKLETSCODESOMETHIN/VeloGrad" rel="noopener noreferrer">https://github.com/IDKLETSCODESOMETHIN/VeloGrad</a></p> opensource machinelearning cpp rust