DEV Community

IDKLETSCODESOMETHIN
IDKLETSCODESOMETHIN

Posted on

How I Built VeloGrad (v1.1.0-beta): A C++20, Rust, and CUDA Deep Learning Engine from Scratch

​

GitHub logo IDKLETSCODESOMETHIN / VeloGrad

Bare-metal, native-first deep learning & autoregressive inference engine in C++20, Rust, CUDA & Python.

VeloGrad (Beta)

Status: Beta License: AGPLv3 Architecture: C++20 / Rust / CUDA / Python Version: 1.1.0-beta Compatibility: NumPy | PyTorch | JAX | GGUF | HF

VeloGrad is a bare-metal, native-first, pure-compiled deep learning and autoregressive inference engine engineered from scratch in C++20, Rust, CUDA, and Python by Ayushman Singh Rathore.

It eliminates Python runtime overhead, framework memory bloat, and garbage collection pauses by introducing an aligned static arena memory architecture, ahead-of-time (AOT) fused compute kernels, an empirical ~66.7% KV-cache and activation VRAM reduction engine, enterprise ZeRO-3 / FSDP distributed scaling with CPU/NVMe offload, and a natural-language autonomous training orchestrator (velograd.instruct).


Important

Public Beta Notice (v1.1.0-beta)

VeloGrad v1.1.0-beta is now in public beta.
All core compute kernels, hardware-fused operators, universal frontier model architectures (from 100M edge models up to 1T+ parameter MoE networks), Paged KV-cache VRAM reduction, ZeRO-3 FSDP parameter sharding, and natural-language autonomous training directives (velograd.instruct) have been verified with complete zero-defect automated test and benchmark suites. Community testing, empirical benchmarks, and feedback are…







​Hey DEV community! My name is Ayushman Singh Rathore, and I’ve been building VeloGrad—an open-source tensor, autograd, and distributed training framework written in C++20, Rust, and CUDA with Python bindings.

​Today I’m releasing v1.1.0-beta into Public Beta. Before diving into the architecture, I want to share an honest builder lesson from v1.0.0 that completely changed how I ship code.

​The v1.0 Wake-Up Call: Auditing AI-Generated Scaffolding

​When I first pushed v1.0.0, I leaned too heavily on an AI coding agent to draft the initial scaffolding and README.md. Right after publishing, I audited the repository line by line and realized the agent had hallucinated benchmark claims and left broken Python bindings and stubbed functions.

​Instead of leaving an unverified repo up, I pulled back, stripped out every unverified claim, added a clear Public Beta status, and spent days rewriting, compiling, and testing the actual systems code so that every single line in the README is backed by real, executable code.

​What’s Inside VeloGrad v1.1.0-beta

​VeloGrad is designed to strip away Python dispatch overhead and GPU memory fragmentation during model training and fine-tuning by keeping the memory arena, autograd tape, and fused kernels inside a tight native runtime—while automatically falling back to AVX2/NEON SIMD kernels on CPU-only Linux, macOS, and Windows machines.

​1. Fused Forward & Backward Kernels (CUDA + SIMD Fallback)

​Instead of launching separate memory-bound kernels for every operation, VeloGrad fuses common transformer primitives into single passes:

​Fused GEMM + Bias + Activation (Forward & Backward)

​Fused RMSNorm + Residual Add

​Fused RoPE + Tiled Flash-Style Self-Attention

​Single-Pass Fused Cross-Entropy Loss (computes log-softmax and gradients without materializing massive [batch, seq, vocab] logit tensors in VRAM)

​Fused Multi-Tensor AdamW

​2. Static Arena Allocation & ~66.7% KV/VRAM Compression

​Dynamic allocations (cudaMalloc) during training loops cause fragmentation and latency spikes. VeloGrad pre-allocates a deterministic static memory arena and pairs block-quantized Paged KV-Cache with activation checkpointing (recompute_backward), cutting KV-cache and activation VRAM overhead by up to ~66.7% compared to unquantized baselines.

​3. ZeRO-3 Sharding + CPU Parameter Offloading

​To let developers train and fine-tune larger models on consumer GPUs without hitting OOM walls, VeloGrad implements ZeRO-3 (FSDP) parameter sharding and CPU parameter offloading over NCCL (with a cross-platform TCP/Ring fallback when NCCL isn't present), supporting SafeTensors and GGUF weights across Llama, Qwen, DeepSeek, Gemma, and Mistral model families.

​4. Natural-Language Training Orchestration (velograd.instruct)

​Alongside standard Python Tensor and nn.Module APIs, v1.1.0-beta introduces velograd.instruct, which auto-detects your local GPU/CPU topology and maps plain-English training directives directly into a configured sharding and training pipeline:

import velograd

Auto-detects available GPUs/RAM, loads weights, and configures sharding + VRAM compression

velograd.instruct(
"Fine-tune Llama-3.2-3B on local_dataset.jsonl with ZeRO-3 offloading and 66.7% KV compression"
)
How to Test v1.1.0-beta on Your Machine
​If you want to compile it and run the reproducible PyTorch comparison and VRAM profiling scripts on your own GPU or CPU laptop:
git clone https://github.com/IDKLETSCODESOMETHIN/VeloGrad.git
cd VeloGrad
pip install -e .

Run the verification suite and benchmarks

pytest
python benchmarks/bench_vs_pytorch.py
python benchmarks/profile_vram.py
What’s Next & Call for Beta Testers
​Because this is a Public Beta, real-world hardware diversity is what matters most right now. If you test VeloGrad on NVIDIA RTX cards, multi-GPU workstations, Apple Silicon, or Windows/Linux CPUs, please open an issue on GitHub with your build logs or benchmark numbers!
​Repository: https://github.com/IDKLETSCODESOMETHIN/VeloGrad

Top comments (0)