If you work in AI today, you are almost certainly using PyTorch or TensorFlow. They are incredible tools, but they are also massive black boxes.
As an AI developer, I realized that relying solely on these frameworks meant I didn't truly understand the underlying hardware realities, memory mechanics, or how the math actually maps to GPU acceleration.
So, I decided to strip away the abstractions. Over the last few months, I built Aakaar—a deep learning framework developed completely from scratch using native C++ and CUDA, wrapped in Python for ease of use.
Here is how I built it, the engineering challenges I faced, and how it surprisingly edged out PyTorch in a direct benchmark.
The Architecture: Hand-Coding the Math
Building a framework from scratch means you don't get autograd for free.
The core architecture of Aakaar relies on:
- Backend: A purely native C++ implementation.
- Components: 18 hand-coded native loss modules and 11 custom optimizers built directly into the C++ backend.
- Memory Management: Explicit contiguity management during tensor transpositions and custom backpropagation. Instead of relying on an automated computational graph, I had to manually track and allocate memory layouts during the backward passes.
- Frontend: A Python wrapper so models can be defined intuitively, similar to standard frameworks.
The Hardest Part: Memory on the GPU
The primary engineering hurdle wasn't the forward pass—it was the backward pass. Mapping abstract mathematical shapes directly to physical GPU memory layouts during backpropagation is brutally unforgiving.
If your memory isn't strictly contiguous when a tensor transposes during a CUDA kernel execution, the entire system bottleneck slows to a crawl or crashes. Managing that contiguity explicitly in C++ without standard autograd overhead was a massive exercise in systems engineering.
The EMNIST Benchmark: Aakaar vs. PyTorch
To see if Aakaar was structurally viable, I didn't want to just run a toy matrix multiplication. I set up a 5-epoch training loop on the EMNIST dataset and benchmarked Aakaar directly against PyTorch on my local system (Intel i7, RTX 4060, 8GB VRAM).
To ensure a fair test, the model architecture, hyperparameters, and weights were identical across both frameworks.
The Results (Average of multiple test runs):
- Aakaar: 127.76 seconds(83.30% acc)
- PyTorch: 131.23 seconds(82.55% acc)
Why was it faster?
Aakaar consistently maintained absolute convergence parity while delivering a slight edge in runtime speed. This performance advantage stems directly from bypassing the Python runtime overhead during the low-level C++ optimizer steps. By keeping the optimizer operations strictly in the native backend, Aakaar shaved off the milliseconds of overhead that accumulate over thousands of batches.
What's Next and How to Contribute
Seeing the math align perfectly with the hardware performance made the struggle worth every line of code.
Aakaar is fully open-source. I am currently looking for feedback from the systems engineering and AI infrastructure communities. Specifically, if you have experience with CUDA kernel optimization or C++ memory allocation strategies, I would love for you to audit the architecture.
- Test Run (Source Code): https://github.com/aaravaggarwal3535/aakaar-wheels/blob/main/main.ipynb
- Documentation: https://aakaar.readthedocs.io
Have you ever tried building ML components from scratch? Let me know your thoughts or optimization ideas in the comments below!
Top comments (1)
Congrats on shipping this — The contiguity part is what got me. Everyone talks about autograd being the hard bit, but it's the memory layout during the backward pass that actually hurts — a transpose mid-backward and suddenly your strides don't line up with anything. Frameworks hide that so well you forget it's happening until you sit down and write it yourself. Respect for going through that.
One thought that might make the benchmark even stronger: PyTorch has foreach and fused optimizer variants that pull the per-parameter loop out of Python — basically the same overhead you engineered around. If your run was on the default path, then re-running against fused=True would let you say you beat PyTorch at its best, which is a much bigger claim than the one you're making now. Might already be true, and if it is, that headline writes itself.
Same thought on the 83.30 vs 82.55 — if you can get both runs seeded so the accuracies land on top of each other, then the time difference becomes clean and unarguable. Exact convergence parity plus a speed edge is a genuinely strong result to be able to state flatly.
18 loss modules and 11 optimizers hand-written in C++ is a serious amount of work either way. Following along to see where this goes.