In 2001, in the final year of my Computer Science degree, I chose A_I and Neural Networks_ as an elective. The theory was beautiful — backpropagation, perceptrons, gradient descent — and it felt completely impossible. Not in a discouraging way, but in the way that astronomy feels impossible: real, rigorous, and yet impossibly far from anything you could touch. I remember thinking: this will matter someday, but not in my lifetime.
As it turned out, I did not have to wait that long.
In 2012, a graduate student running two GTX 580 GPUs at the University of Toronto changed everything. AlexNet — a neural network trained on gaming graphics cards — won the ImageNet competition by a margin so large it didn’t just win the contest, it ended the era of hand-crafted computer vision and started the era of deep learning. The algorithms had not changed. What changed was that matrix multiplications — the core operation inside every neural network — could suddenly be parallelised across thousands of GPU cores simultaneously. Theory became engineering. Impossible became inevitable.
Everything since has been the answer to one question: what happens if we keep throwing more GPU compute at this?
I have spent the years between 1998 and today building systems — writing C, C++, and Java, working with the JVM, understanding memory, processors, and parallelism at the level where the abstraction runs out. I did not know I was building the foundation for understanding AI. But it turns out that is exactly what I was doing.
This article is my attempt to connect everything in between — from what I learned writing systems code close to the metal, to what I taught myself about NumPy, PyTorch, and how modern ML frameworks actually work under the hood. The ideas underlying modern AI may be newer in scale, but the engineering principles underneath them are not. I just had not seen them assembled this way — and called machine learning.
01 — Foundation
Memory, Pointers, and Why Layout Is Everything
Let's start at the bottom. Consider a simple 2D matrix in C:
This is not just syntax — it is the physical layout of data in DRAM/Address space. Every decision about memory layout has direct consequences for CPU performance through one mechanism: cache behaviour.
This matters hugely for ML. NumPy defaults to C-order (row-major) for exactly this reason. When you call np.dot(A, B), NumPy knows the layout and can process rows sequentially — but matrix multiply of A·B still needs column access of B. That's why NumPy transposes B internally before calling its optimised BLAS routine.
🔧 C++ Connection This is identical to the reason you always iterate matrix[row][col] in C++. NumPy simply enforces this systematically across every operation, and the performance cliff for getting it wrong is the same 10–100× penalty you already know from cache profiling.
02 — Data Structures
The ndarray: A Struct with Superpowers
At the C level, a NumPy ndarray is roughly this:
The genius is in strides. A transpose in NumPy is O(1) — it just swaps two numbers in the strides array. No data is moved. This is the same trick you'd use writing a cache-oblivious matrix library in C++.
Algorithmic complexity in tensor operations
Big-O complexity maps directly to understanding why ML code is slow or fast — and for anyone with a systems background, this table will feel immediately familiar:
The matrix multiply case is particularly important. Your mental model of O(n³) is correct for general matmul — but modern GPU BLAS (cuBLAS) achieves this with extraordinarily high hardware utilisation through tiling, register-level blocking, and warp-level parallelism. The algorithm is the same; what changes is the constant factor.
03 — Parallelism
From Pthreads to SIMD to GPU Warps
Coming from systems programming, CPU parallelism is already familiar territory — threads via pthreads, SIMD with SSE/AVX intrinsics, and the cache-coherence headaches that come with them. The bridge to GPU parallelism is shorter than it looks.
CPU SIMD — what NumPy actually calls
💡 Key Insight NumPy's vectorised operations are not slow Python loops with a C wrapper. They dispatch to hand-tuned SIMD kernels compiled with Intel MKL or OpenBLAS. When you write A + B in NumPy, you're calling the same hardware instructions you'd use with AVX intrinsics — without writing a single line of C.
GPU thread hierarchy — the real architecture
The leap from CPU to GPU is a change in philosophy, not just scale. Your CPU has 8–32 high-frequency, complex cores. An NVIDIA A100 has 6912 simpler cores, but they execute in lockstep groups of 32 — called warps. This is SIMT: Single Instruction, Multiple Threads.
Why matrix multiply maps perfectly to GPUs
Think about computing C = A × B where A is 1024×1024 and B is 1024×1024. Each element of C requires a dot product of a row of A with a column of B — 1024 multiply-accumulate operations. There are 1024×1024 = ~1M such elements to compute. These computations are completely independent of each other.
That independence is the key. The GPU assigns one thread to each output element, or one warp to a tile of output elements. All 6912 cores are busy simultaneously, each doing the arithmetic for their assigned output tile. This is why a GPU can do a 1024×1024 matmul in microseconds.
🔧 C++ Connection Warp divergence on a GPU is the exact analogue of branch misprediction on a CPU. If threads in a warp take different paths through an if/else, the GPU serialises both branches — every thread executes both paths, masking the results. This is why neural network activation functions like ReLU, which have a simple conditional (max(0, x)), are preferred over complex branching logic in ML kernels.
04 — NumPy → PyTorch
The Abstraction Stack and Autograd
Here is the full abstraction stack from hardware to PyTorch, shown as the layers we already understand:
The remarkable thing about PyTorch’s autograd — automatic differentiation — is that it’s a graph data structure problem already known: a Directed Acyclic Graph (DAG) where nodes are tensor operations and edges carry gradients.
💡 Key Insight Backpropagation is not magic. It is the chain rule of calculus applied via a reverse post-order traversal of a DAG. Anyone who has implemented topological sort can implement backprop. The gradients flow backward through the same graph the forward pass built — this is exactly why PyTorch calls it a "dynamic" computational graph: the DAG is constructed at runtime, not at compile time.
05 — Neural Networks
Layers Are Just Matrix Multiplications
Strip away the terminology and a neural network layer is:
This is why GPUs dominate ML. Those 134 million operations per layer are entirely parallelisable — each output element is independent. The GPU executes them in parallel across its thousands of cores.
Transformers: Attention is a Weighted Matrix Operation
The attention mechanism that powers every large language model (GPT, Claude, Gemini) reduces to this:
06 — GPU Scaling
Why GPUs and Tensors Are a Perfect Match
The reason ML scales on GPUs reduces to three properties that are true simultaneously:
Arithmetic intensity is high. For a matmul of two N×N matrices, you do O(N³) operations on O(N²) data. As N grows, the ratio of compute to memory load increases — you get more work done per byte fetched. GPUs love this: they can keep their compute units busy while memory catches up.
Operations are embarrassingly parallel. Independent output elements map directly to independent GPU threads. No synchronisation required between threads computing different output tiles.
The memory access pattern is predictable. GPUs hide memory latency through massive oversubscription — when one warp is waiting on a memory fetch, the SM switches to another ready warp. This requires many in-flight warps, which requires many independent work items, which tensor operations provide naturally.
⚡ Performance Note An A100 GPU delivers ~77 TFLOPS for fp32 matmul. The same computation on a 32-core CPU at 3 GHz with AVX-512 delivers roughly ~3 TFLOPS. The GPU wins by ~25× — not because it's faster per core, but because it has orders of magnitude more cores executing the same instruction simultaneously.
Data types: fp16 and bfloat16
ML uses reduced-precision floats — not for accuracy, but for speed and memory. Knowledge of IEEE 754 is directly applicable here:
bfloat16 (Google Brain float) has the same exponent as fp32, so it handles the same numerical range — it just sacrifices mantissa precision. Training works because SGD is already noisy; an extra bit of gradient noise is irrelevant. Meanwhile you get 2× the throughput and 2× the memory capacity.
07 — The Full Picture
Everything Connected
To make the stack concrete, here is a single forward pass through a transformer — every step mapped to its hardware call:
The entire field of "large model training" is, at its core, the engineering problem of keeping GPU cores busy with matrix multiplications for as many hours as possible without running out of memory.
08 — Edge
When systems knowledge meets ML
Most ML resources teach PyTorch from the top down — tensors, models, training loops — and treat the performance as magic happening somewhere underneath. Coming from systems programming, I couldn’t leave it there.
When a model runs slowly, my instinct is to ask: is it compute-bound or memory-bound? I know the difference between arithmetic intensity and memory bandwidth, and I can diagnose which one is the bottleneck using tools like nvprof or PyTorch’s own profiler — the same way I would reach for perf or vtune on a CPU workload.
When I read about Flash Attention — the algorithmic trick that made long-context transformers practical — I immediately recognised it as a tiling strategy that keeps intermediate results in shared memory rather than writing them back to global VRAM. It’s a cache-oblivious algorithm. I’d just never seen it applied to attention before.
When PyTorch warns about non-contiguous tensors, I know exactly what it means: the stride arithmetic no longer produces a sequential layout, so .view() fails. The fix — .contiguous() — forces a fresh row-major copy. I’ve hit the same issue in C with misaligned buffers.
The foundation was already there. The ML ecosystem was speaking a language I already knew — I just needed to learn the vocabulary.
💡 Where I am going next The natural next steps from here are: (1) CUDA C++ — already feels familiar, it is essentially C with thread-block annotations; (2) Writing custom PyTorch ops in C++ via torch::Tensor; (3) Triton — Python-syntax GPU kernel programming that compiles to PTX; (4) The Flash Attention paper (Dao et al. 2022) — a beautiful example of memory-hierarchy-aware algorithm design applied to ML.
Closing
The Stack Was Always There
Machine learning did not invent new computer science. It assembled existing computer science — linear algebra, graph algorithms, memory hierarchy optimisation, parallel computing — into a workflow that scales with data and hardware.
Every concept in this article was already in the toolkit. NumPy’s ndarray is a fat pointer with stride metadata. Autograd is topological sort plus chain rule. GPU training is embarrassingly-parallel GEMM across thousands of ALUs. Transformers are three matrix multiplies per layer, repeated N times.
Writing this article made that clearer than anything else: I had been doing machine learning — in spirit — for years. I was just calling it by different names.
Foot Note
Top comments (0)