Autograd Engine from Scratch: 200 Lines to Beat PyTorch

The Core Idea: Reverse-Mode Differentiation

You don't actually need PyTorch to train neural networks. The entire autograd mechanism — the thing that makes gradient descent possible — fits in about 200 lines of Python. I built one to see what PyTorch is really doing under the hood, and the result was faster than I expected on small models.

The core insight: every operation you perform (addition, multiplication, ReLU) needs to remember two things. The forward pass result, and how to compute gradients flowing backward. That's it. The rest is bookkeeping.

Here's what a minimal Value class looks like:

class Value:
    def __init__(self, data, _children=(), _op=''):
        self.data = float(data)
        self.grad = 0.0
        self._backward = lambda: None
        self._prev = set(_children)
        self._op = _op

    def __repr__(self):
        return f"Value(data={self.data}, grad={self.grad})"

Every Value wraps a scalar. When you perform operations, you create new Value objects that remember their parent nodes in _prev. The _backward function gets defined per operation — it's how gradients propagate.

---

Continue reading the full article on TildAlice