PyTorch from Scratch — Part 1: Tensors, Gradients & Activations

Most people use PyTorch without really knowing what's happening underneath. This series breaks the foundations down into the simplest possible explanations — one concept at a time, with code you can run and exactly what goes in and comes out.

This is Part 1 of 5. By the end you'll understand the five building blocks every neural network is made of: creating tensors, doing math on them, reshaping them, computing gradients, and bending them with activation functions.

No assumed knowledge. Let's go.

---

1. What a tensor actually is

Everything in deep learning is built from one object: the tensor. Don't let the name scare you — a tensor is just a box of numbers.

• 1 number → a scalar
• a row of numbers → a vector
• a grid → a matrix
• stacked grids → a tensor

An image is literally a 3D tensor: height × width × colour.

The first skill is creating them — filled with zeros, ones, or any value you want. Then .tolist() reads the tensor back as a plain Python list.

import torch

def create_tensor(method, shape, value=0.0):
    if method == "zeros":
        t = torch.zeros(shape)
    elif method == "ones":
        t = torch.ones(shape)
    else:                       # "full"
        t = torch.full(shape, value)
    return t.tolist()

What goes in and what comes out:

create_tensor("zeros", [2, 3])        -> [[0.0, 0.0, 0.0], [0.0, 0.0, 0.0]]
create_tensor("full", [2, 2], 7.0)    -> [[7.0, 7.0], [7.0, 7.0]]

💡 Gotcha: a function only hands back a value if you write return. Forget it and your function silently returns None — one of the most common beginner bugs.

---

2. Doing math on tensors

There are two kinds of math you'll use constantly, and mixing them up is the #1 beginner mistake.

1. Element-wise — same position meets same position. [1, 2, 3] + [4, 5, 6] = [5, 7, 9].
2. Matrix multiplication (@) — rows × columns. This one mixes values together, and it's the single most-used operation in all of deep learning. Every layer of every model is a matmul.

import torch

def tensor_op(x, y, op):
    a = torch.tensor(x, dtype=torch.float32)
    b = torch.tensor(y, dtype=torch.float32)
    if op == "add":
        result = a + b
    elif op == "multiply":
        result = a * b
    elif op == "matmul":
        result = a @ b
    elif op == "power":
        result = a ** b
    else:                       # "max"
        result = torch.maximum(a, b)
    return result.tolist()

Input → output:

tensor_op([1,2,3], [4,5,6], "add")              -> [5.0, 7.0, 9.0]
tensor_op([[1,2],[3,4]], [[5,6],[7,8]], "matmul") -> [[19.0, 22.0], [43.0, 50.0]]

💡 Two traps: * is element-wise multiply, @ is matrix multiply — completely different operations. And for the element-wise maximum of two tensors, use torch.maximum(a, b), not Python's built-in max() (that one can't compare tensors position-by-position).

---

3. Reshaping tensors

Reshaping means: same numbers, new shape. The data never changes — only how it's arranged. This matters because data arrives in one shape and the next layer expects another. Reshaping is the quiet glue holding a network together.

• flatten → squash a grid into a single line
• squeeze → drop useless size-1 dimensions
• transpose (.T) → flip rows and columns

import torch

def reshape_tensor(x, op):
    t = torch.tensor(x, dtype=torch.float32)
    if op == "flatten":
        result = torch.flatten(t)
    elif op == "squeeze":
        result = torch.squeeze(t)
    else:                       # "transpose"
        result = t.T
    return result.tolist()

Input → output:

reshape_tensor([[1,2,3],[4,5,6]], "flatten")   -> [1.0, 2.0, 3.0, 4.0, 5.0, 6.0]
reshape_tensor([[1,2],[3,4]], "transpose")     -> [[1.0, 3.0], [2.0, 4.0]]

💡 Gotcha: else is a catch-all — it never takes a condition. Writing else op == "transpose": is a syntax error. Just else:.

---

4. Autograd — the engine that trains everything

This is the most important idea in deep learning, and PyTorch does the hard part for you.

A gradient is just a slope. It answers one question: "if I nudge this input a little, does the output go up or down, and how steeply?" That slope is what tells a network which direction to adjust its weights to reduce its error.

The trick: mark a tensor with requires_grad=True, do your math, then call .backward(). PyTorch quietly records every operation and computes all the slopes automatically — no calculus by hand.

import torch

def compute_gradient(values):
    x = torch.tensor(values, dtype=torch.float32, requires_grad=True)
    y = (x**3 + 2*x).sum()      # collapse to a single number
    y.backward()                # walk backward, fill in the slopes
    return x.grad.tolist()

Input → output:

compute_gradient([1, 2, 3])    -> [5.0, 14.0, 29.0]

Why those numbers? The slope of x³ + 2x is 3x² + 2. At x = 1 that's 3(1) + 2 = 5. At x = 2 it's 3(4) + 2 = 14. PyTorch produced the exact analytical answer — automatically. That's the whole point: it works even for equations far too big to differentiate by hand.

💡 Two gotchas: requires_grad needs floats (integers can't track gradients), and .backward() must start from a single number — that's why we call .sum() first.

---

5. Activation functions — adding the bend

Here's a surprising fact: if you stack layers that each compute weight × input + bias, the whole stack collapses into a single straight line — even if it's a hundred layers deep. A straight line can't model faces, language, or anything interesting.

Activation functions fix this by adding a bend (a "nonlinearity") after each layer. That bend is what lets a network learn curves and complex patterns.

The common ones:

• ReLU → cut off negatives: max(0, x)
• Sigmoid → squash any number into 0…1
• Tanh → squash into −1…1
• LeakyReLU → like ReLU, but lets a tiny bit of negatives through so neurons don't "die"

import torch

def activation(x, method):
    t = torch.tensor(x, dtype=torch.float32)
    if method == "relu":
        result = torch.clamp(t, min=0)
    elif method == "sigmoid":
        result = 1 / (1 + torch.exp(-t))
    elif method == "tanh":
        result = torch.tanh(t)
    else:                       # "leaky_relu"
        result = torch.where(t > 0, t, 0.01 * t)
    return result.tolist()

Input → output:

activation([-2,-1,0,1,2], "relu")     -> [0.0, 0.0, 0.0, 1.0, 2.0]
activation([-1,0,1], "sigmoid")       -> [0.269, 0.5, 0.731]

💡 Gotcha: in 1 / (1 + torch.exp(-t)), the parentheses matter. Without them, Python computes (1/1) + exp(-t) because division runs before addition. When a whole expression is the denominator, wrap it in brackets.

---

The big picture

Put it together and you have the entire core loop of deep learning:

numbers → make a guess → measure the error → compute the slopes → adjust → repeat

That's it. Tensors hold the numbers. Matrix multiply and activations make the guess. Autograd computes the slopes. Everything else — CNNs, transformers, LLMs — is a remix of these same five ideas.

Coming in Part 2: we take these pieces and assemble them into a real, working neural network from scratch.

If this was useful, follow along — I'm building the whole thing in public, one part at a time.

🔗 I post the short version of each part on X: @Meclin_A_Francis

Comments

[Wesam Khallaf — Author of PyTorch From Ground Up]

This is a genuinely good breakdown — "same numbers, new shape" is the clearest one-sentence version of reshaping I've read.
One thing I'd add to the squeeze note, because it cost me an evening once: bare squeeze() removes every size-1 dimension, batch dimension included. So it behaves perfectly all through an epoch and then breaks on the final batch when the data doesn't divide evenly and you're left with one sample. Passing an explicit dim — squeeze(-1) — sidesteps the whole class of bug.
Small one on the sigmoid too: 1/(1+torch.exp(-t)) is perfect for showing the formula, but torch.sigmoid(t) is numerically stable for large negative inputs where exp(-t) overflows. Probably worth a footnote, since beginners tend to copy the hand-rolled version into real code.