Phase 2 of 8: Enough ML to understand inference. Week 7 of 32.
The goal: train the same temperature converter while following each step:
make a prediction, measure its error, work out which way to adjust the weight
and bias, then try again. Think of adjusting two knobs and checking whether
the answer gets closer.
Week 6 trained a tiny model to convert Celsius to Fahrenheit using PyTorch's
ready-made layers and optimizer. That is a normal way to write a training loop,
and it remains useful beyond this example. We explained the prediction formula
and worked through the first update by hand. This week traces how
loss.backward() calculates the gradients that tell that update which way to go:
| Call in Week 6 | Already explained | What Week 7 adds |
|---|---|---|
nn.Linear(1, 1) |
Weight, bias, multiply and add | Plain trainable tensors |
optimizer.step() |
Subtract learning rate times gradient | Write the update directly |
loss.backward() |
Computes gradients | Trace how they are calculated |
I replace the layer and optimizer with plain tensors and a hand-written update
as a learning and debugging exercise. PyTorch still computes the gradients;
I check them against slopes worked out by hand. This goes deeper into Week 6's
loop, rather than replacing the normal PyTorch workflow.
The problem is the same as Week 6 on purpose. Same six examples, same raw Celsius
inputs, same learning rate. We can compare where the two loops end up.1
Follow one prediction to its error
Start with one prediction, before training anything. I chose w=2.0, b=1.0,
and x=3.0 by hand: the prediction is
, against a target of 10.
The squared loss is
. With w=3.0, the prediction would already be
10, with zero loss and zero gradients. I use 2.0 so there is an error to explain.
The lab checks both choices; training later uses random starts.2
To work out how each parameter caused that error, PyTorch keeps a record of the
calculation, like a trail of intermediate results in a debugger. That record is
the computational graph. Setting requires_grad=True on the weight and bias
asks PyTorch to record operations it can differentiate, so it can calculate their
gradients. This automatic gradient calculation is called autograd.
First, look at the record. Each result of a tracked operation carries a
grad_fn, an object naming the operation that produced it:
This code runs on Spark:
ssh spark '~/venvs/w1/bin/python -' <<'PY'
import torch
# Create a one-value weight, shape (1,), with gradient tracking enabled.
w = torch.tensor([2.0], requires_grad=True)
# Make a separate one-value bias with tracking enabled too.
b = torch.tensor([1.0], requires_grad=True)
# Omitted requires_grad defaults to False: this one-value input is fixed data.
x = torch.tensor([3.0])
# Compute the model prediction from input, weight, and bias.
pred = w * x + b
# Choose 10 as this tiny example target, unrelated to the Fahrenheit task.
# Score the squared prediction error against the target: (7 - 10)**2 = 9.
loss = (pred - 10.0)**2
# Show the prediction and loss with their recorded operations.
print("pred:", pred)
print("loss:", loss)
# Compare gradient tracking for the fixed input and trainable weight.
print("x.requires_grad:", x.requires_grad, " w.requires_grad:", w.requires_grad)
PY
pred: tensor([7.], grad_fn=<AddBackward0>)
loss: tensor([9.], grad_fn=<PowBackward0>)
x.requires_grad: False w.requires_grad: True
Two things to read here. pred was built with a multiply and an add, so its
grad_fn is AddBackward0 (the last step). loss was built with a power, so its
grad_fn is PowBackward0. Each grad_fn names only the tensor's immediate
producing step, but that object links back to the steps that fed it, so together
they form the full chain PyTorch will walk in reverse:3
And x.requires_grad is False, because I never asked it to be tracked. Inputs
are fixed data, so they do not need gradients; only the parameters I want to
train do.
For now, follow the weight and bias: backward() will put their gradients in
w.grad and b.grad. The optional gradient-storage check
explains which tensors receive stored gradients by default.
Work out which way to adjust the parameters
A quick reminder from Week 6, part 2: a gradient tells us the slope of the
loss for each parameter. If increasing a parameter slightly lowers the loss,
its gradient is negative; if it raises the loss, its gradient is positive.
The size tells us how steeply the loss changes at the current values.
Backpropagation is how we calculate those gradients: start at the loss and
follow the recorded calculation backward to the weight and bias. In PyTorch,
loss.backward() does this work. It calculates the slopes; the separate update
step uses them to adjust the parameters.
Take the tiny example above: the loss is
with
,
,
, so the prediction is 7. Here is how fast the loss changes if I nudge each
parameter a tiny amount from the current point:
- Nudging the prediction up changes the squared loss at a rate of loss units per prediction unit.
- Nudging the weight up changes the prediction at a rate of .
- Multiply those two rates: . That is the weight's slope.
- Nudging the bias up changes the prediction at a rate of 1, so its slope is .
Multiplying the connected rates along the chain is the chain rule. Written
in calculus notation, those two slopes are:
The symbol
just means "how much the loss
changes
when
changes." I ran backward() and compared it to those hand values:
ssh spark '~/venvs/w1/bin/python -' <<'PY'
import torch
# Create one-value parameters, shape (1,), with gradient tracking enabled.
w = torch.tensor([2.0], requires_grad=True)
b = torch.tensor([1.0], requires_grad=True) # Bias also needs a gradient.
x = torch.tensor([3.0]) # Fixed input; requires_grad defaults to False.
# Compute the model prediction from input, weight, and bias.
pred = w * x + b
# 10 is the deliberately chosen toy target, not a Fahrenheit label.
loss = (pred - 10.0)**2 # Score the squared prediction error against the target.
loss.backward() # Follow the graph to fill w.grad and b.grad.
print("w.grad:", w.grad, " b.grad:", b.grad)
# Apply the chain-rule formulas using the input and prediction values.
print("hand dL/dw:", 2*(pred.item()-10)*x.item(),
" hand dL/db:", 2*(pred.item()-10))
PY
w.grad: tensor([-18.]) b.grad: tensor([-6.])
hand dL/dw: -18.0 hand dL/db: -6.0
They match. backward() filled w.grad and b.grad with the same numbers I got
from the derivative by hand. That is autograd's core job, just for graphs far too
large to differentiate by hand.
Next we use the same idea on the six temperature examples. Their loss is the
average squared error, so the gradient averages their contributions too. The
training transcript compares autograd with that hand calculation; the optional
six-example check shows every
contribution.
The training loop, without the helpers
Now the real thing. This script trains Celsius to Fahrenheit with no nn.Linear
and no optimizer. The weight and bias are plain tensors with
requires_grad=True. The update param -= lr * param.grad is written out. It
also prints the weight and bias before and after training, and checks the first
gradient against the hand-computed derivative.
The inputs and targets have shape (6, 1): six examples, one value per example.
Each parameter has shape (1,). Broadcasting reuses that one weight and bias
for every row, giving predictions of shape (6, 1). The comments mark those
shapes in the code. Element-wise weight * celsius + bias works for this
one-input, one-output model; a general nn.Linear uses matrix multiplication
and then adds its bias.4
Here is the core loop, an exact excerpt, not a standalone program, from the
lab's training script.5
The complete script is at the end;
the next section runs that full version. Read this excerpt as predict and score,
calculate gradients, update, then clear. The two safeguards, no_grad() and
zero_(), get their own small checks after the training result.
# Each epoch uses all six examples for one parameter update.
for epoch in range(1, args.epochs + 1):
# Compute the model prediction from input, weight, and bias.
# The forward pass produces six predictions, shape (6, 1).
pred = weight * celsius + bias
# Score the squared prediction error against each target, then average.
loss = ((pred - fahrenheit) ** 2).mean()
# Backward pass fills weight.grad and bias.grad.
loss.backward()
# Move each parameter opposite its gradient, scaled by the learning
# rate. This is the mathematical plain-SGD rule, using the full batch.
# no_grad keeps the in-place update out of the gradient graph.
with torch.no_grad():
# Adjust the weight using its gradient and the learning rate.
weight -= args.lr * weight.grad
# Apply the same update rule to the bias, using its own gradient.
bias -= args.lr * bias.grad
# Clear the gradients, or the next backward() would add onto these.
weight.grad.zero_()
# Both accumulators must be cleared, not just the weight's.
bias.grad.zero_()
The loop is four steps: predict and score, calculate gradients, update, and clear
the gradients. The full program also sets up the data and parameters,
checks the first gradient, reports progress, and saves and reloads the result.
Watching the hand loop run
The output below is the recorded verification run from the lab. The script's
comments have since been clarified; its calculations and reporting are unchanged.
From the parent repository root on my control Mac, I run the code on Spark in a
fresh directory to preserve earlier checkpoints. Keep run_dir for the later
load command. For a public-only clone, drop public/ from the local source path.
I expect loss to fall and the parameters to approach 1.8 and 32:
run_dir=$(ssh spark 'mktemp -d /tmp/week07-reviewed.XXXXXXXX')
printf '%s\n' "$run_dir"
ssh spark "cd '$run_dir' && ~/venvs/w1/bin/python - \
--epochs 20000 --lr 0.0003 --seed 0 \
--checkpoint manual_celsius_to_fahrenheit.pt --output results.json" \
< public/week-07-autograd-training-loop/train_manual.py
/tmp/week07-reviewed.RXvW4Ygo
device=cpu
before training: weight=1.540996 bias=-0.293429
loss.grad_fn type=MeanBackward0
first grad backward(): weight=-2314.6404 bias=-73.8247
first grad by hand : weight=-2314.6404 bias=-73.8247
epoch 1 loss= 965.8369 weight=2.2354 bias=-0.2713
epoch 50 loss= 849.9656 weight=2.0501 bias=0.5317
epoch 1000 loss= 319.4210 weight=1.9533 bias=12.7090
epoch 5000 loss= 5.1848 weight=1.8195 bias=29.5422
epoch 10000 loss= 0.0300 weight=1.8015 bias=31.8129
epoch 20000 loss= 0.0000 weight=1.8000 bias=31.9981
after training: weight=1.8000 bias=31.9981 (true weight 1.8, bias 32)
final loss (post-update, unrounded)=2.9371251457632752e-06
saved checkpoint to manual_celsius_to_fahrenheit.pt
inference: 25 C -> 76.9985 F (true 77)
Your temporary directory name will differ. The grad_fn type is MeanBackward0,
the final mean operation in the loss. The last table row prints 0.0000 because
it rounds to four decimal places; the raw final loss is about
, not zero. JSON now preserves raw values returned by PyTorch.
The reported epoch losses are measured after each selected update, unlike
the loss used by backward(), which was calculated before that update.
The gradient check passes on the six-example loss. backward() reported a
first weight gradient of -2314.6404, and the hand-computed derivative gave the
same -2314.6404. Same for the bias, -73.8247 from both. So on the real training
loss, not just the toy one, the analytical derivative and autograd agree to
every displayed digit. These computations use floating-point arithmetic.
Before and after, spelled out. The weight started at 1.540996 and ended at
1.8000. The bias started at -0.293429 and ended at 31.9981. That is the roadmap
deliverable: each parameter shown before and after training.
Same answer as Week 6. Week 6's nn.Linear plus optim.SGD ended at weight
1.8000, bias 31.9981, and predicted 76.9985 F for 25 C. This hand-written loop
ended at the same weight 1.8000, bias 31.9981, and predicted the same 76.9985 F,
to every displayed digit. The starting weights differ because a seed makes a
random draw reproducible, but nn.Linear and torch.randn use different rules to
turn that draw into numbers, so seed 0 does not give the same start. Both runs
still end near 1.8 and 32 because they minimize the same loss on the same
straight-line data. Testing numerical agreement of the updates would require
identical starting parameters and comparing intermediate results with a stated
tolerance. Equivalent mathematical rules can round differently; this experiment
shows matching displayed endpoints, not bit-for-bit identical updates.
Two lines that look optional but are not
The loop has two calls that a newcomer might drop. Both matter, and both are easy
to show.
Why the update is inside torch.no_grad(). The line
weight -= lr * weight.grad is itself a tensor operation on a tracked tensor. The
graph should hold only the operations that produced the current loss; the update
just changes the starting values for the next loop, so it must not be recorded.
An in-place change modifies the existing tensor instead of creating a new
one. Without torch.no_grad(), PyTorch refuses this in-place change and errors; with
it, the update succeeds and the weight is still a trainable parameter afterward:
The error calls w a leaf: here that means a trainable tensor created
directly, rather than the result of another tracked calculation.
ssh spark '~/venvs/w1/bin/python -' <<'PY'
import torch
w = torch.tensor([2.0], requires_grad=True) # One tracked value, shape (1,).
loss = (w * 3.0 - 1.0)**2 # Score squared prediction error for input 3, target 1.
loss.backward() # Compute the gradient and store it in w.grad.
try: # Demonstrate the failure when updating a tracked leaf without no_grad.
w -= 0.1 * w.grad # Attempt an in-place gradient update with learning rate 0.1.
except RuntimeError as e:
print("without no_grad:") # Label the failed attempt.
print(str(e)) # Show PyTorch's reason for rejecting the update.
with torch.no_grad(): # Keep the parameter update out of the gradient graph.
w -= 0.1 * w.grad # The same update is now allowed.
print("with no_grad, new w:", w.item())
# Verify the weight still requires gradients for future training steps.
print("w.requires_grad still:", w.requires_grad)
PY
without no_grad:
a leaf Variable that requires grad is being used in an in-place operation.
with no_grad, new w: -1.0
w.requires_grad still: True
Here no_grad() keeps the update out of the graph. It does the same job during
inference: suppressing gradient recording for the operations inside its block.
It leaves the parameter's requires_grad=True flag intact.
Why gradients must be cleared. backward() adds new gradients onto whatever
is already sitting in .grad. It does not overwrite. Run a fresh forward and
backward() twice without clearing and the gradient doubles:
ssh spark '~/venvs/w1/bin/python -' <<'PY'
import torch
w = torch.tensor([2.0], requires_grad=True) # One tracked value, shape (1,).
# Run two backward passes without clearing gradients or updating the weight.
for i in range(1, 3):
loss = w * 3.0 # Use a simple loss whose weight gradient is always 3.
loss.backward() # Add the new gradient to whatever w.grad already holds.
print("backward", i, "-> w.grad:", w.grad.item())
w.grad.zero_() # Clear the accumulated gradient before the next backward pass.
loss = w * 3.0 # Make a fresh forward calculation with the unchanged weight.
loss.backward() # Accumulate onto zero this time.
print("after zero_ and fresh backward -> w.grad:", w.grad.item())
PY
backward 1 -> w.grad: 3.0
backward 2 -> w.grad: 6.0
after zero_ and fresh backward -> w.grad: 3.0
The true gradient is 3.0, but the second backward() without clearing reads 6.0.
That is why the loop calls weight.grad.zero_() and bias.grad.zero_() every
step; zero_() fills the existing gradient tensor with zeros. In Week 6 the
optimizer's zero_grad() did this for me; here I do it directly.
Train mode versus evaluation mode
Our tensor-only model has no .train() or .eval() method; those belong to
nn.Module, PyTorch's base class for model layers. Week 6's plain linear layer
behaved the same in either mode. But
some layers do behave differently, and the clearest example is dropout, a layer
that randomly zeros some values during training so the model does not lean too
hard on any single value. During evaluation it must stop doing that. You switch
with .train() and .eval():
ssh spark '~/venvs/w1/bin/python -' <<'PY'
import torch
from torch import nn
torch.manual_seed(0) # Repeat this environment's random dropout choices.
# Give each input value a 50% chance of being zeroed during training.
drop = nn.Dropout(p=0.5)
x = torch.ones(8) # Eight ones, shape (8,); requires_grad defaults to False.
drop.train() # Enable random dropping and scaling of surviving values.
print("train mode :", drop(x)) # Show which values survived and their scaling.
drop.eval() # Switch that same layer to evaluation behavior.
print("eval mode :", drop(x)) # Run it again on the unchanged input.
PY
train mode : tensor([0., 0., 2., 0., 0., 0., 2., 2.])
eval mode : tensor([1., 1., 1., 1., 1., 1., 1., 1.])
In train mode dropout zeroed several values (five of the eight here) and scaled
the survivors up. Each value independently has a 50% chance of being dropped
because p=0.5, and survivors are multiplied by
so the average
output stays about the same across many dropout samples. In eval mode it passed
all eight values through unchanged, doing neither the dropping nor the scaling.
That is why real inference code calls .eval() first: you do not want random
dropout altering predictions. My Week 6 eval() call was setting exactly this
mode, it just had nothing to change for a plain linear layer.
Mode and gradient recording are separate switches
Use evaluation mode when you want a module's inference behavior. Use no_grad()
when you do not need a gradient graph, such as ordinary prediction or our manual
parameter update. For inference with a module, normally use both.6
| Control | Changes | Leaves alone |
|---|---|---|
.train() / .eval()
|
Mode-sensitive layer behavior | Gradient recording |
torch.no_grad() |
Recording inside the block | Module mode |
param.grad.zero_() |
Stored gradient values | Parameters and module mode |
None of these calls updates the parameters. The optional
combined check tests the switches together
and follows what happens to the graph and stored gradients between steps.
The checkpoint is just the learned tensors
The training program saves a dictionary: the keys weight and bias name its
two learned tensors. weight.detach() shares the weight's storage but has no
gradient history and does not require gradients; the original weight remains
trainable. This is a minimal inference parameter checkpoint, not a full training
restart: it contains no model code, optimizer state, epoch, or module mode.
The printed directory and checkpoint filename locate the actual file:
/tmp/week07-reviewed.RXvW4Ygo/manual_celsius_to_fahrenheit.pt in my run.
run_dir still holds that directory in my control shell. This next SSH command
starts a fresh Python process there. It loads only the checkpoint, selects
the two named tensors, creates a new input, and applies the prediction formula.
I expect the same prediction as the training script's reload check:
ssh spark "cd '$run_dir' && ~/venvs/w1/bin/python -" <<'PY'
import torch
# Load the saved parameter dictionary, restricting loaded types.
# Only load a checkpoint you trust.
state = torch.load("manual_celsius_to_fahrenheit.pt", weights_only=True)
# Verify the checkpoint contains the expected weight and bias entries.
print(f"keys={list(state.keys())}")
# Inspect each saved parameter's shape and gradient flag.
for name, value in state.items():
print(f"{name}: shape={tuple(value.shape)} requires_grad={value.requires_grad}")
with torch.no_grad(): # Prediction needs no gradient graph.
# One new Celsius input, shape (1, 1).
test_c = torch.tensor([[25.0]])
# Compute the model prediction from input, weight, and bias.
# Use the checkpoint's learned tensors rather than training-process variables.
prediction = state["weight"] * test_c + state["bias"]
# Verify the prediction's shape and that it has no gradient tracking.
print(f"prediction: shape={tuple(prediction.shape)} "
f"requires_grad={prediction.requires_grad}")
# Compare the reloaded model's prediction with the known answer.
print(f"inference: 25 C -> {prediction.item()} F (true 77)")
PY
keys=['weight', 'bias']
weight: shape=(1,) requires_grad=False
bias: shape=(1,) requires_grad=False
prediction: shape=(1, 1) requires_grad=False
inference: 25 C -> 76.99851989746094 F (true 77)
That is 76.9985 F at the training transcript's precision, from disk rather than
variables left over from training. The loader supplies the formula; the file
supplies its learned numbers. weights_only=True restricts loading to tensors
and supported simple data types. Only load checkpoints you trust.7
These detached tensors already have tracking disabled; the no_grad() block
also makes the inference intent explicit. No .eval() call is needed because
there is no module here. Temporary directories may be cleaned up by the system;
the lab shows how to retrieve results without replacing a captured baseline.
Results
Here is Week 7 in one place:
| Item | Verified value |
|---|---|
| Task | Celsius to Fahrenheit, F = C * 1.8 + 32 |
| Model | two plain tensors, no nn.Linear, no optimizer |
| Device | CPU |
| Before training (weight / bias) | 1.540996 / -0.293429 |
| First gradient, backward() | weight -2314.6404, bias -73.8247 |
| First gradient, by hand | weight -2314.6404, bias -73.8247 |
| After training (weight / bias) | 1.8000 / 31.9981 |
| Inference: 25 C | 76.9985 F (rounds to 77) |
| Learning rate / epochs | 0.0003 / 20000 |
| Final loss, unrounded rerun | 2.9371251457632752e-06 |
| Week 6 comparison | Same displayed endpoint, not a stepwise test |
What surprised me
I knew backward() computed gradients, but I did not expect the hand-computed
derivative to match it to every printed digit on the six-example loss. Seeing
-2314.6404 twice made autograd feel concrete instead of mysterious. It is not
estimating slopes by trying small changes; it computes the exact chain-rule
derivative, in floating-point numbers.
The other surprise was how similar the result is with and without nn.Linear and
optim.SGD. Removing both changed the starting weights but not the endpoint. The
loop underneath is four steps.
Mistakes and troubleshooting
The first trap is forgetting torch.no_grad() around the update. Without it,
PyTorch tries to track the subtraction and you get errors about modifying a tensor
that requires grad in place. The fix is to wrap the update, which is exactly what
an optimizer does internally.
The second trap is forgetting to clear gradients, which silently mixes gradients
from different steps. They can reinforce or cancel each other. The
accumulation demo above is the quickest way to see it.
The third is expecting celsius.grad to exist. Our inputs have
requires_grad=False; the weight and bias are the values we ask PyTorch to
differentiate. The leaf-tensor check shows their
gradient fields side by side.
Production implications
Production training usually reaches for an optimizer like optim.SGD or a
higher-level trainer instead of updating each parameter by hand, but they run this
same cycle underneath with more parameters and smarter update rules. When a
training run diverges, or a gradient goes to zero, or a fine-tune does nothing,
the thing to picture is these four steps: forward, backward, update, clear.
Knowing that backward() is plain calculus and that gradients accumulate unless
cleared is enough to debug a surprising number of training problems.
What I will learn next
Week 8 leaves the training loop and starts on how models turn text into numbers:
token IDs, embedding vectors, the embedding table, and cosine similarity. That is
the first step toward understanding how a language model represents meaning, and
it connects back to the token IDs I met in Week 2.8
Run it yourself
The public Week 7 lab has the manual training script, the captured run,
observations, and troubleshooting notes.5
I reused the Spark environment from Week 1. Before the experiments, I checked
the machine architecture and PyTorch version:
ssh spark 'uname -m; ~/venvs/w1/bin/python -c "import torch; print(torch.__version__)"'
aarch64
2.13.0+cu130
aarch64 is ARM64. This is a CUDA-enabled PyTorch build, but the tiny training
program uses the CPU. Seeded outputs are tied to this environment, not a
promise that every PyTorch version or machine produces identical values.
Optional: a closer look
The main loop is complete. These checks look more closely at where gradients
are stored, how the six examples contribute, and how mode and recording interact.
Read whichever answers a question you still have, or go straight to the full script.
Where gradients are stored
A trainable leaf tensor has requires_grad=True and was not produced by a
recorded operation. Our directly created weight and bias are leaves; pred and
loss are recorded results. Tensors with requires_grad=False are also called
leaves by convention, even when calculated from other untracked tensors.
By default, backward() accumulates gradients in .grad for leaves that require
gradients and contribute to the loss. Here is that distinction in this model:
ssh spark '~/venvs/w1/bin/python -' <<'PY'
import torch
torch.manual_seed(0) # Repeat the random start in this PyTorch environment.
# Six Celsius inputs, one temperature per row: shape (6, 1).
C = torch.tensor([[-40.],[-10.],[0.],[20.],[37.],[100.]])
F = C * 1.8 + 32 # Build the six target Fahrenheit answers, shape (6, 1).
# Draw a one-value weight, shape (1,), with gradient tracking enabled.
w = torch.randn(1, requires_grad=True)
b = torch.randn(1, requires_grad=True) # A separate random bias.
# Compute the model prediction from input, weight, and bias.
pred = w * C + b # Reuse each parameter across all six rows, shape (6, 1).
# Score the squared prediction error against each target, then average.
loss = ((pred - F) ** 2).mean()
loss.backward() # Compute gradients and accumulate them in leaf .grad fields.
# Compare which leaves require gradients and actually received them.
for name, t in [("weight ", w), ("bias ", b), ("celsius", C)]:
print(name, "is_leaf", t.is_leaf, "req_grad", t.requires_grad, "grad", t.grad)
# Contrast those leaves with the prediction's recorded producing operation.
print("pred ", "is_leaf", pred.is_leaf, "grad_fn", type(pred.grad_fn).__name__)
PY
weight is_leaf True req_grad True grad tensor([-2314.6404])
bias is_leaf True req_grad True grad tensor([-73.8247])
celsius is_leaf True req_grad False grad None
pred is_leaf False grad_fn AddBackward0
The weight and bias are leaves with requires_grad=True, so they get gradients.
celsius is a leaf but has requires_grad=False, so its gradient is None.
pred is not a leaf at all; it carries a grad_fn instead. Those weight and bias
gradients, -2314.6404 and -73.8247, are the same ones the training run reports
above.
How six examples contribute one gradient
The six-example loss works the same way, one more step. The loss averages the
squared error over six examples, so each example contributes its own
term and the weight gradient is their mean.
Here are the six contributions and their average, at the random starting point:
ssh spark '~/venvs/w1/bin/python -' <<'PY'
import torch
# Repeat this environment's random starting parameters.
torch.manual_seed(0)
# Six Celsius inputs, one temperature per row: shape (6, 1).
C = torch.tensor([[-40.],[-10.],[0.],[20.],[37.],[100.]])
F = C * 1.8 + 32 # Build the six target Fahrenheit answers, shape (6, 1).
w = torch.randn(1, requires_grad=True) # One random weight, tracking enabled.
b = torch.randn(1, requires_grad=True) # One random bias, tracking enabled.
# Compute the model prediction from input, weight, and bias.
pred = w * C + b # The shape-(1,) parameters serve all six rows.
# Calculate each example's weight-gradient contribution: 2 * error * input.
contrib = (2.0 * (pred - F) * C)
print("per-example weight-grad contributions:")
# detach() drops tracking for this view; flatten() lays six rows out as (6,).
print(contrib.detach().flatten())
# Average the contributions to compare with the full-batch weight gradient.
print("mean =", contrib.mean().item())
PY
per-example weight-grad contributions:
tensor([ 1754.6619, 594.0678, -0.0000, -1498.9403, -3098.8667,
-11638.7637])
mean = -2314.640380859375
The mean of the six contributions is -2314.6404, exactly the weight gradient the
training run reports. So the full gradient is not a mystery number; it is the
chain-rule slope averaged over the six examples. The bias works the same way with
replaced by 1, so each contribution is
, and
their mean is -73.8247, the reported bias gradient. (The .detach() in the
command just drops gradient tracking so the numbers print cleanly; there is more
on detach() in the checkpoint section above.)
Mode switches and fresh graphs
This checks .eval(), no_grad(), and gradient clearing from the main table.
None of these calls performs a parameter update. Each forward pass builds a new
graph when recording is enabled. backward() uses it and normally releases the
saved intermediate values needed for another backward pass. The leaf .grad
buffers remain. Clearing those buffers changes their numbers, not the parameters
or the graph. Think of a fresh calculation with a separate accumulator for its
answer.
Here is a direct check. I set dropout to p=1.0 so it drops every value in train
mode, making the comparison deterministic. I expect eval mode to preserve the
input's gradient flag, and no_grad() to leave the module in train mode. Then
two fresh forward passes should each produce gradient 36, with zero between them:
ssh spark '~/venvs/w1/bin/python -' <<'PY'
import torch
from torch import nn
# Drop every value during training to make the mode comparison deterministic.
drop = nn.Dropout(p=1.0)
x = torch.ones(3, requires_grad=True) # Three ones, shape (3,), tracking on.
drop.eval() # Switch the layer to evaluation behavior.
y = drop(x) # Evaluation passes the input through unchanged.
# Check module mode and gradient tracking separately.
print("eval:", "training=", drop.training, "requires_grad=", y.requires_grad)
print("values:", y) # Verify evaluation preserved the input values.
drop.train() # Switch the same layer back to training behavior.
with torch.no_grad(): # Disable gradient recording without changing module mode.
y = drop(x) # Training dropout still zeros the same input.
# Check that no_grad left training mode enabled but suppressed tracking.
print("train + no_grad:", "training=", drop.training,
"requires_grad=", y.requires_grad)
print("values:", y) # Show whether training behavior still zeroed the values.
w = torch.tensor([2.0], requires_grad=True) # One tracked value, shape (1,).
# Repeat with the same weight, clearing gradients between fresh graphs.
for step in range(1, 3):
loss = (w * 3.0) ** 2 # Build a fresh graph for the same squared loss.
print("forward", step, "grad_fn:", type(loss.grad_fn).__name__)
loss.backward() # Compute and accumulate the new weight gradient.
print("after backward:", w.grad.item()) # Report the newly computed gradient.
w.grad.zero_() # Clear the gradient in place, without changing w.
print("after zero_:", w.grad.item(), "w:", w.item()) # Show both values.
PY
eval: training= False requires_grad= True
values: tensor([1., 1., 1.], requires_grad=True)
train + no_grad: training= True requires_grad= False
values: tensor([0., 0., 0.])
forward 1 grad_fn: PowBackward0
after backward: 36.0
after zero_: 0.0 w: 2.0
forward 2 grad_fn: PowBackward0
after backward: 36.0
after zero_: 0.0 w: 2.0
Eval left requires_grad=True; no_grad() left training=True and dropout
still zeroed the inputs. In the second check,
, so the weight gradient
at
is
. Both passes report 36
because I clear the accumulator between them. The weight stays 2 throughout:
there is no update in this check. The equal grad_fn type names describe the
operation, not reuse of the same graph object.
Reference: the complete training script
This is the full lab program used by the training command above. The core loop
is unchanged; here you can also follow setup, the gradient check, reporting,
checkpoint saving, and reloading.
#!/usr/bin/env python3
"""Week 7 - the training loop written by hand for Celsius to Fahrenheit.
Week 6 used nn.Linear and torch.optim.SGD. This version removes both. The weight
and bias are plain tensors with requires_grad=True, and the parameter update is
one line I write myself: param -= lr * param.grad. It uses the same mathematical
forward rule and plain-SGD update as Week 6, from a different random start.
This is not a claim of bitwise-identical floating-point results.
The problem is identical to Week 6: learn F = C * 1.8 + 32 from six examples,
with raw Celsius inputs and a small learning rate.
"""
from __future__ import annotations
import argparse
import json
from pathlib import Path
import torch
def parse_args() -> argparse.Namespace:
# Configure the training run and where to save its checkpoint and results.
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--epochs", type=int, default=20000)
parser.add_argument("--lr", type=float, default=3e-4)
parser.add_argument("--seed", type=int, default=0)
parser.add_argument("--checkpoint", type=Path,
default=Path("manual_celsius_to_fahrenheit.pt"))
parser.add_argument("--output", type=Path)
return parser.parse_args()
def main() -> None:
args = parse_args()
# Make the random starting parameters repeatable in this environment.
torch.manual_seed(args.seed)
# Six Celsius inputs, one temperature per row: shape (6, 1).
celsius = torch.tensor([[-40.0], [-10.0], [0.0], [20.0],
[37.0], [100.0]])
# Labels are the six correct answers, also shape (6, 1). Only label creation
# uses the known formula; training receives inputs and labels, not the rule.
# Inputs and labels are fixed data; neither needs gradients.
fahrenheit = celsius * 1.8 + 32.0
# Each parameter has shape (1,): one randomly drawn number. requires_grad
# asks PyTorch to track operations so backward() can compute derivatives.
weight = torch.randn(1, requires_grad=True)
# Draw a separate one-value bias with tracking enabled too.
bias = torch.randn(1, requires_grad=True)
print(f"device={weight.device}")
# item() extracts a one-element tensor as a Python number for reporting.
init_w = weight.item()
init_b = bias.item()
print(f"before training: weight={init_w:.6f} bias={init_b:.6f}")
# One manual forward and backward to show two things:
# 1. loss carries a grad_fn, so it remembers how it was built.
# 2. backward() and a hand-written derivative can be compared numerically.
# Compute the model prediction from input, weight, and bias.
# Broadcasting reuses each shape-(1,) parameter across all six input rows.
pred = weight * celsius + bias
# Score the squared prediction error against each target, then average
# all six errors into one scalar loss, shape ().
loss = ((pred - fahrenheit) ** 2).mean()
print(f"loss.grad_fn type={type(loss.grad_fn).__name__}")
# Follow the recorded graph to fill the two parameters' shape-(1,) .grad.
loss.backward()
# Store the starting loss as a Python number for JSON, before any updates.
first_loss = loss.item()
# Check the MSE derivatives without building another gradient graph:
# dloss/dweight = mean(2 * error * input), dloss/dbias = mean(2 * error).
with torch.no_grad():
# Average the six weight contributions, then extract the scalar number.
hand_w_grad = (2.0 * (pred - fahrenheit) * celsius).mean().item()
# Bias contributions omit the input multiplier; average those too.
hand_b_grad = (2.0 * (pred - fahrenheit)).mean().item()
print(f"first grad backward(): weight={weight.grad.item():.4f} "
f"bias={bias.grad.item():.4f}")
print(f"first grad by hand : weight={hand_w_grad:.4f} "
f"bias={hand_b_grad:.4f}")
# Save both gradients before clearing them; .grad holds a tensor.
first_w_grad = weight.grad.item()
first_b_grad = bias.grad.item()
# Discard this demonstration's gradients before the actual training loop.
weight.grad.zero_()
bias.grad.zero_()
# Keep selected post-update measurements, not every training step.
history = []
# Report progress at these epochs, including the final requested epoch.
log_at = {1, 50, 1000, 5000, 10000, args.epochs}
# Each epoch uses all six examples for one parameter update.
for epoch in range(1, args.epochs + 1):
# Compute the model prediction from input, weight, and bias.
# The forward pass produces six predictions, shape (6, 1).
pred = weight * celsius + bias
# Score the squared prediction error against each target, then average.
loss = ((pred - fahrenheit) ** 2).mean()
# Backward pass fills weight.grad and bias.grad.
loss.backward()
# Move each parameter opposite its gradient, scaled by the learning
# rate. This is the mathematical plain-SGD rule, using the full batch.
# no_grad keeps the in-place update out of the gradient graph.
with torch.no_grad():
# Adjust the weight using its gradient and the learning rate.
weight -= args.lr * weight.grad
# Apply the same update rule to the bias, using its own gradient.
bias -= args.lr * bias.grad
# Clear the gradients, or the next backward() would add onto these.
weight.grad.zero_()
# Both accumulators must be cleared, not just the weight's.
bias.grad.zero_()
# Capture progress only at the selected reporting epochs.
if epoch in log_at:
# Measure again after the update, using the new weight and bias.
with torch.no_grad():
post_loss = ((weight * celsius + bias - fahrenheit) ** 2
).mean().item()
print(f"epoch {epoch:>5} loss={post_loss:>12.4f} "
f"weight={weight.item():.4f} bias={bias.item():.4f}")
# Format stdout for reading, but retain unrounded floats in JSON.
history.append({"epoch": epoch, "loss": post_loss,
"weight": weight.item(), "bias": bias.item()})
final_w = weight.item()
final_b = bias.item()
# The loop's loss precedes its last update. Recompute the final measurement
# from the saved parameters, without changing those parameters.
with torch.no_grad():
final_loss = ((weight * celsius + bias - fahrenheit) ** 2).mean().item()
print(f"after training: weight={final_w:.4f} bias={final_b:.4f} "
f"(true weight 1.8, bias 32)")
# Show the raw final loss so display rounding does not make it look zero.
print(f"final loss (post-update, unrounded)={final_loss}")
# Save a dictionary with two named shape-(1,) tensors. detach() removes
# gradient tracking; the checkpoint contains values, not the training graph.
# torch.save writes this dictionary to the path supplied by --checkpoint.
torch.save({"weight": weight.detach(), "bias": bias.detach()},
args.checkpoint)
print(f"saved checkpoint to {args.checkpoint}")
# Tiny same-process reload smoke test. The README also tests a fresh process.
# weights_only restricts loading to tensors and supported simple data types.
state = torch.load(args.checkpoint, weights_only=True)
# Do not record operations for this prediction-only check.
with torch.no_grad():
# One row, one input: shape (1, 1). The result also has one element.
test_c = torch.tensor([[25.0]])
# Predict Fahrenheit using the reloaded weight and bias, not training state.
predicted = (state["weight"] * test_c + state["bias"]).item()
print(f"inference: 25 C -> {predicted:.4f} F (true 77)")
# Optionally save the run settings and measurements for later comparison.
if args.output:
# Keep Python float values as returned by item(), with no extra rounding.
args.output.write_text(json.dumps({
"seed": args.seed,
"epochs": args.epochs,
"lr": args.lr,
"initial_weight": init_w,
"initial_bias": init_b,
"first_loss": first_loss,
"first_weight_grad_backward": first_w_grad,
"first_weight_grad_by_hand": hand_w_grad,
"first_bias_grad_backward": first_b_grad,
"first_bias_grad_by_hand": hand_b_grad,
"learned_weight": final_w,
"learned_bias": final_b,
"final_loss": final_loss,
"history": history,
"inference_25c_f": predicted,
}, indent=2) + "\n")
if __name__ == "__main__":
main()
-
Week 6 companion lab and recorded training results:
https://github.com/dramasamy/from-api-to-gpu/tree/main/week-06-neural-network-basics ↩ -
Toy weight comparison, command and actual Spark output:
https://github.com/dramasamy/from-api-to-gpu/blob/main/week-07-autograd-training-loop/system-report.md ↩ -
PyTorch autograd mechanics:
https://docs.pytorch.org/docs/stable/notes/autograd.html ↩ -
PyTorch Linear layer and its matrix formula:
https://docs.pytorch.org/docs/stable/generated/torch.nn.Linear.html ↩ -
Week 7 companion lab:
https://github.com/dramasamy/from-api-to-gpu/tree/main/week-07-autograd-training-loop ↩ -
PyTorch gradient modes and evaluation mode:
https://docs.pytorch.org/docs/stable/notes/autograd.html#locally-disabling-gradient-computation ↩ -
PyTorch checkpoint loading and security warning:
https://docs.pytorch.org/docs/stable/generated/torch.load.html ↩ -
Week 8 roadmap:
https://github.com/dramasamy/from-api-to-gpu/blob/main/roadmap/week-08.md ↩
Top comments (0)