DEV Community

Dinesh Kumar Ramasamy
Dinesh Kumar Ramasamy

Posted on

From API to GPU, Week 6 (Part 2): Watching a Neural Network Learn

Phase 2 of 8: Enough ML to understand inference. Week 6 of 32, part 2 of 2.

Part 1 built a one-neuron model that converts Celsius to Fahrenheit, made it
predict, and scored how wrong it was with a single number, the loss. With random
starting values the loss was 10352.21, because every prediction was nonsense. This
post takes that one number and turns it into learning: the weight and bias climb
from random noise to the true 1.8 and 32, and I watch each step happen on screen.

Training is one loop repeated many times, and each pass through it does four
things:

  1. Forward pass: run the inputs through the model to get predictions.
  2. Loss: score how wrong those predictions are, as a single number.
  3. Gradient: work out which direction to move each parameter to lower the loss.
  4. Optimizer step: nudge each parameter a small amount in that direction.

Part 1 covered the first two, prediction and loss. This post covers the last two,
the steps that actually change the weight and bias, and then the full loop that
repeats all four until the model has learned. So the two new ideas here are the
gradient (step 3) and the optimizer (step 4).

From error to a direction: the gradient

The gradient is the slope of the loss for each parameter: which direction, and
how steeply, the loss changes if that parameter moves. If nudging the weight up
makes the loss go down, the gradient is what tells the optimizer to move it up.

Backpropagation computes those gradients for every parameter in one pass. In
PyTorch it is the single call loss.backward(). For our one-neuron model there
are only two gradients to compute, one for the weight and one for the bias, but
the same call scales to the billions of parameters in a large model.

The optimizer and the learning rate

The optimizer nudges each parameter using its gradient. The gradient points in
the direction that increases the loss, so the optimizer subtracts it:

new=oldlearning rate×gradient\text{new} = \text{old} - \text{learning rate} \times \text{gradient}

That minus sign is why subtracting a negative gradient makes a parameter grow. I
use plain SGD. The class is named SGD (stochastic gradient descent), but
"stochastic" normally means updating from random subsets of the data. This run
uses all six examples for every update, so it is really full-batch gradient descent
using the SGD class.1

The learning rate is how big each nudge is. Too small learns slowly, so it
needs more updates; too big overshoots and blows up, which I show below.

Epoch and batch

An epoch is one pass over all the training data. A batch is how many
examples the model sees before each update. Here the dataset is six
Celsius/Fahrenheit pairs and I use all of them at once, so one batch equals one
epoch.

Putting the four moves together, the loop is: forward pass, compute loss,
backpropagate to get gradients, optimizer step, repeat. That loop is training.
Using the finished model to predict without changing it is inference, which is
what the earlier text-generation weeks were doing with pretrained models.

The training script

Here is the whole thing. It builds the data, defines the one-layer model, shows the
first predictions and the first gradient before any learning, runs the loop, and
saves a checkpoint (a copy of the trained numbers on disk). A separate predict.py
loads that checkpoint for inference, shown right after this run:

#!/usr/bin/env python3
"""Week 6 - train the smallest useful neural network: Celsius to Fahrenheit.

The true rule is F = C * 1.8 + 32. A single linear layer y = w*x + b should
learn a weight near 1.8 and a bias near 32. The point is not the model. It is to
watch the weight and bias start random, the loss fall, the gradients drive the
change, and the trained parameters get saved to a .pt checkpoint. A separate
predict.py loads that checkpoint to run inference.

The inputs are raw Celsius, so the printed weight is the model's real parameter.
Because the raw inputs are large, the first gradient is large too, which is why
the learning rate has to be small (0.0003). A large learning rate diverges.
"""

from __future__ import annotations

import argparse
import json
from pathlib import Path

import torch
from torch import nn


def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--epochs", type=int, default=20000)
    parser.add_argument("--lr", type=float, default=3e-4)  # learning rate 0.0003
    parser.add_argument("--seed", type=int, default=0)
    parser.add_argument("--checkpoint", type=Path,
                        default=Path("celsius_to_fahrenheit.pt"))
    parser.add_argument("--output", type=Path)
    return parser.parse_args()


def main() -> None:
    args = parse_args()
    torch.manual_seed(args.seed)

    # Six training inputs in Celsius. Each inner [ ] is one example with one
    # value, so the shape is 6 rows by 1 column (six examples, one feature).
    celsius = torch.tensor([[-40.0], [-10.0], [0.0], [20.0],
                            [37.0], [100.0]])
    # The correct answer for each input. This is the only place the true rule
    # appears, and it only builds the labels; the model never sees it.
    fahrenheit = celsius * 1.8 + 32.0

    model = nn.Linear(1, 1)  # one layer: prediction = weight * C + bias
    loss_fn = nn.MSELoss()  # scores how wrong the predictions are
    optimizer = torch.optim.SGD(model.parameters(), lr=args.lr)  # updates w, b

    print(f"device={model.weight.device}")
    init_w = model.weight.item()
    init_b = model.bias.item()
    print(f"initial weight={init_w:.6f} bias={init_b:.6f}")

    # Show where the first loss comes from: predict, then average the
    # squared errors across the six examples.
    with torch.no_grad():
        first_preds = model(celsius)
    print("initial predictions vs targets:")
    for c, p, f in zip(celsius.tolist(), first_preds.tolist(),
                       fahrenheit.tolist()):
        print(f"  C={c[0]:>6.1f}  pred={p[0]:>8.3f}  target={f[0]:>7.1f}")

    # One manual step to expose the first gradient before the optimizer moves.
    optimizer.zero_grad()
    first_loss_t = loss_fn(model(celsius), fahrenheit)
    first_loss_t.backward()
    first_loss = first_loss_t.item()
    w_grad = model.weight.grad.item()
    b_grad = model.bias.grad.item()
    print(f"first loss={first_loss:.4f} "
          f"weight_grad={w_grad:.4f} bias_grad={b_grad:.4f}")
    print(f"first weight update: {init_w:.6f} - {args.lr} * {w_grad:.4f} "
          f"= {init_w - args.lr * w_grad:.6f}")

    history = []
    # log_at picks the handful of epochs to print, so the output stays short.
    log_at = {1, 50, 200, 1000, 5000, 10000, args.epochs}
    for epoch in range(1, args.epochs + 1):
        # One pass of the four steps from the intro (plus a housekeeping reset):
        optimizer.zero_grad()  # reset: clear gradients left from the last step
        loss = loss_fn(model(celsius), fahrenheit)  # steps 1-2: predict, then score
        loss.backward()  # step 3: backprop fills each parameter's gradient
        optimizer.step()  # step 4: nudge weight and bias down the loss
        if epoch in log_at:
            # This block only logs progress; the learning already happened above.
            # Recompute the loss after the step so the printed loss and the
            # printed weight and bias all describe the same post-update model.
            with torch.no_grad():
                post_loss = loss_fn(model(celsius), fahrenheit).item()
            w = model.weight.item()  # current weight, as a plain Python float
            b = model.bias.item()  # current bias
            print(f"epoch {epoch:>5} loss={post_loss:>12.4f} "
                  f"weight={w:.4f} bias={b:.4f}")
            # Keep this snapshot so results.json can chart the run later.
            history.append({"epoch": epoch, "loss": round(post_loss, 4),
                            "weight": round(w, 4), "bias": round(b, 4)})

    # Training is finished. Read the final learned weight and bias, and the
    # loss of the finished model, which is now tiny.
    final_w = model.weight.item()
    final_b = model.bias.item()
    with torch.no_grad():
        final_loss = loss_fn(model(celsius), fahrenheit).item()
    print(f"\nlearned weight={final_w:.4f} bias={final_b:.4f} "
          f"final_loss={final_loss:.4e} (true weight 1.8, bias 32)")

    torch.save(model.state_dict(), args.checkpoint)
    print(f"saved state_dict to {args.checkpoint}")

    if args.output:
        args.output.write_text(json.dumps({
            "seed": args.seed,
            "epochs": args.epochs,
            "lr": args.lr,
            "initial_weight": round(init_w, 6),
            "initial_bias": round(init_b, 6),
            "first_loss": round(first_loss, 4),
            "first_weight_grad": round(w_grad, 4),
            "first_bias_grad": round(b_grad, 4),
            "learned_weight": round(final_w, 4),
            "learned_bias": round(final_b, 4),
            "final_loss": final_loss,
            "history": history,
        }, indent=2) + "\n")


if __name__ == "__main__":
    main()
Enter fullscreen mode Exit fullscreen mode

Three details are worth calling out before running it.

The celsius * 1.8 + 32.0 line is the only place the true rule appears, and it
only builds the answer labels. That gives pairs like (-40, -40), (0, 32), and
(100, 212). The model receives the Celsius inputs and those Fahrenheit answers,
never the formula, so recovering 1.8 and 32 is real learning, not copying.

optimizer.zero_grad() clears the gradient buffers before each new
forward/backward cycle. PyTorch adds new gradients onto whatever is already
stored, so without clearing, the next cycle's gradients would pile on top of the
last one. It does not touch the weight or bias, and it does not perform an update.

The inputs are raw Celsius, so the printed weight is the model's actual parameter,
with no rescaling. The catch is that raw inputs up to 100 make the first gradient
large, which forces a small learning rate. That trade-off is the subject of its own
section below.

The script also uses torch.no_grad() and saves a state_dict; the eval() call
lives in predict.py. All three are explained in a deferred section so they do not
clutter the main story.2

Watching it learn

I ran the exact command below on the Spark. The < ... train_tiny.py part reads
the script file on my control Mac and pipes it in; Python and the training run on
the Spark. The path starts with public/ because I run from the private parent
repo; if you cloned only the public companion repo, drop that prefix and use
week-06-neural-network-basics/train_tiny.py:

ssh spark '~/venvs/w1/bin/python -' \
    < public/week-06-neural-network-basics/train_tiny.py
Enter fullscreen mode Exit fullscreen mode
device=cpu
initial weight=-0.007487 bias=0.536444
initial predictions vs targets:
  C= -40.0  pred=   0.836  target=  -40.0
  C= -10.0  pred=   0.611  target=   14.0
  C=   0.0  pred=   0.536  target=   32.0
  C=  20.0  pred=   0.387  target=   68.0
  C=  37.0  pred=   0.259  target=   98.6
  C= 100.0  pred=  -0.212  target=  212.0
first loss=10352.2070 weight_grad=-9237.2129 bias_grad=-127.3941
first weight update: -0.007487 - 0.0003 * -9237.2129 = 2.763677
epoch     1 loss=   1992.1444 weight=2.7637 bias=0.5747
epoch    50 loss=    806.2087 weight=2.0436 bias=1.3524
epoch   200 loss=    690.7745 weight=2.0255 bias=3.6312
epoch  1000 loss=    302.9773 weight=1.9493 bias=13.2121
epoch  5000 loss=      4.9179 weight=1.8190 bias=29.6063
epoch 10000 loss=      0.0285 weight=1.8014 bias=31.8178
epoch 20000 loss=      0.0000 weight=1.8000 bias=31.9981

learned weight=1.8000 bias=31.9981 final_loss=2.9334e-06 (true weight 1.8, bias 32)
saved state_dict to celsius_to_fahrenheit.pt
Enter fullscreen mode Exit fullscreen mode

This is the whole point of the week.

The start is random and wrong. The initial weight is about -0.007 and the bias
is about 0.54. Those come straight from torch.manual_seed(0). Every one of the
six initial predictions is far from its target: at 100 C the model guesses -0.212
instead of 212. Squaring those six errors and averaging them gives the first loss,
10352.21, the same number Part 1 ended on. That is where the big first number comes
from.

The gradient points the way. Before any learning, loss.backward() reported a
weight gradient of -9237.2 and a bias gradient of -127.4. Both negative means the
loss would drop if both parameters increased, which is correct: they need to climb
toward 1.8 and 32.

One update, traced by hand. The optimizer's rule is
new = old - learning_rate * gradient, applied to every parameter. The 0.0003
in the run's first weight update: line is that learning rate (the script's
--lr, which defaults to 3e-4, another way to write 0.0003). For the weight
that is -0.007487 - 0.0003 * -9237.2129 = 2.763677, and epoch 1 indeed shows
weight=2.7637. The same arithmetic on the bias is
0.536444 - 0.0003 * -127.3941 = 0.574662, and epoch 1 shows bias=0.5747. So the
first step is not magic; it is that one line of arithmetic done twice.

Why the learning rate is so small. That first weight gradient is huge, about
-9237, because the raw Celsius inputs are large (with these inputs, parameters, and
MSE loss). Even multiplied by the tiny learning rate 0.0003, the weight still jumps
from about 0 to 2.76 in one step, overshooting its target of 1.8. A larger learning
rate would overshoot so hard the numbers explode, which I show below.

The loss falls fast, then slows. Reading the recomputed post-update loss, it
drops from 1992 at epoch 1 to 806 by epoch 50, then grinds down to about 0.0000 by
epoch 20000. The weight moves close to its target much sooner than the bias: by
epoch 50 the weight is 2.04 and still drifting, while the bias is only 1.35 and has
a long climb ahead. Why one parameter moves faster than the other is a gradient
question I leave for Week 7.

It lands on the real rule, almost exactly. The final weight is 1.8000 and the
bias is 31.9981, with a final loss of 2.9e-06. It is not perfectly 32: the run
stopped at 20000 epochs, and running to 40000 or 100000 epochs gives the exact same
values, because the remaining SGD updates are smaller than what FP32 can represent,
so the parameters stop changing. The network rediscovered F = C * 1.8 + 32 from
six examples, to the displayed precision, without ever being told the formula.

The checkpoint is saved. The last line writes the model's state_dict to
celsius_to_fahrenheit.pt, a saved copy of the trained weight and bias. That is
the same idea as the model files from Week 3, at the smallest possible scale.
Loading it back and predicting is the job of the next section. The .pt file is
written in the current directory on the Spark and is not committed to the repo.

One timing detail explains the numbers. The first loss, 10352.21, is measured
before any update. The epoch 1 loss, 1992.14, is recomputed after the update, so it
already reflects the improved weight. Each logged row shows the loss and the
parameters for the same post-update state, which is why they line up.

Loading the checkpoint in a standalone script

Training and inference are separate jobs, so I keep them in separate scripts.
Training, above, produced celsius_to_fahrenheit.pt. Inference needs none of the
training machinery, no loss, no gradients, no optimizer. It only rebuilds the
model, loads the saved numbers into it, and calls it. Here is the whole
predict.py:

#!/usr/bin/env python3
"""Week 6 - load the trained checkpoint and run one inference.

train_tiny.py trains the Celsius-to-Fahrenheit model and saves its parameters to
a .pt file. This script is the other half, with no training code at all: rebuild
the same model shape, load the saved parameters into it, and make a prediction.
Run train_tiny.py first so the checkpoint exists.
"""

from __future__ import annotations

import argparse
from pathlib import Path

import torch
from torch import nn


def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--checkpoint", type=Path,
                        default=Path("celsius_to_fahrenheit.pt"))
    parser.add_argument("--celsius", type=float, default=25.0)
    return parser.parse_args()


def main() -> None:
    args = parse_args()

    # Build the SAME architecture the checkpoint was trained with. A state_dict
    # stores only numbers, not the model shape, so the layer must exist first.
    model = nn.Linear(1, 1)
    # Load the saved weight and bias from the .pt file into this layer.
    model.load_state_dict(torch.load(args.checkpoint))
    model.eval()  # inference mode (no visible effect here, but the right habit)

    print(f"loaded {args.checkpoint}")
    print(f"weight={model.weight.item():.4f} bias={model.bias.item():.4f}")

    # One input, wrapped to shape 1 row by 1 column like the training data.
    celsius = torch.tensor([[args.celsius]])
    with torch.no_grad():  # predicting only, so do not track gradients
        fahrenheit = model(celsius).item()
    print(f"inference: {args.celsius:.1f} C -> {fahrenheit:.4f} F")


if __name__ == "__main__":
    main()
Enter fullscreen mode Exit fullscreen mode

Two things make the load work. First, a state_dict stores only the numbers, not
the model's shape, so the script must build the same nn.Linear(1, 1) before
loading, the same point Part 1 made about a state_dict. Second, eval() and
torch.no_grad() put the model in inference mode. Run it right after training, from
the same directory so it finds the .pt:

ssh spark '~/venvs/w1/bin/python -' \
    < public/week-06-neural-network-basics/predict.py
Enter fullscreen mode Exit fullscreen mode
loaded celsius_to_fahrenheit.pt
weight=1.8000 bias=31.9981
inference: 25.0 C -> 76.9985 F
Enter fullscreen mode Exit fullscreen mode

The loaded weight and bias are 1.8000 and 31.9981, the exact values training landed
on, and 25 C predicts 76.9985 F, which rounds to the true 77 (25 * 1.8 + 32 = 77).
Nothing was retrained. predict.py only read the parameters train_tiny.py wrote
and applied the same weight * C + bias from Part 1.

Learning rate: too big blows up, and the run repeats

The small learning rate is not a random choice. Raw Celsius inputs make the
gradients large, and at learning rate 0.1 the optimizer overshoots so far the loss
explodes. Here is a short run at learning rate 0.1 that prints the loss in
scientific notation each epoch:

ssh spark '~/venvs/w1/bin/python - <<PY
import torch
from torch import nn
torch.manual_seed(0)
# Same six Celsius inputs (C) and their Fahrenheit answers (F) as before.
C=torch.tensor([[-40.],[-10.],[0.],[20.],[37.],[100.]]); F=C*1.8+32
# Same model, loss, and optimizer, but a large learning rate of 0.1.
m=nn.Linear(1,1); lf=nn.MSELoss(); opt=torch.optim.SGD(m.parameters(),lr=0.1)
# Run 15 training steps and print the loss (%.4e = scientific notation).
for e in range(1,16):
    opt.zero_grad(); l=lf(m(C),F); l.backward(); opt.step()
    print("epoch %2d loss=%.4e"%(e,l.item()))
PY'
Enter fullscreen mode Exit fullscreen mode
epoch  1 loss=1.0352e+04
epoch  2 loss=1.9073e+09
epoch  3 loss=3.8280e+14
epoch  4 loss=7.6828e+19
epoch  5 loss=1.5419e+25
epoch  6 loss=3.0946e+30
epoch  7 loss=6.2109e+35
epoch  8 loss=inf
epoch  9 loss=inf
epoch 10 loss=inf
epoch 11 loss=inf
epoch 12 loss=inf
epoch 13 loss=inf
epoch 14 loss=inf
epoch 15 loss=inf
Enter fullscreen mode Exit fullscreen mode

The loss grows fast, gaining about five orders of magnitude every step, and by
epoch 8 it is inf (floating-point infinity, the result of a number too large to
represent). Once a value is inf or nan (short for "not a number") the run is
dead. So the small learning rate is doing real work: it keeps the huge first
gradient from throwing the model off a cliff.

The run is also reproducible on this setup. torch.manual_seed(0) fixes the
initial random weight and bias, so on this Spark with PyTorch 2.13.0+cu130 the two
tested seed-0 runs gave the same displayed values, and seeds 1 and 2 reached the
same displayed 1.8 and 32 as well.2

Training versus inference, made concrete

The two scripts are the two modes I have been using loosely. train_tiny.py does
training: every step calls loss.backward() and optimizer.step() to change
the weights. predict.py does inference: it loads finished weights, calls
eval() and torch.no_grad(), and never changes anything. These two calls do
different jobs, and the deferred section spells them out.2 The earlier
text-generation weeks ran models in this inference mode. This week is the first
time I ran the training half of the loop, and splitting the scripts makes the line
between the two obvious.

Results

Here is the training run in one place:

Item Verified value
Task Celsius to Fahrenheit, F = C * 1.8 + 32
Model one linear layer, nn.Linear(1, 1), no activation
Device CPU
Initial weight / bias -0.007487 / 0.536444
First loss 10352.21 (squared Fahrenheit)
First gradients weight -9237.21, bias -127.39
First weight after one step 2.7637
Loss at epoch 5000 (post-update) 4.9179
Learned weight / bias 1.8000 / 31.9981
Final loss 2.9e-06
Inference: 25 C 76.9985 F (rounds to 77)
Optimizer / learning rate SGD / 0.0003
Epochs / batch 20000 / full batch of 6

What surprised me

I expected the loss to fall. I did not expect the weight and the bias to learn at
such different speeds. The weight moves close to its target within the first
handful of updates, while the bias takes thousands more epochs to crawl from 0.5 up
to nearly 32. The optimizer is not solving an equation; it is taking small steps
downhill on the loss, and the two parameters travel at very different speeds. Week
7 digs into why.

The other satisfying part was watching six examples be enough to recover the rule
to the displayed precision. A large language model can have billions of parameters
and messy data, but the loop is the same one I just watched: predict, measure loss,
get gradients, step.

Mistakes and troubleshooting

My first version scaled the inputs down by 100 to keep SGD stable, then unscaled the
weight before printing. That worked, but it created two different meanings of
"weight": the model's internal parameter and the display value. The printed gradient
belonged to the internal parameter, so it did not line up with the printed weight,
which made the single most important step impossible to follow. Switching to raw
inputs with a small learning rate removed the confusion. Now the printed weight is
the real parameter, and the first-update arithmetic checks out exactly.

The lesson that survived is about input size. Large inputs make large gradients,
which force a small learning rate. Scaling inputs to a small range is the usual fix
in real training; I left it out here only to keep every printed number literal.

Production implications

I will not train models this way in production, but this loop is the foundation
under everything that does. Fine-tuning, which this series reaches in Phase 8, is
the same forward, loss, backward, step cycle, just with a pretrained model, far more
parameters, and real data. Knowing what a gradient and a learning rate actually do
makes the later fine-tuning weeks far less mysterious.

It also explains two settings I will keep seeing. The learning rate is a key
training setting: set it too high and training diverges, as the --lr 0.1 run
showed. And a checkpoint here is a saved state_dict, the same idea as the model
files I inspected in Week 3, at the smallest possible scale.

What I will learn next

Week 7 stays on the training loop and goes one level deeper: the computational
graph, requires_grad, and writing the forward, backward(), and optimizer step by
hand instead of leaning on the tidy loop above. That turns this week's
loss.backward() from a black box into something I can trace.3

The training questions, answered

A few terms in the run reward a closer look. They are not needed to follow the main
story, so I pulled them here.

What exactly is a state_dict? For this model it is a mapping from parameter
name to tensor, holding just the learned weight and bias. You can print each
name with its tensor shape and value:

ssh spark '~/venvs/w1/bin/python -c "
import torch; from torch import nn
torch.manual_seed(0)
# state_dict() returns the layer's saved parameters as name -> tensor pairs.
sd=nn.Linear(1,1).state_dict()
# Print each parameter name, its shape, and its values.
for name, tensor in sd.items():
    print(name, tuple(tensor.shape), tensor)"'
Enter fullscreen mode Exit fullscreen mode
weight (1, 1) tensor([[-0.0075]])
bias (1,) tensor([0.5364])
Enter fullscreen mode Exit fullscreen mode

So the two entries really are tensors carrying the parameter values (here the
random start, before training). The state_dict stores the numbers, not the code.
It does not know it came from an nn.Linear(1, 1), which is why the script must
build that same layer again before loading the values into it. Calling every
checkpoint "just a state_dict" would be too broad; this is what a PyTorch
checkpoint holds, not every packaging format from Week 3.

What do eval() and no_grad() each do? They sound like one setting but have
separate jobs. eval() switches layers whose behavior differs between training and
inference into inference mode. A plain linear layer behaves the same either way, so
eval() has no visible effect on this model; I call it because it is the correct
habit for real models that do have such layers. no_grad() tells PyTorch to stop
recording operations for gradient computation. (Not recording is expected to save
memory and time during inference, though I do not measure that here.) Neither one
freezes the weights. The weights stay put simply because there is no backward()
and no optimizer.step() during inference.

How do I know other seeds still converge? Because I ran them. Seed 0 twice, then
seeds 1 and 2, all 20000 epochs:

ssh spark '~/venvs/w1/bin/python - <<PY
import torch
from torch import nn
# Train the model from scratch for one random seed and return the final w, b.
def run(seed):
    torch.manual_seed(seed)  # a different seed = a different random start
    C=torch.tensor([[-40.],[-10.],[0.],[20.],[37.],[100.]]); F=C*1.8+32
    m=nn.Linear(1,1); lf=nn.MSELoss(); opt=torch.optim.SGD(m.parameters(),lr=3e-4)
    for e in range(20000):  # the full training loop, 20000 steps
        opt.zero_grad(); l=lf(m(C),F); l.backward(); opt.step()
    return m.weight.item(),m.bias.item()
# Seed 0 twice (to show it repeats), then seeds 1 and 2 (different starts).
for s in [0,0,1,2]:
    w,b=run(s); print("seed %d -> w=%.4f b=%.4f"%(s,w,b))
PY'
Enter fullscreen mode Exit fullscreen mode
seed 0 -> w=1.8000 b=31.9981
seed 0 -> w=1.8000 b=31.9981
seed 1 -> w=1.8000 b=31.9981
seed 2 -> w=1.8000 b=31.9981
Enter fullscreen mode Exit fullscreen mode

Seed 0 gave the same displayed values across runs, and seeds 1 and 2 reached the
same displayed 1.8 and 32. This task has a single best fit, and on this setup the
tested seeds all reached it.

Run it yourself

The public Week 6 lab has the training and inference scripts, the captured runs,
observations, and troubleshooting notes.4


  1. PyTorch torch.optim.SGD:
    https://docs.pytorch.org/docs/stable/generated/torch.optim.SGD.html 

  2. See the deferred section,
    The training questions, answered

  3. Week 7 roadmap:
    https://github.com/dramasamy/from-api-to-gpu/blob/main/roadmap/week-07.md 

  4. Week 6 companion lab:
    https://github.com/dramasamy/from-api-to-gpu/tree/main/week-06-neural-network-basics 

Top comments (0)