DEV Community: Dinesh Kumar Ramasamy

From API to GPU, Week 7: The Training Loop, Written by Hand

Dinesh Kumar Ramasamy — Tue, 08 Sep 2026 03:51:40 +0000

Phase 2 of 8: Enough ML to understand inference. Week 7 of 32.

The goal: train the same temperature converter while following each step:
make a prediction, measure its error, work out which way to adjust the weight
and bias, then try again. Think of adjusting two knobs and checking whether
the answer gets closer.

Week 6 trained a tiny model to convert Celsius to Fahrenheit using PyTorch's
ready-made layers and optimizer. That is a normal way to write a training loop,
and it remains useful beyond this example. We explained the prediction formula
and worked through the first update by hand. This week traces how
loss.backward() calculates the gradients that tell that update which way to go:

Call in Week 6	Already explained	What Week 7 adds
`nn.Linear(1, 1)`	Weight, bias, multiply and add	Plain trainable tensors
`optimizer.step()`	Subtract learning rate times gradient	Write the update directly
`loss.backward()`	Computes gradients	Trace how they are calculated

I replace the layer and optimizer with plain tensors and a hand-written update
as a learning and debugging exercise. PyTorch still computes the gradients;
I check them against slopes worked out by hand. This goes deeper into Week 6's
loop, rather than replacing the normal PyTorch workflow.

The problem is the same as Week 6 on purpose. Same six examples, same raw Celsius
inputs, same learning rate. We can compare where the two loops end up.¹

Follow one prediction to its error

Start with one prediction, before training anything. I chose w=2.0, b=1.0,
and x=3.0 by hand: the prediction is $2 \times 3 + 1 = 7$ , against a target of 10.
The squared loss is $(7 - 10)^{2} = 9$ . With w=3.0, the prediction would already be
10, with zero loss and zero gradients. I use 2.0 so there is an error to explain.
The lab checks both choices; training later uses random starts.²

To work out how each parameter caused that error, PyTorch keeps a record of the
calculation, like a trail of intermediate results in a debugger. That record is
the computational graph. Setting requires_grad=True on the weight and bias
asks PyTorch to record operations it can differentiate, so it can calculate their
gradients. This automatic gradient calculation is called autograd.

First, look at the record. Each result of a tracked operation carries a
grad_fn, an object naming the operation that produced it:

This code runs on Spark:

ssh spark '~/venvs/w1/bin/python -' <<'PY'
import torch
# Create a one-value weight, shape (1,), with gradient tracking enabled.
w = torch.tensor([2.0], requires_grad=True)
# Make a separate one-value bias with tracking enabled too.
b = torch.tensor([1.0], requires_grad=True)
# Omitted requires_grad defaults to False: this one-value input is fixed data.
x = torch.tensor([3.0])
# Compute the model prediction from input, weight, and bias.
pred = w * x + b
# Choose 10 as this tiny example target, unrelated to the Fahrenheit task.
# Score the squared prediction error against the target: (7 - 10)**2 = 9.
loss = (pred - 10.0)**2
# Show the prediction and loss with their recorded operations.
print("pred:", pred)
print("loss:", loss)
# Compare gradient tracking for the fixed input and trainable weight.
print("x.requires_grad:", x.requires_grad, " w.requires_grad:", w.requires_grad)
PY

pred: tensor([7.], grad_fn=<AddBackward0>)
loss: tensor([9.], grad_fn=<PowBackward0>)
x.requires_grad: False  w.requires_grad: True

Two things to read here. pred was built with a multiply and an add, so its
grad_fn is AddBackward0 (the last step). loss was built with a power, so its
grad_fn is PowBackward0. Each grad_fn names only the tensor's immediate
producing step, but that object links back to the steps that fed it, so together
they form the full chain PyTorch will walk in reverse:³

w, x \to w x \to w x + b \to w x + b - 10 \to (\cdot)^{2} \to L

And x.requires_grad is False, because I never asked it to be tracked. Inputs
are fixed data, so they do not need gradients; only the parameters I want to
train do.

For now, follow the weight and bias: backward() will put their gradients in
w.grad and b.grad. The optional gradient-storage check
explains which tensors receive stored gradients by default.

Work out which way to adjust the parameters

A quick reminder from Week 6, part 2: a gradient tells us the slope of the
loss for each parameter. If increasing a parameter slightly lowers the loss,
its gradient is negative; if it raises the loss, its gradient is positive.
The size tells us how steeply the loss changes at the current values.

Backpropagation is how we calculate those gradients: start at the loss and
follow the recorded calculation backward to the weight and bias. In PyTorch,
loss.backward() does this work. It calculates the slopes; the separate update
step uses them to adjust the parameters.

Take the tiny example above: the loss is $(w x + b - 10)^{2}$ with $x = 3$ , $w = 2$ ,
$b = 1$ , so the prediction is 7. Here is how fast the loss changes if I nudge each
parameter a tiny amount from the current point:

Nudging the prediction up changes the squared loss at a rate of $2 (pred - 10) = 2 (7 - 10) = - 6$ loss units per prediction unit.
Nudging the weight up changes the prediction at a rate of $x = 3$ .
Multiply those two rates: $- 6 \times 3 = - 18$ . That is the weight's slope.
Nudging the bias up changes the prediction at a rate of 1, so its slope is $- 6 \times 1 = - 6$ .

Multiplying the connected rates along the chain is the chain rule. Written
in calculus notation, those two slopes are:

\frac{\partial L}{\partial w} = 2 (w x + b - 10) x, \frac{\partial L}{\partial b} = 2 (w x + b - 10)

The symbol $\partial L / \partial w$ just means "how much the loss $L$ changes
when $w$ changes." I ran backward() and compared it to those hand values:

ssh spark '~/venvs/w1/bin/python -' <<'PY'
import torch
# Create one-value parameters, shape (1,), with gradient tracking enabled.
w = torch.tensor([2.0], requires_grad=True)
b = torch.tensor([1.0], requires_grad=True)  # Bias also needs a gradient.
x = torch.tensor([3.0])  # Fixed input; requires_grad defaults to False.
# Compute the model prediction from input, weight, and bias.
pred = w * x + b
# 10 is the deliberately chosen toy target, not a Fahrenheit label.
loss = (pred - 10.0)**2  # Score the squared prediction error against the target.
loss.backward()  # Follow the graph to fill w.grad and b.grad.
print("w.grad:", w.grad, " b.grad:", b.grad)
# Apply the chain-rule formulas using the input and prediction values.
print("hand dL/dw:", 2*(pred.item()-10)*x.item(),
      " hand dL/db:", 2*(pred.item()-10))
PY

w.grad: tensor([-18.])  b.grad: tensor([-6.])
hand dL/dw: -18.0  hand dL/db: -6.0

They match. backward() filled w.grad and b.grad with the same numbers I got
from the derivative by hand. That is autograd's core job, just for graphs far too
large to differentiate by hand.

Next we use the same idea on the six temperature examples. Their loss is the
average squared error, so the gradient averages their contributions too. The
training transcript compares autograd with that hand calculation; the optional
six-example check shows every
contribution.

The training loop, without the helpers

Now the real thing. This script trains Celsius to Fahrenheit with no nn.Linear
and no optimizer. The weight and bias are plain tensors with
requires_grad=True. The update param -= lr * param.grad is written out. It
also prints the weight and bias before and after training, and checks the first
gradient against the hand-computed derivative.

The inputs and targets have shape (6, 1): six examples, one value per example.
Each parameter has shape (1,). Broadcasting reuses that one weight and bias
for every row, giving predictions of shape (6, 1). The comments mark those
shapes in the code. Element-wise weight * celsius + bias works for this
one-input, one-output model; a general nn.Linear uses matrix multiplication
and then adds its bias.⁴

Here is the core loop, an exact excerpt, not a standalone program, from the
lab's training script.⁵
The complete script is at the end;
the next section runs that full version. Read this excerpt as predict and score,
calculate gradients, update, then clear. The two safeguards, no_grad() and
zero_(), get their own small checks after the training result.

    # Each epoch uses all six examples for one parameter update.
    for epoch in range(1, args.epochs + 1):
        # Compute the model prediction from input, weight, and bias.
        # The forward pass produces six predictions, shape (6, 1).
        pred = weight * celsius + bias
        # Score the squared prediction error against each target, then average.
        loss = ((pred - fahrenheit) ** 2).mean()
        # Backward pass fills weight.grad and bias.grad.
        loss.backward()
        # Move each parameter opposite its gradient, scaled by the learning
        # rate. This is the mathematical plain-SGD rule, using the full batch.
        # no_grad keeps the in-place update out of the gradient graph.
        with torch.no_grad():
            # Adjust the weight using its gradient and the learning rate.
            weight -= args.lr * weight.grad
            # Apply the same update rule to the bias, using its own gradient.
            bias -= args.lr * bias.grad
        # Clear the gradients, or the next backward() would add onto these.
        weight.grad.zero_()
        # Both accumulators must be cleared, not just the weight's.
        bias.grad.zero_()

The loop is four steps: predict and score, calculate gradients, update, and clear
the gradients. The full program also sets up the data and parameters,
checks the first gradient, reports progress, and saves and reloads the result.

Watching the hand loop run

The output below is the recorded verification run from the lab. The script's
comments have since been clarified; its calculations and reporting are unchanged.

From the parent repository root on my control Mac, I run the code on Spark in a
fresh directory to preserve earlier checkpoints. Keep run_dir for the later
load command. For a public-only clone, drop public/ from the local source path.
I expect loss to fall and the parameters to approach 1.8 and 32:

run_dir=$(ssh spark 'mktemp -d /tmp/week07-reviewed.XXXXXXXX')
printf '%s\n' "$run_dir"
ssh spark "cd '$run_dir' && ~/venvs/w1/bin/python - \
    --epochs 20000 --lr 0.0003 --seed 0 \
    --checkpoint manual_celsius_to_fahrenheit.pt --output results.json" \
    < public/week-07-autograd-training-loop/train_manual.py

/tmp/week07-reviewed.RXvW4Ygo
device=cpu
before training: weight=1.540996 bias=-0.293429
loss.grad_fn type=MeanBackward0
first grad backward(): weight=-2314.6404 bias=-73.8247
first grad by hand   : weight=-2314.6404 bias=-73.8247
epoch     1 loss=    965.8369 weight=2.2354 bias=-0.2713
epoch    50 loss=    849.9656 weight=2.0501 bias=0.5317
epoch  1000 loss=    319.4210 weight=1.9533 bias=12.7090
epoch  5000 loss=      5.1848 weight=1.8195 bias=29.5422
epoch 10000 loss=      0.0300 weight=1.8015 bias=31.8129
epoch 20000 loss=      0.0000 weight=1.8000 bias=31.9981
after training:  weight=1.8000 bias=31.9981 (true weight 1.8, bias 32)
final loss (post-update, unrounded)=2.9371251457632752e-06
saved checkpoint to manual_celsius_to_fahrenheit.pt
inference: 25 C -> 76.9985 F (true 77)

Your temporary directory name will differ. The grad_fn type is MeanBackward0,
the final mean operation in the loss. The last table row prints 0.0000 because
it rounds to four decimal places; the raw final loss is about
$0.0000029371$ , not zero. JSON now preserves raw values returned by PyTorch.
The reported epoch losses are measured after each selected update, unlike
the loss used by backward(), which was calculated before that update.

The gradient check passes on the six-example loss. backward() reported a
first weight gradient of -2314.6404, and the hand-computed derivative gave the
same -2314.6404. Same for the bias, -73.8247 from both. So on the real training
loss, not just the toy one, the analytical derivative and autograd agree to
every displayed digit. These computations use floating-point arithmetic.

Before and after, spelled out. The weight started at 1.540996 and ended at
1.8000. The bias started at -0.293429 and ended at 31.9981. That is the roadmap
deliverable: each parameter shown before and after training.

Same answer as Week 6. Week 6's nn.Linear plus optim.SGD ended at weight
1.8000, bias 31.9981, and predicted 76.9985 F for 25 C. This hand-written loop
ended at the same weight 1.8000, bias 31.9981, and predicted the same 76.9985 F,
to every displayed digit. The starting weights differ because a seed makes a
random draw reproducible, but nn.Linear and torch.randn use different rules to
turn that draw into numbers, so seed 0 does not give the same start. Both runs
still end near 1.8 and 32 because they minimize the same loss on the same
straight-line data. Testing numerical agreement of the updates would require
identical starting parameters and comparing intermediate results with a stated
tolerance. Equivalent mathematical rules can round differently; this experiment
shows matching displayed endpoints, not bit-for-bit identical updates.

Two lines that look optional but are not

The loop has two calls that a newcomer might drop. Both matter, and both are easy
to show.

Why the update is inside torch.no_grad(). The line
weight -= lr * weight.grad is itself a tensor operation on a tracked tensor. The
graph should hold only the operations that produced the current loss; the update
just changes the starting values for the next loop, so it must not be recorded.
An in-place change modifies the existing tensor instead of creating a new
one. Without torch.no_grad(), PyTorch refuses this in-place change and errors; with
it, the update succeeds and the weight is still a trainable parameter afterward:

The error calls w a leaf: here that means a trainable tensor created
directly, rather than the result of another tracked calculation.

ssh spark '~/venvs/w1/bin/python -' <<'PY'
import torch
w = torch.tensor([2.0], requires_grad=True)  # One tracked value, shape (1,).
loss = (w * 3.0 - 1.0)**2  # Score squared prediction error for input 3, target 1.
loss.backward()  # Compute the gradient and store it in w.grad.
try:  # Demonstrate the failure when updating a tracked leaf without no_grad.
    w -= 0.1 * w.grad  # Attempt an in-place gradient update with learning rate 0.1.
except RuntimeError as e:
    print("without no_grad:")  # Label the failed attempt.
    print(str(e))  # Show PyTorch's reason for rejecting the update.
with torch.no_grad():  # Keep the parameter update out of the gradient graph.
    w -= 0.1 * w.grad  # The same update is now allowed.
print("with no_grad, new w:", w.item())
# Verify the weight still requires gradients for future training steps.
print("w.requires_grad still:", w.requires_grad)
PY

without no_grad:
a leaf Variable that requires grad is being used in an in-place operation.
with no_grad, new w: -1.0
w.requires_grad still: True

Here no_grad() keeps the update out of the graph. It does the same job during
inference: suppressing gradient recording for the operations inside its block.
It leaves the parameter's requires_grad=True flag intact.

Why gradients must be cleared. backward() adds new gradients onto whatever
is already sitting in .grad. It does not overwrite. Run a fresh forward and
backward() twice without clearing and the gradient doubles:

ssh spark '~/venvs/w1/bin/python -' <<'PY'
import torch
w = torch.tensor([2.0], requires_grad=True)  # One tracked value, shape (1,).
# Run two backward passes without clearing gradients or updating the weight.
for i in range(1, 3):
    loss = w * 3.0  # Use a simple loss whose weight gradient is always 3.
    loss.backward()  # Add the new gradient to whatever w.grad already holds.
    print("backward", i, "-> w.grad:", w.grad.item())
w.grad.zero_()  # Clear the accumulated gradient before the next backward pass.
loss = w * 3.0  # Make a fresh forward calculation with the unchanged weight.
loss.backward()  # Accumulate onto zero this time.
print("after zero_ and fresh backward -> w.grad:", w.grad.item())
PY

backward 1 -> w.grad: 3.0
backward 2 -> w.grad: 6.0
after zero_ and fresh backward -> w.grad: 3.0

The true gradient is 3.0, but the second backward() without clearing reads 6.0.
That is why the loop calls weight.grad.zero_() and bias.grad.zero_() every
step; zero_() fills the existing gradient tensor with zeros. In Week 6 the
optimizer's zero_grad() did this for me; here I do it directly.

Train mode versus evaluation mode

Our tensor-only model has no .train() or .eval() method; those belong to
nn.Module, PyTorch's base class for model layers. Week 6's plain linear layer
behaved the same in either mode. But
some layers do behave differently, and the clearest example is dropout, a layer
that randomly zeros some values during training so the model does not lean too
hard on any single value. During evaluation it must stop doing that. You switch
with .train() and .eval():

ssh spark '~/venvs/w1/bin/python -' <<'PY'
import torch
from torch import nn
torch.manual_seed(0)  # Repeat this environment's random dropout choices.
# Give each input value a 50% chance of being zeroed during training.
drop = nn.Dropout(p=0.5)
x = torch.ones(8)  # Eight ones, shape (8,); requires_grad defaults to False.
drop.train()  # Enable random dropping and scaling of surviving values.
print("train mode :", drop(x))  # Show which values survived and their scaling.
drop.eval()  # Switch that same layer to evaluation behavior.
print("eval  mode :", drop(x))  # Run it again on the unchanged input.
PY

train mode : tensor([0., 0., 2., 0., 0., 0., 2., 2.])
eval  mode : tensor([1., 1., 1., 1., 1., 1., 1., 1.])

In train mode dropout zeroed several values (five of the eight here) and scaled
the survivors up. Each value independently has a 50% chance of being dropped
because p=0.5, and survivors are multiplied by $1/ (1 - p) = 2$ so the average
output stays about the same across many dropout samples. In eval mode it passed
all eight values through unchanged, doing neither the dropping nor the scaling.
That is why real inference code calls .eval() first: you do not want random
dropout altering predictions. My Week 6 eval() call was setting exactly this
mode, it just had nothing to change for a plain linear layer.

Mode and gradient recording are separate switches

Use evaluation mode when you want a module's inference behavior. Use no_grad()
when you do not need a gradient graph, such as ordinary prediction or our manual
parameter update. For inference with a module, normally use both.⁶

Control	Changes	Leaves alone
`.train()` / `.eval()`	Mode-sensitive layer behavior	Gradient recording
`torch.no_grad()`	Recording inside the block	Module mode
`param.grad.zero_()`	Stored gradient values	Parameters and module mode

None of these calls updates the parameters. The optional
combined check tests the switches together
and follows what happens to the graph and stored gradients between steps.

The checkpoint is just the learned tensors

The training program saves a dictionary: the keys weight and bias name its
two learned tensors. weight.detach() shares the weight's storage but has no
gradient history and does not require gradients; the original weight remains
trainable. This is a minimal inference parameter checkpoint, not a full training
restart: it contains no model code, optimizer state, epoch, or module mode.

The printed directory and checkpoint filename locate the actual file:
/tmp/week07-reviewed.RXvW4Ygo/manual_celsius_to_fahrenheit.pt in my run.
run_dir still holds that directory in my control shell. This next SSH command
starts a fresh Python process there. It loads only the checkpoint, selects
the two named tensors, creates a new input, and applies the prediction formula.
I expect the same prediction as the training script's reload check:

ssh spark "cd '$run_dir' && ~/venvs/w1/bin/python -" <<'PY'
import torch

# Load the saved parameter dictionary, restricting loaded types.
# Only load a checkpoint you trust.
state = torch.load("manual_celsius_to_fahrenheit.pt", weights_only=True)
# Verify the checkpoint contains the expected weight and bias entries.
print(f"keys={list(state.keys())}")
# Inspect each saved parameter's shape and gradient flag.
for name, value in state.items():
    print(f"{name}: shape={tuple(value.shape)} requires_grad={value.requires_grad}")
with torch.no_grad():  # Prediction needs no gradient graph.
    # One new Celsius input, shape (1, 1).
    test_c = torch.tensor([[25.0]])
    # Compute the model prediction from input, weight, and bias.
    # Use the checkpoint's learned tensors rather than training-process variables.
    prediction = state["weight"] * test_c + state["bias"]
# Verify the prediction's shape and that it has no gradient tracking.
print(f"prediction: shape={tuple(prediction.shape)} "
      f"requires_grad={prediction.requires_grad}")
# Compare the reloaded model's prediction with the known answer.
print(f"inference: 25 C -> {prediction.item()} F (true 77)")
PY

keys=['weight', 'bias']
weight: shape=(1,) requires_grad=False
bias: shape=(1,) requires_grad=False
prediction: shape=(1, 1) requires_grad=False
inference: 25 C -> 76.99851989746094 F (true 77)

That is 76.9985 F at the training transcript's precision, from disk rather than
variables left over from training. The loader supplies the formula; the file
supplies its learned numbers. weights_only=True restricts loading to tensors
and supported simple data types. Only load checkpoints you trust.⁷
These detached tensors already have tracking disabled; the no_grad() block
also makes the inference intent explicit. No .eval() call is needed because
there is no module here. Temporary directories may be cleaned up by the system;
the lab shows how to retrieve results without replacing a captured baseline.

Results

Here is Week 7 in one place:

Item	Verified value
Task	Celsius to Fahrenheit, F = C * 1.8 + 32
Model	two plain tensors, no `nn.Linear`, no optimizer
Device	CPU
Before training (weight / bias)	1.540996 / -0.293429
First gradient, backward()	weight -2314.6404, bias -73.8247
First gradient, by hand	weight -2314.6404, bias -73.8247
After training (weight / bias)	1.8000 / 31.9981
Inference: 25 C	76.9985 F (rounds to 77)
Learning rate / epochs	0.0003 / 20000
Final loss, unrounded rerun	2.9371251457632752e-06
Week 6 comparison	Same displayed endpoint, not a stepwise test

What surprised me

I knew backward() computed gradients, but I did not expect the hand-computed
derivative to match it to every printed digit on the six-example loss. Seeing
-2314.6404 twice made autograd feel concrete instead of mysterious. It is not
estimating slopes by trying small changes; it computes the exact chain-rule
derivative, in floating-point numbers.

The other surprise was how similar the result is with and without nn.Linear and
optim.SGD. Removing both changed the starting weights but not the endpoint. The
loop underneath is four steps.

Mistakes and troubleshooting

The first trap is forgetting torch.no_grad() around the update. Without it,
PyTorch tries to track the subtraction and you get errors about modifying a tensor
that requires grad in place. The fix is to wrap the update, which is exactly what
an optimizer does internally.

The second trap is forgetting to clear gradients, which silently mixes gradients
from different steps. They can reinforce or cancel each other. The
accumulation demo above is the quickest way to see it.

The third is expecting celsius.grad to exist. Our inputs have
requires_grad=False; the weight and bias are the values we ask PyTorch to
differentiate. The leaf-tensor check shows their
gradient fields side by side.

Production implications

Production training usually reaches for an optimizer like optim.SGD or a
higher-level trainer instead of updating each parameter by hand, but they run this
same cycle underneath with more parameters and smarter update rules. When a
training run diverges, or a gradient goes to zero, or a fine-tune does nothing,
the thing to picture is these four steps: forward, backward, update, clear.
Knowing that backward() is plain calculus and that gradients accumulate unless
cleared is enough to debug a surprising number of training problems.

What I will learn next

Week 8 leaves the training loop and starts on how models turn text into numbers:
token IDs, embedding vectors, the embedding table, and cosine similarity. That is
the first step toward understanding how a language model represents meaning, and
it connects back to the token IDs I met in Week 2.⁸

Run it yourself

The public Week 7 lab has the manual training script, the captured run,
observations, and troubleshooting notes.⁵

I reused the Spark environment from Week 1. Before the experiments, I checked
the machine architecture and PyTorch version:

ssh spark 'uname -m; ~/venvs/w1/bin/python -c "import torch; print(torch.__version__)"'

aarch64
2.13.0+cu130

aarch64 is ARM64. This is a CUDA-enabled PyTorch build, but the tiny training
program uses the CPU. Seeded outputs are tied to this environment, not a
promise that every PyTorch version or machine produces identical values.

Optional: a closer look

The main loop is complete. These checks look more closely at where gradients
are stored, how the six examples contribute, and how mode and recording interact.
Read whichever answers a question you still have, or go straight to the full script.

Where gradients are stored

A trainable leaf tensor has requires_grad=True and was not produced by a
recorded operation. Our directly created weight and bias are leaves; pred and
loss are recorded results. Tensors with requires_grad=False are also called
leaves by convention, even when calculated from other untracked tensors.
By default, backward() accumulates gradients in .grad for leaves that require
gradients and contribute to the loss. Here is that distinction in this model:

ssh spark '~/venvs/w1/bin/python -' <<'PY'
import torch
torch.manual_seed(0)  # Repeat the random start in this PyTorch environment.
# Six Celsius inputs, one temperature per row: shape (6, 1).
C = torch.tensor([[-40.],[-10.],[0.],[20.],[37.],[100.]])
F = C * 1.8 + 32  # Build the six target Fahrenheit answers, shape (6, 1).
# Draw a one-value weight, shape (1,), with gradient tracking enabled.
w = torch.randn(1, requires_grad=True)
b = torch.randn(1, requires_grad=True)  # A separate random bias.
# Compute the model prediction from input, weight, and bias.
pred = w * C + b  # Reuse each parameter across all six rows, shape (6, 1).
# Score the squared prediction error against each target, then average.
loss = ((pred - F) ** 2).mean()
loss.backward()  # Compute gradients and accumulate them in leaf .grad fields.
# Compare which leaves require gradients and actually received them.
for name, t in [("weight ", w), ("bias   ", b), ("celsius", C)]:
    print(name, "is_leaf", t.is_leaf, "req_grad", t.requires_grad, "grad", t.grad)
# Contrast those leaves with the prediction's recorded producing operation.
print("pred   ", "is_leaf", pred.is_leaf, "grad_fn", type(pred.grad_fn).__name__)
PY

weight  is_leaf True req_grad True grad tensor([-2314.6404])
bias    is_leaf True req_grad True grad tensor([-73.8247])
celsius is_leaf True req_grad False grad None
pred    is_leaf False grad_fn AddBackward0

The weight and bias are leaves with requires_grad=True, so they get gradients.
celsius is a leaf but has requires_grad=False, so its gradient is None.
pred is not a leaf at all; it carries a grad_fn instead. Those weight and bias
gradients, -2314.6404 and -73.8247, are the same ones the training run reports
above.

How six examples contribute one gradient

The six-example loss works the same way, one more step. The loss averages the
squared error over six examples, so each example contributes its own
$2 (pred - target) x$ term and the weight gradient is their mean.
Here are the six contributions and their average, at the random starting point:

ssh spark '~/venvs/w1/bin/python -' <<'PY'
import torch
# Repeat this environment's random starting parameters.
torch.manual_seed(0)
# Six Celsius inputs, one temperature per row: shape (6, 1).
C = torch.tensor([[-40.],[-10.],[0.],[20.],[37.],[100.]])
F = C * 1.8 + 32  # Build the six target Fahrenheit answers, shape (6, 1).
w = torch.randn(1, requires_grad=True)  # One random weight, tracking enabled.
b = torch.randn(1, requires_grad=True)  # One random bias, tracking enabled.
# Compute the model prediction from input, weight, and bias.
pred = w * C + b  # The shape-(1,) parameters serve all six rows.
# Calculate each example's weight-gradient contribution: 2 * error * input.
contrib = (2.0 * (pred - F) * C)
print("per-example weight-grad contributions:")
# detach() drops tracking for this view; flatten() lays six rows out as (6,).
print(contrib.detach().flatten())
# Average the contributions to compare with the full-batch weight gradient.
print("mean =", contrib.mean().item())
PY

per-example weight-grad contributions:
tensor([  1754.6619,    594.0678,     -0.0000,  -1498.9403,  -3098.8667,
        -11638.7637])
mean = -2314.640380859375

The mean of the six contributions is -2314.6404, exactly the weight gradient the
training run reports. So the full gradient is not a mystery number; it is the
chain-rule slope averaged over the six examples. The bias works the same way with
$x$ replaced by 1, so each contribution is $2 (pred - target)$ , and
their mean is -73.8247, the reported bias gradient. (The .detach() in the
command just drops gradient tracking so the numbers print cleanly; there is more
on detach() in the checkpoint section above.)

Mode switches and fresh graphs

This checks .eval(), no_grad(), and gradient clearing from the main table.
None of these calls performs a parameter update. Each forward pass builds a new
graph when recording is enabled. backward() uses it and normally releases the
saved intermediate values needed for another backward pass. The leaf .grad
buffers remain. Clearing those buffers changes their numbers, not the parameters
or the graph. Think of a fresh calculation with a separate accumulator for its
answer.

Here is a direct check. I set dropout to p=1.0 so it drops every value in train
mode, making the comparison deterministic. I expect eval mode to preserve the
input's gradient flag, and no_grad() to leave the module in train mode. Then
two fresh forward passes should each produce gradient 36, with zero between them:

ssh spark '~/venvs/w1/bin/python -' <<'PY'
import torch
from torch import nn

# Drop every value during training to make the mode comparison deterministic.
drop = nn.Dropout(p=1.0)
x = torch.ones(3, requires_grad=True)  # Three ones, shape (3,), tracking on.
drop.eval()  # Switch the layer to evaluation behavior.
y = drop(x)  # Evaluation passes the input through unchanged.
# Check module mode and gradient tracking separately.
print("eval:", "training=", drop.training, "requires_grad=", y.requires_grad)
print("values:", y)  # Verify evaluation preserved the input values.
drop.train()  # Switch the same layer back to training behavior.
with torch.no_grad():  # Disable gradient recording without changing module mode.
    y = drop(x)  # Training dropout still zeros the same input.
# Check that no_grad left training mode enabled but suppressed tracking.
print("train + no_grad:", "training=", drop.training,
      "requires_grad=", y.requires_grad)
print("values:", y)  # Show whether training behavior still zeroed the values.

w = torch.tensor([2.0], requires_grad=True)  # One tracked value, shape (1,).
# Repeat with the same weight, clearing gradients between fresh graphs.
for step in range(1, 3):
    loss = (w * 3.0) ** 2  # Build a fresh graph for the same squared loss.
    print("forward", step, "grad_fn:", type(loss.grad_fn).__name__)
    loss.backward()  # Compute and accumulate the new weight gradient.
    print("after backward:", w.grad.item())  # Report the newly computed gradient.
    w.grad.zero_()  # Clear the gradient in place, without changing w.
    print("after zero_:", w.grad.item(), "w:", w.item())  # Show both values.
PY

eval: training= False requires_grad= True
values: tensor([1., 1., 1.], requires_grad=True)
train + no_grad: training= True requires_grad= False
values: tensor([0., 0., 0.])
forward 1 grad_fn: PowBackward0
after backward: 36.0
after zero_: 0.0 w: 2.0
forward 2 grad_fn: PowBackward0
after backward: 36.0
after zero_: 0.0 w: 2.0

Eval left requires_grad=True; no_grad() left training=True and dropout
still zeroed the inputs. In the second check, $L = (3 w)^{2}$ , so the weight gradient
at $w = 2$ is $2 (3 w) \times 3 = 2 \times 6 \times 3 = 36$ . Both passes report 36
because I clear the accumulator between them. The weight stays 2 throughout:
there is no update in this check. The equal grad_fn type names describe the
operation, not reuse of the same graph object.

Reference: the complete training script

This is the full lab program used by the training command above. The core loop
is unchanged; here you can also follow setup, the gradient check, reporting,
checkpoint saving, and reloading.

#!/usr/bin/env python3
"""Week 7 - the training loop written by hand for Celsius to Fahrenheit.

Week 6 used nn.Linear and torch.optim.SGD. This version removes both. The weight
and bias are plain tensors with requires_grad=True, and the parameter update is
one line I write myself: param -= lr * param.grad. It uses the same mathematical
forward rule and plain-SGD update as Week 6, from a different random start.
This is not a claim of bitwise-identical floating-point results.

The problem is identical to Week 6: learn F = C * 1.8 + 32 from six examples,
with raw Celsius inputs and a small learning rate.
"""

from __future__ import annotations

import argparse
import json
from pathlib import Path

import torch


def parse_args() -> argparse.Namespace:
    # Configure the training run and where to save its checkpoint and results.
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--epochs", type=int, default=20000)
    parser.add_argument("--lr", type=float, default=3e-4)
    parser.add_argument("--seed", type=int, default=0)
    parser.add_argument("--checkpoint", type=Path,
                        default=Path("manual_celsius_to_fahrenheit.pt"))
    parser.add_argument("--output", type=Path)
    return parser.parse_args()


def main() -> None:
    args = parse_args()
    # Make the random starting parameters repeatable in this environment.
    torch.manual_seed(args.seed)

    # Six Celsius inputs, one temperature per row: shape (6, 1).
    celsius = torch.tensor([[-40.0], [-10.0], [0.0], [20.0],
                            [37.0], [100.0]])
    # Labels are the six correct answers, also shape (6, 1). Only label creation
    # uses the known formula; training receives inputs and labels, not the rule.
    # Inputs and labels are fixed data; neither needs gradients.
    fahrenheit = celsius * 1.8 + 32.0

    # Each parameter has shape (1,): one randomly drawn number. requires_grad
    # asks PyTorch to track operations so backward() can compute derivatives.
    weight = torch.randn(1, requires_grad=True)
    # Draw a separate one-value bias with tracking enabled too.
    bias = torch.randn(1, requires_grad=True)
    print(f"device={weight.device}")
    # item() extracts a one-element tensor as a Python number for reporting.
    init_w = weight.item()
    init_b = bias.item()
    print(f"before training: weight={init_w:.6f} bias={init_b:.6f}")

    # One manual forward and backward to show two things:
    #   1. loss carries a grad_fn, so it remembers how it was built.
    #   2. backward() and a hand-written derivative can be compared numerically.
    # Compute the model prediction from input, weight, and bias.
    # Broadcasting reuses each shape-(1,) parameter across all six input rows.
    pred = weight * celsius + bias
    # Score the squared prediction error against each target, then average
    # all six errors into one scalar loss, shape ().
    loss = ((pred - fahrenheit) ** 2).mean()
    print(f"loss.grad_fn type={type(loss.grad_fn).__name__}")
    # Follow the recorded graph to fill the two parameters' shape-(1,) .grad.
    loss.backward()
    # Store the starting loss as a Python number for JSON, before any updates.
    first_loss = loss.item()
    # Check the MSE derivatives without building another gradient graph:
    # dloss/dweight = mean(2 * error * input), dloss/dbias = mean(2 * error).
    with torch.no_grad():
        # Average the six weight contributions, then extract the scalar number.
        hand_w_grad = (2.0 * (pred - fahrenheit) * celsius).mean().item()
        # Bias contributions omit the input multiplier; average those too.
        hand_b_grad = (2.0 * (pred - fahrenheit)).mean().item()
    print(f"first grad backward(): weight={weight.grad.item():.4f} "
          f"bias={bias.grad.item():.4f}")
    print(f"first grad by hand   : weight={hand_w_grad:.4f} "
          f"bias={hand_b_grad:.4f}")
    # Save both gradients before clearing them; .grad holds a tensor.
    first_w_grad = weight.grad.item()
    first_b_grad = bias.grad.item()
    # Discard this demonstration's gradients before the actual training loop.
    weight.grad.zero_()
    bias.grad.zero_()

    # Keep selected post-update measurements, not every training step.
    history = []
    # Report progress at these epochs, including the final requested epoch.
    log_at = {1, 50, 1000, 5000, 10000, args.epochs}
    # Each epoch uses all six examples for one parameter update.
    for epoch in range(1, args.epochs + 1):
        # Compute the model prediction from input, weight, and bias.
        # The forward pass produces six predictions, shape (6, 1).
        pred = weight * celsius + bias
        # Score the squared prediction error against each target, then average.
        loss = ((pred - fahrenheit) ** 2).mean()
        # Backward pass fills weight.grad and bias.grad.
        loss.backward()
        # Move each parameter opposite its gradient, scaled by the learning
        # rate. This is the mathematical plain-SGD rule, using the full batch.
        # no_grad keeps the in-place update out of the gradient graph.
        with torch.no_grad():
            # Adjust the weight using its gradient and the learning rate.
            weight -= args.lr * weight.grad
            # Apply the same update rule to the bias, using its own gradient.
            bias -= args.lr * bias.grad
        # Clear the gradients, or the next backward() would add onto these.
        weight.grad.zero_()
        # Both accumulators must be cleared, not just the weight's.
        bias.grad.zero_()
        # Capture progress only at the selected reporting epochs.
        if epoch in log_at:
            # Measure again after the update, using the new weight and bias.
            with torch.no_grad():
                post_loss = ((weight * celsius + bias - fahrenheit) ** 2
                             ).mean().item()
            print(f"epoch {epoch:>5} loss={post_loss:>12.4f} "
                  f"weight={weight.item():.4f} bias={bias.item():.4f}")
            # Format stdout for reading, but retain unrounded floats in JSON.
            history.append({"epoch": epoch, "loss": post_loss,
                            "weight": weight.item(), "bias": bias.item()})

    final_w = weight.item()
    final_b = bias.item()
    # The loop's loss precedes its last update. Recompute the final measurement
    # from the saved parameters, without changing those parameters.
    with torch.no_grad():
        final_loss = ((weight * celsius + bias - fahrenheit) ** 2).mean().item()

    print(f"after training:  weight={final_w:.4f} bias={final_b:.4f} "
          f"(true weight 1.8, bias 32)")
    # Show the raw final loss so display rounding does not make it look zero.
    print(f"final loss (post-update, unrounded)={final_loss}")

    # Save a dictionary with two named shape-(1,) tensors. detach() removes
    # gradient tracking; the checkpoint contains values, not the training graph.
    # torch.save writes this dictionary to the path supplied by --checkpoint.
    torch.save({"weight": weight.detach(), "bias": bias.detach()},
               args.checkpoint)
    print(f"saved checkpoint to {args.checkpoint}")

    # Tiny same-process reload smoke test. The README also tests a fresh process.
    # weights_only restricts loading to tensors and supported simple data types.
    state = torch.load(args.checkpoint, weights_only=True)
    # Do not record operations for this prediction-only check.
    with torch.no_grad():
        # One row, one input: shape (1, 1). The result also has one element.
        test_c = torch.tensor([[25.0]])
        # Predict Fahrenheit using the reloaded weight and bias, not training state.
        predicted = (state["weight"] * test_c + state["bias"]).item()
    print(f"inference: 25 C -> {predicted:.4f} F (true 77)")

    # Optionally save the run settings and measurements for later comparison.
    if args.output:
        # Keep Python float values as returned by item(), with no extra rounding.
        args.output.write_text(json.dumps({
            "seed": args.seed,
            "epochs": args.epochs,
            "lr": args.lr,
            "initial_weight": init_w,
            "initial_bias": init_b,
            "first_loss": first_loss,
            "first_weight_grad_backward": first_w_grad,
            "first_weight_grad_by_hand": hand_w_grad,
            "first_bias_grad_backward": first_b_grad,
            "first_bias_grad_by_hand": hand_b_grad,
            "learned_weight": final_w,
            "learned_bias": final_b,
            "final_loss": final_loss,
            "history": history,
            "inference_25c_f": predicted,
        }, indent=2) + "\n")


if __name__ == "__main__":
    main()

Week 6 companion lab and recorded training results:
https://github.com/dramasamy/from-api-to-gpu/tree/main/week-06-neural-network-basics ↩
Toy weight comparison, command and actual Spark output:
https://github.com/dramasamy/from-api-to-gpu/blob/main/week-07-autograd-training-loop/system-report.md ↩
PyTorch autograd mechanics:
https://docs.pytorch.org/docs/stable/notes/autograd.html ↩
PyTorch Linear layer and its matrix formula:
https://docs.pytorch.org/docs/stable/generated/torch.nn.Linear.html ↩
Week 7 companion lab:
https://github.com/dramasamy/from-api-to-gpu/tree/main/week-07-autograd-training-loop ↩
PyTorch gradient modes and evaluation mode:
https://docs.pytorch.org/docs/stable/notes/autograd.html#locally-disabling-gradient-computation ↩
PyTorch checkpoint loading and security warning:
https://docs.pytorch.org/docs/stable/generated/torch.load.html ↩
Week 8 roadmap:
https://github.com/dramasamy/from-api-to-gpu/blob/main/roadmap/week-08.md ↩

From API to GPU, Week 6 (Part 2): Watching a Neural Network Learn

Dinesh Kumar Ramasamy — Sat, 05 Sep 2026 19:11:51 +0000

Phase 2 of 8: Enough ML to understand inference. Week 6 of 32, part 2 of 2.

Part 1 built a one-neuron model that converts Celsius to Fahrenheit, made it
predict, and scored how wrong it was with a single number, the loss. With random
starting values the loss was 10352.21, because every prediction was nonsense. This
post takes that one number and turns it into learning: the weight and bias climb
from random noise to the true 1.8 and 32, and I watch each step happen on screen.

Training is one loop repeated many times, and each pass through it does four
things:

Forward pass: run the inputs through the model to get predictions.
Loss: score how wrong those predictions are, as a single number.
Gradient: work out which direction to move each parameter to lower the loss.
Optimizer step: nudge each parameter a small amount in that direction.

Part 1 covered the first two, prediction and loss. This post covers the last two,
the steps that actually change the weight and bias, and then the full loop that
repeats all four until the model has learned. So the two new ideas here are the
gradient (step 3) and the optimizer (step 4).

From error to a direction: the gradient

The gradient is the slope of the loss for each parameter: which direction, and
how steeply, the loss changes if that parameter moves. If nudging the weight up
makes the loss go down, the gradient is what tells the optimizer to move it up.

Backpropagation computes those gradients for every parameter in one pass. In
PyTorch it is the single call loss.backward(). For our one-neuron model there
are only two gradients to compute, one for the weight and one for the bias, but
the same call scales to the billions of parameters in a large model.

The optimizer and the learning rate

The optimizer nudges each parameter using its gradient. The gradient points in
the direction that increases the loss, so the optimizer subtracts it:

new = old - learning rate \times gradient

That minus sign is why subtracting a negative gradient makes a parameter grow. I
use plain SGD. The class is named SGD (stochastic gradient descent), but
"stochastic" normally means updating from random subsets of the data. This run
uses all six examples for every update, so it is really full-batch gradient descent
using the SGD class.¹

The learning rate is how big each nudge is. Too small learns slowly, so it
needs more updates; too big overshoots and blows up, which I show below.

Epoch and batch

An epoch is one pass over all the training data. A batch is how many
examples the model sees before each update. Here the dataset is six
Celsius/Fahrenheit pairs and I use all of them at once, so one batch equals one
epoch.

Putting the four moves together, the loop is: forward pass, compute loss,
backpropagate to get gradients, optimizer step, repeat. That loop is training.
Using the finished model to predict without changing it is inference, which is
what the earlier text-generation weeks were doing with pretrained models.

The training script

Here is the whole thing. It builds the data, defines the one-layer model, shows the
first predictions and the first gradient before any learning, runs the loop, and
saves a checkpoint (a copy of the trained numbers on disk). A separate predict.py
loads that checkpoint for inference, shown right after this run:

#!/usr/bin/env python3
"""Week 6 - train the smallest useful neural network: Celsius to Fahrenheit.

The true rule is F = C * 1.8 + 32. A single linear layer y = w*x + b should
learn a weight near 1.8 and a bias near 32. The point is not the model. It is to
watch the weight and bias start random, the loss fall, the gradients drive the
change, and the trained parameters get saved to a .pt checkpoint. A separate
predict.py loads that checkpoint to run inference.

The inputs are raw Celsius, so the printed weight is the model's real parameter.
Because the raw inputs are large, the first gradient is large too, which is why
the learning rate has to be small (0.0003). A large learning rate diverges.
"""

from __future__ import annotations

import argparse
import json
from pathlib import Path

import torch
from torch import nn


def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--epochs", type=int, default=20000)
    parser.add_argument("--lr", type=float, default=3e-4)  # learning rate 0.0003
    parser.add_argument("--seed", type=int, default=0)
    parser.add_argument("--checkpoint", type=Path,
                        default=Path("celsius_to_fahrenheit.pt"))
    parser.add_argument("--output", type=Path)
    return parser.parse_args()


def main() -> None:
    args = parse_args()
    torch.manual_seed(args.seed)

    # Six training inputs in Celsius. Each inner [ ] is one example with one
    # value, so the shape is 6 rows by 1 column (six examples, one feature).
    celsius = torch.tensor([[-40.0], [-10.0], [0.0], [20.0],
                            [37.0], [100.0]])
    # The correct answer for each input. This is the only place the true rule
    # appears, and it only builds the labels; the model never sees it.
    fahrenheit = celsius * 1.8 + 32.0

    model = nn.Linear(1, 1)  # one layer: prediction = weight * C + bias
    loss_fn = nn.MSELoss()  # scores how wrong the predictions are
    optimizer = torch.optim.SGD(model.parameters(), lr=args.lr)  # updates w, b

    print(f"device={model.weight.device}")
    init_w = model.weight.item()
    init_b = model.bias.item()
    print(f"initial weight={init_w:.6f} bias={init_b:.6f}")

    # Show where the first loss comes from: predict, then average the
    # squared errors across the six examples.
    with torch.no_grad():
        first_preds = model(celsius)
    print("initial predictions vs targets:")
    for c, p, f in zip(celsius.tolist(), first_preds.tolist(),
                       fahrenheit.tolist()):
        print(f"  C={c[0]:>6.1f}  pred={p[0]:>8.3f}  target={f[0]:>7.1f}")

    # One manual step to expose the first gradient before the optimizer moves.
    optimizer.zero_grad()
    first_loss_t = loss_fn(model(celsius), fahrenheit)
    first_loss_t.backward()
    first_loss = first_loss_t.item()
    w_grad = model.weight.grad.item()
    b_grad = model.bias.grad.item()
    print(f"first loss={first_loss:.4f} "
          f"weight_grad={w_grad:.4f} bias_grad={b_grad:.4f}")
    print(f"first weight update: {init_w:.6f} - {args.lr} * {w_grad:.4f} "
          f"= {init_w - args.lr * w_grad:.6f}")

    history = []
    # log_at picks the handful of epochs to print, so the output stays short.
    log_at = {1, 50, 200, 1000, 5000, 10000, args.epochs}
    for epoch in range(1, args.epochs + 1):
        # One pass of the four steps from the intro (plus a housekeeping reset):
        optimizer.zero_grad()  # reset: clear gradients left from the last step
        loss = loss_fn(model(celsius), fahrenheit)  # steps 1-2: predict, then score
        loss.backward()  # step 3: backprop fills each parameter's gradient
        optimizer.step()  # step 4: nudge weight and bias down the loss
        if epoch in log_at:
            # This block only logs progress; the learning already happened above.
            # Recompute the loss after the step so the printed loss and the
            # printed weight and bias all describe the same post-update model.
            with torch.no_grad():
                post_loss = loss_fn(model(celsius), fahrenheit).item()
            w = model.weight.item()  # current weight, as a plain Python float
            b = model.bias.item()  # current bias
            print(f"epoch {epoch:>5} loss={post_loss:>12.4f} "
                  f"weight={w:.4f} bias={b:.4f}")
            # Keep this snapshot so results.json can chart the run later.
            history.append({"epoch": epoch, "loss": round(post_loss, 4),
                            "weight": round(w, 4), "bias": round(b, 4)})

    # Training is finished. Read the final learned weight and bias, and the
    # loss of the finished model, which is now tiny.
    final_w = model.weight.item()
    final_b = model.bias.item()
    with torch.no_grad():
        final_loss = loss_fn(model(celsius), fahrenheit).item()
    print(f"\nlearned weight={final_w:.4f} bias={final_b:.4f} "
          f"final_loss={final_loss:.4e} (true weight 1.8, bias 32)")

    torch.save(model.state_dict(), args.checkpoint)
    print(f"saved state_dict to {args.checkpoint}")

    if args.output:
        args.output.write_text(json.dumps({
            "seed": args.seed,
            "epochs": args.epochs,
            "lr": args.lr,
            "initial_weight": round(init_w, 6),
            "initial_bias": round(init_b, 6),
            "first_loss": round(first_loss, 4),
            "first_weight_grad": round(w_grad, 4),
            "first_bias_grad": round(b_grad, 4),
            "learned_weight": round(final_w, 4),
            "learned_bias": round(final_b, 4),
            "final_loss": final_loss,
            "history": history,
        }, indent=2) + "\n")


if __name__ == "__main__":
    main()

Three details are worth calling out before running it.

The celsius * 1.8 + 32.0 line is the only place the true rule appears, and it
only builds the answer labels. That gives pairs like (-40, -40), (0, 32), and
(100, 212). The model receives the Celsius inputs and those Fahrenheit answers,
never the formula, so recovering 1.8 and 32 is real learning, not copying.

optimizer.zero_grad() clears the gradient buffers before each new
forward/backward cycle. PyTorch adds new gradients onto whatever is already
stored, so without clearing, the next cycle's gradients would pile on top of the
last one. It does not touch the weight or bias, and it does not perform an update.

The inputs are raw Celsius, so the printed weight is the model's actual parameter,
with no rescaling. The catch is that raw inputs up to 100 make the first gradient
large, which forces a small learning rate. That trade-off is the subject of its own
section below.

The script also uses torch.no_grad() and saves a state_dict; the eval() call
lives in predict.py. All three are explained in a deferred section so they do not
clutter the main story.²

Watching it learn

I ran the exact command below on the Spark. The < ... train_tiny.py part reads
the script file on my control Mac and pipes it in; Python and the training run on
the Spark. The path starts with public/ because I run from the private parent
repo; if you cloned only the public companion repo, drop that prefix and use
week-06-neural-network-basics/train_tiny.py:

ssh spark '~/venvs/w1/bin/python -' \
    < public/week-06-neural-network-basics/train_tiny.py

device=cpu
initial weight=-0.007487 bias=0.536444
initial predictions vs targets:
  C= -40.0  pred=   0.836  target=  -40.0
  C= -10.0  pred=   0.611  target=   14.0
  C=   0.0  pred=   0.536  target=   32.0
  C=  20.0  pred=   0.387  target=   68.0
  C=  37.0  pred=   0.259  target=   98.6
  C= 100.0  pred=  -0.212  target=  212.0
first loss=10352.2070 weight_grad=-9237.2129 bias_grad=-127.3941
first weight update: -0.007487 - 0.0003 * -9237.2129 = 2.763677
epoch     1 loss=   1992.1444 weight=2.7637 bias=0.5747
epoch    50 loss=    806.2087 weight=2.0436 bias=1.3524
epoch   200 loss=    690.7745 weight=2.0255 bias=3.6312
epoch  1000 loss=    302.9773 weight=1.9493 bias=13.2121
epoch  5000 loss=      4.9179 weight=1.8190 bias=29.6063
epoch 10000 loss=      0.0285 weight=1.8014 bias=31.8178
epoch 20000 loss=      0.0000 weight=1.8000 bias=31.9981

learned weight=1.8000 bias=31.9981 final_loss=2.9334e-06 (true weight 1.8, bias 32)
saved state_dict to celsius_to_fahrenheit.pt

This is the whole point of the week.

The start is random and wrong. The initial weight is about -0.007 and the bias
is about 0.54. Those come straight from torch.manual_seed(0). Every one of the
six initial predictions is far from its target: at 100 C the model guesses -0.212
instead of 212. Squaring those six errors and averaging them gives the first loss,
10352.21, the same number Part 1 ended on. That is where the big first number comes
from.

The gradient points the way. Before any learning, loss.backward() reported a
weight gradient of -9237.2 and a bias gradient of -127.4. Both negative means the
loss would drop if both parameters increased, which is correct: they need to climb
toward 1.8 and 32.

One update, traced by hand. The optimizer's rule is
new = old - learning_rate * gradient, applied to every parameter. The 0.0003
in the run's first weight update: line is that learning rate (the script's
--lr, which defaults to 3e-4, another way to write 0.0003). For the weight
that is -0.007487 - 0.0003 * -9237.2129 = 2.763677, and epoch 1 indeed shows
weight=2.7637. The same arithmetic on the bias is
0.536444 - 0.0003 * -127.3941 = 0.574662, and epoch 1 shows bias=0.5747. So the
first step is not magic; it is that one line of arithmetic done twice.

Why the learning rate is so small. That first weight gradient is huge, about
-9237, because the raw Celsius inputs are large (with these inputs, parameters, and
MSE loss). Even multiplied by the tiny learning rate 0.0003, the weight still jumps
from about 0 to 2.76 in one step, overshooting its target of 1.8. A larger learning
rate would overshoot so hard the numbers explode, which I show below.

The loss falls fast, then slows. Reading the recomputed post-update loss, it
drops from 1992 at epoch 1 to 806 by epoch 50, then grinds down to about 0.0000 by
epoch 20000. The weight moves close to its target much sooner than the bias: by
epoch 50 the weight is 2.04 and still drifting, while the bias is only 1.35 and has
a long climb ahead. Why one parameter moves faster than the other is a gradient
question I leave for Week 7.

It lands on the real rule, almost exactly. The final weight is 1.8000 and the
bias is 31.9981, with a final loss of 2.9e-06. It is not perfectly 32: the run
stopped at 20000 epochs, and running to 40000 or 100000 epochs gives the exact same
values, because the remaining SGD updates are smaller than what FP32 can represent,
so the parameters stop changing. The network rediscovered F = C * 1.8 + 32 from
six examples, to the displayed precision, without ever being told the formula.

The checkpoint is saved. The last line writes the model's state_dict to
celsius_to_fahrenheit.pt, a saved copy of the trained weight and bias. That is
the same idea as the model files from Week 3, at the smallest possible scale.
Loading it back and predicting is the job of the next section. The .pt file is
written in the current directory on the Spark and is not committed to the repo.

One timing detail explains the numbers. The first loss, 10352.21, is measured
before any update. The epoch 1 loss, 1992.14, is recomputed after the update, so it
already reflects the improved weight. Each logged row shows the loss and the
parameters for the same post-update state, which is why they line up.

Loading the checkpoint in a standalone script

Training and inference are separate jobs, so I keep them in separate scripts.
Training, above, produced celsius_to_fahrenheit.pt. Inference needs none of the
training machinery, no loss, no gradients, no optimizer. It only rebuilds the
model, loads the saved numbers into it, and calls it. Here is the whole
predict.py:

#!/usr/bin/env python3
"""Week 6 - load the trained checkpoint and run one inference.

train_tiny.py trains the Celsius-to-Fahrenheit model and saves its parameters to
a .pt file. This script is the other half, with no training code at all: rebuild
the same model shape, load the saved parameters into it, and make a prediction.
Run train_tiny.py first so the checkpoint exists.
"""

from __future__ import annotations

import argparse
from pathlib import Path

import torch
from torch import nn


def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--checkpoint", type=Path,
                        default=Path("celsius_to_fahrenheit.pt"))
    parser.add_argument("--celsius", type=float, default=25.0)
    return parser.parse_args()


def main() -> None:
    args = parse_args()

    # Build the SAME architecture the checkpoint was trained with. A state_dict
    # stores only numbers, not the model shape, so the layer must exist first.
    model = nn.Linear(1, 1)
    # Load the saved weight and bias from the .pt file into this layer.
    model.load_state_dict(torch.load(args.checkpoint))
    model.eval()  # inference mode (no visible effect here, but the right habit)

    print(f"loaded {args.checkpoint}")
    print(f"weight={model.weight.item():.4f} bias={model.bias.item():.4f}")

    # One input, wrapped to shape 1 row by 1 column like the training data.
    celsius = torch.tensor([[args.celsius]])
    with torch.no_grad():  # predicting only, so do not track gradients
        fahrenheit = model(celsius).item()
    print(f"inference: {args.celsius:.1f} C -> {fahrenheit:.4f} F")


if __name__ == "__main__":
    main()

Two things make the load work. First, a state_dict stores only the numbers, not
the model's shape, so the script must build the same nn.Linear(1, 1) before
loading, the same point Part 1 made about a state_dict. Second, eval() and
torch.no_grad() put the model in inference mode. Run it right after training, from
the same directory so it finds the .pt:

ssh spark '~/venvs/w1/bin/python -' \
    < public/week-06-neural-network-basics/predict.py

loaded celsius_to_fahrenheit.pt
weight=1.8000 bias=31.9981
inference: 25.0 C -> 76.9985 F

The loaded weight and bias are 1.8000 and 31.9981, the exact values training landed
on, and 25 C predicts 76.9985 F, which rounds to the true 77 (25 * 1.8 + 32 = 77).
Nothing was retrained. predict.py only read the parameters train_tiny.py wrote
and applied the same weight * C + bias from Part 1.

Learning rate: too big blows up, and the run repeats

The small learning rate is not a random choice. Raw Celsius inputs make the
gradients large, and at learning rate 0.1 the optimizer overshoots so far the loss
explodes. Here is a short run at learning rate 0.1 that prints the loss in
scientific notation each epoch:

ssh spark '~/venvs/w1/bin/python - <<PY
import torch
from torch import nn
torch.manual_seed(0)
# Same six Celsius inputs (C) and their Fahrenheit answers (F) as before.
C=torch.tensor([[-40.],[-10.],[0.],[20.],[37.],[100.]]); F=C*1.8+32
# Same model, loss, and optimizer, but a large learning rate of 0.1.
m=nn.Linear(1,1); lf=nn.MSELoss(); opt=torch.optim.SGD(m.parameters(),lr=0.1)
# Run 15 training steps and print the loss (%.4e = scientific notation).
for e in range(1,16):
    opt.zero_grad(); l=lf(m(C),F); l.backward(); opt.step()
    print("epoch %2d loss=%.4e"%(e,l.item()))
PY'

epoch  1 loss=1.0352e+04
epoch  2 loss=1.9073e+09
epoch  3 loss=3.8280e+14
epoch  4 loss=7.6828e+19
epoch  5 loss=1.5419e+25
epoch  6 loss=3.0946e+30
epoch  7 loss=6.2109e+35
epoch  8 loss=inf
epoch  9 loss=inf
epoch 10 loss=inf
epoch 11 loss=inf
epoch 12 loss=inf
epoch 13 loss=inf
epoch 14 loss=inf
epoch 15 loss=inf

The loss grows fast, gaining about five orders of magnitude every step, and by
epoch 8 it is inf (floating-point infinity, the result of a number too large to
represent). Once a value is inf or nan (short for "not a number") the run is
dead. So the small learning rate is doing real work: it keeps the huge first
gradient from throwing the model off a cliff.

The run is also reproducible on this setup. torch.manual_seed(0) fixes the
initial random weight and bias, so on this Spark with PyTorch 2.13.0+cu130 the two
tested seed-0 runs gave the same displayed values, and seeds 1 and 2 reached the
same displayed 1.8 and 32 as well.²

Training versus inference, made concrete

The two scripts are the two modes I have been using loosely. train_tiny.py does
training: every step calls loss.backward() and optimizer.step() to change
the weights. predict.py does inference: it loads finished weights, calls
eval() and torch.no_grad(), and never changes anything. These two calls do
different jobs, and the deferred section spells them out.² The earlier
text-generation weeks ran models in this inference mode. This week is the first
time I ran the training half of the loop, and splitting the scripts makes the line
between the two obvious.

Results

Here is the training run in one place:

Item	Verified value
Task	Celsius to Fahrenheit, F = C * 1.8 + 32
Model	one linear layer, `nn.Linear(1, 1)`, no activation
Device	CPU
Initial weight / bias	-0.007487 / 0.536444
First loss	10352.21 (squared Fahrenheit)
First gradients	weight -9237.21, bias -127.39
First weight after one step	2.7637
Loss at epoch 5000 (post-update)	4.9179
Learned weight / bias	1.8000 / 31.9981
Final loss	2.9e-06
Inference: 25 C	76.9985 F (rounds to 77)
Optimizer / learning rate	SGD / 0.0003
Epochs / batch	20000 / full batch of 6

What surprised me

I expected the loss to fall. I did not expect the weight and the bias to learn at
such different speeds. The weight moves close to its target within the first
handful of updates, while the bias takes thousands more epochs to crawl from 0.5 up
to nearly 32. The optimizer is not solving an equation; it is taking small steps
downhill on the loss, and the two parameters travel at very different speeds. Week
7 digs into why.

The other satisfying part was watching six examples be enough to recover the rule
to the displayed precision. A large language model can have billions of parameters
and messy data, but the loop is the same one I just watched: predict, measure loss,
get gradients, step.

Mistakes and troubleshooting

My first version scaled the inputs down by 100 to keep SGD stable, then unscaled the
weight before printing. That worked, but it created two different meanings of
"weight": the model's internal parameter and the display value. The printed gradient
belonged to the internal parameter, so it did not line up with the printed weight,
which made the single most important step impossible to follow. Switching to raw
inputs with a small learning rate removed the confusion. Now the printed weight is
the real parameter, and the first-update arithmetic checks out exactly.

The lesson that survived is about input size. Large inputs make large gradients,
which force a small learning rate. Scaling inputs to a small range is the usual fix
in real training; I left it out here only to keep every printed number literal.

Production implications

I will not train models this way in production, but this loop is the foundation
under everything that does. Fine-tuning, which this series reaches in Phase 8, is
the same forward, loss, backward, step cycle, just with a pretrained model, far more
parameters, and real data. Knowing what a gradient and a learning rate actually do
makes the later fine-tuning weeks far less mysterious.

It also explains two settings I will keep seeing. The learning rate is a key
training setting: set it too high and training diverges, as the --lr 0.1 run
showed. And a checkpoint here is a saved state_dict, the same idea as the model
files I inspected in Week 3, at the smallest possible scale.

What I will learn next

Week 7 stays on the training loop and goes one level deeper: the computational
graph, requires_grad, and writing the forward, backward(), and optimizer step by
hand instead of leaning on the tidy loop above. That turns this week's
loss.backward() from a black box into something I can trace.³

The training questions, answered

A few terms in the run reward a closer look. They are not needed to follow the main
story, so I pulled them here.

What exactly is a state_dict? For this model it is a mapping from parameter
name to tensor, holding just the learned weight and bias. You can print each
name with its tensor shape and value:

ssh spark '~/venvs/w1/bin/python -c "
import torch; from torch import nn
torch.manual_seed(0)
# state_dict() returns the layer's saved parameters as name -> tensor pairs.
sd=nn.Linear(1,1).state_dict()
# Print each parameter name, its shape, and its values.
for name, tensor in sd.items():
    print(name, tuple(tensor.shape), tensor)"'

weight (1, 1) tensor([[-0.0075]])
bias (1,) tensor([0.5364])

So the two entries really are tensors carrying the parameter values (here the
random start, before training). The state_dict stores the numbers, not the code.
It does not know it came from an nn.Linear(1, 1), which is why the script must
build that same layer again before loading the values into it. Calling every
checkpoint "just a state_dict" would be too broad; this is what a PyTorch
checkpoint holds, not every packaging format from Week 3.

What do eval() and no_grad() each do? They sound like one setting but have
separate jobs. eval() switches layers whose behavior differs between training and
inference into inference mode. A plain linear layer behaves the same either way, so
eval() has no visible effect on this model; I call it because it is the correct
habit for real models that do have such layers. no_grad() tells PyTorch to stop
recording operations for gradient computation. (Not recording is expected to save
memory and time during inference, though I do not measure that here.) Neither one
freezes the weights. The weights stay put simply because there is no backward()
and no optimizer.step() during inference.

How do I know other seeds still converge? Because I ran them. Seed 0 twice, then
seeds 1 and 2, all 20000 epochs:

ssh spark '~/venvs/w1/bin/python - <<PY
import torch
from torch import nn
# Train the model from scratch for one random seed and return the final w, b.
def run(seed):
    torch.manual_seed(seed)  # a different seed = a different random start
    C=torch.tensor([[-40.],[-10.],[0.],[20.],[37.],[100.]]); F=C*1.8+32
    m=nn.Linear(1,1); lf=nn.MSELoss(); opt=torch.optim.SGD(m.parameters(),lr=3e-4)
    for e in range(20000):  # the full training loop, 20000 steps
        opt.zero_grad(); l=lf(m(C),F); l.backward(); opt.step()
    return m.weight.item(),m.bias.item()
# Seed 0 twice (to show it repeats), then seeds 1 and 2 (different starts).
for s in [0,0,1,2]:
    w,b=run(s); print("seed %d -> w=%.4f b=%.4f"%(s,w,b))
PY'

seed 0 -> w=1.8000 b=31.9981
seed 0 -> w=1.8000 b=31.9981
seed 1 -> w=1.8000 b=31.9981
seed 2 -> w=1.8000 b=31.9981

Seed 0 gave the same displayed values across runs, and seeds 1 and 2 reached the
same displayed 1.8 and 32. This task has a single best fit, and on this setup the
tested seeds all reached it.

Run it yourself

The public Week 6 lab has the training and inference scripts, the captured runs,
observations, and troubleshooting notes.⁴

PyTorch torch.optim.SGD:
https://docs.pytorch.org/docs/stable/generated/torch.optim.SGD.html ↩
See the deferred section,
The training questions, answered. ↩
Week 7 roadmap:
https://github.com/dramasamy/from-api-to-gpu/blob/main/roadmap/week-07.md ↩
Week 6 companion lab:
https://github.com/dramasamy/from-api-to-gpu/tree/main/week-06-neural-network-basics ↩

From API to GPU, Week 6 (Part 1): A Model That Predicts, and How Wrong It Is

Dinesh Kumar Ramasamy — Sat, 05 Sep 2026 19:11:00 +0000

Phase 2 of 8: Enough ML to understand inference. Week 6 of 32, part 1 of 2.

Every week so far, the model already existed. I ran Phi-4, read Qwen's files, and
measured tensors, but I never made a model learn anything. This week I build one
from scratch and train it. That is a lot for one sitting, so I split the week into
two posts and go slowly.

The goal: the model starts with a random weight and bias, which are just
wrong guesses. Training is what adjusts them, step by step, until they land on
the values that actually convert Celsius to Fahrenheit: a weight of 1.8 and a
bias of 32. Everything in these two posts is in service of that one idea.

This first post builds the model and answers two questions, nothing more: how does
the model make a prediction, and how do we measure how wrong that prediction is?
No learning yet. Part 2 takes the error measured here and turns it into learning.

The model is a one-neuron linear model: one output value computed from one weight
and one bias. It is deliberately tiny, so nothing hides the machinery. It runs on
the CPU through ssh spark, because the problem is far too small to need a GPU.

What "learning" means here

If you have never trained a model before, "learning" can sound like magic. It is
not. It is trial and error run in a tight loop, the same way you would
reverse-engineer an unknown function from its input and output logs.

Picture a black box that turns numbers in into numbers out. You cannot see inside
it, but you have a few examples of what it did: it turned -40 into -40, 0 into 32,
and 100 into 212. You want a formula that reproduces those outputs. So you guess
one, check how far each guess is from the real answer, nudge the guess in the
direction that shrinks the error, and repeat until the answers line up. That loop
is all that "training" is.

A neural network does exactly this, with two differences. First, the "formula" is
a set of adjustable numbers called parameters (the weights and biases from Week
4), and learning means finding good values for them. Second, the nudging is
automatic: the network measures its own error and works out which way to move each
parameter, so nobody tweaks anything by hand.

That loop has four moves: make a guess, measure how wrong it is, work out which way
to adjust, and take a small step, then repeat. This post covers only the first
two, making a guess (the forward pass) and measuring how wrong it is (the
loss). Part 2 covers the adjusting. Splitting it this way keeps each idea tied
to a real number you can see, instead of a pile of new words up front.

The black box in this post is Celsius to Fahrenheit. The true rule is
F = C * 1.8 + 32. The model never sees that rule. I only use it to build the
correct answers, then show the model input-output examples and let it discover on
its own that the weight should be 1.8 and the bias should be 32.

The model: one neuron

Week 4 said a weight multiplies an input and a bias is added after. This
model is exactly that, one weight and one bias:

prediction = weight \times Celsius + bias

That single weight-and-bias unit is a neuron, the smallest building block of a
neural network. A neural network is just many neurons wired together, so the
output of some becomes the input of others. Real networks stack thousands or
millions of them; this week's has exactly one, which is why nothing hides the
mechanics.

In PyTorch that is a single linear layer, nn.Linear(1, 1): one input
feature (the one input value) and one output value per example. A layer is
one processing step, and a linear transformation is the multiply-and-add above.
(Because there is a nonzero bias, a mathematician would call this an affine
transformation, but PyTorch names the layer Linear, so I use that word.) If the
weight lands on 1.8 and the bias on 32, the layer computes Fahrenheit exactly.

Why is this called linear? Picture plotting the model's output against its
input on a graph. A weight-times-input-plus-bias rule always traces a straight
line. The weight sets the line's slope: how steeply the output climbs as the
input grows. The bias slides the whole line up or down. Celsius to Fahrenheit is a
straight line with slope 1.8, because every extra degree Celsius adds exactly 1.8
degrees Fahrenheit, the same amount at every temperature. That fixed,
never-changing slope is what "linear" and "straight line" mean here, and a single
linear layer can match it perfectly.

This model has no activation function, the extra piece that would let it bend a
straight line into a curve. Because Celsius to Fahrenheit is already straight, it
does not need one. What an activation function is, and what a curve looks like, I
unpack at the end, in Wait, why is there no activation function?, so the main
thread stays on the model.

Making a prediction: the forward pass

A forward pass runs the inputs through the layer to get predictions. In
PyTorch it is the single expression model(celsius). Before any training the
weight and bias are random, so the predictions will be nonsense, which is exactly
what I want to see first.

The six Celsius inputs are paired with their correct Fahrenheit answers. Each
correct answer is called a label or target, and the line
fahrenheit = celsius * 1.8 + 32.0 builds all six at once. There is no loop here.
celsius is a tensor, the small array type from Week 5, and multiplying a tensor
by a single number multiplies every element by that number; adding 32 then adds it
to every element. That is an element-wise operation, and it is why the result is
another tensor of six answers, not one number. Note that celsius is the input
data, not the model. The model is the nn.Linear layer built a few lines down.

Each value in celsius is written with a trailing dot, so -40. means the
floating-point number -40.0, not the integer -40. The dot keeps the tensor in
decimals, which is what the model's math needs. And each value sits in its own
brackets, like [-40.], so stacking the six of them makes a tensor shaped 6 rows
by 1 column: six examples, one value each.

To be clear, those six numbers are six separate Celsius inputs (-40, -10, 0, 20,
37, and 100 degrees). [-40.] and [-10.] are two different examples, not an
input-and-answer pair. Each input's Fahrenheit answer lives at the matching
position in the separate fahrenheit tensor, and the run below prints them side
by side. The value -40 shows up as both the first input and its own answer only
because -40 C equals -40 F, the one temperature where the Celsius and Fahrenheit
scales meet, which is a coincidence of the scales, not the data layout.

The command below builds the model, prints the device it runs on and its random
starting weight and bias, then does one forward pass and lays each prediction next
to its target:

ssh spark '~/venvs/w1/bin/python - <<PY
import torch  # PyTorch: the tensor and neural-network library
from torch import nn  # nn holds ready-made layers such as Linear

# Fix the random starting values so this run is repeatable and matches the post.
torch.manual_seed(0)

# The six training inputs, in Celsius. Each inner [ ] is one example holding one
# value, so the tensor shape is 6 rows by 1 column: six examples, one feature.
celsius = torch.tensor([[-40.], [-10.], [0.], [20.], [37.], [100.]])

# The correct answer for each input, built from the real rule F = C * 1.8 + 32.
# The model never sees this rule; it only sees the inputs and these answers.
fahrenheit = celsius * 1.8 + 32.0

# One linear layer with 1 input feature and 1 output: prediction = w * C + b.
model = nn.Linear(1, 1)

# weight and bias start at random values; .device shows they live on the CPU.
print(f"device={model.weight.device}")
print(f"initial weight={model.weight.item():.6f} bias={model.bias.item():.6f}")

# The forward pass: run all six inputs through the layer to get predictions.
# no_grad() means "just predict, do not record anything for training".
with torch.no_grad():
    preds = model(celsius)

# Show each prediction next to the answer it should have produced.
print("initial predictions vs targets:")
for c, p, f in zip(celsius.tolist(), preds.tolist(), fahrenheit.tolist()):
    print(f"  C={c[0]:>6.1f}  pred={p[0]:>8.3f}  target={f[0]:>7.1f}")
PY'

device=cpu
initial weight=-0.007487 bias=0.536444
initial predictions vs targets:
  C= -40.0  pred=   0.836  target=  -40.0
  C= -10.0  pred=   0.611  target=   14.0
  C=   0.0  pred=   0.536  target=   32.0
  C=  20.0  pred=   0.387  target=   68.0
  C=  37.0  pred=   0.259  target=   98.6
  C= 100.0  pred=  -0.212  target=  212.0

Two things to notice. device=cpu confirms this runs on the CPU, the .device
attribute from Week 5, and the problem is far too small to need a GPU. And the
random start makes every prediction wrong: the weight is about -0.007 and the bias
about 0.54, straight from torch.manual_seed(0) (a fixed seed, so you get these
same numbers), so at 100 C the model guesses -0.212 instead of 212. The model has
no idea what Fahrenheit is yet.

So why are the predictions wrong when the correct answers are sitting right there
in fahrenheit? Because the model never looks at those answers when it predicts.
A prediction is only weight * celsius + bias, computed from the model's own
weight and bias, which right now are the random -0.007 and 0.54. The fahrenheit
values are a separate answer key: they are used to score the predictions, and in
Part 2 to teach the model, but the model does not see them while it predicts. It
is like a student taking a test with the answer key face down. Until they learn
the rule, the answers are guesses, no matter that the key is in the room.

The answers are not inside the input tensor at all. celsius and fahrenheit
are two separate tensors, two different variables. The call model(celsius)
passes only celsius, so fahrenheit never enters the model. Inside, the linear
layer computes exactly weight * celsius + bias from its own two numbers. The one and only place fahrenheit is handed to
anything is the loss, loss_fn(preds, fahrenheit), in the next section. You can
prove the model uses only its weight, bias, and the input by recomputing its
prediction by hand and comparing:

ssh spark '~/venvs/w1/bin/python - <<PY
import torch
from torch import nn
torch.manual_seed(0)
celsius = torch.tensor([[-40.], [-10.], [0.], [20.], [37.], [100.]])
fahrenheit = celsius * 1.8 + 32.0   # the answer key: a separate tensor

model = nn.Linear(1, 1)
w = model.weight.item()             # weight and bias: the model owns these
b = model.bias.item()

with torch.no_grad():
    preds = model(celsius)          # pass ONLY celsius, never fahrenheit
by_hand = celsius * w + b           # weight * input + bias, computed by hand

print(f"weight={w:.6f} bias={b:.6f}")
print("model(celsius) matches weight*celsius+bias:",
      torch.allclose(preds, by_hand))
print("prediction for 100 C:", round(preds[-1].item(), 4))
PY'

weight=-0.007487 bias=0.536444
model(celsius) matches weight*celsius+bias: True
prediction for 100 C: -0.2122

model(celsius) and the hand-computed weight * celsius + bias come out
identical, so the model's output depends only on its two parameters and the
Celsius input. The value -0.2122 is the same wrong guess for 100 C from the
table above. fahrenheit plays no part in producing it.

Measuring how wrong it is: the loss

Now that the model can make predictions, we need a single number that says how good
or bad those six predictions are as a group. That number is the loss. A smaller
loss means better predictions; a loss of zero means every prediction is exactly
right.

Building it starts with the error for one example: how far a single prediction
is from its target, which is just prediction - target. For the 100 C example the
prediction was -0.212 and the target is 212, so the error is about -212. Some
errors come out negative (the guess was too low) and some positive (too high).

To turn six errors into one score, I use mean squared error (MSE). The name is
the recipe read backwards: take each error, square it, then take the mean, which is
the average.

MSE = \frac{1}{6} i = 1 \sum 6 (prediction_{i} - Fahrenheit_{i})^{2}

Squaring does two useful things. It makes every error positive, so a guess that is
too low and one that is too high both count as wrong instead of cancelling out. And
it punishes big misses far more than small ones: an error of 200 becomes 40000,
while an error of 2 becomes just 4. The command below does this by hand, one step at
a time, then checks the result against PyTorch's built-in nn.MSELoss:

ssh spark '~/venvs/w1/bin/python - <<PY
import torch
from torch import nn
torch.manual_seed(0)
celsius = torch.tensor([[-40.], [-10.], [0.], [20.], [37.], [100.]])
fahrenheit = celsius * 1.8 + 32.0     # the correct answers

model = nn.Linear(1, 1)
with torch.no_grad():
    preds = model(celsius)            # the six predictions

# Step 1: the error for each example is prediction minus target.
errors = preds - fahrenheit
# Step 2: square each error so big misses count more and signs do not cancel.
squared = errors ** 2
print("per-example error and squared error:")
for e, s in zip(errors.flatten().tolist(), squared.flatten().tolist()):
    print(f"  error={e:>10.3f}  squared={s:>12.3f}")

# Step 3: the loss is the average of those six squared errors.
by_hand = squared.mean().item()
builtin = nn.MSELoss()(preds, fahrenheit).item()
print(f"average of squared errors (by hand) = {by_hand:.4f}")
print(f"nn.MSELoss() gives the same number  = {builtin:.4f}")
PY'

per-example error and squared error:
  error=    40.836  squared=    1667.572
  error=   -13.389  squared=     179.257
  error=   -31.464  squared=     989.955
  error=   -67.613  squared=    4571.558
  error=   -98.341  squared=    9670.867
  error=  -212.212  squared=   45034.031
average of squared errors (by hand) = 10352.2070
nn.MSELoss() gives the same number  = 10352.2070

Read the output top to bottom and the loss stops being a mystery number. Each row
is one example's error and that error squared. The 100 C example dominates: its
error of -212 squares to 45034, more than all the other five combined, which is
exactly the effect of punishing big misses. Add the six squared errors and divide
by six and you get 10352.2070, and PyTorch's nn.MSELoss returns the identical
value. So the built-in loss really is just the average of the squared errors,
nothing more.

One consequence of squaring: the loss is in squared Fahrenheit units, not degrees.
A loss of 10352.21 does not mean the model is off by 10352 F. The number is not a
temperature to read; it is a score to drive down. Right now it is huge, which fits,
because the predictions are still random. In Part 2, every training step pushes this
number lower.

This is the whole job of Part 1. The model can make a prediction (the forward
pass), and we can score how wrong it is with one number (the loss). Part 2 uses
that one number to improve the model.

Results so far

Item	Verified value
Task	Celsius to Fahrenheit, F = C * 1.8 + 32
Model	one linear layer, `nn.Linear(1, 1)`, no activation
Device	CPU
Initial weight / bias	-0.007487 / 0.536444
Worst initial prediction	100 C predicts -0.212 (target 212)
First loss (MSE)	10352.21 (squared Fahrenheit)

What Part 2 covers

Part 2 takes the loss from here and turns it into learning. It introduces the
gradient (which direction to move each parameter), backpropagation (how
PyTorch computes those directions), the optimizer and learning rate (how
big a step to take), and epoch and batch (how the loop repeats). Then it
runs the full training loop and watches the weight and bias climb from random noise
to 1.8 and 32.

Wait, why is there no activation function?

Most neural networks include an activation function: a step that reshapes a
layer's output so the network can learn relationships that are not straight lines.
This model leaves it out, and seeing why is a good way to learn what one does.

Many relationships are not straight. When the output climbs quickly in one place
and slowly in another, plotting it draws a curve: a line whose slope keeps
changing instead of staying fixed. A bend is just a point where the slope
changes. A few everyday examples make it concrete:

A savings account with compound interest. The balance grows slowly at first, then faster and faster, because the interest itself earns interest. Over time the line curves upward.
A phone battery charging. It races from 0 to 80 percent, then crawls through the last stretch to 100. The rate keeps changing, so the line bends.
A web server's response time as load rises. Lightly loaded, a few more requests barely change it. Near full capacity, each extra request adds much more delay and the response time shoots up. Nearly flat, then steep.

Straight-line relationships, by contrast, hold one constant rate: total cost is the
price per item times the number of items, distance is speed times time at a steady
speed, and Celsius to Fahrenheit adds the same 1.8 for every degree. A single
linear layer matches those exactly. The moment the rate has to change, a straight
line cannot follow it, and that is the job an activation function does.

The most common activation function is ReLU. The rule is simple: replace every
negative value with zero, and leave positive values unchanged. Here it is applied
to five sample numbers:

ssh spark '~/venvs/w1/bin/python -c "
import torch; from torch import nn
# Five sample numbers, two negative, a zero, and two positive.
x=torch.tensor([-2.0, -0.5, 0.0, 1.5, 3.0])
# ReLU replaces every negative with 0 and leaves the rest unchanged.
print(nn.ReLU()(x))"'

tensor([0.0000, 0.0000, 0.0000, 1.5000, 3.0000])

The two negative inputs came out as zero, the zero stayed zero, and the two
positive ones passed through unchanged. That bend at zero is the ingredient that,
stacked across many neurons and layers, lets a network trace curves like the ones
above. Celsius to Fahrenheit is a straight line, though, so this model needs no
activation function. Adding a ReLU here would actually break it: its Fahrenheit
answers go negative (-40 C is -40 F), and ReLU would clamp those to zero. When a
task does need curves, an activation function is what provides them, and a later
week puts one to work.

So what does a curve mean for a model in practice? It sets a ceiling on what the
model can learn. A model built only from linear layers can draw straight lines and
nothing else, no matter how its weights are set or how many layers you stack,
because stacking straight-line layers just produces another straight line. If the
real relationship bends, that model can never fit it well. An activation function
is what lifts the ceiling: the bends let the network shape its output to follow
curved data.

That gives a simple rule for when to use one. If the relationship you are modeling
really is a straight line, like this Celsius-to-Fahrenheit toy or a plain linear
regression, you do not need an activation function. For almost everything else,
recognizing an image, predicting a price from many features, understanding text,
the relationship bends, so real networks add an activation function after every
hidden layer by default. This model is the rare exception, which is exactly why it
is a clean place to see what an activation function does by watching a model that
works fine without one.

Run it yourself

The public Week 6 lab has the training script used in Part 2, the captured runs,
observations, and troubleshooting notes.¹

Week 6 companion lab:
https://github.com/dramasamy/from-api-to-gpu/tree/main/week-06-neural-network-basics ↩

From API to GPU, Week 5: Tensors, the Data Structure Behind Every Model

Dinesh Kumar Ramasamy — Mon, 17 Aug 2026 01:12:44 +0000

Phase 2 of 8: Enough ML to understand inference. Week 5 of 32.

Phase 1 was about running models. Phase 2 is about understanding what happens
inside them, starting with the one data structure they are all built from: the
tensor. This week I stop talking about model files and start touching the
actual numbers, in PyTorch, on the GPU.

If you write software, a tensor is a typed, multi-dimensional array that lives on
a specific device. By the end of this post I can create tensors, read their shape
and byte size, move them to the GPU, and measure how precision changes both speed
and memory.

This is also where the parallel CUDA track starts. I am not writing GPU
kernels yet, the small programs that run on the GPU. Level 1 is just being a
competent CUDA user: picking a device, moving data to it, and timing GPU work
correctly.¹

Everything runs on the DGX Spark over ssh spark. I reuse the Week 1 PyTorch
environment, ~/venvs/w1, which already has a CUDA build of PyTorch.

A tensor is a typed array with a shape and a device

Here are the names used for different array dimensions:

A scalar is a single number. Rank 0.
A vector is a list of numbers. Rank 1.
A matrix is a grid of numbers. Rank 2.
A tensor is the general word for any of these, including 3-D and higher.

Rank is the number of dimensions. Shape is the size along each dimension.
Dtype is the number format, the same FP32, FP16, and BF16 I measured in
Week 4. Device is where the tensor lives, the CPU or the GPU.

Instead of describing this, I print it. The first script builds a scalar, vector,
matrix, and 3-D tensor and reports each one's rank, shape, dtype, byte size, and
device. It then shows three operations I explain right after: an element-wise
add (position by position), a broadcast (a smaller tensor applied across a
larger one), and a host-to-device transfer (moving a tensor to the GPU):

#!/usr/bin/env python3
"""Week 5 - tensor basics: rank, shape, dtype, bytes, and device.

Shows a scalar, vector, matrix, and 3D tensor, then one element-wise
operation, one broadcast, and one CPU-to-GPU transfer. Every printed size and
byte count comes from PyTorch, not from a hand estimate.
"""

from __future__ import annotations

import torch


def describe(name: str, tensor: torch.Tensor) -> None:
    print(
        f"{name:8} rank={tensor.ndim} "
        f"shape={tuple(tensor.shape)} "
        f"dtype={str(tensor.dtype).replace('torch.', '')} "
        f"elem={tensor.element_size()}B "
        f"total={tensor.nbytes}B "
        f"dev={tensor.device}"
    )


def main() -> None:
    scalar = torch.tensor(3.0)
    vector = torch.tensor([1.0, 2.0, 3.0])
    matrix = torch.zeros(2, 3)
    tensor3d = torch.zeros(2, 3, 4)

    print("=== rank, shape, dtype, bytes, device ===")
    describe("scalar", scalar)
    describe("vector", vector)
    describe("matrix", matrix)
    describe("tensor3d", tensor3d)

    print("\n=== same shape, three precisions ===")
    for dtype in (torch.float32, torch.float16, torch.bfloat16):
        describe(str(dtype).replace("torch.", ""), torch.zeros(1024, 1024,
                                                                dtype=dtype))

    print("\n=== element-wise operation ===")
    a = torch.tensor([1.0, 2.0, 3.0])
    b = torch.tensor([10.0, 20.0, 30.0])
    print("a + b =", (a + b).tolist())

    print("\n=== broadcasting ===")
    matrix = torch.ones(2, 3)
    row = torch.tensor([1.0, 2.0, 3.0])
    print("matrix shape", tuple(matrix.shape), "+ row shape", tuple(row.shape))
    print("result:\n", (matrix + row).tolist())

    print("\n=== device transfer ===")
    if torch.cuda.is_available():
        cpu_tensor = torch.ones(3)
        gpu_tensor = cpu_tensor.to("cuda")
        print("cpu_tensor.device", cpu_tensor.device)
        print("gpu_tensor.device", gpu_tensor.device)
    else:
        print("CUDA not available; skipping GPU transfer")


if __name__ == "__main__":
    main()

I ran it on the Spark. These commands use public/... paths because I run them
from the parent repository. If you cloned the public companion repo, drop the
public/ prefix and run from that repo root.

ssh spark '~/venvs/w1/bin/python -' \
    < public/week-05-pytorch-tensors/tensor_basics.py

=== rank, shape, dtype, bytes, device ===
scalar   rank=0 shape=() dtype=float32 elem=4B total=4B dev=cpu
vector   rank=1 shape=(3,) dtype=float32 elem=4B total=12B dev=cpu
matrix   rank=2 shape=(2, 3) dtype=float32 elem=4B total=24B dev=cpu
tensor3d rank=3 shape=(2, 3, 4) dtype=float32 elem=4B total=96B dev=cpu

=== same shape, three precisions ===
float32  rank=2 shape=(1024, 1024) dtype=float32 elem=4B total=4194304B dev=cpu
float16  rank=2 shape=(1024, 1024) dtype=float16 elem=2B total=2097152B dev=cpu
bfloat16 rank=2 shape=(1024, 1024) dtype=bfloat16 elem=2B total=2097152B dev=cpu

=== element-wise operation ===
a + b = [11.0, 22.0, 33.0]

=== broadcasting ===
matrix shape (2, 3) + row shape (3,)
result:
 [[2.0, 3.0, 4.0], [2.0, 3.0, 4.0]]

=== device transfer ===
cpu_tensor.device cpu
gpu_tensor.device cuda:0

There is a lot in that output, so here is what matters.

Reading rank, shape, dtype, and device

The shape notation is worth decoding once. () means zero dimensions, a single
number. (3,) means one dimension holding three values, the trailing comma
marking it as a shape rather than a plain number. (2, 3) means two rows and
three columns. (2, 3, 4) adds a third dimension.

I never set a dtype in those constructors, and every tensor still came out
float32. That is PyTorch's default floating-point type. The precision block
below sets the dtype on purpose.

Byte size is Week 4's formula, now measured

Look at the vector: shape (3,), 4 bytes per element, 12 total bytes. That is
just 3 times 4. The matrix is 2 times 3 times 4, which is 24 bytes. The rule is
the same one from Week 4: total bytes equals element count times bytes per
element. PyTorch reports it directly with nbytes, so I no longer have to
estimate.

The three-precision block makes the point sharper. The same (1024, 1024) shape
is 4,194,304 bytes at FP32 but 2,097,152 bytes at FP16 and BF16, exactly half. A
dtype decides two things at once: the byte size of every number, and which values
the number can represent. This experiment measures the byte size. The next one
looks at the values.

FP16 and BF16 are the same size but not the same numbers

FP16 and BF16 both report 2 bytes, so they use the same memory. The difference is
how they split those 16 bits between range (how large a number can get) and step
size (how finely close numbers can be told apart). I read both directly with
torch.finfo:

ssh spark '~/venvs/w1/bin/python - <<PY
import torch
for name, dt in (("fp32", torch.float32),
                 ("fp16", torch.float16),
                 ("bf16", torch.bfloat16)):
    fi = torch.finfo(dt)
    print(f"{name:5} max={fi.max:.3e} smallest_step_near_1={fi.eps:.3e}")
PY'

fp32  max=3.403e+38 smallest_step_near_1=1.192e-07
fp16  max=6.550e+04 smallest_step_near_1=9.766e-04
bf16  max=3.390e+38 smallest_step_near_1=7.812e-03

FP16 tops out at about 65,500 but has the finer step. BF16 keeps almost FP32's
huge range but has a coarser step. That is the tradeoff in one line: FP16 gives
finer detail over a small range, BF16 gives a wide range with coarser detail.
Which one is better for training and inference is a later-week topic. This week I
only need to see that same size does not mean same numbers.

Element-wise operations and broadcasting

Two everyday operations show up constantly, so they are worth naming.

An element-wise operation works position by position. Adding [1, 2, 3] and
[10, 20, 30] gives [11, 22, 33]. Nothing mixes across positions.

Broadcasting lets PyTorch treat compatible dimensions as repeated, without
making a full copy. Here the row's length 3 matches the matrix's last dimension
of 3, so PyTorch adds the row to every row of the matrix. The output shows
[1, 2, 3] added to both rows of ones, giving [2, 3, 4] twice. This is how
model code adds the same bias vector to every row of a result in one line.

Device: CPU tensor versus CUDA tensor

The last block shows the CUDA-track idea in miniature. A tensor's device is
where PyTorch will run its operations, either the host (the CPU side) or a
CUDA GPU. cpu_tensor.device is cpu. After
.to("cuda"), gpu_tensor.device is cuda:0. That .to("cuda") is a
host-to-device transfer, and cuda:0 is device selection, naming the
first GPU. On the DGX Spark the CPU and GPU share one physical memory pool, so
this is not a copy across a separate VRAM. The output proves the device label
changed to cuda:0, which is what the GPU needs before it will compute on the
tensor. It does not measure transfer cost.

This week begins CUDA-track Level 1: pick a device, put your data on it, measure
correctly, and know what happens on out-of-memory. Level 1 continues over the
next few weeks. Week 1 already covered the driver, toolkit, and CUDA-enabled
PyTorch underneath it.

Precision changes speed, not just memory

Week 4 showed precision changes storage. Week 1 showed the GPU running a large
matrix multiply much faster than the CPU. Week 5 puts those together and adds the
missing dimension: precision also changes how fast the same operation runs.

The benchmark multiplies two square matrices many times per precision. Two
details keep GPU timing honest, both from the CUDA track. First a warmup: the
first GPU call pays a one-time setup cost, so I run several throwaway iterations
before timing. Second synchronization: CUDA runs work asynchronously, so
without torch.cuda.synchronize() I would be timing how long it takes to queue
the work, not to finish it. Here is the script:

#!/usr/bin/env python3
"""Week 5 - measure matmul speed and memory for FP32, FP16, and BF16.

Times a square matrix multiply on the GPU for three precisions with warmup and
CUDA synchronization, then compares one CPU run with one GPU run at FP32. All
timings use time.perf_counter around synchronized GPU work. Results print as a
table and optionally save to JSON.
"""

from __future__ import annotations

import argparse
import json
import statistics
import time
from pathlib import Path

import torch

DTYPES = {
    "fp32": torch.float32,
    "fp16": torch.float16,
    "bf16": torch.bfloat16,
}


def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--size", type=int, default=4096)
    parser.add_argument("--iters", type=int, default=20)
    parser.add_argument("--warmup", type=int, default=5)
    parser.add_argument("--output", type=Path)
    return parser.parse_args()


def time_matmul(a: torch.Tensor, b: torch.Tensor, iters: int,
                warmup: int) -> float:
    cuda = a.device.type == "cuda"
    for _ in range(warmup):
        a @ b
    if cuda:
        torch.cuda.synchronize()
    samples = []
    for _ in range(iters):
        start = time.perf_counter()
        a @ b
        if cuda:
            torch.cuda.synchronize()
        samples.append(time.perf_counter() - start)
    return statistics.median(samples)


def gflops(size: int, seconds: float) -> float:
    return (2 * size ** 3) / seconds / 1e9


def main() -> None:
    args = parse_args()
    size = args.size
    has_cuda = torch.cuda.is_available()
    device = "cuda" if has_cuda else "cpu"

    rows = []
    print(f"matmul {size}x{size}, iters={args.iters}, device={device}\n")
    print(f"{'precision':10} {'median_ms':>10} {'gflops':>10} "
          f"{'peak_mib':>10}")
    for name, dtype in DTYPES.items():
        if not has_cuda:
            continue
        torch.cuda.empty_cache()
        torch.cuda.reset_peak_memory_stats()
        a = torch.randn(size, size, device=device, dtype=dtype)
        b = torch.randn(size, size, device=device, dtype=dtype)
        seconds = time_matmul(a, b, args.iters, args.warmup)
        peak_mib = torch.cuda.max_memory_allocated() / (1024 ** 2)
        row = {
            "precision": name,
            "median_ms": round(seconds * 1000, 3),
            "gflops": round(gflops(size, seconds), 1),
            "peak_mib": round(peak_mib, 1),
        }
        rows.append(row)
        print(f"{name:10} {row['median_ms']:>10.3f} {row['gflops']:>10.1f} "
              f"{row['peak_mib']:>10.1f}")
        del a, b

    print("\n=== CPU versus GPU at FP32 ===")
    a_cpu = torch.randn(size, size)
    b_cpu = torch.randn(size, size)
    cpu_iters = max(3, args.iters // 4)
    cpu_s = time_matmul(a_cpu, b_cpu, cpu_iters, 1)
    print(f"cpu_fp32   {cpu_s * 1000:10.3f} ms   {gflops(size, cpu_s):8.1f} "
          f"gflops")
    speedup = None
    if has_cuda:
        gpu_fp32 = next(r for r in rows if r["precision"] == "fp32")
        speedup = round(cpu_s * 1000 / gpu_fp32["median_ms"], 1)
        print(f"gpu_fp32   {gpu_fp32['median_ms']:10.3f} ms   "
              f"{gpu_fp32['gflops']:8.1f} gflops")
        print(f"gpu_is     {speedup}x faster than cpu at fp32")

    if args.output:
        payload = {
            "size": size,
            "warmup": args.warmup,
            "gpu_iters": args.iters,
            "cpu_iters": cpu_iters,
            "device": device,
            "precisions": rows,
            "cpu_fp32_ms": round(cpu_s * 1000, 3),
            "gpu_speedup_fp32": speedup,
        }
        args.output.write_text(json.dumps(payload, indent=2) + "\n")


if __name__ == "__main__":
    main()

gflops reports billions of floating-point operations per second. A square
matmul of size n produces n^2 output cells, and each cell does about n
multiplications and n additions, so the total is about 2 * n^3 operations.
Dividing that by the seconds taken gives a speed I can compare across precisions.
The GPU results below are the median of 20 timed iterations after warmup; the CPU
result is the median of 5, since each CPU run is much slower. I ran it at 4096:

ssh spark '~/venvs/w1/bin/python - --size 4096 --iters 20' \
    < public/week-05-pytorch-tensors/benchmark_matmul.py

matmul 4096x4096, iters=20, device=cuda

precision   median_ms     gflops   peak_mib
fp32            7.440    18473.0      224.0
fp16            1.548    88807.9      128.0
bf16            1.576    87202.3      128.0

=== CPU versus GPU at FP32 ===
cpu_fp32      168.281 ms      816.7 gflops
gpu_fp32        7.440 ms    18473.0 gflops
gpu_is     22.6x faster than cpu at fp32

Two clear results.

First, the CPU-versus-GPU gap. The CPU does the FP32 matmul in about 168 ms. The
GB10 GPU does it in about 7.4 ms, roughly 22.6 times faster. That matches the
22.5x I measured in Week 1, a good sign the setup is consistent.

Second, the new result: precision changes speed. FP16 and BF16 finished the same
matmul in about 1.55 ms versus 7.44 ms for FP32, roughly 4.8 times faster, on the
exact same shape and hardware, just by using 16-bit numbers.

Memory dropped too. The two input matrices plus the temporary output matrix hold
192 MiB of payload at FP32 and 96 MiB at FP16 or BF16, exactly half. PyTorch's
measured peak, which is the most memory it had allocated at once (1 MiB is
1024 x 1024 bytes), fell from 224 MiB to 128 MiB. That is about 43 percent, not
exactly half, because the peak includes roughly 32 MiB of extra working
allocation on top of the three matrices.

FP16 and BF16 landed nearly equal in this run, about 1.55 versus 1.58 ms. They
are the same size and run at nearly the same speed here, and they can swap order
between runs. The choice between them is about the range-and-step tradeoff shown
earlier, not about performance on this test.

Results

Here is Week 5 in one place:

Item	Verified value
FP32 bytes per element	4
FP16 / BF16 bytes per element	2
1024x1024 FP32 tensor	4,194,304 bytes
1024x1024 FP16 / BF16 tensor	2,097,152 bytes
GPU FP32 matmul (4096)	7.44 ms, ~18,473 GFLOP/s
GPU FP16 matmul (4096)	1.55 ms, ~88,808 GFLOP/s
GPU BF16 matmul (4096)	1.58 ms, ~87,202 GFLOP/s
CPU FP32 matmul (4096)	168.28 ms, ~817 GFLOP/s
GPU vs CPU at FP32	about 22.6x faster
FP16 vs FP32 speed	about 4.8x faster
FP32 vs FP16 measured peak	224 MiB vs 128 MiB

What surprised me

The byte sizes matching Week 4 exactly was reassuring rather than surprising: a
1024x1024 FP32 tensor really is 4,194,304 bytes, no rounding.

The real surprise was that precision buys speed, not only memory. I already knew
16-bit numbers use half the bytes. Seeing the same matmul run almost five times
faster in FP16 made it clear why production inference leans on lower precision. It
saves memory and time at once, at least on this one operation.

Mistakes and troubleshooting

My first timing attempt forgot torch.cuda.synchronize(). Because CUDA runs work
asynchronously, I was timing how long it took to launch the matmul, not to finish
it, so the GPU looked far faster than it really is. Adding a warmup and a
synchronize per timed iteration fixed it. Week 1 shows the related effect: its
cold first GPU run took much longer than its warmed-up runs. GPU work is
asynchronous, so time it with a warmup and a sync or you time the wrong thing.

Single timings are also noisy, so the benchmark reports the median of 20 GPU
iterations and 5 CPU iterations. The absolute milliseconds shift between runs,
and FP16 and BF16 can trade places, but the groupings hold: 16-bit beats FP32,
and the GPU beats the CPU.

Production implications

This week explains a choice I will keep seeing: serving models in FP16 or BF16
instead of FP32. Lower precision halves the tensor payload and, on this hardware,
ran this dense matmul almost five times faster. That is one operation, not a
whole model. A real model also reads weights from memory and runs other steps, so
it will not simply be 4.8 times faster end to end. Even so, the direction is why
production inference leans on 16-bit precision: it saves both memory and time.

The CUDA-track habits matter for the rest of the series too. Correct GPU timing,
explicit device placement, and watching peak memory are the starting tools.
They measure elapsed time and capacity, not utilization or bandwidth, so they
cannot yet classify a workload as compute-bound or memory-bound. Later CUDA-track
profiling adds those measurements.

What I will learn next

Week 6 steps up from single tensors to a tiny neural network: inputs, a linear
layer, an activation, a loss, and a training loop small enough to watch the
weights change. That connects this week's tensor operations to how a model
actually learns.²

Run it yourself

The public Week 5 lab has the tensor-basics script, the matmul benchmark, the
captured results, observations, and troubleshooting notes.³

CUDA track (parallel), Level 1:
https://github.com/dramasamy/from-api-to-gpu/blob/main/roadmap/cuda-track.md ↩
Week 6 roadmap:
https://github.com/dramasamy/from-api-to-gpu/blob/main/roadmap/week-06.md ↩
Week 5 companion lab:
https://github.com/dramasamy/from-api-to-gpu/tree/main/week-05-pytorch-tensors ↩

From API to GPU, Week 4: What Model Parameters Actually Cost in Memory

Dinesh Kumar Ramasamy — Mon, 20 Jul 2026 00:54:57 +0000

Phase 1 of 8: Comfortable running local models. Week 4 of 32.

People throw around "7B", "14B", and "70B" like everyone knows what they mean.
For a long time I nodded along without really knowing. This week I make those
numbers concrete. By the end I can look at a model size and a precision and say
how much memory the weights need, then explain why the running model needs more.

This is the last week of Phase 1. In Week 1 I learned the machine, in Week 2 I
ran a model through Ollama, and in Week 3 I read a model's files on Hugging Face.
Now I connect the parameter count I saw in Week 3 to real memory on the DGX
Spark.

The calculator part runs anywhere with Python. The memory measurements run on
the Spark over ssh spark, the same setup as earlier weeks.

What a parameter actually is

A parameter is a single learned number inside the model. Training adjusts
billions of these numbers until the model predicts text well. When a model is
called "3B", it has about 3 billion of these numbers.

Wait, is a parameter the same thing as a token? No. A token is a piece of
text that enters or leaves the model. Tokens are the request and response data;
parameters are the learned numbers stored in the model files.

	Token	Parameter
What it is	a piece of input or generated text	a learned number
Where it comes from	the prompt or model response	training
What happens during inference	input tokens arrive and output tokens are added	values stay fixed
What its count affects here	context and KV-cache use	weight memory

They are connected, though. The tokenizer turns each token into an ID. The
model uses learned parameters to turn those IDs into internal values, process
them, and predict the next token. A 3B model therefore has about 3 billion
parameters, not 3 billion tokens. The same parameters are reused for every
token the model processes.

Parameter is the umbrella term. Two kinds sit under it:

A weight multiplies an input value. Most parameters are weights.
A bias is another learned value added after the multiply. Models usually have far fewer bias entries than weight entries.

The reported parameter count includes both. So when I say "weight memory" in this
post, I mean the storage for all parameters, weights and biases together. The
count is what drives the memory math.

A few more words show up when you look inside a model, so here they are in plain
terms:

A tensor is a block of numbers. A single number is a scalar, a list is a vector, a grid is a matrix, and a tensor is the general name for any of these.
A matrix is a two-dimensional grid of numbers, the most common shape for a block of weights.
A layer is one processing stage. A model stacks many layers, and each one holds its own tensors.
A shape is the size of a tensor along each dimension, like 2048 by 2048.

stack of layers that forms a language model." width="800" height="533">

I do not need the internals of these this week. Week 5 works with tensors
directly in PyTorch, and Week 11 explains layers. Here they are just the words
for "the numbers we are about to measure".

In Week 3 I read the parameter count from the Hugging Face API. Here I pin the
API request to the same commit used for the weight index later in this post:

$ MODEL=Qwen/Qwen2.5-3B-Instruct
$ REV=aa8e72537993ba99e69dfaafa59ed015b17504d1
$ curl -s "https://huggingface.co/api/models/$MODEL/revision/$REV" | jq '{
    revision: .sha,
    parameters: .safetensors.total,
    precision_groups: .safetensors.parameters
}'
{
  "revision": "aa8e72537993ba99e69dfaafa59ed015b17504d1",
  "parameters": 3085938688,
  "precision_groups": {
    "BF16": 3085938688
  }
}

So this model has 3,085,938,688 parameters, and every one of them is stored in
the BF16 format. That second fact is the other half of the memory question.

Precision: how many bytes each number takes

A parameter is a number, and a number needs storage. Precision is the format
used to store each one. The format decides how many bits, and therefore how many
bytes, every parameter takes.

The short forms are worth spelling out. FP means floating point, a format for
fractional numbers. BF16 means bfloat16, a 16-bit floating-point format. INT
means integer. You saw torch_dtype: bfloat16 in the Week 3 config. Here is what
the common formats cost per parameter:

Precision	Bits	Bytes
FP32	32	4
FP16	16	2
BF16	16	2
INT8	8	1
INT4	4	0.5

FP32 is the 32-bit baseline, using 4 bytes per number. FP16 and BF16 are both
16-bit formats, so both use 2 bytes. INT8 and INT4 are the low-bit formats
produced by quantization, which is a whole phase later in this series (weeks 14
to 16). INT4's 0.5 byte is an average: two 4-bit values pack into one byte. For
now the only thing that matters is the bytes column.

GB versus GiB, before the numbers start

Memory numbers come in two units, and mixing them causes confusion, so I define
them once up front:

GB is decimal: divide bytes by 1,000,000,000.
GiB is binary: divide bytes by 1024 three times (1024 x 1024 x 1024).

The calculator prints both. Ollama later prints its own rounded GB labels, so
those are approximate runtime reports, not byte-exact comparisons.

The weight-memory formula

Put the two facts together and the weight memory is simple multiplication:

weight bytes = parameter count \times \frac{bits per parameter}{8}

Dividing by 8 converts bits to bytes. That is the whole formula. The Week 4
deliverable is a small calculator that applies it, so I do not redo the
arithmetic by hand each time.

Here is the complete script:

#!/usr/bin/env python3
"""Estimate model weight memory from a parameter count and a precision.

The math is exact for a single uniform precision:

    weight_bytes = parameter_count * bits_per_parameter / 8

This is the tensor payload only. Real runtime memory is larger because of the
KV cache, activations, and framework overhead, and it cannot be derived from the
parameter count. The blog measures that runtime memory separately.
"""

from __future__ import annotations

import argparse

BITS_PER_PARAMETER = {
    "fp32": 32,
    "fp16": 16,
    "bf16": 16,
    "int8": 8,
    "int4": 4,
}

GB = 1_000_000_000
GIB = 1024 ** 3


def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser(description=__doc__)
    group = parser.add_mutually_exclusive_group(required=True)
    group.add_argument(
        "--parameters",
        type=float,
        help="Parameter count in billions, e.g. 70 or 3.2",
    )
    group.add_argument(
        "--count",
        type=int,
        help="Exact parameter count, e.g. 3085938688",
    )
    parser.add_argument(
        "--precision",
        choices=sorted(BITS_PER_PARAMETER),
        required=True,
    )
    return parser.parse_args()


def weight_bytes(parameter_count: int, precision: str) -> float:
    return parameter_count * BITS_PER_PARAMETER[precision] / 8


def main() -> None:
    args = parse_args()
    if args.count is not None:
        parameter_count = args.count
    else:
        parameter_count = round(args.parameters * 1_000_000_000)

    raw_bytes = weight_bytes(parameter_count, args.precision)

    print(f"parameter_count    {parameter_count:,}")
    print(f"precision          {args.precision}")
    print(f"bits_per_parameter {BITS_PER_PARAMETER[args.precision]}")
    print(f"weight_bytes       {raw_bytes:,.0f}")
    print(f"weight_gb          {raw_bytes / GB:.2f} GB (decimal, / 1e9)")
    print(f"weight_gib         {raw_bytes / GIB:.2f} GiB (binary, / 1024^3)")


if __name__ == "__main__":
    main()

I run the roadmap example first: a 70-billion-parameter model at INT4.

$ python3 public/week-04-parameters-memory/model_memory.py \
  --parameters 70 --precision int4
parameter_count    70,000,000,000
precision          int4
bits_per_parameter 4
weight_bytes       35,000,000,000
weight_gb          35.00 GB (decimal, / 1e9)
weight_gib         32.60 GiB (binary, / 1024^3)

70 billion parameters at half a byte each is 35 GB of weights. The same 70B
model at BF16 would be four times that, about 140 GB. That is roughly 130 GiB,
already more than the Spark's 121 GiB pool before any runtime memory. This is
why big models are so often quantized before anyone tries to run them locally.

Checking the formula against a real file

A formula is only trustworthy if it matches reality. I ran the calculator with
the exact Qwen count and its real BF16 precision:

$ python3 public/week-04-parameters-memory/model_memory.py \
  --count 3085938688 --precision bf16
parameter_count    3,085,938,688
precision          bf16
bits_per_parameter 16
weight_bytes       6,171,877,376
weight_gb          6.17 GB (decimal, / 1e9)
weight_gib         5.75 GiB (binary, / 1024^3)

The calculator says 6,171,877,376 bytes. Now the model's own Safetensors index,
read directly with the pinned revision from Week 3:

$ MODEL=Qwen/Qwen2.5-3B-Instruct
$ REV=aa8e72537993ba99e69dfaafa59ed015b17504d1
$ curl -sL "https://huggingface.co/$MODEL/raw/$REV/model.safetensors.index.json" \
    | jq '.metadata.total_size'
6171877376

Identical. The formula is not a guess. For a uniform precision it lands exactly
on the tensor-data bytes the index reports. That total is the parameter payload,
not the whole file with its headers, but for the weight math it is the number I
want.

Why the running model needs more than its weights

Everything so far is weight storage: the model sitting on disk. A model doing
work needs more memory than that. The extra memory has three main parts:

Activation memory is the temporary numbers created while processing a request. They come and go as the model runs.
KV-cache memory stores saved intermediate data from earlier tokens, so the model does not recompute them for every new token. This grows with context length. The KV cache is a Week 18 topic; here I only need to see that it exists and costs memory.
Framework overhead is the runtime's own working memory and buffers.

The exact-file check used BF16 Qwen. This runtime experiment switches to
Ollama's quantized llama3.2:3b, the model already available from Week 2. Its
2.0 GB package, 2.6 GB load, and 4.1 GB load belong to one comparison; they
should not be compared with Qwen's 6.17 GB BF16 weights.

Ollama reports the loaded size of a model, so I can compare it with the 2.0 GB
package size. I expect the 4,096-token load to exceed 2.0 GB because the runtime
needs memory beyond the package. I expect the 16,384-token load to be larger
again because it reserves more KV-cache capacity. ollama ps reports the total,
so this experiment can show the change but cannot assign an exact number of
bytes to the KV cache.

I connect to the Spark once, then run each check directly:

$ ssh spark
$ ollama list | awk 'NR == 1 || $1 == "llama3.2:3b"'
NAME                       ID              SIZE      MODIFIED
llama3.2:3b                a80c4f17acd5    2.0 GB    3 days ago

Next I unload any existing copy, send one short request with a 4,096-token
context, and inspect the loaded model. The API's HTTP 200 confirms that the
otherwise silent request completed:

$ ollama stop llama3.2:3b >/dev/null 2>&1 || true
$ curl -fsS -o /dev/null -w 'HTTP %{http_code}\n' \
    http://localhost:11434/api/generate \
    -d '{"model":"llama3.2:3b","prompt":"hi","stream":false,
    "options":{"num_ctx":4096,"num_predict":1}}'
HTTP 200
$ ollama ps
NAME           ID              SIZE      PROCESSOR    CONTEXT    UNTIL
llama3.2:3b    a80c4f17acd5    2.6 GB    100% GPU     4096       4 minutes from now

I repeat the same request with a 16,384-token context:

$ ollama stop llama3.2:3b >/dev/null 2>&1 || true
$ curl -fsS -o /dev/null -w 'HTTP %{http_code}\n' \
    http://localhost:11434/api/generate \
    -d '{"model":"llama3.2:3b","prompt":"hi","stream":false,
    "options":{"num_ctx":16384,"num_predict":1}}'
HTTP 200
$ ollama ps
NAME           ID              SIZE      PROCESSOR    CONTEXT    UNTIL
llama3.2:3b    a80c4f17acd5    4.1 GB    100% GPU     16384      4 minutes from now

Ollama lists the package as 2.0 GB. Loaded with a 4,096-token context, it reports
2.6 GB. Loaded with a 16,384-token context, it reports 4.1 GB. The model did not
change. Only the configured context capacity changed, and the reported loaded
size grew by 1.5 GB. The KV cache grows with context, so it is the expected main
cause, but ollama ps reports one total and does not split KV cache, activations,
and framework buffers.

This is the practical lesson: weight memory is the floor, not the total. Runtime
memory is larger, and context length is one of the settings that pushes it up.

Results

Here is Week 4 in one place:

Item	Verified value
Formula	parameters × bits / 8
70B at INT4	35.00 GB of weights
Qwen2.5-3B-Instruct at BF16	6,171,877,376 bytes
Formula versus Safetensors index	exact match
Loaded at 4,096 context	2.6 GB
Loaded at 16,384 context	4.1 GB
Spark unified memory	121 GiB

What surprised me

The formula matching the Safetensors index to the exact byte was satisfying.
Weight memory really is just multiplication once you know the count and the
precision.

The context effect was the most useful. I knew the KV cache existed from Week 2,
but seeing a 2.0 GB package report 4.1 GB of loaded memory just by raising
context made it concrete.

Mistakes and troubleshooting

I also had to be careful about GB versus GiB. The calculator prints both, and
Ollama prints its own rounded GB labels, so I keep the units visible to avoid
comparing the wrong things.

Production implications

Weight memory sets the floor for whether a model fits. For capacity planning I
would:

Estimate weights from the count and precision, then confirm against the real file for quantized models.
Measure total runtime memory with the real prompt lengths and workload, since the KV cache and activations depend on how the model is actually used, not on the parameter count.
Treat the Spark's 121 GiB unified pool as shared by the model, its runtime, and everything else on the box.

The formula tells me if the weights fit. The runtime measurement tells me if the
working model fits. Both matter before scheduling hardware.

What I will learn next

Week 5 moves into PyTorch tensors directly. I will allocate tensors on the CPU
and GPU, compare FP32, FP16, and BF16, and watch allocated memory change. That
turns this week's byte math into something I can measure inside a running Python
process.¹

Weeks 14 to 16 cover quantization properly, which is where the Q4_K_M mixed
format and its real bit cost get explained in full.

Run it yourself

The public Week 4 lab has the calculator, direct runtime commands, captured
results, observations, and troubleshooting notes.² The
quantized-file note holds the optional Q4_K_M storage investigation with its
full commands and output.³

Optional depth: why Q4 uses more than four bits on disk

The clean formula assumes every parameter uses the same number of bits. A
quantized file can mix several tensor formats and store extra scale data, so a
model labeled "4-bit" may use more than 4 bits per parameter on disk.

I checked Ollama's llama3.2:3b package. Its API reports 3,212,749,888
parameters and Q4_K_M quantization.⁴ A clean INT4 calculation predicts
1,606,374,944 bytes, but the model blob is 2,019,377,376 bytes. That works out
to about 5.03 on-disk bits per parameter. I also matched the blob's byte size
and SHA-256 digest to its manifest record, so this is the exact file Ollama
references, not only a filename that looks right.

This was the opposite of the uniform BF16 result. I expected a "4-bit" model to
cost 4 bits per parameter, but this file is about 26 percent larger than the
clean estimate. The fix was to use the manifest and real blob instead of treating
the quantization label as an exact file-size promise.

The companion note keeps the full manifest-to-blob chain and tensor-type
counts.³ For Week 4, the rule is enough: use the formula for a
first estimate, then read the real file size for a quantized model.

Week 5 roadmap:
https://github.com/dramasamy/from-api-to-gpu/blob/main/roadmap/week-05.md ↩
Week 4 companion lab:
https://github.com/dramasamy/from-api-to-gpu/tree/main/week-04-parameters-memory ↩
Why a Q4 model uses more than four bits per parameter:
https://github.com/dramasamy/from-api-to-gpu/blob/main/week-04-parameters-memory/quantized-file-size.md ↩
GGUF and its K-quant mixed formats are from the llama.cpp project:
https://github.com/ggml-org/ggml/blob/master/docs/gguf.md ↩

From API to GPU, Week 3: Reading a Hugging Face Model Repository

Dinesh Kumar Ramasamy — Sun, 19 Jul 2026 23:00:44 +0000

Phase 1 of 8: Comfortable running local models. Week 3 of 32.

In Week 2 I ran Phi-4 through Ollama. Ollama gave me a friendly model name and
hid most of the files underneath it. This week I remove that layer and inspect a
model at its source: its Hugging Face page.

The natural way to get to know a model is to open its web page, read the model
card, and click through the files. So that is exactly how I start here, in an
ordinary browser. By the end I can answer four questions before I download
anything:

What architecture and size is this model?
Is it a base model or an instruction-tuned model?
How much data will I download?
Am I legally allowed to use it for my product?

I inspect Qwen/Qwen2.5-3B-Instruct. Everything in this post is done by
browsing its Hugging Face page: no GPU, no API key, no account. The companion
lab repeats every one of these answers as reproducible curl and jq commands
plus a small Python inspector, the "API way", for anyone who wants to script
it.¹

The model card page is the front door

I open the model's page in a browser.² Before reading any file,
the top of the page already answers a lot.

name, the tag row, the license badge, and the sidebar showing model size and
tensor type." width="800" height="320">

Reading the header top to bottom:

Qwen / Qwen2.5-3B-Instruct is the repository: the owner (Qwen) and the model name. This is the exact ID I hand to any tool that downloads the model.
The tag row shows the task (Text Generation), the framework (Transformers), the weight format (Safetensors, a model-weight file format), and a chat tag.
License: qwen-research is a license label, not the license itself. I come back to it below, because the label alone does not tell me whether I can use this commercially.
The three tabs are Model card, Files and versions, and Community. The whole week lives in the first two.
The right sidebar already says Model size 3B params, Tensor type BF16 (bfloat16, a 16-bit number format), and Chat template. So before opening a single file I know this is a three-billion-parameter model, stored in a 16-bit format, that ships a chat template.

The four names I kept mixing up

The page uses four words for different parts of the same release, and I kept
mixing them up at first:

Term	Plain meaning	Where it shows on the page
Model family	related models released together	the `Qwen2.5` name and collection
Architecture	the neural-network layout	the `config.json` file (opened below)
Checkpoint	the learned parameter values	the `.safetensors` weight files
Repository	the versioned files that ship the model	the whole page, owner plus name

The repository is the page itself: the owner Qwen plus the name
Qwen2.5-3B-Instruct. A Hugging Face repository is a Git repository designed to
hold large ML files, so it carries a full commit history like any other
repo.³

The model family is Qwen2.5. The name and its linked collection show the
family includes several sizes plus base and instruction-tuned
variants.⁴

The architecture is the neural-network layout. The page does not print it in
the header; it lives in config.json, which I open later. Architecture describes
the shape (layers, hidden width, attention heads), not the learned knowledge.

A checkpoint is the learned numbers stored in the weight files. Two models
can share the same architecture but hold different checkpoints, just as two
containers can run the same application build with different data. The base and
instruct models are exactly that: same shape, different learned weights.

Base model versus instruct model

A base model learns to predict the next token from a large training set. It
is useful as a starting point for more training, but it is not automatically a
good chat assistant.

An instruct model starts from a base checkpoint and receives post-training
that teaches it to follow requests and behave better in conversations. The
model card is the README.md shown on the Model card tab. It describes what
the model is, how it was trained, and its limits. Scrolling down the card, the
model states plainly what it is:

parameter count, layer count, attention-head layout, and context
length." width="799" height="434">

The card says "This repo contains the instruction-tuned 3B Qwen2.5 model" and
lists Training Stage: Pretraining & Post-training. That post-training step is
what makes it an instruct model. The same block answers most of question 1 too:
3.09B parameters, 36 layers, 16 query heads and 2 key/value heads
(more on that split in Week 10 and Week 13), and a native 32,768-token
context length.

The card is a claim by the publisher. The page backs it with a machine-readable
link. The Model tree box on the same page names the base model this one was
fine-tuned from:

repository as a fine-tune of it." width="800" height="343">

So the lineage is explicit: Qwen/Qwen2.5-3B-Instruct is a fine-tune of
Qwen/Qwen2.5-3B.⁵² Fine-tuning means continuing
training from an existing checkpoint; here it is the post-training that turns the
base model into an instruction follower. The architecture stays the same while
post-training changes the weights. The companion lab confirms the weights really
differ by comparing the two checkpoints' Hub-reported file fingerprints, but the
model card and the Model tree already answer the question for a normal user.

The license is the first thing to check

Question 4 (am I allowed to use this?) is the one I answer before running
anything, because it can stop a project before it starts. The header badge says
qwen-research, a non-standard license, so the badge alone is not enough. I
click the LICENSE file on the Files tab and read it:

NON-COMMERCIAL PURPOSES ONLY and requiring a separate license for commercial
use." width="800" height="600">

The LICENSE file is the Qwen Research License Agreement. Line 19 grants use
"FOR NON-COMMERCIAL PURPOSES ONLY", and the next line says commercial use
requires a separate license from Alibaba Cloud.⁶

That is the most important result this week. The model page looks ready to use,
but this 3B release is not licensed for commercial use by default. License review
has to happen before model evaluation, not after a prototype is built. The Hub's
machine-readable label for this is license: other, a non-standard license, a
detail I show in the lab.

The Files and versions tab

The Files and versions tab lists everything the repository ships. This is
where question 3 (how much will I download?) gets answered:

of 6.18 GB, and the latest commit on the main branch." width="800" height="600">

The tab shows a total of 6.18 GB across 12 files, and the header line
records the branch (main), the contributor count, and the latest commit. To
make the list easier to learn, I group the twelve files into six groups, which
is just my reading order, not anything Hugging Face labels:

Docs and license: README.md (the model card) and LICENSE.
Model shape: config.json.
Generation defaults: generation_config.json.
Tokenizer: tokenizer.json, tokenizer_config.json, vocab.json, merges.txt.
Weights: model-00001-of-00002.safetensors, model-00002-of-00002.safetensors, and their index model.safetensors.index.json.
Git plumbing: .gitattributes, which configures Git LFS and is not model data.

Clicking any file opens it in the browser. The rest of this post opens the small
ones and reads them.

`config.json` is the architecture contract

config.json tells a runtime how to build the model before loading any weights.
It is tiny, 661 bytes, so I open it and read the whole thing:

{
  "architectures": [
    "Qwen2ForCausalLM"
  ],
  "attention_dropout": 0.0,
  "bos_token_id": 151643,
  "eos_token_id": 151645,
  "hidden_act": "silu",
  "hidden_size": 2048,
  "initializer_range": 0.02,
  "intermediate_size": 11008,
  "max_position_embeddings": 32768,
  "max_window_layers": 70,
  "model_type": "qwen2",
  "num_attention_heads": 16,
  "num_hidden_layers": 36,
  "num_key_value_heads": 2,
  "rms_norm_eps": 1e-06,
  "rope_theta": 1000000.0,
  "sliding_window": 32768,
  "tie_word_embeddings": true,
  "torch_dtype": "bfloat16",
  "transformers_version": "4.43.1",
  "use_cache": true,
  "use_sliding_window": false,
  "vocab_size": 151936
}

This is where the architecture name lives: Qwen2ForCausalLM. The header did
not print it, but the file does. I do not need to understand every field in
Week 3. My goal today is to find and record them, and their deeper lessons are
already scheduled:

Week 4 turns torch_dtype and the parameter count into weight-memory math. Week 5 compares FP32, FP16, and BF16 tensors on CPU and GPU.
Week 9 explains vocab_size, tokenizer files, and special token IDs.
Week 10 explains attention heads. Week 13 returns to why this model has 16 query heads (num_attention_heads) but only 2 key/value heads (num_key_value_heads).
Week 11 explains num_hidden_layers, hidden_size, and intermediate_size, and how the architecture class builds transformer blocks from them.
Week 18 compares max_position_embeddings with the tokenizer limit, the runtime setting, and the context length that is practical in memory.

The important distinction is simpler than the individual fields: config.json
describes the model's shape, while the checkpoint files contain the learned
numbers that fill that shape.

`generation_config.json` supplies defaults, not hard limits

This file holds the default sampling settings for text generation. It is 242
bytes, so again I open the whole thing:

{
  "bos_token_id": 151643,
  "pad_token_id": 151643,
  "do_sample": true,
  "eos_token_id": [
    151645,
    151643
  ],
  "repetition_penalty": 1.05,
  "temperature": 0.7,
  "top_p": 0.8,
  "top_k": 20,
  "transformers_version": "4.37.0"
}

These are defaults, not training facts, and a runtime or API request can override
them. I changed generation temperature in Week 2 and briefly introduced top-p
there. Week 12 is the full generation-and-sampling lesson, where I change top-k,
top-p, temperature, repetition penalty, and seed one at a time. In Week 3 I only
need to know that generation_config.json is where this repository stores its
defaults, and that it names IDs 151643 and 151645 as end-of-generation
markers. The field names spell out the roles: bos_token_id is the
beginning-of-sequence (BOS) marker, eos_token_id the end-of-sequence (EOS)
marker, and pad_token_id the padding marker. One ID can fill several roles:
here 151643 is BOS and padding and is also one of the two accepted EOS IDs.

The tokenizer is several files working together

The tokenizer turns text into token IDs. This repository ships it as several
files, all visible on the Files tab:

vocab.json maps token text to IDs.
merges.txt holds byte-pair encoding merge rules.
tokenizer_config.json stores the special tokens and the chat template.
tokenizer.json packages the whole tokenizer into one file for fast loading.

Byte-pair encoding (BPE) builds tokens by repeatedly joining common text
pieces according to the merge rules.⁷ This is one tokenizer method, not a
universal standard. Other model families can use different algorithms and file
layouts.

tokenizer_config.json is larger than the config files, so instead of reading it
top to bottom I look for two things: the special tokens and the chat template.
Opening it and scrolling to the tokenizer class and the added_tokens_decoder
map, the fields that matter here are:

{
  "tokenizer_class": "Qwen2Tokenizer",
  "model_max_length": 131072,
  "added_tokens_decoder": {
    "151643": { "content": "<|endoftext|>", "special": true },
    "151644": { "content": "<|im_start|>", "special": true },
    "151645": { "content": "<|im_end|>", "special": true }
  }
}

That maps the three special tokens that matter here:

Token ID	Text	Role
151643	`<	endoftext
151644	{% raw %}`<	im_start
151645	{% raw %}`<	im_end

This answers a question left over from Week 2. With Phi-4 in Ollama I saw
{% raw %}<|im_start|>, <|im_end|>, and <|im_sep|> inside the packaged GGUF model.
Now I can see where the source repository keeps that information:
generation_config.json names which IDs act as start, end, and padding markers,
and tokenizer_config.json maps those IDs to text and stores the chat template
that places them around messages. The values belong to each model's tokenizer:
Phi-4 used IDs 100264 to 100266, Qwen uses 151643 to 151645, and Qwen has no
<|im_sep|>. I should never copy special-token IDs or assume the same chat
format across model families.

The chat template is a template written in Jinja, a text templating language,
stored inside tokenizer_config.json.⁸ It converts role-based messages
into the special-token sequence the model expects, the same job as the ChatML
markers I inspected in Week 2.⁹ The full template is worth a
look, so I show it after the file tour in
the chat-template walkthrough.

One number on the tokenizer looks contradictory at first. tokenizer_config.json
sets model_max_length to 131,072, while config.json says 32,768. The two
fields describe different things: model_max_length is tokenizer metadata (a
safety cap on input length), while max_position_embeddings in config.json is
this checkpoint's native architecture limit. The model card's introduction
mentions the wider Qwen2.5 family supports up to 128K tokens, but this specific
3B checkpoint lists a native 32,768-token context, which matches config.json.
The 131,072 setting is not proof that this checkpoint can safely use that length,
so I treat 32,768 as the verified native limit.²

Safetensors files are the checkpoint

The two .safetensors files hold the learned tensors, the multidimensional
arrays of numbers that are the model's parameters. Safetensors is a weight-file
format designed to load those arrays without the code-execution risk of Python's
Pickle format.¹⁰ It is one option, not the only one. Week 2 used
GGUF, a different format that packages weights and metadata together.

The checkpoint is split into two shards, model-00001-of-00002.safetensors
(3.97 GB) and model-00002-of-00002.safetensors (2.2 GB). The third weight file,
model.safetensors.index.json, is not weights at all: it is a map from each
tensor name to the shard that holds it. Opening it, the top is a metadata block
followed by a weight_map with one entry per tensor:

{
  "metadata": { "total_size": 6171877376 },
  "weight_map": {
    "model.embed_tokens.weight": "model-00001-of-00002.safetensors",
    "model.layers.0.input_layernorm.weight": "model-00001-of-00002.safetensors"
  }
}

The real weight_map has 434 entries (I show two); that count is how many
tensors a loader must place. Splitting one checkpoint into shards just makes it
easier to upload, cache, and download; both shards are required, and two files do
not mean two models.

The metadata.total_size field records the total tensor size: 6,171,877,376
bytes. That number is a good check on the "3B params" and "BF16" labels from the
sidebar. BF16 means each parameter uses 16 bits, or 2 bytes,¹¹ so:

6, 171, 877, 376 bytes \div 2 = 3, 085, 938, 688 parameters

That is 3.09 billion parameters, exactly what the model card rounds to. Next week
I turn this calculation into a reusable memory tool.

How do I know it is really an instruct model?

There is no universal is_instruct: true field. Here is the evidence the page
gives, strongest first:

Publisher statement: the model card says "instruction-tuned" and "Pretraining & Post-training". That is why I call it an instruct model.
Model relationship: the Model tree names the base model as Qwen/Qwen2.5-3B, and the repository carries the chat tag.
Different learned checkpoint: the companion lab reads only the file fingerprints and shows the base and instruct first shards have different SHA-256 hashes, which proves the weight bytes differ.¹²
Chat-oriented defaults: the instruct generation_config.json enables sampling and lists <|im_end|> as an end token, which the base model does not.

Most architecture values are identical between the two, because post-training
changes the learned weights, not the model's shape. Both still have 36 layers
and the same parameter count. Confirming that with real file hashes is the lab's
job; the model card and the Model tree already answer it for day-to-day use.

The Qwen chat template, connected to Week 2

This section is optional depth. The file tour already gave me what I need for
Week 3, but the full template makes the Week 2 roles and markers concrete.

The template is stored as one long string inside tokenizer_config.json. Opening
that file in the browser shows it escaped on a single line; the lab prints it
cleanly with one jq command. Formatted, it reads:

{%- if tools %}
    {{- '<|im_start|>system\n' }}
    {%- if messages[0]['role'] == 'system' %}
        {{- messages[0]['content'] }}
    {%- else %}
        {{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }}
    {%- endif %}
    {{- "\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
    {%- for tool in tools %}
        {{- "\n" }}
        {{- tool | tojson }}
    {%- endfor %}
    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
{%- else %}
    {%- if messages[0]['role'] == 'system' %}
        {{- '<|im_start|>system\n' + messages[0]['content'] + '<|im_end|>\n' }}
    {%- else %}
        {{- '<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n' }}
    {%- endif %}
{%- endif %}
{%- for message in messages %}
    {%- if (message.role == "user") or (message.role == "system" and not loop.first) or (message.role == "assistant" and not message.tool_calls) %}
        {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
    {%- elif message.role == "assistant" %}
        {{- '<|im_start|>' + message.role }}
        {%- if message.content %}
            {{- '\n' + message.content }}
        {%- endif %}
        {%- for tool_call in message.tool_calls %}
            {%- if tool_call.function is defined %}
                {%- set tool_call = tool_call.function %}
            {%- endif %}
            {{- '\n<tool_call>\n{"name": "' }}
            {{- tool_call.name }}
            {{- '", "arguments": ' }}
            {{- tool_call.arguments | tojson }}
            {{- '}\n</tool_call>' }}
        {%- endfor %}
        {{- '<|im_end|>\n' }}
    {%- elif message.role == "tool" %}
        {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != "tool") %}
            {{- '<|im_start|>user' }}
        {%- endif %}
        {{- '\n<tool_response>\n' }}
        {{- message.content }}
        {{- '\n</tool_response>' }}
        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
            {{- '<|im_end|>\n' }}
        {%- endif %}
    {%- endif %}
{%- endfor %}
{%- if add_generation_prompt %}
    {{- '<|im_start|>assistant\n' }}
{%- endif %}

I do not need to understand every branch yet. The four parts that connect to
Week 2 are:

A system message becomes <|im_start|>system, its content, then <|im_end|>.
The loop does the same for user and assistant messages.
If no system message is supplied, the template inserts Qwen's default system message.
add_generation_prompt ends with <|im_start|>assistant, telling the model which role should speak next.

The first branch also formats tool calls. I leave that branch for Week 27. The
important point here is that the API's role-based messages do not go straight to
the model. The tokenizer's chat template turns them into the model-specific text
and special-token sequence first.

Results

Browsing the page answered every item from the Week 3 plan:

Question	Verified answer
Repository	`Qwen/Qwen2.5-3B-Instruct`
Latest commit	`aa8e725` (shown on the Files tab)
Architecture	`Qwen2ForCausalLM`
Parameters	3,085,938,688 BF16 values
Layers	36
Hidden size	2,048
Attention configuration	16 query heads, 2 shared key/value heads
Vocabulary	151,936 token IDs
Native configured context	32,768 tokens
BOS ID	151643
EOS IDs	151645 and 151643
Weight shards	2 files, 6,171,877,376 tensor bytes
License	Qwen Research License, non-commercial

The machine-readable copy is saved as results.json in the public lab.

What surprised me

The biggest surprise was the license. A page that looks production-ready can
still be non-commercial by default, and only the LICENSE file says so. Reading
it first is now a habit, not an afterthought.

The second surprise was how much the web page alone answered. Without a GPU, an
API key, or a line of code, I read the architecture, the size, the base-model
lineage, the tokenizer's special tokens, and a 6.18 GB download estimate straight
off the page.

The third surprise was that the round "3B" label and the exact file size agree.
The index reports 6,171,877,376 bytes; divided by 2 bytes per BF16 value, that is
exactly 3,085,938,688 parameters, which is what the card rounds to 3.09B.

Mistakes and troubleshooting

My first instinct was to trust the license badge. The header says qwen-research,
which sounds like a name I could look up and move on. Opening the actual LICENSE
file is what surfaced the non-commercial restriction. The badge is a label; the
file is the contract.

I also assumed the sidebar's "3B params" and the card's "3.09B" were rounded
marketing numbers. The index's exact byte count showed otherwise: the round label
and the precise file size describe the same 3,085,938,688 parameters.

The lab's curl and jq version hits a few API-specific snags (the Hub reports
the license as other, the card object needs to_dict(), and unauthenticated
requests print a rate-limit warning). Those are written up in the lab's
troubleshooting notes, since they only matter for the scripted path.

Production implications

Model selection is dependency management with much larger artifacts. I would
apply the same controls used for containers and packages:

Pin the full repository revision, not mutable main.
Record every weight hash and verify it after download.
Review the model card, then read the actual license.
Estimate download and memory size before scheduling hardware.
Review repository code before enabling trust_remote_code.
Keep tokenizer and generation files with the exact checkpoint they came from.

This repository contains no Python model files, so the metadata inspection did
not execute repository code. That is a useful security property, but I would
check it again for every new model.

What I will learn next

Week 4 turns the parameter count and BF16 data type into memory math. I will
separate raw weight storage from runtime memory, activations, the KV cache, and
framework overhead, then build a calculator for different precisions.¹³

Run it yourself: the API way in the lab

Everything above was done by browsing the web page. The companion Week 3 lab does
the same inspection the "API way", so you can automate or script it:

The curl and jq one-liners that read each field from the Hub API and the raw files, pinned to an exact repository revision.
model-inspector.py, which pulls the file list, config, tokenizer, license, and shard index for any model in one command and can write results.json.
compare-models.py, which compares the base and instruct repositories field by field and checks the Hub-reported SHA-256 (Git LFS) fingerprint of their first weight shard, proving the checkpoints differ.

Its default and verified target is Qwen/Qwen2.5-3B-Instruct. The --model
option works for compatible repositories, but different model families can use
different files and config keys.¹

Week 3 companion lab:
https://github.com/dramasamy/from-api-to-gpu/tree/main/week-03-model-files ↩
Qwen2.5-3B-Instruct model card:
https://huggingface.co/Qwen/Qwen2.5-3B-Instruct ↩
Hugging Face repository documentation:
https://huggingface.co/docs/hub/repositories ↩
Qwen2.5 model collection:
https://huggingface.co/collections/Qwen/qwen25 ↩
Qwen2.5-3B base model card:
https://huggingface.co/Qwen/Qwen2.5-3B ↩
Qwen2.5-3B-Instruct license:
https://huggingface.co/Qwen/Qwen2.5-3B-Instruct/blob/main/LICENSE ↩
Hugging Face Tokenizers model documentation:
https://huggingface.co/docs/tokenizers/main/en/components#models ↩
Jinja template-language documentation:
https://jinja.palletsprojects.com/ ↩
Hugging Face chat-template documentation:
https://huggingface.co/docs/transformers/chat_templating ↩
Safetensors documentation:
https://huggingface.co/docs/safetensors/index ↩
PyTorch data-type documentation:
https://pytorch.org/docs/stable/tensor_attributes.html#torch.dtype ↩
NIST Secure Hash Standard:
https://csrc.nist.gov/pubs/fips/180-4/upd1/final ↩
Week 4 roadmap:
https://github.com/dramasamy/from-api-to-gpu/blob/main/roadmap/week-04.md ↩

From API to GPU, Week 2: What Actually Happens Behind the API

Dinesh Kumar Ramasamy — Sun, 19 Jul 2026 20:19:40 +0000

Phase 1 of 8: Comfortable running local models. Week 2 of 32.

In Week 1 I went through the machine and proved the GPU works. This week I run an
actual language model on it and, for the first time, look at what sits behind the
"AI API" I have called for years. The short version: the API is not magic. It is
an HTTP layer over a local process that loads a file of numbers into memory and
does the matrix math from Week 1. By the end of this post that sentence will be
clear, backed by real commands and real output you can reproduce.

I am using Ollama, a runtime that makes running a local model about as easy as
running a container. Other names you will see in this space are llama.cpp for
lightweight local inference, vLLM for high-throughput serving, and NVIDIA
TensorRT-LLM for optimized NVIDIA inference. Hugging Face Transformers is also
common, but it is a broader Python framework for running and training models,
not a ready-made local model service. These tools overlap, but they are not exact
replacements. I chose Ollama because it gives me a CLI and local HTTP API with
very little setup.¹

Everything below runs on the DGX Spark and is reached over SSH as spark, the
same setup as Week 1.

Concept 1: a model is not a runtime

The first idea to keep straight, because it holds for the rest of the series, is
the split between the model and the runtime.

The model is trained data: weights plus the metadata needed to use them. On its own it does nothing. It is data on disk.
The runtime is the program that loads those weights into memory and runs the math to turn your prompt into text. Ollama is the runtime here.

If you know Docker, there is a useful analogy, but it is not exact. A model
package is like an image made of versioned, content-addressed layers. Ollama is
like the engine that pulls those layers and starts the workload. Unlike a
container image, the main model layer is trained numeric data, not an app and
its operating-system files.

Ollama calls each stored package file a blob. Here, a blob is just a file
kept under a name derived from its content digest. The inspection below follows
the package index to the model blob and checks that it is the expected file.

Rather than run a wrapper script, I inspected the model with a few direct
commands, one at a time. Each command answers a single question, so you can
paste it, read the output, then move to the next.

First, list what Ollama has downloaded locally:

ssh spark 'ollama list'

NAME                       ID              SIZE      MODIFIED
nomic-embed-text:latest    0a109f422b47    274 MB    2 days ago
llama3.2:3b                a80c4f17acd5    2.0 GB    3 days ago
phi4:latest                ac896e5b8b34    9.1 GB    7 days ago
qwen3.6:35b-a3b-bf16       94061ddd23a7    71 GB     8 days ago

Next, ask Ollama for the focused facts about Phi-4. The Ollama server listens on
localhost:11434 on the Spark, so run this in a shell on the Spark
(ssh spark):

curl -s http://localhost:11434/api/show -d '{"model": "phi4"}' | jq '{
  format: .details.format,
    architecture: .details.family,
    parameters: .details.parameter_size,
    context_length: (.model_info | to_entries
      | map(select(.key | endswith(".context_length"))) | first.value),
    embedding_length: (.model_info | to_entries
      | map(select(.key | endswith(".embedding_length"))) | first.value),
    quantization: .details.quantization_level,
    capabilities,
    runtime_parameters: (.parameters | split("\n")
      | map(select(length > 0) | gsub(" +"; " ")))
  }'

{
  "format": "gguf",
  "architecture": "phi3",
  "parameters": "14.7B",
  "context_length": 16384,
  "embedding_length": 5120,
  "quantization": "Q4_K_M",
  "capabilities": [
    "completion"
  ],
  "runtime_parameters": [
    "stop \"<|im_start|>\"",
    "stop \"<|im_end|>\"",
    "stop \"<|im_sep|>\""
  ]
}

That call gives readable model facts, including "format": "gguf". For a
normal check, this is how I know Ollama identifies the model as GGUF. I do not
need to locate the raw file and inspect its bytes just to answer that question.

To see the exact files that make up the package, read Ollama's manifest. It is a
small JSON file on the Spark, so jq can read it directly. I select just the
package version and the file list:

jq '{schemaVersion, layers: [.layers[] | {mediaType, digest, size}]}' \
  /usr/share/ollama/.ollama/models/manifests/registry.ollama.ai/library/phi4/latest

{
  "schemaVersion": 2,
  "layers": [
    {
      "mediaType": "application/vnd.ollama.image.model",
      "digest": "sha256:fd7b6731c33c57f61767612f56517460ec2d1e2e5a3f0163e0eb3d8d8cb5df20",
      "size": 9053114464
    },
    {
      "mediaType": "application/vnd.ollama.image.template",
      "digest": "sha256:32695b892af87ef8fca6e13a1a31c67c1441d7398be037e366e2fc763857c06a",
      "size": 275
    },
    {
      "mediaType": "application/vnd.ollama.image.license",
      "digest": "sha256:fa8235e5b48faca34e3ca98cf4f694ef08bd216d28b58071a1f85b1d50cb814d",
      "size": 1084
    },
    {
      "mediaType": "application/vnd.ollama.image.params",
      "digest": "sha256:45a1c652dddc9efdcefa977ab81cfbe26b6e52bc8e78f2f4c698538783e0ac80",
      "size": 82
    }
  ]
}

The registry path ends in library/phi4/latest, so this manifest is the index
for the model name phi4 and tag latest. It does not hold the 9.1 GB of
weights. It lists the files that make up the Ollama package, much like a lock
file maps package names to exact artifacts.

Each entry under layers is one file in the package (one blob). It has three
useful fields:

mediaType says what the file's role is. This package has a model, a prompt template, a license, and default parameters.
digest is the file's content identity. The sha256 prefix names the hash algorithm, and the characters after the colon are the hash of the file's bytes.
size says exactly how many bytes that file should contain.

The model layer's digest is also how Ollama names the file on disk. It stores
each blob under a blobs directory using the digest as the filename, with the
colon changed to a hyphen, so sha256:fd7b...df20 becomes sha256-fd7b...df20.
That file is 9,053,114,464 bytes, the size shown above, and its SHA-256
matches the digest. The linked investigation independently checks the file
header with xxd: bytes 47 47 55 46 spell GGUF in ASCII. That deeper check
is useful when reusing the raw file in another runtime, but .details.format
is the simple Ollama API answer.²

So ollama show and the manifest answer different questions. The metadata call
gives readable model facts such as architecture and context length. The
manifest tells Ollama which exact files form the runnable package and where to
find them by content ID.

You might now wonder whether another runtime can use the same GGUF file. That is
useful, but it is not part of the main Ollama lesson. I answer it near the end in
Can llama.cpp reuse this Ollama model?.

I am using phi4, Microsoft's 14.7B-parameter model. At 16 bits per weight,
14.7 billion weights alone would need about 29.4 GB. The local package is only
9.1 GB, which leads to the first surprise.

A quick tour of what each line means, since these terms come up constantly:

Field	Value	Plain meaning
format	gguf	model file format reported by Ollama
architecture	phi3	the neural-network design (phi4 reuses the phi3 family)
parameters	14.7B	how many trained weights the model has
context length	16384	the most tokens (prompt + reply) it can consider at once
embedding length	5120	width of each token vector; covered later
quantization	Q4_K_M	the weights are stored at about 4 bits each, not 16

That last row answers the 9.1 GB surprise. This build is quantized: its
weights use a mixed low-bit Q4_K_M representation instead of 16-bit values.
Four bits per weight would be about 7.35 GB before metadata and quantization
overhead, so a 9.1 GB model layer is reasonable.
Quantization is a whole phase later in this series (weeks 14 to 16). For now the
only thing to take away is that Ollama runs lower-precision weights, and that is
why this 14.7B model takes much less space than its 16-bit source model.

The three stop entries (<|im_start|>, <|im_end|>, <|im_sep|>) are
special tokens: markers the model was trained to use as turn boundaries in a
chat. The runtime watches for them to know when the model has finished its reply.
They show up again in the next section, so keep them in mind.

Concept 2: the API is just HTTP over a local process

Ollama exposes a local HTTP API on port 11434,³ the same shape of
thing as any cloud AI API. The difference is that the "server" is a process on
my own machine.
I called the generate endpoint with curl and formatted the reply with jq.
Run it directly on the Spark:

curl -s http://localhost:11434/api/generate -d '{
  "model": "phi4",
  "prompt": "Explain Kubernetes scheduling in three sentences.",
  "stream": false
}' | jq

Here is the complete response, nothing removed. It is long, but seeing all of it
is the point, because two parts of it explain how the model actually works:

{
  "model": "phi4",
  "created_at": "2026-07-14T03:11:50.862545331Z",
  "response": "Kubernetes scheduling is the process of assigning workloads, such as pods, to appropriate nodes within a cluster based on resource availability and constraints. The scheduler evaluates factors like CPU, memory requirements, affinity/anti-affinity rules, taints/tolerations, and node selectors to determine the best fit for each pod. This ensures optimal utilization of resources, workload distribution, and adherence to specified policies, thereby maintaining efficient cluster operation and performance.",
  "done": true,
  "done_reason": "stop",
  "context": [
    100264, 882, 100266, 198, 849, 21435, 67474, 38952, 304, 2380,
    23719, 13, 100265, 198, 100264, 78191, 100266, 198, 42, 30927,
    38952, 374, 279, 1920, 315, 61853, 990, 33785, 11, 1778,
    439, 55687, 11, 311, 8475, 7954, 2949, 264, 10879, 3196,
    389, 5211, 18539, 323, 17413, 13, 578, 29909, 67349, 9547,
    1093, 14266, 11, 5044, 8670, 11, 51552, 14, 15719, 71260,
    13797, 5718, 11, 259, 1673, 82, 5640, 22847, 811, 11,
    323, 2494, 57137, 311, 8417, 279, 1888, 5052, 369, 1855,
    7661, 13, 1115, 26420, 23669, 50549, 315, 5070, 11, 54696,
    8141, 11, 323, 71628, 311, 5300, 10396, 11, 28592, 20958,
    11297, 10879, 5784, 323, 5178, 13
  ],
  "total_duration": 3974942694,
  "load_duration": 100037676,
  "prompt_eval_count": 18,
  "prompt_eval_duration": 46132000,
  "eval_count": 89,
  "eval_duration": 3817895000
}

Reading it top to bottom:

The response field is the answer, a real reply from a model running on my
hardware with no network call leaving the box.

The context field is the part worth staring at. You might be wondering what
those numbers are. They are tokens. A model does not read text; it reads
token IDs, which are integers. Before anything runs, a tokenizer splits the
text into tokens and maps each one to an integer. And this array is not just the
prompt, it is the whole conversation as tokens: the chat template, my question,
and the model's full answer. Decoding the first several so it is not a mystery:

100264 is the special marker <|im_start|> (start of a turn), and 882 is the word "user". So the conversation begins "start of turn, user".
849, 21435, 67474, 38952 are "Ex", "plain", " Kubernetes", " scheduling". Notice "Explain" is split into two tokens. Tokens are often sub-word pieces, not whole words.
304, 2380, 23719, 13 are " in", " three", " sentences", ".", finishing my prompt.
100265 is <|im_end|> (end of turn), then 100264 78191 100266 is the start of the assistant's turn ("start, assistant, separator").
42, 30927 onward ("K", "ubernetes", ...) is the model's answer, token by token, which is exactly the text in the response field above.

The special IDs 100264, 100265, 100266 are the chat markers
(<|im_start|>, <|im_end|>, <|im_sep|>), the turn boundaries from the
model's embedded tokenizer and chat template. Two questions probably popped into
your head reading that, the same ones that popped into mine: wait, how do I even
know 849 is "Ex" or that 100264 is <|im_start|>? And how does Ollama know
which tokenizer to use? Both have clear answers, but they would break the flow
here, so I answer them at the end. If you want them now, jump to
the token questions, answered.

This is why token counts matter everywhere. On a cloud API you pay per token. In
a model, tokens are what fill the context window, and the whole conversation is
carried forward as this growing list of integers. When people say "tokens per
second," this array is the unit being counted.

Concept 3: inference has two measured phases

The bottom of that same output has the timing, and it turns vague words like
"latency" into measured numbers. All the durations are in nanoseconds, so here
they are converted:

Field	Raw value	Converted value
`load_duration`	100,037,676 ns	0.10 s
`prompt_eval_count`	18	18 input tokens
`prompt_eval_duration`	46,132,000 ns	46 ms
`eval_count`	89	89 output tokens
`eval_duration`	3,817,895,000 ns	3.82 s
`total_duration`	3,974,942,694 ns	3.97 s

prompt_eval_duration is the time spent reading the tokenized input. That is
the prefill phase. eval_duration is the time spent generating output
tokens. That is the decode phase. load_duration is time Ollama spent
loading or preparing the model for this request.

One headline number can be calculated directly:

Tokens per second (generation speed): 89 tokens divided by 3.82 s is about 23.3 tokens/sec.

I cannot honestly calculate time to first token (TTFT) from this
non-streaming response. Adding load and prefill gives 146 ms of server work
before decode, but that is not the same as observing when the first token reaches
the client. To measure TTFT, the client must request a streamed response and
timestamp the first non-empty token. I do that in the benchmark later in this
post.

The timing split also shows that generating text has two phases that behave
differently:

Prefill (prompt_eval): the model reads all 18 prompt tokens at once. This is fast (46 ms) because the tokens are processed together, which is the parallel, compute-heavy work the GPU from Week 1 is good at.
Decode (eval): the model generates output tokens one at a time, each one depending on the previous. This is the slow, sequential phase, and it is where memory bandwidth matters. Each token passes through the model layers and uses their weights again, while the key-value cache avoids recalculating all prior tokens from scratch.

That is the core of Week 2. The API is HTTP over a local process, the "text" is
really a stream of integer tokens, and a request is load, then prefill, then
decode, each one measurable. Now I can start changing the settings that shape the
output.

Concept 4: temperature, or how random the model is allowed to be

Back in Concept 2 I ran the same prompt twice and got slightly different wording
each time. That is not a bug. It is a setting called temperature, and it
controls how random the model is allowed to be when it picks each next token.

The setting is options.temperature. I sent the same coffee-shop prompt twice
at each temperature to watch it change. I changed the value in this direct call
and ran it twice:

curl -s http://localhost:11434/api/generate -d '{
  "model": "phi4",
  "prompt": "Give me a one-sentence tagline for a coffee shop.",
  "stream": false,
  "options": { "temperature": 0 }
}' | jq -r .response

Twice at temperature 0:

temperature=0
run 1: "Awaken Your Senses, One Cup at a Time."
run 2: "Awaken Your Senses, One Cup at a Time."

These two runs matched. The same call with "temperature": 1.2 gives variety:

temperature=1.2
run 1: "Where every sip is a moment of delight."
run 2: "Where Every Cup is a Perfect Brew—Awaken Your Senses."

Same input, two different answers.

So what is temperature actually doing? At each step, the model does not just pick
one next token. It produces a probability for every token in its vocabulary, like
"there is a 40% chance the next token is Awaken, 8% chance it is Where, and so
on". Temperature reshapes that list before one token is picked. Top-p and seed
also affect the choice, but each control has a different job:

Control	Changes	Lower/same	Higher/different
Temperature	Probability shape	Favors likely tokens	Allows more variety
Top-p	Candidate set	Fewer likely tokens	More possible tokens
Seed	Random sequence	Same starting sequence	Another sequence

Top-p is also called nucleus sampling. At 0.9, it keeps the most likely
tokens whose probabilities add up to 90%, then samples only from that set.
Temperature and top-p can be used together, but while learning I would change
one at a time so I know which setting changed the output.

You might also see a seed beside temperature. A model runtime uses a
pseudorandom number generator when it samples from the possible next tokens.
Pseudorandom means the values look random, but they are generated by a formula.
The seed is the starting number for that formula, like starting with the same
shuffle of a deck. The same seed gives the sampler the same sequence of random
choices when the model, prompt, options, runtime, and execution conditions stay
the same.

I tested that with temperature 1.2, where sampling has room to vary:

for RUN in 1 2 3 4; do
  printf "run=%s seed=42: " "$RUN"
  curl -s http://localhost:11434/api/generate -d '{
    "model": "phi4",
    "prompt": "Reply with one invented coffee shop name and nothing else.",
    "stream": false,
    "options": {"temperature": 1.2, "seed": 42, "num_predict": 16}
  }' | jq -r .response | tr '\n' ' '
  printf "\n"
done

run=1 seed=42: Espresso Enchantments
run=2 seed=42: Espresso Enchantments
run=3 seed=42: Espresso Enchantments
run=4 seed=42: Espresso Enchantments

These four warm-model runs matched. A fixed seed improves repeatability, but it
is not a universal guarantee. A different runtime version, model build,
hardware path, parallel execution order, or other nondeterministic GPU behavior
can still change the result. A seed controls the sampler's random sequence; it
does not freeze the whole software and hardware stack.

The practical takeaway is simple. Start near zero when repeatability matters,
and raise temperature when you want variety. Add a fixed seed when you want more
repeatable comparisons, and record the runtime and model version too. Everything
else this week uses the same options object in the API call.

Concept 5: chat is a list of role-tagged messages

So far I used /api/generate, which takes one plain prompt. Ollama also has
chat interfaces. I will use them in increasing depth:

The CLI proves I can have a one-shot conversation.
The HTTP API exposes the role-tagged message structure.
The Python client keeps that structure across multiple turns.

First, chat through the CLI

I keep using phi4, already pulled in Concept 1, so there is nothing new to
download. Ollama's CLI is terminal-aware, so I forced SSH to allocate a terminal
with -tt:

ssh -tt spark \
  'ollama run --nowordwrap phi4 \
  "Reply with exactly these three words: local model ready"'

Local model ready.

Connection to spark closed.

This proves the CLI can send a prompt and display a reply. It does not show the
message roles or how a client maintains a conversation. For that, I need the
HTTP API.

Next, inspect messages through the API

The /api/chat endpoint takes a messages array. Each item has a role that
says who wrote it and content that holds the text:

Role	Who writes it	What it does
`system`	Application	Sets behavior, style, or limits
`user`	Person using the app	Carries the user's prompt or follow-up
`assistant`	Model	Stores a previous model reply in the history

The system prompt is an instruction the application places before the user
conversation. It sets the expected role, style, or limits of the assistant. It
is configuration, not a security boundary. A user can still send conflicting
instructions, so production systems need checks outside the prompt too.

The user prompt is simply the content of a message with role: "user".
After the model replies, the client stores that answer as an assistant message.
The next request sends the earlier messages plus the new user message.

Why resend them? The Ollama API does not remember this client's conversation
between independent HTTP requests. I sent two consecutive requests to the same
running Ollama process. First, I included the earlier user question and
assistant reply:

curl -s http://localhost:11434/api/chat -d '{
  "model": "phi4",
  "stream": false,
  "messages": [
    {"role": "system", "content": "Reply with only a name or UNKNOWN."},
    {"role": "user", "content": "My cluster is called Atlas."},
    {"role": "assistant", "content": "Atlas"},
    {"role": "user", "content": "What name did I give it?"}
  ],
  "options": {"temperature": 0, "seed": 42}
}' | jq '{reply: .message, prompt_tokens: .prompt_eval_count}'

{
  "reply": {
    "role": "assistant",
    "content": "Atlas"
  },
  "prompt_tokens": 50
}

Then I sent only the follow-up, with no earlier turns:

curl -s http://localhost:11434/api/chat -d '{
  "model": "phi4",
  "stream": false,
  "messages": [
    {"role": "system", "content": "Reply with only a name or UNKNOWN."},
    {"role": "user", "content": "What name did I give it?"}
  ],
  "options": {"temperature": 0, "seed": 42}
}' | jq '{reply: .message, prompt_tokens: .prompt_eval_count}'

{
  "reply": {
    "role": "assistant",
    "content": "UNKNOWN"
  },
  "prompt_tokens": 31
}

Request	Reply	Prompt tokens
Earlier turns included	`Atlas`	50
Earlier turns omitted	`UNKNOWN`	31

What changed between the two calls? Only the messages array.

In the first request, the JSON body included the earlier line My cluster is called Atlas, the model's earlier answer, and the new follow-up. Ollama could
therefore pass all of that text to the model, so the answer was Atlas.

In the second request, the conversation history contained only What name did I give it?. There was no mention of Atlas in its messages, so the model answered
UNKNOWN. Ollama did not look up the earlier HTTP call and add it automatically.

That is what the client owns the conversation state means. The application,
such as this Python client, keeps the message list in memory or a database. For
each new turn, it appends the latest user message and sends the relevant history
again. Ollama processes the messages it receives in that request.

Resending history has a cost. The request with earlier turns contained 50 prompt
tokens; the request without them contained 31. The 19-token difference includes
the added message text, role labels, and chat-template separators. I measured
the total difference here; I did not split it token by token. All 19 additional
tokens used positions in the context window.

Finally, implement the conversation in Python

The required Python client now implements the message flow I just tested. It
uses /api/chat, keeps prior turns in the messages
list, and streams the reply as it arrives. With "stream": true, Ollama sends
one JSON object per line; each text fragment is in message.content. The loop
prints each fragment and joins them into the completed assistant message. Here
is the complete file:

#!/usr/bin/env python3
"""Small command-line chat client for Ollama's streaming /api/chat endpoint."""

import argparse
import json
import os
import urllib.error
import urllib.request


def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--model", default=os.getenv("MODEL", "phi4"))
    parser.add_argument("--url", default="http://localhost:11434")
    parser.add_argument("--system", default="You are a concise assistant.")
    parser.add_argument("--temperature", type=float, default=0.0)
    parser.add_argument("--top-p", type=float, default=0.9)
    parser.add_argument("--num-ctx", type=int, default=4096)
    parser.add_argument("--once", help="Send one prompt and exit")
    return parser.parse_args()


def request_chat(
    url: str,
    model: str,
    messages: list[dict[str, str]],
    options: dict[str, float | int],
) -> str:
    body = {
        "model": model,
        "messages": messages,
        "stream": True,
        "options": options,
    }
    request = urllib.request.Request(
        f"{url.rstrip('/')}/api/chat",
        data=json.dumps(body).encode(),
        headers={"Content-Type": "application/json"},
    )
    parts: list[str] = []
    try:
        with urllib.request.urlopen(request) as response:
            for line in response:
                event = json.loads(line)
                text = event.get("message", {}).get("content", "")
                if text:
                    print(text, end="", flush=True)
                    parts.append(text)
    except urllib.error.URLError as exc:
        raise SystemExit(f"Ollama request failed: {exc}") from exc
    print()
    return "".join(parts)


def main() -> None:
    args = parse_args()
    messages: list[dict[str, str]] = []
    if args.system:
        messages.append({"role": "system", "content": args.system})
    options = {
        "temperature": args.temperature,
        "top_p": args.top_p,
        "num_ctx": args.num_ctx,
    }

    if args.once:
        messages.append({"role": "user", "content": args.once})
        request_chat(args.url, args.model, messages, options)
        return

    print(f"Chatting with {args.model}. Type /exit to quit.")
    while True:
        try:
            prompt = input("you> ").strip()
        except (EOFError, KeyboardInterrupt):
            print()
            return
        if prompt.lower() in {"/exit", "/quit"}:
            return
        if not prompt:
            continue
        messages.append({"role": "user", "content": prompt})
        print("model> ", end="", flush=True)
        answer = request_chat(args.url, args.model, messages, options)
        messages.append({"role": "assistant", "content": answer})


if __name__ == "__main__":
    main()

I used --once for a repeatable blog example and passed a system prompt. If I
leave off --once, the same program opens an interactive loop and keeps each
user and assistant turn in the next request.

ssh spark '~/venvs/w1/bin/python - \
  --model phi4 \
  --system "Answer in exactly one short sentence." \
  --once "What does the Kubernetes scheduler do?"' \
  < public/week-02-first-local-model/chat_client.py

The Kubernetes scheduler assigns pods to nodes based on resource availability,
constraints, and policies.

This is the same local runtime and the same phi4 model through three
interfaces: Ollama's CLI, /api/generate with curl, and /api/chat from
Python. Only the client interface changed.

Concept 6: measure cold starts and verify context allocation

Imagine Phi-4 is behind a chat application. No one has used it for a while, so
Ollama has unloaded it from memory. A user sends a message and waits about six
seconds before seeing the first word. They send another message a minute later
and see the first word in a fraction of a second.

That difference is what this experiment measures:

Request state	What Ollama must do	What the user notices
Cold	Load the model, then process the prompt	Long wait for first text
Warm	Reuse the model already in memory	First text arrives quickly

A cold request starts with the model unloaded, so Ollama must load it before
inference. A warm request starts with the model already resident in memory
and ready for reuse.

Before running anything, I expected the first request to have a much higher
time to first token (TTFT) because it includes model loading. I expected the
next requests to start quickly. I did not expect the output generation rate to
change much, because all runs use the same model and output length.

I also used the experiment to answer a separate configuration question:

What does num_ctx change? It changes how much token capacity Ollama allocates. It does not make a short prompt longer or automatically faster.
How different is the first request from later requests? A streamed client can measure real time to first token for both a cold and warm model.

These are two checks in one script, not a claim that context size causes the
cold-start difference. The cold-versus-warm runs measure model loading. The
2,048-versus-4,096 setting verifies how I request context capacity.

The model metadata showed a maximum context length of 16,384 tokens. The
configured context size tells Ollama how much of that capacity to allocate
for a run. The prompt does not become 4,096 tokens just because I set
num_ctx to 4,096. It means the prompt, chat history, and generated reply may
use up to that budget.

A larger context also needs a larger key-value cache, often shortened to KV
cache. That cache holds intermediate values from earlier tokens so decode does
not repeat all prior work for every new token. More context gives the cache room
for more tokens, which uses more memory.

The benchmark changes num_ctx from 2,048 to 4,096. At each setting it runs
three requests in this exact order:

Run	Model state	Purpose
1	Cold	Measure the user-visible cost of loading Phi-4
2	Warm	Measure a request with Phi-4 already loaded
3	Warm	Repeat the warm measurement once more

It fixes temperature and seed, limits each reply to 96 tokens, and calls
ollama stop phi4 before run 1 at each context setting. Runs 2 and 3 do not stop
the model, so they reuse it in memory.

Most importantly, it requests a streamed response. The client starts a clock
before the HTTP request and stops the TTFT clock when the first non-empty text
chunk arrives. That is a real client-observed TTFT measurement.

Here is the full benchmark:

#!/usr/bin/env python3
"""Measure client TTFT and server timings from Ollama's streaming API."""

import argparse
import json
import subprocess
import time
import urllib.request
from pathlib import Path
from typing import Any


def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--model", default="phi4")
    parser.add_argument(
        "--prompt",
        default="Explain Kubernetes scheduling in three sentences.",
    )
    parser.add_argument("--contexts", default="2048,4096")
    parser.add_argument("--runs", type=int, default=3)
    parser.add_argument("--url", default="http://localhost:11434")
    parser.add_argument("--output", type=Path)
    return parser.parse_args()


def unload(model: str) -> None:
    subprocess.run(
        ["ollama", "stop", model],
        stdout=subprocess.DEVNULL,
        stderr=subprocess.DEVNULL,
        check=False,
    )
    time.sleep(1)


def run_once(
    url: str,
    model: str,
    prompt: str,
    num_ctx: int,
    cold: bool,
) -> dict[str, Any]:
    if cold:
        unload(model)
    body = {
        "model": model,
        "prompt": prompt,
        "stream": True,
        "options": {
            "temperature": 0,
            "seed": 42,
            "num_ctx": num_ctx,
            "num_predict": 96,
        },
    }
    request = urllib.request.Request(
        f"{url.rstrip('/')}/api/generate",
        data=json.dumps(body).encode(),
        headers={"Content-Type": "application/json"},
    )

    started = time.perf_counter()
    first_token_at: float | None = None
    final: dict[str, Any] = {}
    with urllib.request.urlopen(request) as response:
        for line in response:
            event = json.loads(line)
            if event.get("response") and first_token_at is None:
                first_token_at = time.perf_counter()
            if event.get("done"):
                final = event
    finished = time.perf_counter()

    if first_token_at is None or not final:
        raise RuntimeError("Ollama stream ended without timing data")
    eval_seconds = final["eval_duration"] / 1e9
    return {
        "context": num_ctx,
        "cold": cold,
        "ttft_ms": round((first_token_at - started) * 1000, 1),
        "client_total_ms": round((finished - started) * 1000, 1),
        "load_ms": round(final["load_duration"] / 1e6, 1),
        "prompt_eval_ms": round(final["prompt_eval_duration"] / 1e6, 1),
        "prompt_tokens": final["prompt_eval_count"],
        "output_tokens": final["eval_count"],
        "tokens_per_second": round(final["eval_count"] / eval_seconds, 1),
    }


def main() -> None:
    args = parse_args()
    contexts = [int(value) for value in args.contexts.split(",")]
    rows: list[dict[str, Any]] = []
    print("ctx  run  cold  ttft_ms  total_ms  load_ms  out_tok  tok/s")
    for context in contexts:
        for run_number in range(1, args.runs + 1):
            row = run_once(
                args.url,
                args.model,
                args.prompt,
                context,
                cold=run_number == 1,
            )
            rows.append(row)
            print(
                f"{context:>4}  {run_number:>3}  {str(row['cold']):>5}  "
                f"{row['ttft_ms']:>7.1f}  {row['client_total_ms']:>8.1f}  "
                f"{row['load_ms']:>7.1f}  {row['output_tokens']:>7}  "
                f"{row['tokens_per_second']:>5.1f}"
            )

    result = {
        "model": args.model,
        "prompt": args.prompt,
        "runs_per_context": args.runs,
        "measurements": rows,
    }
    if args.output:
        args.output.write_text(json.dumps(result, indent=2) + "\n")


if __name__ == "__main__":
    main()

I ran three requests per context size. The first is cold because the script
stops Phi-4 before it. The next two reuse the loaded model.

ssh spark '~/venvs/w1/bin/python - --contexts 2048,4096 --runs 3' \
  < public/week-02-first-local-model/benchmark.py

ctx  run  cold  ttft_ms  total_ms  load_ms  out_tok  tok/s
2048    1   True   6034.1   10129.6   5908.4       96   23.4
2048    2  False    196.1    4288.0    140.4       96   23.5
2048    3  False    151.9    4244.7    104.1       96   23.5
4096    1   True   6260.8   10369.2   6150.1       96   23.4
4096    2  False    174.3    4262.8    117.8       96   23.5
4096    3  False    173.5    4260.8    124.8       96   23.5

The cold requests took about 6.0 to 6.3 seconds before the first token. The warm
requests took 151.9 to 196.1 ms. This is the cold-start effect I could not
measure from the earlier non-streaming call.

Generation stayed at 23.4 to 23.5 tokens/sec. The short 18-token input did not
use the extra context capacity, so doubling the setting did not make this test
faster. It just allowed a larger KV cache. ollama ps confirmed the last
requested allocation:

ssh spark 'ollama ps'

NAME           ID              SIZE      PROCESSOR    CONTEXT    UNTIL
phi4:latest    ac896e5b8b34    9.7 GB    100% GPU     4096       4 minutes from now

What did this experiment prove?

A cold Phi-4 request took about 6 seconds to produce its first token because Ollama had to load the model.
Warm requests produced the first token in about 0.15 to 0.20 seconds.
Ollama accepted the final 4,096-token setting and reported CONTEXT 4096.
Doubling the requested context did not change throughput for this 18-token prompt. Both settings stayed near 23.5 tokens/sec.

It did not measure the memory difference between the two context settings,
and it did not test a long prompt. Those need a separate experiment. The lesson
here is simpler: keep models warm for latency, and treat context size as a
capacity setting rather than a speed setting.

The token questions, answered

Earlier, in the token section (Concept 2), I parked two questions to keep the
main thread moving: how do I even know that 849 is "Ex", and how does Ollama
know which tokenizer to use? Here are the answers. You do not need them to follow
the rest of the post, so read on only if you want to poke at the internals.

Wait, how do I even know what each token ID means?

I did not guess. A runnable model package needs a tokenizer, the exact
rulebook that maps text to token IDs and back. The ordinary Phi-4 token IDs in
the API response match OpenAI's cl100k_base encoding. I verified that match
with decode_tokens.py instead of inferring it from the size of the IDs.

#!/usr/bin/env python3
"""Week 2 - show how text becomes tokens for a cl100k model like phi4.

Encodes the prompt and decodes individual token IDs, so you can match the
`context` array from the generate API back to real words.

Run with a Python that has tiktoken installed, e.g. the Week 1 venv:
    ~/venvs/w1/bin/python -m pip install -q tiktoken
    ~/venvs/w1/bin/python decode_tokens.py
"""
import tiktoken

PROMPT = "Explain Kubernetes scheduling in three sentences."
# A few IDs taken from the generate API's context array.
SAMPLE_IDS = [849, 21435, 67474, 38952, 304, 2380, 23719, 13, 882, 78191]


def main() -> None:
    enc = tiktoken.get_encoding("cl100k_base")

    print("encode:", enc.encode(PROMPT))
    print("decode individual IDs:")
    for t in SAMPLE_IDS:
        print(f"  {t:>6}  {enc.decode([t])!r}")


if __name__ == "__main__":
    main()

Then I ran the file with the pinned tiktoken 0.13.0 package:

ssh spark '~/venvs/w1/bin/python -' \
  < public/week-02-first-local-model/decode_tokens.py

encode: [849, 21435, 67474, 38952, 304, 2380, 23719, 13]
decode individual IDs:
     849  'Ex'
   21435  'plain'
   67474  ' Kubernetes'
   38952  ' scheduling'
     304  ' in'
    2380  ' three'
   23719  ' sentences'
      13  '.'
     882  'user'
   78191  'assistant'

The first line reproduces the exact prompt tokens from the context array, which
confirms phi4 really does use this vocabulary. Decoding the individual IDs gives
back the words, including user and assistant. This proves the ordinary
prompt and role IDs match cl100k_base. It does not prove what the added special
IDs mean. I verify those from the model file in the next section.

Is cl100k a universal standard? No. It is one of a small set of named
vocabularies from OpenAI's tiktoken library, where each vocabulary (such as
cl100k_base for GPT-4 and the newer o200k_base for GPT-4o) is tied to specific
models.⁴ Other model families use entirely different tokenizers. Llama
models, for example, use a different scheme with different token IDs, so the same
word gets a different number there. Each model comes with its own vocabulary, and
you can always load that exact tokenizer to see how it splits text.

And how does Ollama know which tokenizer to use?

The runtime is what turns the prompt into tokens, so this is a fair question. It
does not keep a list of models and guess. The tokenizer is packed inside the
model file. Ollama stores models in the GGUF format,⁵ and a GGUF file
carries the tokenizer metadata next to the weights.

There are two files involved:

The small manifest file is /usr/share/ollama/.ollama/models/manifests/registry.ollama.ai/library/phi4/latest. It is a JSON index for the Phi-4 package. It does not contain the tokenizer. Its model-layer entry tells Ollama which blob file to open.
The model blob is in /usr/share/ollama/.ollama/models/blobs. Its filename is sha256-fd7b6731c33c57f61767612f56517460ec2d1e2e5a3f0163e0eb3d8d8cb5df20. This 9 GB file is the actual GGUF model. It contains the model weights, tokenizer vocabulary, special tokens, and chat template.

I resolved that path directly before using Python:

ssh spark '
  MODELS_DIR=/usr/share/ollama/.ollama/models
  MANIFEST_FILE=manifests/registry.ollama.ai/library/phi4/latest
  MANIFEST="$MODELS_DIR/$MANIFEST_FILE"
  DIGEST="$(jq -r '\''.layers[]
    | select(.mediaType == "application/vnd.ollama.image.model")
    | .digest'\'' "$MANIFEST")"
  BLOB_FILE="blobs/${DIGEST/:/-}"
  printf "models directory: %s\n" "$MODELS_DIR"
  printf "manifest file: %s\n" "$MANIFEST_FILE"
  printf "model digest: %s\n" "$DIGEST"
  printf "GGUF file: %s\n" "$BLOB_FILE"
  stat -c "GGUF bytes: %s" "$MODELS_DIR/$BLOB_FILE"
'

models directory: /usr/share/ollama/.ollama/models
manifest file: manifests/registry.ollama.ai/library/phi4/latest
model digest: sha256:fd7b6731c33c57f61767612f56517460ec2d1e2e5a3f0163e0eb3d8d8cb5df20
GGUF file: blobs/sha256-fd7b6731c33c57f61767612f56517460ec2d1e2e5a3f0163e0eb3d8d8cb5df20
GGUF bytes: 9053114464

The digest changes from sha256:... in the manifest to sha256-... in the
blob filename. Only the colon becomes a hyphen. The Python script automates
these same steps, opens that GGUF blob, and reads a few tokenizer fields so I do
not have to inspect a 9 GB binary file by hand:

#!/usr/bin/env python3
"""Read selected tokenizer metadata directly from an Ollama GGUF model layer."""

import argparse
import json
import os
from pathlib import Path

from gguf import GGUFReader


def parse_args() -> argparse.Namespace:
  parser = argparse.ArgumentParser(description=__doc__)
  parser.add_argument("--model", default=os.getenv("MODEL", "phi4"))
  parser.add_argument(
    "--models-dir",
    type=Path,
    default=Path("/usr/share/ollama/.ollama/models"),
  )
  return parser.parse_args()


def field(reader: GGUFReader, name: str):
  value = reader.get_field(name)
  if value is None:
    raise KeyError(f"Missing GGUF field: {name}")
  return value.contents()


def main() -> None:
  args = parse_args()
  name, separator, tag = args.model.partition(":")
  if not separator:
    tag = "latest"
  manifest = (
    args.models_dir
    / "manifests"
    / "registry.ollama.ai"
    / "library"
    / name
    / tag
  )
  data = json.loads(manifest.read_text())
  digest = next(
    layer["digest"]
    for layer in data["layers"]
    if layer["mediaType"].endswith("image.model")
  )
  blob = args.models_dir / "blobs" / digest.replace(":", "-")
  reader = GGUFReader(blob)
  tokens = field(reader, "tokenizer.ggml.tokens")
  template = field(reader, "tokenizer.chat_template")

  print("tokenizer model:", field(reader, "tokenizer.ggml.model"))
  print("tokenizer pre:", field(reader, "tokenizer.ggml.pre"))
  print("token count:", len(tokens))
  for token_id in (100257, 100264, 100265, 100266):
    print(f"token {token_id}: {tokens[token_id]!r}")
  for marker in ("<|im_start|>", "<|im_end|>", "<|im_sep|>"):
    print(f"template contains {marker}: {marker in template}")


if __name__ == "__main__":
  main()

I ran it with the pinned gguf 0.19.0 package:

ssh spark '~/venvs/w1/bin/python -' \
  < public/week-02-first-local-model/inspect_tokenizer.py

tokenizer model: gpt2
tokenizer pre: dbrx
token count: 100352
token 100257: '<|endoftext|>'
token 100264: '<|im_start|>'
token 100265: '<|im_end|>'
token 100266: '<|im_sep|>'
template contains <|im_start|>: True
template contains <|im_end|>: True
template contains <|im_sep|>: True

Everything the runtime needs is in the file:

tokenizer.ggml.model = 'gpt2' is the tokenizer type. It is a byte-pair encoding (BPE), the same family GPT-2 and GPT-4 use.
tokenizer.ggml.tokens is the full vocabulary, all 100,352 entries, the list that maps text pieces to IDs.
The three IDs are the exact special-token strings claimed earlier.
The embedded chat template contains those same markers. Phi-4's model card uses this ChatML-style prompt layout.⁶

So Ollama reads the vocabulary from the model file itself and uses it to turn my
prompt into token IDs. The ordinary IDs used in this prompt are compatible with
cl100k_base, while the GGUF adds Phi-4's special chat tokens. A Llama model's
GGUF carries a different vocabulary, and Ollama uses that instead. Same runtime,
different tokenizer per model, because the tokenizer travels with the model.

Results

Here is the Week 2 result in one place:

Item	Measured result
Runtime	Ollama 0.31.2
Main model	Phi-4, 14.7B parameters, Q4_K_M
Model layer	9,053,114,464 bytes
Context settings tested	2,048 and 4,096 tokens
Cold TTFT	6,034.1 to 6,260.8 ms
Warm TTFT	151.9 to 196.1 ms
Generation speed	23.4 to 23.5 tokens/sec
Python deliverable	Streaming command-line chat client
Benchmark deliverable	Cold/warm TTFT and throughput benchmark

I also used every interface in the plan: interactive CLI, curl, and Python.
The commands changed temperature and context size. The Python client used a
system prompt, and the benchmark compared repeatable runs with measured timing.

What surprised me

The first surprise was the model size. Phi-4 has 14.7 billion parameters, but
its quantized model layer is 9.1 GB instead of the roughly 29.4 GB needed for
16-bit weights alone.

The bigger surprise was the cold start. Generation speed stayed near 23.5
tokens/sec, but the first token took about 6 seconds when Ollama had to load the
model. Once warm, it arrived in about 0.15 to 0.20 seconds.

Changing context capacity from 2,048 to 4,096 did not improve this short
request. A larger limit gives the KV cache room for more tokens. It does not make
an 18-token prompt larger or automatically faster.

Mistakes and troubleshooting

I first called the output of ollama show a manifest. That was wrong. Model
metadata is a readable summary. The manifest is the JSON index that points to
the model, template, license, and parameter layers.

I also tried to estimate TTFT by adding load_duration and
prompt_eval_duration from a non-streaming response. That is useful server
timing, but it is not client-observed TTFT. The fixed benchmark streams the
response and timestamps the first non-empty text chunk.

One smaller issue came from Ollama's terminal-aware CLI. Its one-shot response
was not visible through plain SSH capture, so I used ssh -tt to allocate a
pseudo-terminal. The Python and curl clients did not need that workaround.

Production implications

A production service should keep commonly used models warm. Otherwise, a user
can wait seconds for model loading before generation even starts. Streaming also
matters because it lets the user see the first token instead of waiting for the
whole reply.

Context size is a capacity and memory decision. Setting it to the model maximum
for every request can reserve more KV-cache memory than the workload needs. The
right setting comes from measured prompt and conversation lengths.

The system prompt is not an access-control system. If I expose Ollama beyond
localhost, I still need a gateway with authentication, authorization, TLS, rate
limits, request-size limits, and logging. Prompt instructions do not replace
those controls.

What I will learn next

Week 3 moves from running a packaged model to opening a Hugging Face repository
and inspecting its files. I will compare base and instruct models, read the
model card and license, then inspect configuration, tokenizer, chat-template,
and Safetensors files.⁷

Run it yourself

The public Week 2 lab contains the chat client, benchmark, scripts, pinned
requirements, raw results, and model-run record.⁸

Can llama.cpp reuse this Ollama model?

This is optional reading. It is useful if you want to understand whether a model
file belongs to one runtime or can move between compatible runtimes.

For this Phi-4 download, yes. Ollama reports details.format = "gguf", and the
independent file-header check agrees. llama.cpp loads local GGUF files, so it
can use this same 9,053,114,464-byte model blob without downloading the weights
again.⁵ The content matters, not the missing .gguf extension on Ollama's
content-addressed filename.

A file header alone does not prove another runtime can load the model and
generate text, so I installed a pinned CUDA build of llama.cpp and pointed it
at the exact model blob named by Ollama's manifest digest. It loaded the file
and generated a reply:

Check	Result
Model blob	same file, Ollama manifest digest `fd7b...df20`
`llama.cpp` build	pinned commit `571d0d54`
Device used	CUDA0 (NVIDIA GB10)
GPU layers offloaded	41/41
Generated reply	`Shared model works.`

That is a concrete yes. The full investigation records the failed first check,
CUDA toolkit discovery, pinned ARM64 build, scripts, commands, raw output, and
the apparent terminal hang.²

One limit remains. The model blob is only one layer of the Ollama package. The
manifest also lists a prompt template, license, and default parameters as
separate layers. llama.cpp does not read those Ollama layers just because it
can read the model blob. This Phi-4 GGUF carries its own tokenizer and chat
template, but I still set the seed, temperature, context, and GPU layers
explicitly. Same weights do not guarantee the same output from two runtimes.

This result is specific to this Phi-4 file and this pinned llama.cpp commit.
For another Ollama download, I would repeat the test because an older runtime
may not support a newer model architecture.

Official project docs for the alternatives named here:
llama.cpp at https://github.com/ggml-org/llama.cpp, vLLM at
https://docs.vllm.ai, NVIDIA TensorRT-LLM at
https://nvidia.github.io/TensorRT-LLM/, and Hugging Face Transformers at
https://huggingface.co/docs/transformers/index ↩
Full Ollama-to-llama.cpp investigation, including the pinned
CUDA build and raw command output:
https://github.com/dramasamy/from-api-to-gpu/blob/main/week-02-first-local-model/ollama-model-in-llamacpp.md ↩
Ollama API reference:
https://docs.ollama.com/api/introduction ↩
tiktoken, OpenAI's tokenizer library, lists the standard vocabularies
and which model each one belongs to. See the model-to-encoding table at
https://github.com/openai/tiktoken ↩
GGUF is the model file format from the llama.cpp project. It stores the
weights plus metadata such as the tokenizer and chat template. See
https://github.com/ggml-org/ggml/blob/master/docs/gguf.md ↩
Microsoft Phi-4 model card, including its chat input format:
https://huggingface.co/microsoft/phi-4#input-formats ↩
Week 3 roadmap:
https://github.com/dramasamy/from-api-to-gpu/blob/main/roadmap/week-03.md ↩
Week 2 companion lab:
https://github.com/dramasamy/from-api-to-gpu/tree/main/week-02-first-local-model ↩

From API to GPU, Week 1: Understanding Your Local GPU Environment

Dinesh Kumar Ramasamy — Sat, 11 Jul 2026 20:57:28 +0000

Phase 1 of 8: Comfortable running local models. Week 1 of 32.

I've used AI through APIs to ship features: POST a prompt, get tokens back,
move on. I have never once deployed a model myself. No PyTorch, no GPU memory
math, no idea what actually happens between my HTTP request and the text that
comes back. This series is me closing that gap on purpose, one week at a time,
on an NVIDIA DGX Spark.

I'm a software engineer and technical program manager. I'm comfortable with
Linux, Python, Docker, Kubernetes, and APIs. I'm a complete beginner at machine
learning. So Week 1 is deliberately unglamorous: before running any model, I
want to know the machine: what CPU and GPU it has, how its memory works, and
what the NVIDIA software stack underneath is actually made of. Every claim below
is backed by a real command and its real output, so you can run the same thing
on your own box and compare.

About this series

From API to GPU is a 32-week journey from AI-API consumer to
local LLM systems architect: running, optimizing, and eventually
fine-tuning models on local hardware, documenting each week as a hands-on lab
plus a blog post. The full week-by-week plan lives in the roadmap¹, and
every week's runnable code lands in the companion GitHub repo². If you have
a similar machine, you can follow along and reproduce every result.

The plan runs in eight phases, with a parallel CUDA track starting around week 5:

Phase	Weeks	Focus
1	1–4	Comfortable running local models
2	5–8	Enough ML to understand inference
3	9–13	Transformers and terminology
4	14–16	Quantization and model formats
5	17–20	Inference engineering
6	21–24	Production model services
7	25–28	RAG and application integration
8	29–32	Fine-tuning

The goal of this first post is narrow on purpose: stand up and understand
the environment. By the end you'll be able to inventory a DGX Spark, read what
nvidia-smi tells you, explain how its unified memory differs from a normal
GPU, untangle the NVIDIA driver/CUDA-runtime/toolkit layers, and prove the GPU
is usable from PyTorch with a measured CPU-vs-GPU speedup. No model yet; that's
week 2. This is the foundation everything else builds on.

All the commands in this post are packaged as a runnable script in the companion
repo under week-01-environment/³. Clone it, set up an SSH alias spark
that reaches your DGX Spark, and run ./inventory.sh to reproduce everything
here. The repo holds the commands and scripts; this post holds the explanations,
so neither repeats the other.

Here's the whole inventory script, so you can see exactly what it runs without
cloning anything. It just wraps each command in an SSH call to the Spark and
prints a labelled section:

#!/usr/bin/env bash
# Week 1: DGX Spark machine inventory. Prereq: an SSH alias `spark`.
set -euo pipefail
SPARK_HOST="${SPARK_HOST:-spark}"

run() { echo "=== {% katex inline %}1 ==="; ssh "{% endkatex %}SPARK_HOST" "$2" 2>&1 || true; echo; }

run "uname"      'uname -a'
run "os-release" 'cat /etc/os-release'
run "lscpu"      'lscpu'
run "memory"     'free -h'
run "storage"    'lsblk'
run "gpu"        'nvidia-smi'
run "cuda-nvcc"  'nvcc --version || /usr/local/cuda/bin/nvcc --version'
run "cuda-dirs"  'ls -d /usr/local/cuda*'
run "python"     'python3 --version'
run "docker"     'docker version'

The rest of this post walks through the interesting parts of that output one
section at a time.

Where I started: two machines, and a rule I broke immediately

The setup is two machines. A MacBook Pro is the control machine, for writing,
editing, and opening SSH sessions. An NVIDIA DGX Spark is the workhorse, where
every model and every GPU command actually runs. I reach it over SSH as spark.

A theme for this whole series is that I verify facts with a command instead of
assuming them, even the obvious ones. So rather than start by stating what the
machines are, I'll show them. First the control machine:

# On the control MacBook
uname -m && sysctl -n machdep.cpu.brand_string

x86_64
Intel(R) Core(TM) i5-1038NG7 CPU @ 2.00GHz

An Intel x86_64 Mac. This matters more than it looks, because it's a
different architecture from the Spark, so I want it on the record, not assumed.
Now the Spark:

ssh spark 'uname -a'

Linux spark-66b9 6.17.0-1026-nvidia ... aarch64 aarch64 aarch64 GNU/Linux

The Spark is aarch64: ARM64, the same CPU family as phones and Apple
Silicon, but a different architecture from the Intel Mac.

Machine	Role	Architecture	Verified by
MacBook Pro	control / authoring	x86_64 (Intel i5)	`uname -m`
DGX Spark	model + GPU work	aarch64 (ARM64)	`uname -a`

This is not trivia. Because the two machines are different architectures, a
Python wheel or Docker image built for the Intel Mac will not necessarily run on
the ARM64 Spark. That is exactly why all the real work in this series happens
over ssh spark, on the box itself, and why "it works on my laptop" means
nothing here.

CPU versus GPU, and why models love the GPU

The Spark's CPU is a 20-core ARM chip:

ssh spark 'lscpu'

Architecture: aarch64
CPU(s):       20
Model name:   Cortex-X925   (10 performance cores)
Model name:   Cortex-A725   (10 efficiency cores)

Twenty cores sounds like a lot, but for running a model the CPU is mostly a
traffic director: it runs the OS, your Python, and the data loading. The heavy
lifting happens on the GPU. Here's what finally made that click for me.

A neural-network layer boils down to one operation repeated endlessly: multiply
a big grid of numbers (the model's weights, $W$ ) by a list of numbers (the
input, $x$ ) and add them up, a matrix multiplication. One output value is:

y_{i} = j \sum W_{ij} x_{j}

A model with billions of parameters does billions of these multiply-adds for a
single token. The magic property is that they're independent: computing
$y_{1}$ doesn't need $y_{2}$ . In terms I already know, that's embarrassingly
parallel, like the map phase of a MapReduce where no shard waits on another.

That's the whole reason a GPU wins:

Aspect	CPU (the 20 ARM cores)	GPU (the NVIDIA GB10)
Parallel workers	a few strong cores	thousands of small cores
Good at	branching logic, one-at-a-time	the same math on huge data at once
Analogy	a few expert chefs	a stadium of line cooks

A CPU is a handful of very smart workers doing tasks in sequence. A GPU is
thousands of simpler workers all doing the identical multiply-add on different
numbers simultaneously. Since a model is nothing but that identical operation
repeated, the GPU is the right tool.

One caveat I'm carrying forward: those cores are useless if you can't feed
them numbers fast enough, so memory bandwidth, not raw compute, usually
limits how fast a model runs. I'll test the CPU-vs-GPU speed difference directly
with a matmul benchmark once PyTorch is installed; my prediction is a 10x–50x
GPU speedup, with a slow first GPU run due to one-time warmup.

Reading `nvidia-smi`: my new `top` for the GPU

nvidia-smi is the command I'll run every day from now on. It's the GPU
equivalent of top or docker stats. Here's the real output, lightly trimmed:

ssh spark 'nvidia-smi'

NVIDIA-SMI 580.159.03   Driver Version: 580.159.03   CUDA Version: 13.0
GPU 0: NVIDIA GB10   Persistence-M: On
Temp  Perf  Pwr:Usage/Cap   Memory-Usage    GPU-Util  Compute M.
35C   P8    4W / N/A        Not Supported   0%        Default

Field by field, and why each one will matter later:

Field	Value	Meaning
Driver Version	580.159.03	kernel driver talking to the GPU
CUDA Version	13.0	max CUDA the driver supports
GPU name	NVIDIA GB10	the device (Grace-Blackwell)
Temp	35C	die temperature (heat → throttling)
Perf	P8	clock state, P0 = max … P8 = idle
Pwr:Usage/Cap	4W / N/A	current vs max power draw
Memory-Usage	Not Supported	would show VRAM used/total (see below)
GPU-Util	0%	% of last sample the GPU was busy

The bottom of the output also has a process table listing every PID holding the
GPU. Right now that's just Xorg, GNOME, and Firefox using it for the desktop.
That's my first stop whenever I hit "out of memory": find the offender, like
lsof on a stuck port.

For Week 1 I'm using nvidia-smi purely as an inventory tool: what GPU,
what driver, what max CUDA, who's using it. The deeper use (streaming monitors,
reading utilization and memory bandwidth to decide if a workload is
compute-bound or memory-bound) is a profiling skill I'm deliberately saving for
the CUDA track around Week 5, so I don't tangle the two learning tracks.

The surprise: unified memory, and a blank that isn't a bug

The one field that stopped me was Memory-Usage: Not Supported. On a normal PC
with a discrete GPU, that column is how you answer "did my model fit? how much
VRAM is left?" On the Spark it's blank, and that's not a bug; it's the whole
point of the machine.

A traditional GPU has its own separate memory (VRAM), physically distinct from
system RAM. The DGX Spark's GB10 is a Grace-Blackwell superchip that fuses the
ARM CPU and the GPU onto one package and gives them one shared memory pool:

ssh spark 'free -h'

               total   used   free   shared  buff/cache   available
Mem:           121Gi   6.2Gi   67Gi     36Mi        48Gi        115Gi
Swap:           15Gi   152Ki   15Gi

Traditional discrete GPU	DGX Spark (GB10)
System RAM (e.g. 64 GB) + separate VRAM (e.g. 24 GB)	one shared 121 GiB pool
Copy data RAM → VRAM over PCIe	CPU and GPU read the same memory
"Will it fit?" limited by VRAM (24 GB)	limited by total RAM (121 GB)

This is unified memory. nvidia-smi reports "Not Supported" for GPU memory
because there is no separate VRAM to report: the GPU's memory is the system's
121 GiB. In infra terms, a discrete GPU pays a "copy tax" moving weights across
the PCIe bus, like shuffling data between two services with separate caches;
unified memory removes that hop, like two services sharing one in-memory cache.

The practical consequence for me: on the Spark, the ceiling on model size isn't
a stingy 24 GB of VRAM; it's 121 GB. But I have to track model memory
differently, via free -h or PyTorch's own counters, not the nvidia-smi
memory column. The trade-off (which I'll measure later) is that shared memory
usually has lower peak bandwidth than a top-end discrete card's dedicated VRAM,
so the Spark trades some raw speed for the ability to fit much larger models.

Driver vs CUDA runtime vs CUDA toolkit

The most confusing part of the NVIDIA stack for a newcomer is that "CUDA" isn't
one thing: it's three separate layers, installed and versioned independently.
Mapping each to infrastructure I already understand finally made it stick:

Layer	What it is	Infra analogy	Who needs it
NVIDIA driver	kernel module that talks to the GPU	a device driver	everyone using the GPU
CUDA runtime (libcudart)	shared libs an app calls to run GPU work	the `.so` libs you link	anyone running GPU programs
CUDA toolkit	the `nvcc` compiler, headers, profilers	gcc + headers + build tools	only people compiling CUDA

The insight that unblocked me: you can run GPU code without the toolkit.
PyTorch ships its own copy of the CUDA runtime inside its wheel. So to run
models I need the driver (system-level) plus a CUDA-enabled PyTorch (which
brings its own runtime). I do not need nvcc; that's only for compiling
custom CUDA kernels, a much-later CUDA-track activity.

With that lens, two clues from the inventory make sense. First, the header line
CUDA Version: 13.0 is the maximum CUDA the driver supports, a ceiling, not
what's installed. That's the number that matters this week: when I install
PyTorch, I must pick a CUDA build ≤ 13.0 so the driver can run it.

Second, the alarming-looking one:

ssh spark 'nvcc --version'

bash: nvcc: command not found

This looks broken but isn't. It only means the toolkit's compiler isn't on my
PATH. The driver and runtime clearly work; nvidia-smi talks to the GPU. And
the toolkit is physically installed:

ssh spark 'ls -d /usr/local/cuda*'

/usr/local/cuda  /usr/local/cuda-13  /usr/local/cuda-13.0

So nvcc exists; it's just at /usr/local/cuda/bin/nvcc, not on PATH. Rather
than assert that, I confirmed it by calling the full path:

ssh spark '/usr/local/cuda/bin/nvcc --version'

Cuda compilation tools, release 13.0, V13.0.88

The toolkit is version 13.0.88, matching the driver's CUDA 13.0 ceiling, a
healthy, consistent stack. Nothing is broken; command not found was a PATH
issue, not a missing install. If I ever want nvcc on PATH, it's one line:
export PATH=/usr/local/cuda/bin:$PATH. But to run models, I never need it.

Installing PyTorch, the right way: a virtual environment

PyTorch is the Python library I'll use to talk to the GPU. Before installing it,
one habit worth keeping: never install into the system Python. I use a
virtual environment (venv): an isolated per-project Python with its own
packages, exactly like a per-service dependency sandbox so one project's
libraries can't break another's. On the Spark that's why the earlier
python3 -c "import torch" failed: the system Python genuinely has no torch, and
I want to keep it that way.

If you're following along, first check whether you already have PyTorch, since
many setups ship with it:

ssh spark 'python3 -c "import torch, sys; print(torch.__version__)" \
  || echo "no torch in this Python"'

I didn't, so I made a clean venv and installed torch into it:

ssh spark 'python3 -m venv ~/venvs/w1'
ssh spark '~/venvs/w1/bin/python -m pip install --upgrade pip'
ssh spark '~/venvs/w1/bin/python -m pip install torch numpy'

The install is packaged as setup.sh in the companion repo, which just runs the
three steps above against requirements.txt:

#!/usr/bin/env bash
set -euo pipefail
VENV="{% katex inline %}{VENV:-{% endkatex %}HOME/venvs/w1}"
HERE="{% katex inline %}(cd "{% endkatex %}(dirname "$0")" && pwd)"
python3 -m venv "$VENV"
"$VENV/bin/python" -m pip install --upgrade pip
"{% katex inline %}VENV/bin/python" -m pip install -r "{% endkatex %}HERE/requirements.txt"

The interesting part is what pip pulled in. Remember the stack tops out at CUDA
13.0, and without me specifying any special index, PyPI served an ARM64 +
CUDA 13 build automatically:

Successfully installed torch-2.13.0 nvidia-cuda-runtime-13.0.96
  nvidia-cudnn-cu13-9.20.0.48 nvidia-cublas-13.1.1.3 ... (aarch64 wheels)

This is the payoff of the "PyTorch ships its own CUDA runtime" point from
earlier: I did not install the CUDA toolkit or touch nvcc. Torch brought
its own CUDA 13 runtime libraries (libcudart, cuDNN, cuBLAS) as ordinary
Python wheels, matched to the ARM64 architecture and the driver's CUDA 13
ceiling. The driver was the only piece I needed pre-installed.

Proving the GPU actually works from Python

Now the moment this whole week builds to: does Python see the GPU? The
validation script (validate_gpu.py in the repo) is deliberately tiny:

import torch

print("PyTorch:", torch.__version__)
print("CUDA available:", torch.cuda.is_available())
print("CUDA version (torch):", torch.version.cuda)

if torch.cuda.is_available():
    print("Device:", torch.cuda.get_device_name(0))
    print("Capability:", torch.cuda.get_device_capability(0))
    x = torch.rand(3, 3, device="cuda")   # put a real tensor on the GPU
    print("Tensor on:", x.device)

Run it with the venv's Python:

ssh spark '~/venvs/w1/bin/python validate_gpu.py'

PyTorch: 2.13.0+cu130
CUDA available: True
CUDA version (torch): 13.0
Device: NVIDIA GB10
Capability: (12, 1)
Tensor on: cuda:0

Every line here is worth reading. CUDA available: True means PyTorch found a
usable GPU through the driver. 2.13.0+cu130 confirms it's a CUDA 13 build.
Device: NVIDIA GB10 is our chip. Capability: (12, 1) is the GPU's compute
capability, NVIDIA's versioning for GPU features; 12.x is the Blackwell
generation. And Tensor on: cuda:0 is the real proof: the tensor physically
lives in GPU memory, not on the CPU. That's the difference between "installed"
and "actually usable."

Testing the prediction: CPU vs GPU

Earlier I predicted the GPU would beat the CPU by 10x-50x on a large matrix
multiply. Time to measure instead of hand-wave. The benchmark
(benchmark.py) multiplies two 4096x4096 matrices 20 times on each device and
averages. Two details make the GPU timing honest: a warmup (the first GPU
call pays a one-time kernel-load cost, so I run a few throwaway iterations
first), and torch.cuda.synchronize() before stopping the clock (CUDA launches
kernels asynchronously, so without a sync I'd be timing queueing, not
computing):

def bench(a, b, sync=False):
    if sync: torch.cuda.synchronize()
    t0 = time.perf_counter()
    for _ in range(ITERS):
        c = a @ b
    if sync: torch.cuda.synchronize()
    return (time.perf_counter() - t0) / ITERS

cpu_s = bench(a, b)                       # on CPU
ag, bg = a.cuda(), b.cuda()
for _ in range(3): _ = ag @ bg            # warmup
gpu_s = bench(ag, bg, sync=True)          # on GPU

That is the core idea; the full benchmark.py also times the very first (cold)
GPU call and runs a TF32 pass, which is where the extra numbers below come from.

ssh spark '~/venvs/w1/bin/python benchmark.py'

Matrix: 4096x4096 float32, iters: 20
CPU        172.57 ms   0.80 TFLOP/s
GPU cold     135.73 ms   (first call: one-time kernel load)
GPU FP32       7.67 ms   17.9 TFLOP/s   22.5x faster than CPU
GPU TF32       3.42 ms   40.1 TFLOP/s   50.4x faster than CPU

A single 4096x4096 float32 matmul is about 137 billion floating-point
operations. The 20-core ARM CPU works through it in ~173 ms (~0.80 TFLOP/s); the
GB10 does it in ~7.7 ms (~17.9 TFLOP/s), a 22.5x speedup, inside my predicted
10x-50x range. That gap is why GPUs exist: the same math, done thousands at a
time instead of a few at a time.

Two details in that output are worth calling out. First, the cold GPU call
took 135 ms, nearly as long as the whole CPU run, then every warm call took 7.7
ms. That is the one-time kernel-load cost I predicted; it is exactly why the
benchmark warms up before timing. Second, I measured a TF32 number too:
40 TFLOP/s, a 50x speedup. TF32 is a lower-precision mode NVIDIA GPUs use on
tensor cores; PyTorch leaves it off for matmul by default, so the honest FP32
number is the 22.5x one. TF32 and other reduced-precision formats are a whole
topic for later weeks, but it is worth seeing early that precision is a dial you
can trade for speed.

One caveat so the number is not oversold: this matmul is compute-bound (it
keeps the cores busy), which is why the GPU looks so good. When a model generates
text token by token, it is usually memory-bandwidth-bound instead, and the
speedups are smaller. That distinction is a big theme later. (All these values
are in results.json in the repo; yours will differ by machine.)

What surprised me

A GPU with no VRAM number is a feature, not a fault. Unified memory reframes "will it fit?" from ~24 GB to 121 GB.
nvcc: command not found is not an error state; the CUDA that matters for running models lives inside PyTorch, not in the toolkit. Installing torch pulled its own CUDA 13 runtime as plain wheels; I never touched the toolkit.
Verifying every fact with a command, even the "obvious" ones, already caught assumptions I would otherwise have carried into later weeks.
The GPU's 22.5x matmul win was real and measurable on day one, not a slide.

What I'll do next

Week 1 is done: I know the machine, I understand the NVIDIA stack, and I've proven
the GPU is usable from Python with a real speedup. Week 2 leaves inventory behind
and runs an actual language model with Ollama, a model runtime with a local HTTP
API, where I'll start measuring the things that matter for serving: tokens per
second and time to first token.

32-week roadmap:
https://github.com/dramasamy/from-api-to-gpu/tree/main/roadmap ↩
Companion code repository:
https://github.com/dramasamy/from-api-to-gpu ↩
Week 1 lab:
https://github.com/dramasamy/from-api-to-gpu/tree/main/week-01-environment ↩