DEV Community: Dinesh Kumar Ramasamy

From API to GPU, Week 4: What Model Parameters Actually Cost in Memory

Dinesh Kumar Ramasamy — Mon, 20 Jul 2026 00:54:57 +0000

Phase 1 of 8: Comfortable running local models. Week 4 of 32.

People throw around "7B", "14B", and "70B" like everyone knows what they mean.
For a long time I nodded along without really knowing. This week I make those
numbers concrete. By the end I can look at a model size and a precision and say
how much memory the weights need, then explain why the running model needs more.

This is the last week of Phase 1. In Week 1 I learned the machine, in Week 2 I
ran a model through Ollama, and in Week 3 I read a model's files on Hugging Face.
Now I connect the parameter count I saw in Week 3 to real memory on the DGX
Spark.

The calculator part runs anywhere with Python. The memory measurements run on
the Spark over ssh spark, the same setup as earlier weeks.

What a parameter actually is

A parameter is a single learned number inside the model. Training adjusts
billions of these numbers until the model predicts text well. When a model is
called "3B", it has about 3 billion of these numbers.

Wait, is a parameter the same thing as a token? No. A token is a piece of
text that enters or leaves the model. Tokens are the request and response data;
parameters are the learned numbers stored in the model files.

	Token	Parameter
What it is	a piece of input or generated text	a learned number
Where it comes from	the prompt or model response	training
What happens during inference	input tokens arrive and output tokens are added	values stay fixed
What its count affects here	context and KV-cache use	weight memory

They are connected, though. The tokenizer turns each token into an ID. The
model uses learned parameters to turn those IDs into internal values, process
them, and predict the next token. A 3B model therefore has about 3 billion
parameters, not 3 billion tokens. The same parameters are reused for every
token the model processes.

Parameter is the umbrella term. Two kinds sit under it:

A weight multiplies an input value. Most parameters are weights.
A bias is another learned value added after the multiply. Models usually have far fewer bias entries than weight entries.

The reported parameter count includes both. So when I say "weight memory" in this
post, I mean the storage for all parameters, weights and biases together. The
count is what drives the memory math.

A few more words show up when you look inside a model, so here they are in plain
terms:

A tensor is a block of numbers. A single number is a scalar, a list is a vector, a grid is a matrix, and a tensor is the general name for any of these.
A matrix is a two-dimensional grid of numbers, the most common shape for a block of weights.
A layer is one processing stage. A model stacks many layers, and each one holds its own tensors.
A shape is the size of a tensor along each dimension, like 2048 by 2048.

stack of layers that forms a language model." width="800" height="533">

I do not need the internals of these this week. Week 5 works with tensors
directly in PyTorch, and Week 11 explains layers. Here they are just the words
for "the numbers we are about to measure".

In Week 3 I read the parameter count from the Hugging Face API. Here I pin the
API request to the same commit used for the weight index later in this post:

$ MODEL=Qwen/Qwen2.5-3B-Instruct
$ REV=aa8e72537993ba99e69dfaafa59ed015b17504d1
$ curl -s "https://huggingface.co/api/models/$MODEL/revision/$REV" | jq '{
    revision: .sha,
    parameters: .safetensors.total,
    precision_groups: .safetensors.parameters
}'
{
  "revision": "aa8e72537993ba99e69dfaafa59ed015b17504d1",
  "parameters": 3085938688,
  "precision_groups": {
    "BF16": 3085938688
  }
}

So this model has 3,085,938,688 parameters, and every one of them is stored in
the BF16 format. That second fact is the other half of the memory question.

Precision: how many bytes each number takes

A parameter is a number, and a number needs storage. Precision is the format
used to store each one. The format decides how many bits, and therefore how many
bytes, every parameter takes.

The short forms are worth spelling out. FP means floating point, a format for
fractional numbers. BF16 means bfloat16, a 16-bit floating-point format. INT
means integer. You saw torch_dtype: bfloat16 in the Week 3 config. Here is what
the common formats cost per parameter:

Precision	Bits	Bytes
FP32	32	4
FP16	16	2
BF16	16	2
INT8	8	1
INT4	4	0.5

FP32 is the 32-bit baseline, using 4 bytes per number. FP16 and BF16 are both
16-bit formats, so both use 2 bytes. INT8 and INT4 are the low-bit formats
produced by quantization, which is a whole phase later in this series (weeks 14
to 16). INT4's 0.5 byte is an average: two 4-bit values pack into one byte. For
now the only thing that matters is the bytes column.

GB versus GiB, before the numbers start

Memory numbers come in two units, and mixing them causes confusion, so I define
them once up front:

GB is decimal: divide bytes by 1,000,000,000.
GiB is binary: divide bytes by 1024 three times (1024 x 1024 x 1024).

The calculator prints both. Ollama later prints its own rounded GB labels, so
those are approximate runtime reports, not byte-exact comparisons.

The weight-memory formula

Put the two facts together and the weight memory is simple multiplication:

weight bytes = parameter count \times \frac{bits per parameter}{8}

Dividing by 8 converts bits to bytes. That is the whole formula. The Week 4
deliverable is a small calculator that applies it, so I do not redo the
arithmetic by hand each time.

Here is the complete script:

#!/usr/bin/env python3
"""Estimate model weight memory from a parameter count and a precision.

The math is exact for a single uniform precision:

    weight_bytes = parameter_count * bits_per_parameter / 8

This is the tensor payload only. Real runtime memory is larger because of the
KV cache, activations, and framework overhead, and it cannot be derived from the
parameter count. The blog measures that runtime memory separately.
"""

from __future__ import annotations

import argparse

BITS_PER_PARAMETER = {
    "fp32": 32,
    "fp16": 16,
    "bf16": 16,
    "int8": 8,
    "int4": 4,
}

GB = 1_000_000_000
GIB = 1024 ** 3


def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser(description=__doc__)
    group = parser.add_mutually_exclusive_group(required=True)
    group.add_argument(
        "--parameters",
        type=float,
        help="Parameter count in billions, e.g. 70 or 3.2",
    )
    group.add_argument(
        "--count",
        type=int,
        help="Exact parameter count, e.g. 3085938688",
    )
    parser.add_argument(
        "--precision",
        choices=sorted(BITS_PER_PARAMETER),
        required=True,
    )
    return parser.parse_args()


def weight_bytes(parameter_count: int, precision: str) -> float:
    return parameter_count * BITS_PER_PARAMETER[precision] / 8


def main() -> None:
    args = parse_args()
    if args.count is not None:
        parameter_count = args.count
    else:
        parameter_count = round(args.parameters * 1_000_000_000)

    raw_bytes = weight_bytes(parameter_count, args.precision)

    print(f"parameter_count    {parameter_count:,}")
    print(f"precision          {args.precision}")
    print(f"bits_per_parameter {BITS_PER_PARAMETER[args.precision]}")
    print(f"weight_bytes       {raw_bytes:,.0f}")
    print(f"weight_gb          {raw_bytes / GB:.2f} GB (decimal, / 1e9)")
    print(f"weight_gib         {raw_bytes / GIB:.2f} GiB (binary, / 1024^3)")


if __name__ == "__main__":
    main()

I run the roadmap example first: a 70-billion-parameter model at INT4.

$ python3 public/week-04-parameters-memory/model_memory.py \
  --parameters 70 --precision int4
parameter_count    70,000,000,000
precision          int4
bits_per_parameter 4
weight_bytes       35,000,000,000
weight_gb          35.00 GB (decimal, / 1e9)
weight_gib         32.60 GiB (binary, / 1024^3)

70 billion parameters at half a byte each is 35 GB of weights. The same 70B
model at BF16 would be four times that, about 140 GB. That is roughly 130 GiB,
already more than the Spark's 121 GiB pool before any runtime memory. This is
why big models are so often quantized before anyone tries to run them locally.

Checking the formula against a real file

A formula is only trustworthy if it matches reality. I ran the calculator with
the exact Qwen count and its real BF16 precision:

$ python3 public/week-04-parameters-memory/model_memory.py \
  --count 3085938688 --precision bf16
parameter_count    3,085,938,688
precision          bf16
bits_per_parameter 16
weight_bytes       6,171,877,376
weight_gb          6.17 GB (decimal, / 1e9)
weight_gib         5.75 GiB (binary, / 1024^3)

The calculator says 6,171,877,376 bytes. Now the model's own Safetensors index,
read directly with the pinned revision from Week 3:

$ MODEL=Qwen/Qwen2.5-3B-Instruct
$ REV=aa8e72537993ba99e69dfaafa59ed015b17504d1
$ curl -sL "https://huggingface.co/$MODEL/raw/$REV/model.safetensors.index.json" \
    | jq '.metadata.total_size'
6171877376

Identical. The formula is not a guess. For a uniform precision it lands exactly
on the tensor-data bytes the index reports. That total is the parameter payload,
not the whole file with its headers, but for the weight math it is the number I
want.

Why the running model needs more than its weights

Everything so far is weight storage: the model sitting on disk. A model doing
work needs more memory than that. The extra memory has three main parts:

Activation memory is the temporary numbers created while processing a request. They come and go as the model runs.
KV-cache memory stores saved intermediate data from earlier tokens, so the model does not recompute them for every new token. This grows with context length. The KV cache is a Week 18 topic; here I only need to see that it exists and costs memory.
Framework overhead is the runtime's own working memory and buffers.

The exact-file check used BF16 Qwen. This runtime experiment switches to
Ollama's quantized llama3.2:3b, the model already available from Week 2. Its
2.0 GB package, 2.6 GB load, and 4.1 GB load belong to one comparison; they
should not be compared with Qwen's 6.17 GB BF16 weights.

Ollama reports the loaded size of a model, so I can compare it with the 2.0 GB
package size. I expect the 4,096-token load to exceed 2.0 GB because the runtime
needs memory beyond the package. I expect the 16,384-token load to be larger
again because it reserves more KV-cache capacity. ollama ps reports the total,
so this experiment can show the change but cannot assign an exact number of
bytes to the KV cache.

I connect to the Spark once, then run each check directly:

$ ssh spark
$ ollama list | awk 'NR == 1 || $1 == "llama3.2:3b"'
NAME                       ID              SIZE      MODIFIED
llama3.2:3b                a80c4f17acd5    2.0 GB    3 days ago

Next I unload any existing copy, send one short request with a 4,096-token
context, and inspect the loaded model. The API's HTTP 200 confirms that the
otherwise silent request completed:

$ ollama stop llama3.2:3b >/dev/null 2>&1 || true
$ curl -fsS -o /dev/null -w 'HTTP %{http_code}\n' \
    http://localhost:11434/api/generate \
    -d '{"model":"llama3.2:3b","prompt":"hi","stream":false,
    "options":{"num_ctx":4096,"num_predict":1}}'
HTTP 200
$ ollama ps
NAME           ID              SIZE      PROCESSOR    CONTEXT    UNTIL
llama3.2:3b    a80c4f17acd5    2.6 GB    100% GPU     4096       4 minutes from now

I repeat the same request with a 16,384-token context:

$ ollama stop llama3.2:3b >/dev/null 2>&1 || true
$ curl -fsS -o /dev/null -w 'HTTP %{http_code}\n' \
    http://localhost:11434/api/generate \
    -d '{"model":"llama3.2:3b","prompt":"hi","stream":false,
    "options":{"num_ctx":16384,"num_predict":1}}'
HTTP 200
$ ollama ps
NAME           ID              SIZE      PROCESSOR    CONTEXT    UNTIL
llama3.2:3b    a80c4f17acd5    4.1 GB    100% GPU     16384      4 minutes from now

Ollama lists the package as 2.0 GB. Loaded with a 4,096-token context, it reports
2.6 GB. Loaded with a 16,384-token context, it reports 4.1 GB. The model did not
change. Only the configured context capacity changed, and the reported loaded
size grew by 1.5 GB. The KV cache grows with context, so it is the expected main
cause, but ollama ps reports one total and does not split KV cache, activations,
and framework buffers.

This is the practical lesson: weight memory is the floor, not the total. Runtime
memory is larger, and context length is one of the settings that pushes it up.

Results

Here is Week 4 in one place:

Item	Verified value
Formula	parameters × bits / 8
70B at INT4	35.00 GB of weights
Qwen2.5-3B-Instruct at BF16	6,171,877,376 bytes
Formula versus Safetensors index	exact match
Loaded at 4,096 context	2.6 GB
Loaded at 16,384 context	4.1 GB
Spark unified memory	121 GiB

What surprised me

The formula matching the Safetensors index to the exact byte was satisfying.
Weight memory really is just multiplication once you know the count and the
precision.

The context effect was the most useful. I knew the KV cache existed from Week 2,
but seeing a 2.0 GB package report 4.1 GB of loaded memory just by raising
context made it concrete.

Mistakes and troubleshooting

I also had to be careful about GB versus GiB. The calculator prints both, and
Ollama prints its own rounded GB labels, so I keep the units visible to avoid
comparing the wrong things.

Production implications

Weight memory sets the floor for whether a model fits. For capacity planning I
would:

Estimate weights from the count and precision, then confirm against the real file for quantized models.
Measure total runtime memory with the real prompt lengths and workload, since the KV cache and activations depend on how the model is actually used, not on the parameter count.
Treat the Spark's 121 GiB unified pool as shared by the model, its runtime, and everything else on the box.

The formula tells me if the weights fit. The runtime measurement tells me if the
working model fits. Both matter before scheduling hardware.

What I will learn next

Week 5 moves into PyTorch tensors directly. I will allocate tensors on the CPU
and GPU, compare FP32, FP16, and BF16, and watch allocated memory change. That
turns this week's byte math into something I can measure inside a running Python
process.¹

Weeks 14 to 16 cover quantization properly, which is where the Q4_K_M mixed
format and its real bit cost get explained in full.

Run it yourself

The public Week 4 lab has the calculator, direct runtime commands, captured
results, observations, and troubleshooting notes.² The
quantized-file note holds the optional Q4_K_M storage investigation with its
full commands and output.³

Optional depth: why Q4 uses more than four bits on disk

The clean formula assumes every parameter uses the same number of bits. A
quantized file can mix several tensor formats and store extra scale data, so a
model labeled "4-bit" may use more than 4 bits per parameter on disk.

I checked Ollama's llama3.2:3b package. Its API reports 3,212,749,888
parameters and Q4_K_M quantization.⁴ A clean INT4 calculation predicts
1,606,374,944 bytes, but the model blob is 2,019,377,376 bytes. That works out
to about 5.03 on-disk bits per parameter. I also matched the blob's byte size
and SHA-256 digest to its manifest record, so this is the exact file Ollama
references, not only a filename that looks right.

This was the opposite of the uniform BF16 result. I expected a "4-bit" model to
cost 4 bits per parameter, but this file is about 26 percent larger than the
clean estimate. The fix was to use the manifest and real blob instead of treating
the quantization label as an exact file-size promise.

The companion note keeps the full manifest-to-blob chain and tensor-type
counts.³ For Week 4, the rule is enough: use the formula for a
first estimate, then read the real file size for a quantized model.

Week 5 roadmap:
https://github.com/dramasamy/from-api-to-gpu/blob/main/roadmap/week-05.md ↩
Week 4 companion lab:
https://github.com/dramasamy/from-api-to-gpu/tree/main/week-04-parameters-memory ↩
Why a Q4 model uses more than four bits per parameter:
https://github.com/dramasamy/from-api-to-gpu/blob/main/week-04-parameters-memory/quantized-file-size.md ↩
GGUF and its K-quant mixed formats are from the llama.cpp project:
https://github.com/ggml-org/ggml/blob/master/docs/gguf.md ↩

From API to GPU, Week 3: Reading a Hugging Face Model Repository

Dinesh Kumar Ramasamy — Sun, 19 Jul 2026 23:00:44 +0000

Phase 1 of 8: Comfortable running local models. Week 3 of 32.

In Week 2 I ran Phi-4 through Ollama. Ollama gave me a friendly model name and
hid most of the files underneath it. This week I remove that layer and inspect a
model at its source: its Hugging Face page.

The natural way to get to know a model is to open its web page, read the model
card, and click through the files. So that is exactly how I start here, in an
ordinary browser. By the end I can answer four questions before I download
anything:

What architecture and size is this model?
Is it a base model or an instruction-tuned model?
How much data will I download?
Am I legally allowed to use it for my product?

I inspect Qwen/Qwen2.5-3B-Instruct. Everything in this post is done by
browsing its Hugging Face page: no GPU, no API key, no account. The companion
lab repeats every one of these answers as reproducible curl and jq commands
plus a small Python inspector, the "API way", for anyone who wants to script
it.¹

The model card page is the front door

I open the model's page in a browser.² Before reading any file,
the top of the page already answers a lot.

name, the tag row, the license badge, and the sidebar showing model size and
tensor type." width="800" height="320">

Reading the header top to bottom:

Qwen / Qwen2.5-3B-Instruct is the repository: the owner (Qwen) and the model name. This is the exact ID I hand to any tool that downloads the model.
The tag row shows the task (Text Generation), the framework (Transformers), the weight format (Safetensors, a model-weight file format), and a chat tag.
License: qwen-research is a license label, not the license itself. I come back to it below, because the label alone does not tell me whether I can use this commercially.
The three tabs are Model card, Files and versions, and Community. The whole week lives in the first two.
The right sidebar already says Model size 3B params, Tensor type BF16 (bfloat16, a 16-bit number format), and Chat template. So before opening a single file I know this is a three-billion-parameter model, stored in a 16-bit format, that ships a chat template.

The four names I kept mixing up

The page uses four words for different parts of the same release, and I kept
mixing them up at first:

Term	Plain meaning	Where it shows on the page
Model family	related models released together	the `Qwen2.5` name and collection
Architecture	the neural-network layout	the `config.json` file (opened below)
Checkpoint	the learned parameter values	the `.safetensors` weight files
Repository	the versioned files that ship the model	the whole page, owner plus name

The repository is the page itself: the owner Qwen plus the name
Qwen2.5-3B-Instruct. A Hugging Face repository is a Git repository designed to
hold large ML files, so it carries a full commit history like any other
repo.³

The model family is Qwen2.5. The name and its linked collection show the
family includes several sizes plus base and instruction-tuned
variants.⁴

The architecture is the neural-network layout. The page does not print it in
the header; it lives in config.json, which I open later. Architecture describes
the shape (layers, hidden width, attention heads), not the learned knowledge.

A checkpoint is the learned numbers stored in the weight files. Two models
can share the same architecture but hold different checkpoints, just as two
containers can run the same application build with different data. The base and
instruct models are exactly that: same shape, different learned weights.

Base model versus instruct model

A base model learns to predict the next token from a large training set. It
is useful as a starting point for more training, but it is not automatically a
good chat assistant.

An instruct model starts from a base checkpoint and receives post-training
that teaches it to follow requests and behave better in conversations. The
model card is the README.md shown on the Model card tab. It describes what
the model is, how it was trained, and its limits. Scrolling down the card, the
model states plainly what it is:

parameter count, layer count, attention-head layout, and context
length." width="799" height="434">

The card says "This repo contains the instruction-tuned 3B Qwen2.5 model" and
lists Training Stage: Pretraining & Post-training. That post-training step is
what makes it an instruct model. The same block answers most of question 1 too:
3.09B parameters, 36 layers, 16 query heads and 2 key/value heads
(more on that split in Week 10 and Week 13), and a native 32,768-token
context length.

The card is a claim by the publisher. The page backs it with a machine-readable
link. The Model tree box on the same page names the base model this one was
fine-tuned from:

repository as a fine-tune of it." width="800" height="343">

So the lineage is explicit: Qwen/Qwen2.5-3B-Instruct is a fine-tune of
Qwen/Qwen2.5-3B.⁵² Fine-tuning means continuing
training from an existing checkpoint; here it is the post-training that turns the
base model into an instruction follower. The architecture stays the same while
post-training changes the weights. The companion lab confirms the weights really
differ by comparing the two checkpoints' Hub-reported file fingerprints, but the
model card and the Model tree already answer the question for a normal user.

The license is the first thing to check

Question 4 (am I allowed to use this?) is the one I answer before running
anything, because it can stop a project before it starts. The header badge says
qwen-research, a non-standard license, so the badge alone is not enough. I
click the LICENSE file on the Files tab and read it:

NON-COMMERCIAL PURPOSES ONLY and requiring a separate license for commercial
use." width="800" height="600">

The LICENSE file is the Qwen Research License Agreement. Line 19 grants use
"FOR NON-COMMERCIAL PURPOSES ONLY", and the next line says commercial use
requires a separate license from Alibaba Cloud.⁶

That is the most important result this week. The model page looks ready to use,
but this 3B release is not licensed for commercial use by default. License review
has to happen before model evaluation, not after a prototype is built. The Hub's
machine-readable label for this is license: other, a non-standard license, a
detail I show in the lab.

The Files and versions tab

The Files and versions tab lists everything the repository ships. This is
where question 3 (how much will I download?) gets answered:

of 6.18 GB, and the latest commit on the main branch." width="800" height="600">

The tab shows a total of 6.18 GB across 12 files, and the header line
records the branch (main), the contributor count, and the latest commit. To
make the list easier to learn, I group the twelve files into six groups, which
is just my reading order, not anything Hugging Face labels:

Docs and license: README.md (the model card) and LICENSE.
Model shape: config.json.
Generation defaults: generation_config.json.
Tokenizer: tokenizer.json, tokenizer_config.json, vocab.json, merges.txt.
Weights: model-00001-of-00002.safetensors, model-00002-of-00002.safetensors, and their index model.safetensors.index.json.
Git plumbing: .gitattributes, which configures Git LFS and is not model data.

Clicking any file opens it in the browser. The rest of this post opens the small
ones and reads them.

`config.json` is the architecture contract

config.json tells a runtime how to build the model before loading any weights.
It is tiny, 661 bytes, so I open it and read the whole thing:

{
  "architectures": [
    "Qwen2ForCausalLM"
  ],
  "attention_dropout": 0.0,
  "bos_token_id": 151643,
  "eos_token_id": 151645,
  "hidden_act": "silu",
  "hidden_size": 2048,
  "initializer_range": 0.02,
  "intermediate_size": 11008,
  "max_position_embeddings": 32768,
  "max_window_layers": 70,
  "model_type": "qwen2",
  "num_attention_heads": 16,
  "num_hidden_layers": 36,
  "num_key_value_heads": 2,
  "rms_norm_eps": 1e-06,
  "rope_theta": 1000000.0,
  "sliding_window": 32768,
  "tie_word_embeddings": true,
  "torch_dtype": "bfloat16",
  "transformers_version": "4.43.1",
  "use_cache": true,
  "use_sliding_window": false,
  "vocab_size": 151936
}

This is where the architecture name lives: Qwen2ForCausalLM. The header did
not print it, but the file does. I do not need to understand every field in
Week 3. My goal today is to find and record them, and their deeper lessons are
already scheduled:

Week 4 turns torch_dtype and the parameter count into weight-memory math. Week 5 compares FP32, FP16, and BF16 tensors on CPU and GPU.
Week 9 explains vocab_size, tokenizer files, and special token IDs.
Week 10 explains attention heads. Week 13 returns to why this model has 16 query heads (num_attention_heads) but only 2 key/value heads (num_key_value_heads).
Week 11 explains num_hidden_layers, hidden_size, and intermediate_size, and how the architecture class builds transformer blocks from them.
Week 18 compares max_position_embeddings with the tokenizer limit, the runtime setting, and the context length that is practical in memory.

The important distinction is simpler than the individual fields: config.json
describes the model's shape, while the checkpoint files contain the learned
numbers that fill that shape.

`generation_config.json` supplies defaults, not hard limits

This file holds the default sampling settings for text generation. It is 242
bytes, so again I open the whole thing:

{
  "bos_token_id": 151643,
  "pad_token_id": 151643,
  "do_sample": true,
  "eos_token_id": [
    151645,
    151643
  ],
  "repetition_penalty": 1.05,
  "temperature": 0.7,
  "top_p": 0.8,
  "top_k": 20,
  "transformers_version": "4.37.0"
}

These are defaults, not training facts, and a runtime or API request can override
them. I changed generation temperature in Week 2 and briefly introduced top-p
there. Week 12 is the full generation-and-sampling lesson, where I change top-k,
top-p, temperature, repetition penalty, and seed one at a time. In Week 3 I only
need to know that generation_config.json is where this repository stores its
defaults, and that it names IDs 151643 and 151645 as end-of-generation
markers. The field names spell out the roles: bos_token_id is the
beginning-of-sequence (BOS) marker, eos_token_id the end-of-sequence (EOS)
marker, and pad_token_id the padding marker. One ID can fill several roles:
here 151643 is BOS and padding and is also one of the two accepted EOS IDs.

The tokenizer is several files working together

The tokenizer turns text into token IDs. This repository ships it as several
files, all visible on the Files tab:

vocab.json maps token text to IDs.
merges.txt holds byte-pair encoding merge rules.
tokenizer_config.json stores the special tokens and the chat template.
tokenizer.json packages the whole tokenizer into one file for fast loading.

Byte-pair encoding (BPE) builds tokens by repeatedly joining common text
pieces according to the merge rules.⁷ This is one tokenizer method, not a
universal standard. Other model families can use different algorithms and file
layouts.

tokenizer_config.json is larger than the config files, so instead of reading it
top to bottom I look for two things: the special tokens and the chat template.
Opening it and scrolling to the tokenizer class and the added_tokens_decoder
map, the fields that matter here are:

{
  "tokenizer_class": "Qwen2Tokenizer",
  "model_max_length": 131072,
  "added_tokens_decoder": {
    "151643": { "content": "<|endoftext|>", "special": true },
    "151644": { "content": "<|im_start|>", "special": true },
    "151645": { "content": "<|im_end|>", "special": true }
  }
}

That maps the three special tokens that matter here:

Token ID	Text	Role
151643	`<	endoftext
151644	{% raw %}`<	im_start
151645	{% raw %}`<	im_end

This answers a question left over from Week 2. With Phi-4 in Ollama I saw
{% raw %}<|im_start|>, <|im_end|>, and <|im_sep|> inside the packaged GGUF model.
Now I can see where the source repository keeps that information:
generation_config.json names which IDs act as start, end, and padding markers,
and tokenizer_config.json maps those IDs to text and stores the chat template
that places them around messages. The values belong to each model's tokenizer:
Phi-4 used IDs 100264 to 100266, Qwen uses 151643 to 151645, and Qwen has no
<|im_sep|>. I should never copy special-token IDs or assume the same chat
format across model families.

The chat template is a template written in Jinja, a text templating language,
stored inside tokenizer_config.json.⁸ It converts role-based messages
into the special-token sequence the model expects, the same job as the ChatML
markers I inspected in Week 2.⁹ The full template is worth a
look, so I show it after the file tour in
the chat-template walkthrough.

One number on the tokenizer looks contradictory at first. tokenizer_config.json
sets model_max_length to 131,072, while config.json says 32,768. The two
fields describe different things: model_max_length is tokenizer metadata (a
safety cap on input length), while max_position_embeddings in config.json is
this checkpoint's native architecture limit. The model card's introduction
mentions the wider Qwen2.5 family supports up to 128K tokens, but this specific
3B checkpoint lists a native 32,768-token context, which matches config.json.
The 131,072 setting is not proof that this checkpoint can safely use that length,
so I treat 32,768 as the verified native limit.²

Safetensors files are the checkpoint

The two .safetensors files hold the learned tensors, the multidimensional
arrays of numbers that are the model's parameters. Safetensors is a weight-file
format designed to load those arrays without the code-execution risk of Python's
Pickle format.¹⁰ It is one option, not the only one. Week 2 used
GGUF, a different format that packages weights and metadata together.

The checkpoint is split into two shards, model-00001-of-00002.safetensors
(3.97 GB) and model-00002-of-00002.safetensors (2.2 GB). The third weight file,
model.safetensors.index.json, is not weights at all: it is a map from each
tensor name to the shard that holds it. Opening it, the top is a metadata block
followed by a weight_map with one entry per tensor:

{
  "metadata": { "total_size": 6171877376 },
  "weight_map": {
    "model.embed_tokens.weight": "model-00001-of-00002.safetensors",
    "model.layers.0.input_layernorm.weight": "model-00001-of-00002.safetensors"
  }
}

The real weight_map has 434 entries (I show two); that count is how many
tensors a loader must place. Splitting one checkpoint into shards just makes it
easier to upload, cache, and download; both shards are required, and two files do
not mean two models.

The metadata.total_size field records the total tensor size: 6,171,877,376
bytes. That number is a good check on the "3B params" and "BF16" labels from the
sidebar. BF16 means each parameter uses 16 bits, or 2 bytes,¹¹ so:

6, 171, 877, 376 bytes \div 2 = 3, 085, 938, 688 parameters

That is 3.09 billion parameters, exactly what the model card rounds to. Next week
I turn this calculation into a reusable memory tool.

How do I know it is really an instruct model?

There is no universal is_instruct: true field. Here is the evidence the page
gives, strongest first:

Publisher statement: the model card says "instruction-tuned" and "Pretraining & Post-training". That is why I call it an instruct model.
Model relationship: the Model tree names the base model as Qwen/Qwen2.5-3B, and the repository carries the chat tag.
Different learned checkpoint: the companion lab reads only the file fingerprints and shows the base and instruct first shards have different SHA-256 hashes, which proves the weight bytes differ.¹²
Chat-oriented defaults: the instruct generation_config.json enables sampling and lists <|im_end|> as an end token, which the base model does not.

Most architecture values are identical between the two, because post-training
changes the learned weights, not the model's shape. Both still have 36 layers
and the same parameter count. Confirming that with real file hashes is the lab's
job; the model card and the Model tree already answer it for day-to-day use.

The Qwen chat template, connected to Week 2

This section is optional depth. The file tour already gave me what I need for
Week 3, but the full template makes the Week 2 roles and markers concrete.

The template is stored as one long string inside tokenizer_config.json. Opening
that file in the browser shows it escaped on a single line; the lab prints it
cleanly with one jq command. Formatted, it reads:

{%- if tools %}
    {{- '<|im_start|>system\n' }}
    {%- if messages[0]['role'] == 'system' %}
        {{- messages[0]['content'] }}
    {%- else %}
        {{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }}
    {%- endif %}
    {{- "\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
    {%- for tool in tools %}
        {{- "\n" }}
        {{- tool | tojson }}
    {%- endfor %}
    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
{%- else %}
    {%- if messages[0]['role'] == 'system' %}
        {{- '<|im_start|>system\n' + messages[0]['content'] + '<|im_end|>\n' }}
    {%- else %}
        {{- '<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n' }}
    {%- endif %}
{%- endif %}
{%- for message in messages %}
    {%- if (message.role == "user") or (message.role == "system" and not loop.first) or (message.role == "assistant" and not message.tool_calls) %}
        {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
    {%- elif message.role == "assistant" %}
        {{- '<|im_start|>' + message.role }}
        {%- if message.content %}
            {{- '\n' + message.content }}
        {%- endif %}
        {%- for tool_call in message.tool_calls %}
            {%- if tool_call.function is defined %}
                {%- set tool_call = tool_call.function %}
            {%- endif %}
            {{- '\n<tool_call>\n{"name": "' }}
            {{- tool_call.name }}
            {{- '", "arguments": ' }}
            {{- tool_call.arguments | tojson }}
            {{- '}\n</tool_call>' }}
        {%- endfor %}
        {{- '<|im_end|>\n' }}
    {%- elif message.role == "tool" %}
        {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != "tool") %}
            {{- '<|im_start|>user' }}
        {%- endif %}
        {{- '\n<tool_response>\n' }}
        {{- message.content }}
        {{- '\n</tool_response>' }}
        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
            {{- '<|im_end|>\n' }}
        {%- endif %}
    {%- endif %}
{%- endfor %}
{%- if add_generation_prompt %}
    {{- '<|im_start|>assistant\n' }}
{%- endif %}

I do not need to understand every branch yet. The four parts that connect to
Week 2 are:

A system message becomes <|im_start|>system, its content, then <|im_end|>.
The loop does the same for user and assistant messages.
If no system message is supplied, the template inserts Qwen's default system message.
add_generation_prompt ends with <|im_start|>assistant, telling the model which role should speak next.

The first branch also formats tool calls. I leave that branch for Week 27. The
important point here is that the API's role-based messages do not go straight to
the model. The tokenizer's chat template turns them into the model-specific text
and special-token sequence first.

Results

Browsing the page answered every item from the Week 3 plan:

Question	Verified answer
Repository	`Qwen/Qwen2.5-3B-Instruct`
Latest commit	`aa8e725` (shown on the Files tab)
Architecture	`Qwen2ForCausalLM`
Parameters	3,085,938,688 BF16 values
Layers	36
Hidden size	2,048
Attention configuration	16 query heads, 2 shared key/value heads
Vocabulary	151,936 token IDs
Native configured context	32,768 tokens
BOS ID	151643
EOS IDs	151645 and 151643
Weight shards	2 files, 6,171,877,376 tensor bytes
License	Qwen Research License, non-commercial

The machine-readable copy is saved as results.json in the public lab.

What surprised me

The biggest surprise was the license. A page that looks production-ready can
still be non-commercial by default, and only the LICENSE file says so. Reading
it first is now a habit, not an afterthought.

The second surprise was how much the web page alone answered. Without a GPU, an
API key, or a line of code, I read the architecture, the size, the base-model
lineage, the tokenizer's special tokens, and a 6.18 GB download estimate straight
off the page.

The third surprise was that the round "3B" label and the exact file size agree.
The index reports 6,171,877,376 bytes; divided by 2 bytes per BF16 value, that is
exactly 3,085,938,688 parameters, which is what the card rounds to 3.09B.

Mistakes and troubleshooting

My first instinct was to trust the license badge. The header says qwen-research,
which sounds like a name I could look up and move on. Opening the actual LICENSE
file is what surfaced the non-commercial restriction. The badge is a label; the
file is the contract.

I also assumed the sidebar's "3B params" and the card's "3.09B" were rounded
marketing numbers. The index's exact byte count showed otherwise: the round label
and the precise file size describe the same 3,085,938,688 parameters.

The lab's curl and jq version hits a few API-specific snags (the Hub reports
the license as other, the card object needs to_dict(), and unauthenticated
requests print a rate-limit warning). Those are written up in the lab's
troubleshooting notes, since they only matter for the scripted path.

Production implications

Model selection is dependency management with much larger artifacts. I would
apply the same controls used for containers and packages:

Pin the full repository revision, not mutable main.
Record every weight hash and verify it after download.
Review the model card, then read the actual license.
Estimate download and memory size before scheduling hardware.
Review repository code before enabling trust_remote_code.
Keep tokenizer and generation files with the exact checkpoint they came from.

This repository contains no Python model files, so the metadata inspection did
not execute repository code. That is a useful security property, but I would
check it again for every new model.

What I will learn next

Week 4 turns the parameter count and BF16 data type into memory math. I will
separate raw weight storage from runtime memory, activations, the KV cache, and
framework overhead, then build a calculator for different precisions.¹³

Run it yourself: the API way in the lab

Everything above was done by browsing the web page. The companion Week 3 lab does
the same inspection the "API way", so you can automate or script it:

The curl and jq one-liners that read each field from the Hub API and the raw files, pinned to an exact repository revision.
model-inspector.py, which pulls the file list, config, tokenizer, license, and shard index for any model in one command and can write results.json.
compare-models.py, which compares the base and instruct repositories field by field and checks the Hub-reported SHA-256 (Git LFS) fingerprint of their first weight shard, proving the checkpoints differ.

Its default and verified target is Qwen/Qwen2.5-3B-Instruct. The --model
option works for compatible repositories, but different model families can use
different files and config keys.¹

Week 3 companion lab:
https://github.com/dramasamy/from-api-to-gpu/tree/main/week-03-model-files ↩
Qwen2.5-3B-Instruct model card:
https://huggingface.co/Qwen/Qwen2.5-3B-Instruct ↩
Hugging Face repository documentation:
https://huggingface.co/docs/hub/repositories ↩
Qwen2.5 model collection:
https://huggingface.co/collections/Qwen/qwen25 ↩
Qwen2.5-3B base model card:
https://huggingface.co/Qwen/Qwen2.5-3B ↩
Qwen2.5-3B-Instruct license:
https://huggingface.co/Qwen/Qwen2.5-3B-Instruct/blob/main/LICENSE ↩
Hugging Face Tokenizers model documentation:
https://huggingface.co/docs/tokenizers/main/en/components#models ↩
Jinja template-language documentation:
https://jinja.palletsprojects.com/ ↩
Hugging Face chat-template documentation:
https://huggingface.co/docs/transformers/chat_templating ↩
Safetensors documentation:
https://huggingface.co/docs/safetensors/index ↩
PyTorch data-type documentation:
https://pytorch.org/docs/stable/tensor_attributes.html#torch.dtype ↩
NIST Secure Hash Standard:
https://csrc.nist.gov/pubs/fips/180-4/upd1/final ↩
Week 4 roadmap:
https://github.com/dramasamy/from-api-to-gpu/blob/main/roadmap/week-04.md ↩

From API to GPU, Week 2: What Actually Happens Behind the API

Dinesh Kumar Ramasamy — Sun, 19 Jul 2026 20:19:40 +0000

Phase 1 of 8: Comfortable running local models. Week 2 of 32.

In Week 1 I went through the machine and proved the GPU works. This week I run an
actual language model on it and, for the first time, look at what sits behind the
"AI API" I have called for years. The short version: the API is not magic. It is
an HTTP layer over a local process that loads a file of numbers into memory and
does the matrix math from Week 1. By the end of this post that sentence will be
clear, backed by real commands and real output you can reproduce.

I am using Ollama, a runtime that makes running a local model about as easy as
running a container. Other names you will see in this space are llama.cpp for
lightweight local inference, vLLM for high-throughput serving, and NVIDIA
TensorRT-LLM for optimized NVIDIA inference. Hugging Face Transformers is also
common, but it is a broader Python framework for running and training models,
not a ready-made local model service. These tools overlap, but they are not exact
replacements. I chose Ollama because it gives me a CLI and local HTTP API with
very little setup.¹

Everything below runs on the DGX Spark and is reached over SSH as spark, the
same setup as Week 1.

Concept 1: a model is not a runtime

The first idea to keep straight, because it holds for the rest of the series, is
the split between the model and the runtime.

The model is trained data: weights plus the metadata needed to use them. On its own it does nothing. It is data on disk.
The runtime is the program that loads those weights into memory and runs the math to turn your prompt into text. Ollama is the runtime here.

If you know Docker, there is a useful analogy, but it is not exact. A model
package is like an image made of versioned, content-addressed layers. Ollama is
like the engine that pulls those layers and starts the workload. Unlike a
container image, the main model layer is trained numeric data, not an app and
its operating-system files.

Ollama calls each stored package file a blob. Here, a blob is just a file
kept under a name derived from its content digest. The inspection below follows
the package index to the model blob and checks that it is the expected file.

Rather than run a wrapper script, I inspected the model with a few direct
commands, one at a time. Each command answers a single question, so you can
paste it, read the output, then move to the next.

First, list what Ollama has downloaded locally:

ssh spark 'ollama list'

NAME                       ID              SIZE      MODIFIED
nomic-embed-text:latest    0a109f422b47    274 MB    2 days ago
llama3.2:3b                a80c4f17acd5    2.0 GB    3 days ago
phi4:latest                ac896e5b8b34    9.1 GB    7 days ago
qwen3.6:35b-a3b-bf16       94061ddd23a7    71 GB     8 days ago

Next, ask Ollama for the focused facts about Phi-4. The Ollama server listens on
localhost:11434 on the Spark, so run this in a shell on the Spark
(ssh spark):

curl -s http://localhost:11434/api/show -d '{"model": "phi4"}' | jq '{
  format: .details.format,
    architecture: .details.family,
    parameters: .details.parameter_size,
    context_length: (.model_info | to_entries
      | map(select(.key | endswith(".context_length"))) | first.value),
    embedding_length: (.model_info | to_entries
      | map(select(.key | endswith(".embedding_length"))) | first.value),
    quantization: .details.quantization_level,
    capabilities,
    runtime_parameters: (.parameters | split("\n")
      | map(select(length > 0) | gsub(" +"; " ")))
  }'

{
  "format": "gguf",
  "architecture": "phi3",
  "parameters": "14.7B",
  "context_length": 16384,
  "embedding_length": 5120,
  "quantization": "Q4_K_M",
  "capabilities": [
    "completion"
  ],
  "runtime_parameters": [
    "stop \"<|im_start|>\"",
    "stop \"<|im_end|>\"",
    "stop \"<|im_sep|>\""
  ]
}

That call gives readable model facts, including "format": "gguf". For a
normal check, this is how I know Ollama identifies the model as GGUF. I do not
need to locate the raw file and inspect its bytes just to answer that question.

To see the exact files that make up the package, read Ollama's manifest. It is a
small JSON file on the Spark, so jq can read it directly. I select just the
package version and the file list:

jq '{schemaVersion, layers: [.layers[] | {mediaType, digest, size}]}' \
  /usr/share/ollama/.ollama/models/manifests/registry.ollama.ai/library/phi4/latest

{
  "schemaVersion": 2,
  "layers": [
    {
      "mediaType": "application/vnd.ollama.image.model",
      "digest": "sha256:fd7b6731c33c57f61767612f56517460ec2d1e2e5a3f0163e0eb3d8d8cb5df20",
      "size": 9053114464
    },
    {
      "mediaType": "application/vnd.ollama.image.template",
      "digest": "sha256:32695b892af87ef8fca6e13a1a31c67c1441d7398be037e366e2fc763857c06a",
      "size": 275
    },
    {
      "mediaType": "application/vnd.ollama.image.license",
      "digest": "sha256:fa8235e5b48faca34e3ca98cf4f694ef08bd216d28b58071a1f85b1d50cb814d",
      "size": 1084
    },
    {
      "mediaType": "application/vnd.ollama.image.params",
      "digest": "sha256:45a1c652dddc9efdcefa977ab81cfbe26b6e52bc8e78f2f4c698538783e0ac80",
      "size": 82
    }
  ]
}

The registry path ends in library/phi4/latest, so this manifest is the index
for the model name phi4 and tag latest. It does not hold the 9.1 GB of
weights. It lists the files that make up the Ollama package, much like a lock
file maps package names to exact artifacts.

Each entry under layers is one file in the package (one blob). It has three
useful fields:

mediaType says what the file's role is. This package has a model, a prompt template, a license, and default parameters.
digest is the file's content identity. The sha256 prefix names the hash algorithm, and the characters after the colon are the hash of the file's bytes.
size says exactly how many bytes that file should contain.

The model layer's digest is also how Ollama names the file on disk. It stores
each blob under a blobs directory using the digest as the filename, with the
colon changed to a hyphen, so sha256:fd7b...df20 becomes sha256-fd7b...df20.
That file is 9,053,114,464 bytes, the size shown above, and its SHA-256
matches the digest. The linked investigation independently checks the file
header with xxd: bytes 47 47 55 46 spell GGUF in ASCII. That deeper check
is useful when reusing the raw file in another runtime, but .details.format
is the simple Ollama API answer.²

So ollama show and the manifest answer different questions. The metadata call
gives readable model facts such as architecture and context length. The
manifest tells Ollama which exact files form the runnable package and where to
find them by content ID.

You might now wonder whether another runtime can use the same GGUF file. That is
useful, but it is not part of the main Ollama lesson. I answer it near the end in
Can llama.cpp reuse this Ollama model?.

I am using phi4, Microsoft's 14.7B-parameter model. At 16 bits per weight,
14.7 billion weights alone would need about 29.4 GB. The local package is only
9.1 GB, which leads to the first surprise.

A quick tour of what each line means, since these terms come up constantly:

Field	Value	Plain meaning
format	gguf	model file format reported by Ollama
architecture	phi3	the neural-network design (phi4 reuses the phi3 family)
parameters	14.7B	how many trained weights the model has
context length	16384	the most tokens (prompt + reply) it can consider at once
embedding length	5120	width of each token vector; covered later
quantization	Q4_K_M	the weights are stored at about 4 bits each, not 16

That last row answers the 9.1 GB surprise. This build is quantized: its
weights use a mixed low-bit Q4_K_M representation instead of 16-bit values.
Four bits per weight would be about 7.35 GB before metadata and quantization
overhead, so a 9.1 GB model layer is reasonable.
Quantization is a whole phase later in this series (weeks 14 to 16). For now the
only thing to take away is that Ollama runs lower-precision weights, and that is
why this 14.7B model takes much less space than its 16-bit source model.

The three stop entries (<|im_start|>, <|im_end|>, <|im_sep|>) are
special tokens: markers the model was trained to use as turn boundaries in a
chat. The runtime watches for them to know when the model has finished its reply.
They show up again in the next section, so keep them in mind.

Concept 2: the API is just HTTP over a local process

Ollama exposes a local HTTP API on port 11434,³ the same shape of
thing as any cloud AI API. The difference is that the "server" is a process on
my own machine.
I called the generate endpoint with curl and formatted the reply with jq.
Run it directly on the Spark:

curl -s http://localhost:11434/api/generate -d '{
  "model": "phi4",
  "prompt": "Explain Kubernetes scheduling in three sentences.",
  "stream": false
}' | jq

Here is the complete response, nothing removed. It is long, but seeing all of it
is the point, because two parts of it explain how the model actually works:

{
  "model": "phi4",
  "created_at": "2026-07-14T03:11:50.862545331Z",
  "response": "Kubernetes scheduling is the process of assigning workloads, such as pods, to appropriate nodes within a cluster based on resource availability and constraints. The scheduler evaluates factors like CPU, memory requirements, affinity/anti-affinity rules, taints/tolerations, and node selectors to determine the best fit for each pod. This ensures optimal utilization of resources, workload distribution, and adherence to specified policies, thereby maintaining efficient cluster operation and performance.",
  "done": true,
  "done_reason": "stop",
  "context": [
    100264, 882, 100266, 198, 849, 21435, 67474, 38952, 304, 2380,
    23719, 13, 100265, 198, 100264, 78191, 100266, 198, 42, 30927,
    38952, 374, 279, 1920, 315, 61853, 990, 33785, 11, 1778,
    439, 55687, 11, 311, 8475, 7954, 2949, 264, 10879, 3196,
    389, 5211, 18539, 323, 17413, 13, 578, 29909, 67349, 9547,
    1093, 14266, 11, 5044, 8670, 11, 51552, 14, 15719, 71260,
    13797, 5718, 11, 259, 1673, 82, 5640, 22847, 811, 11,
    323, 2494, 57137, 311, 8417, 279, 1888, 5052, 369, 1855,
    7661, 13, 1115, 26420, 23669, 50549, 315, 5070, 11, 54696,
    8141, 11, 323, 71628, 311, 5300, 10396, 11, 28592, 20958,
    11297, 10879, 5784, 323, 5178, 13
  ],
  "total_duration": 3974942694,
  "load_duration": 100037676,
  "prompt_eval_count": 18,
  "prompt_eval_duration": 46132000,
  "eval_count": 89,
  "eval_duration": 3817895000
}

Reading it top to bottom:

The response field is the answer, a real reply from a model running on my
hardware with no network call leaving the box.

The context field is the part worth staring at. You might be wondering what
those numbers are. They are tokens. A model does not read text; it reads
token IDs, which are integers. Before anything runs, a tokenizer splits the
text into tokens and maps each one to an integer. And this array is not just the
prompt, it is the whole conversation as tokens: the chat template, my question,
and the model's full answer. Decoding the first several so it is not a mystery:

100264 is the special marker <|im_start|> (start of a turn), and 882 is the word "user". So the conversation begins "start of turn, user".
849, 21435, 67474, 38952 are "Ex", "plain", " Kubernetes", " scheduling". Notice "Explain" is split into two tokens. Tokens are often sub-word pieces, not whole words.
304, 2380, 23719, 13 are " in", " three", " sentences", ".", finishing my prompt.
100265 is <|im_end|> (end of turn), then 100264 78191 100266 is the start of the assistant's turn ("start, assistant, separator").
42, 30927 onward ("K", "ubernetes", ...) is the model's answer, token by token, which is exactly the text in the response field above.

The special IDs 100264, 100265, 100266 are the chat markers
(<|im_start|>, <|im_end|>, <|im_sep|>), the turn boundaries from the
model's embedded tokenizer and chat template. Two questions probably popped into
your head reading that, the same ones that popped into mine: wait, how do I even
know 849 is "Ex" or that 100264 is <|im_start|>? And how does Ollama know
which tokenizer to use? Both have clear answers, but they would break the flow
here, so I answer them at the end. If you want them now, jump to
the token questions, answered.

This is why token counts matter everywhere. On a cloud API you pay per token. In
a model, tokens are what fill the context window, and the whole conversation is
carried forward as this growing list of integers. When people say "tokens per
second," this array is the unit being counted.

Concept 3: inference has two measured phases

The bottom of that same output has the timing, and it turns vague words like
"latency" into measured numbers. All the durations are in nanoseconds, so here
they are converted:

Field	Raw value	Converted value
`load_duration`	100,037,676 ns	0.10 s
`prompt_eval_count`	18	18 input tokens
`prompt_eval_duration`	46,132,000 ns	46 ms
`eval_count`	89	89 output tokens
`eval_duration`	3,817,895,000 ns	3.82 s
`total_duration`	3,974,942,694 ns	3.97 s

prompt_eval_duration is the time spent reading the tokenized input. That is
the prefill phase. eval_duration is the time spent generating output
tokens. That is the decode phase. load_duration is time Ollama spent
loading or preparing the model for this request.

One headline number can be calculated directly:

Tokens per second (generation speed): 89 tokens divided by 3.82 s is about 23.3 tokens/sec.

I cannot honestly calculate time to first token (TTFT) from this
non-streaming response. Adding load and prefill gives 146 ms of server work
before decode, but that is not the same as observing when the first token reaches
the client. To measure TTFT, the client must request a streamed response and
timestamp the first non-empty token. I do that in the benchmark later in this
post.

The timing split also shows that generating text has two phases that behave
differently:

Prefill (prompt_eval): the model reads all 18 prompt tokens at once. This is fast (46 ms) because the tokens are processed together, which is the parallel, compute-heavy work the GPU from Week 1 is good at.
Decode (eval): the model generates output tokens one at a time, each one depending on the previous. This is the slow, sequential phase, and it is where memory bandwidth matters. Each token passes through the model layers and uses their weights again, while the key-value cache avoids recalculating all prior tokens from scratch.

That is the core of Week 2. The API is HTTP over a local process, the "text" is
really a stream of integer tokens, and a request is load, then prefill, then
decode, each one measurable. Now I can start changing the settings that shape the
output.

Concept 4: temperature, or how random the model is allowed to be

Back in Concept 2 I ran the same prompt twice and got slightly different wording
each time. That is not a bug. It is a setting called temperature, and it
controls how random the model is allowed to be when it picks each next token.

The setting is options.temperature. I sent the same coffee-shop prompt twice
at each temperature to watch it change. I changed the value in this direct call
and ran it twice:

curl -s http://localhost:11434/api/generate -d '{
  "model": "phi4",
  "prompt": "Give me a one-sentence tagline for a coffee shop.",
  "stream": false,
  "options": { "temperature": 0 }
}' | jq -r .response

Twice at temperature 0:

temperature=0
run 1: "Awaken Your Senses, One Cup at a Time."
run 2: "Awaken Your Senses, One Cup at a Time."

These two runs matched. The same call with "temperature": 1.2 gives variety:

temperature=1.2
run 1: "Where every sip is a moment of delight."
run 2: "Where Every Cup is a Perfect Brew—Awaken Your Senses."

Same input, two different answers.

So what is temperature actually doing? At each step, the model does not just pick
one next token. It produces a probability for every token in its vocabulary, like
"there is a 40% chance the next token is Awaken, 8% chance it is Where, and so
on". Temperature reshapes that list before one token is picked. Top-p and seed
also affect the choice, but each control has a different job:

Control	Changes	Lower/same	Higher/different
Temperature	Probability shape	Favors likely tokens	Allows more variety
Top-p	Candidate set	Fewer likely tokens	More possible tokens
Seed	Random sequence	Same starting sequence	Another sequence

Top-p is also called nucleus sampling. At 0.9, it keeps the most likely
tokens whose probabilities add up to 90%, then samples only from that set.
Temperature and top-p can be used together, but while learning I would change
one at a time so I know which setting changed the output.

You might also see a seed beside temperature. A model runtime uses a
pseudorandom number generator when it samples from the possible next tokens.
Pseudorandom means the values look random, but they are generated by a formula.
The seed is the starting number for that formula, like starting with the same
shuffle of a deck. The same seed gives the sampler the same sequence of random
choices when the model, prompt, options, runtime, and execution conditions stay
the same.

I tested that with temperature 1.2, where sampling has room to vary:

for RUN in 1 2 3 4; do
  printf "run=%s seed=42: " "$RUN"
  curl -s http://localhost:11434/api/generate -d '{
    "model": "phi4",
    "prompt": "Reply with one invented coffee shop name and nothing else.",
    "stream": false,
    "options": {"temperature": 1.2, "seed": 42, "num_predict": 16}
  }' | jq -r .response | tr '\n' ' '
  printf "\n"
done

run=1 seed=42: Espresso Enchantments
run=2 seed=42: Espresso Enchantments
run=3 seed=42: Espresso Enchantments
run=4 seed=42: Espresso Enchantments

These four warm-model runs matched. A fixed seed improves repeatability, but it
is not a universal guarantee. A different runtime version, model build,
hardware path, parallel execution order, or other nondeterministic GPU behavior
can still change the result. A seed controls the sampler's random sequence; it
does not freeze the whole software and hardware stack.

The practical takeaway is simple. Start near zero when repeatability matters,
and raise temperature when you want variety. Add a fixed seed when you want more
repeatable comparisons, and record the runtime and model version too. Everything
else this week uses the same options object in the API call.

Concept 5: chat is a list of role-tagged messages

So far I used /api/generate, which takes one plain prompt. Ollama also has
chat interfaces. I will use them in increasing depth:

The CLI proves I can have a one-shot conversation.
The HTTP API exposes the role-tagged message structure.
The Python client keeps that structure across multiple turns.

First, chat through the CLI

I keep using phi4, already pulled in Concept 1, so there is nothing new to
download. Ollama's CLI is terminal-aware, so I forced SSH to allocate a terminal
with -tt:

ssh -tt spark \
  'ollama run --nowordwrap phi4 \
  "Reply with exactly these three words: local model ready"'

Local model ready.

Connection to spark closed.

This proves the CLI can send a prompt and display a reply. It does not show the
message roles or how a client maintains a conversation. For that, I need the
HTTP API.

Next, inspect messages through the API

The /api/chat endpoint takes a messages array. Each item has a role that
says who wrote it and content that holds the text:

Role	Who writes it	What it does
`system`	Application	Sets behavior, style, or limits
`user`	Person using the app	Carries the user's prompt or follow-up
`assistant`	Model	Stores a previous model reply in the history

The system prompt is an instruction the application places before the user
conversation. It sets the expected role, style, or limits of the assistant. It
is configuration, not a security boundary. A user can still send conflicting
instructions, so production systems need checks outside the prompt too.

The user prompt is simply the content of a message with role: "user".
After the model replies, the client stores that answer as an assistant message.
The next request sends the earlier messages plus the new user message.

Why resend them? The Ollama API does not remember this client's conversation
between independent HTTP requests. I sent two consecutive requests to the same
running Ollama process. First, I included the earlier user question and
assistant reply:

curl -s http://localhost:11434/api/chat -d '{
  "model": "phi4",
  "stream": false,
  "messages": [
    {"role": "system", "content": "Reply with only a name or UNKNOWN."},
    {"role": "user", "content": "My cluster is called Atlas."},
    {"role": "assistant", "content": "Atlas"},
    {"role": "user", "content": "What name did I give it?"}
  ],
  "options": {"temperature": 0, "seed": 42}
}' | jq '{reply: .message, prompt_tokens: .prompt_eval_count}'

{
  "reply": {
    "role": "assistant",
    "content": "Atlas"
  },
  "prompt_tokens": 50
}

Then I sent only the follow-up, with no earlier turns:

curl -s http://localhost:11434/api/chat -d '{
  "model": "phi4",
  "stream": false,
  "messages": [
    {"role": "system", "content": "Reply with only a name or UNKNOWN."},
    {"role": "user", "content": "What name did I give it?"}
  ],
  "options": {"temperature": 0, "seed": 42}
}' | jq '{reply: .message, prompt_tokens: .prompt_eval_count}'

{
  "reply": {
    "role": "assistant",
    "content": "UNKNOWN"
  },
  "prompt_tokens": 31
}

Request	Reply	Prompt tokens
Earlier turns included	`Atlas`	50
Earlier turns omitted	`UNKNOWN`	31

What changed between the two calls? Only the messages array.

In the first request, the JSON body included the earlier line My cluster is called Atlas, the model's earlier answer, and the new follow-up. Ollama could
therefore pass all of that text to the model, so the answer was Atlas.

In the second request, the conversation history contained only What name did I give it?. There was no mention of Atlas in its messages, so the model answered
UNKNOWN. Ollama did not look up the earlier HTTP call and add it automatically.

That is what the client owns the conversation state means. The application,
such as this Python client, keeps the message list in memory or a database. For
each new turn, it appends the latest user message and sends the relevant history
again. Ollama processes the messages it receives in that request.

Resending history has a cost. The request with earlier turns contained 50 prompt
tokens; the request without them contained 31. The 19-token difference includes
the added message text, role labels, and chat-template separators. I measured
the total difference here; I did not split it token by token. All 19 additional
tokens used positions in the context window.

Finally, implement the conversation in Python

The required Python client now implements the message flow I just tested. It
uses /api/chat, keeps prior turns in the messages
list, and streams the reply as it arrives. With "stream": true, Ollama sends
one JSON object per line; each text fragment is in message.content. The loop
prints each fragment and joins them into the completed assistant message. Here
is the complete file:

#!/usr/bin/env python3
"""Small command-line chat client for Ollama's streaming /api/chat endpoint."""

import argparse
import json
import os
import urllib.error
import urllib.request


def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--model", default=os.getenv("MODEL", "phi4"))
    parser.add_argument("--url", default="http://localhost:11434")
    parser.add_argument("--system", default="You are a concise assistant.")
    parser.add_argument("--temperature", type=float, default=0.0)
    parser.add_argument("--top-p", type=float, default=0.9)
    parser.add_argument("--num-ctx", type=int, default=4096)
    parser.add_argument("--once", help="Send one prompt and exit")
    return parser.parse_args()


def request_chat(
    url: str,
    model: str,
    messages: list[dict[str, str]],
    options: dict[str, float | int],
) -> str:
    body = {
        "model": model,
        "messages": messages,
        "stream": True,
        "options": options,
    }
    request = urllib.request.Request(
        f"{url.rstrip('/')}/api/chat",
        data=json.dumps(body).encode(),
        headers={"Content-Type": "application/json"},
    )
    parts: list[str] = []
    try:
        with urllib.request.urlopen(request) as response:
            for line in response:
                event = json.loads(line)
                text = event.get("message", {}).get("content", "")
                if text:
                    print(text, end="", flush=True)
                    parts.append(text)
    except urllib.error.URLError as exc:
        raise SystemExit(f"Ollama request failed: {exc}") from exc
    print()
    return "".join(parts)


def main() -> None:
    args = parse_args()
    messages: list[dict[str, str]] = []
    if args.system:
        messages.append({"role": "system", "content": args.system})
    options = {
        "temperature": args.temperature,
        "top_p": args.top_p,
        "num_ctx": args.num_ctx,
    }

    if args.once:
        messages.append({"role": "user", "content": args.once})
        request_chat(args.url, args.model, messages, options)
        return

    print(f"Chatting with {args.model}. Type /exit to quit.")
    while True:
        try:
            prompt = input("you> ").strip()
        except (EOFError, KeyboardInterrupt):
            print()
            return
        if prompt.lower() in {"/exit", "/quit"}:
            return
        if not prompt:
            continue
        messages.append({"role": "user", "content": prompt})
        print("model> ", end="", flush=True)
        answer = request_chat(args.url, args.model, messages, options)
        messages.append({"role": "assistant", "content": answer})


if __name__ == "__main__":
    main()

I used --once for a repeatable blog example and passed a system prompt. If I
leave off --once, the same program opens an interactive loop and keeps each
user and assistant turn in the next request.

ssh spark '~/venvs/w1/bin/python - \
  --model phi4 \
  --system "Answer in exactly one short sentence." \
  --once "What does the Kubernetes scheduler do?"' \
  < public/week-02-first-local-model/chat_client.py

The Kubernetes scheduler assigns pods to nodes based on resource availability,
constraints, and policies.

This is the same local runtime and the same phi4 model through three
interfaces: Ollama's CLI, /api/generate with curl, and /api/chat from
Python. Only the client interface changed.

Concept 6: measure cold starts and verify context allocation

Imagine Phi-4 is behind a chat application. No one has used it for a while, so
Ollama has unloaded it from memory. A user sends a message and waits about six
seconds before seeing the first word. They send another message a minute later
and see the first word in a fraction of a second.

That difference is what this experiment measures:

Request state	What Ollama must do	What the user notices
Cold	Load the model, then process the prompt	Long wait for first text
Warm	Reuse the model already in memory	First text arrives quickly

A cold request starts with the model unloaded, so Ollama must load it before
inference. A warm request starts with the model already resident in memory
and ready for reuse.

Before running anything, I expected the first request to have a much higher
time to first token (TTFT) because it includes model loading. I expected the
next requests to start quickly. I did not expect the output generation rate to
change much, because all runs use the same model and output length.

I also used the experiment to answer a separate configuration question:

What does num_ctx change? It changes how much token capacity Ollama allocates. It does not make a short prompt longer or automatically faster.
How different is the first request from later requests? A streamed client can measure real time to first token for both a cold and warm model.

These are two checks in one script, not a claim that context size causes the
cold-start difference. The cold-versus-warm runs measure model loading. The
2,048-versus-4,096 setting verifies how I request context capacity.

The model metadata showed a maximum context length of 16,384 tokens. The
configured context size tells Ollama how much of that capacity to allocate
for a run. The prompt does not become 4,096 tokens just because I set
num_ctx to 4,096. It means the prompt, chat history, and generated reply may
use up to that budget.

A larger context also needs a larger key-value cache, often shortened to KV
cache. That cache holds intermediate values from earlier tokens so decode does
not repeat all prior work for every new token. More context gives the cache room
for more tokens, which uses more memory.

The benchmark changes num_ctx from 2,048 to 4,096. At each setting it runs
three requests in this exact order:

Run	Model state	Purpose
1	Cold	Measure the user-visible cost of loading Phi-4
2	Warm	Measure a request with Phi-4 already loaded
3	Warm	Repeat the warm measurement once more

It fixes temperature and seed, limits each reply to 96 tokens, and calls
ollama stop phi4 before run 1 at each context setting. Runs 2 and 3 do not stop
the model, so they reuse it in memory.

Most importantly, it requests a streamed response. The client starts a clock
before the HTTP request and stops the TTFT clock when the first non-empty text
chunk arrives. That is a real client-observed TTFT measurement.

Here is the full benchmark:

#!/usr/bin/env python3
"""Measure client TTFT and server timings from Ollama's streaming API."""

import argparse
import json
import subprocess
import time
import urllib.request
from pathlib import Path
from typing import Any


def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--model", default="phi4")
    parser.add_argument(
        "--prompt",
        default="Explain Kubernetes scheduling in three sentences.",
    )
    parser.add_argument("--contexts", default="2048,4096")
    parser.add_argument("--runs", type=int, default=3)
    parser.add_argument("--url", default="http://localhost:11434")
    parser.add_argument("--output", type=Path)
    return parser.parse_args()


def unload(model: str) -> None:
    subprocess.run(
        ["ollama", "stop", model],
        stdout=subprocess.DEVNULL,
        stderr=subprocess.DEVNULL,
        check=False,
    )
    time.sleep(1)


def run_once(
    url: str,
    model: str,
    prompt: str,
    num_ctx: int,
    cold: bool,
) -> dict[str, Any]:
    if cold:
        unload(model)
    body = {
        "model": model,
        "prompt": prompt,
        "stream": True,
        "options": {
            "temperature": 0,
            "seed": 42,
            "num_ctx": num_ctx,
            "num_predict": 96,
        },
    }
    request = urllib.request.Request(
        f"{url.rstrip('/')}/api/generate",
        data=json.dumps(body).encode(),
        headers={"Content-Type": "application/json"},
    )

    started = time.perf_counter()
    first_token_at: float | None = None
    final: dict[str, Any] = {}
    with urllib.request.urlopen(request) as response:
        for line in response:
            event = json.loads(line)
            if event.get("response") and first_token_at is None:
                first_token_at = time.perf_counter()
            if event.get("done"):
                final = event
    finished = time.perf_counter()

    if first_token_at is None or not final:
        raise RuntimeError("Ollama stream ended without timing data")
    eval_seconds = final["eval_duration"] / 1e9
    return {
        "context": num_ctx,
        "cold": cold,
        "ttft_ms": round((first_token_at - started) * 1000, 1),
        "client_total_ms": round((finished - started) * 1000, 1),
        "load_ms": round(final["load_duration"] / 1e6, 1),
        "prompt_eval_ms": round(final["prompt_eval_duration"] / 1e6, 1),
        "prompt_tokens": final["prompt_eval_count"],
        "output_tokens": final["eval_count"],
        "tokens_per_second": round(final["eval_count"] / eval_seconds, 1),
    }


def main() -> None:
    args = parse_args()
    contexts = [int(value) for value in args.contexts.split(",")]
    rows: list[dict[str, Any]] = []
    print("ctx  run  cold  ttft_ms  total_ms  load_ms  out_tok  tok/s")
    for context in contexts:
        for run_number in range(1, args.runs + 1):
            row = run_once(
                args.url,
                args.model,
                args.prompt,
                context,
                cold=run_number == 1,
            )
            rows.append(row)
            print(
                f"{context:>4}  {run_number:>3}  {str(row['cold']):>5}  "
                f"{row['ttft_ms']:>7.1f}  {row['client_total_ms']:>8.1f}  "
                f"{row['load_ms']:>7.1f}  {row['output_tokens']:>7}  "
                f"{row['tokens_per_second']:>5.1f}"
            )

    result = {
        "model": args.model,
        "prompt": args.prompt,
        "runs_per_context": args.runs,
        "measurements": rows,
    }
    if args.output:
        args.output.write_text(json.dumps(result, indent=2) + "\n")


if __name__ == "__main__":
    main()

I ran three requests per context size. The first is cold because the script
stops Phi-4 before it. The next two reuse the loaded model.

ssh spark '~/venvs/w1/bin/python - --contexts 2048,4096 --runs 3' \
  < public/week-02-first-local-model/benchmark.py

ctx  run  cold  ttft_ms  total_ms  load_ms  out_tok  tok/s
2048    1   True   6034.1   10129.6   5908.4       96   23.4
2048    2  False    196.1    4288.0    140.4       96   23.5
2048    3  False    151.9    4244.7    104.1       96   23.5
4096    1   True   6260.8   10369.2   6150.1       96   23.4
4096    2  False    174.3    4262.8    117.8       96   23.5
4096    3  False    173.5    4260.8    124.8       96   23.5

The cold requests took about 6.0 to 6.3 seconds before the first token. The warm
requests took 151.9 to 196.1 ms. This is the cold-start effect I could not
measure from the earlier non-streaming call.

Generation stayed at 23.4 to 23.5 tokens/sec. The short 18-token input did not
use the extra context capacity, so doubling the setting did not make this test
faster. It just allowed a larger KV cache. ollama ps confirmed the last
requested allocation:

ssh spark 'ollama ps'

NAME           ID              SIZE      PROCESSOR    CONTEXT    UNTIL
phi4:latest    ac896e5b8b34    9.7 GB    100% GPU     4096       4 minutes from now

What did this experiment prove?

A cold Phi-4 request took about 6 seconds to produce its first token because Ollama had to load the model.
Warm requests produced the first token in about 0.15 to 0.20 seconds.
Ollama accepted the final 4,096-token setting and reported CONTEXT 4096.
Doubling the requested context did not change throughput for this 18-token prompt. Both settings stayed near 23.5 tokens/sec.

It did not measure the memory difference between the two context settings,
and it did not test a long prompt. Those need a separate experiment. The lesson
here is simpler: keep models warm for latency, and treat context size as a
capacity setting rather than a speed setting.

The token questions, answered

Earlier, in the token section (Concept 2), I parked two questions to keep the
main thread moving: how do I even know that 849 is "Ex", and how does Ollama
know which tokenizer to use? Here are the answers. You do not need them to follow
the rest of the post, so read on only if you want to poke at the internals.

Wait, how do I even know what each token ID means?

I did not guess. A runnable model package needs a tokenizer, the exact
rulebook that maps text to token IDs and back. The ordinary Phi-4 token IDs in
the API response match OpenAI's cl100k_base encoding. I verified that match
with decode_tokens.py instead of inferring it from the size of the IDs.

#!/usr/bin/env python3
"""Week 2 - show how text becomes tokens for a cl100k model like phi4.

Encodes the prompt and decodes individual token IDs, so you can match the
`context` array from the generate API back to real words.

Run with a Python that has tiktoken installed, e.g. the Week 1 venv:
    ~/venvs/w1/bin/python -m pip install -q tiktoken
    ~/venvs/w1/bin/python decode_tokens.py
"""
import tiktoken

PROMPT = "Explain Kubernetes scheduling in three sentences."
# A few IDs taken from the generate API's context array.
SAMPLE_IDS = [849, 21435, 67474, 38952, 304, 2380, 23719, 13, 882, 78191]


def main() -> None:
    enc = tiktoken.get_encoding("cl100k_base")

    print("encode:", enc.encode(PROMPT))
    print("decode individual IDs:")
    for t in SAMPLE_IDS:
        print(f"  {t:>6}  {enc.decode([t])!r}")


if __name__ == "__main__":
    main()

Then I ran the file with the pinned tiktoken 0.13.0 package:

ssh spark '~/venvs/w1/bin/python -' \
  < public/week-02-first-local-model/decode_tokens.py

encode: [849, 21435, 67474, 38952, 304, 2380, 23719, 13]
decode individual IDs:
     849  'Ex'
   21435  'plain'
   67474  ' Kubernetes'
   38952  ' scheduling'
     304  ' in'
    2380  ' three'
   23719  ' sentences'
      13  '.'
     882  'user'
   78191  'assistant'

The first line reproduces the exact prompt tokens from the context array, which
confirms phi4 really does use this vocabulary. Decoding the individual IDs gives
back the words, including user and assistant. This proves the ordinary
prompt and role IDs match cl100k_base. It does not prove what the added special
IDs mean. I verify those from the model file in the next section.

Is cl100k a universal standard? No. It is one of a small set of named
vocabularies from OpenAI's tiktoken library, where each vocabulary (such as
cl100k_base for GPT-4 and the newer o200k_base for GPT-4o) is tied to specific
models.⁴ Other model families use entirely different tokenizers. Llama
models, for example, use a different scheme with different token IDs, so the same
word gets a different number there. Each model comes with its own vocabulary, and
you can always load that exact tokenizer to see how it splits text.

And how does Ollama know which tokenizer to use?

The runtime is what turns the prompt into tokens, so this is a fair question. It
does not keep a list of models and guess. The tokenizer is packed inside the
model file. Ollama stores models in the GGUF format,⁵ and a GGUF file
carries the tokenizer metadata next to the weights.

There are two files involved:

The small manifest file is /usr/share/ollama/.ollama/models/manifests/registry.ollama.ai/library/phi4/latest. It is a JSON index for the Phi-4 package. It does not contain the tokenizer. Its model-layer entry tells Ollama which blob file to open.
The model blob is in /usr/share/ollama/.ollama/models/blobs. Its filename is sha256-fd7b6731c33c57f61767612f56517460ec2d1e2e5a3f0163e0eb3d8d8cb5df20. This 9 GB file is the actual GGUF model. It contains the model weights, tokenizer vocabulary, special tokens, and chat template.

I resolved that path directly before using Python:

ssh spark '
  MODELS_DIR=/usr/share/ollama/.ollama/models
  MANIFEST_FILE=manifests/registry.ollama.ai/library/phi4/latest
  MANIFEST="$MODELS_DIR/$MANIFEST_FILE"
  DIGEST="$(jq -r '\''.layers[]
    | select(.mediaType == "application/vnd.ollama.image.model")
    | .digest'\'' "$MANIFEST")"
  BLOB_FILE="blobs/${DIGEST/:/-}"
  printf "models directory: %s\n" "$MODELS_DIR"
  printf "manifest file: %s\n" "$MANIFEST_FILE"
  printf "model digest: %s\n" "$DIGEST"
  printf "GGUF file: %s\n" "$BLOB_FILE"
  stat -c "GGUF bytes: %s" "$MODELS_DIR/$BLOB_FILE"
'

models directory: /usr/share/ollama/.ollama/models
manifest file: manifests/registry.ollama.ai/library/phi4/latest
model digest: sha256:fd7b6731c33c57f61767612f56517460ec2d1e2e5a3f0163e0eb3d8d8cb5df20
GGUF file: blobs/sha256-fd7b6731c33c57f61767612f56517460ec2d1e2e5a3f0163e0eb3d8d8cb5df20
GGUF bytes: 9053114464

The digest changes from sha256:... in the manifest to sha256-... in the
blob filename. Only the colon becomes a hyphen. The Python script automates
these same steps, opens that GGUF blob, and reads a few tokenizer fields so I do
not have to inspect a 9 GB binary file by hand:

#!/usr/bin/env python3
"""Read selected tokenizer metadata directly from an Ollama GGUF model layer."""

import argparse
import json
import os
from pathlib import Path

from gguf import GGUFReader


def parse_args() -> argparse.Namespace:
  parser = argparse.ArgumentParser(description=__doc__)
  parser.add_argument("--model", default=os.getenv("MODEL", "phi4"))
  parser.add_argument(
    "--models-dir",
    type=Path,
    default=Path("/usr/share/ollama/.ollama/models"),
  )
  return parser.parse_args()


def field(reader: GGUFReader, name: str):
  value = reader.get_field(name)
  if value is None:
    raise KeyError(f"Missing GGUF field: {name}")
  return value.contents()


def main() -> None:
  args = parse_args()
  name, separator, tag = args.model.partition(":")
  if not separator:
    tag = "latest"
  manifest = (
    args.models_dir
    / "manifests"
    / "registry.ollama.ai"
    / "library"
    / name
    / tag
  )
  data = json.loads(manifest.read_text())
  digest = next(
    layer["digest"]
    for layer in data["layers"]
    if layer["mediaType"].endswith("image.model")
  )
  blob = args.models_dir / "blobs" / digest.replace(":", "-")
  reader = GGUFReader(blob)
  tokens = field(reader, "tokenizer.ggml.tokens")
  template = field(reader, "tokenizer.chat_template")

  print("tokenizer model:", field(reader, "tokenizer.ggml.model"))
  print("tokenizer pre:", field(reader, "tokenizer.ggml.pre"))
  print("token count:", len(tokens))
  for token_id in (100257, 100264, 100265, 100266):
    print(f"token {token_id}: {tokens[token_id]!r}")
  for marker in ("<|im_start|>", "<|im_end|>", "<|im_sep|>"):
    print(f"template contains {marker}: {marker in template}")


if __name__ == "__main__":
  main()

I ran it with the pinned gguf 0.19.0 package:

ssh spark '~/venvs/w1/bin/python -' \
  < public/week-02-first-local-model/inspect_tokenizer.py

tokenizer model: gpt2
tokenizer pre: dbrx
token count: 100352
token 100257: '<|endoftext|>'
token 100264: '<|im_start|>'
token 100265: '<|im_end|>'
token 100266: '<|im_sep|>'
template contains <|im_start|>: True
template contains <|im_end|>: True
template contains <|im_sep|>: True

Everything the runtime needs is in the file:

tokenizer.ggml.model = 'gpt2' is the tokenizer type. It is a byte-pair encoding (BPE), the same family GPT-2 and GPT-4 use.
tokenizer.ggml.tokens is the full vocabulary, all 100,352 entries, the list that maps text pieces to IDs.
The three IDs are the exact special-token strings claimed earlier.
The embedded chat template contains those same markers. Phi-4's model card uses this ChatML-style prompt layout.⁶

So Ollama reads the vocabulary from the model file itself and uses it to turn my
prompt into token IDs. The ordinary IDs used in this prompt are compatible with
cl100k_base, while the GGUF adds Phi-4's special chat tokens. A Llama model's
GGUF carries a different vocabulary, and Ollama uses that instead. Same runtime,
different tokenizer per model, because the tokenizer travels with the model.

Results

Here is the Week 2 result in one place:

Item	Measured result
Runtime	Ollama 0.31.2
Main model	Phi-4, 14.7B parameters, Q4_K_M
Model layer	9,053,114,464 bytes
Context settings tested	2,048 and 4,096 tokens
Cold TTFT	6,034.1 to 6,260.8 ms
Warm TTFT	151.9 to 196.1 ms
Generation speed	23.4 to 23.5 tokens/sec
Python deliverable	Streaming command-line chat client
Benchmark deliverable	Cold/warm TTFT and throughput benchmark

I also used every interface in the plan: interactive CLI, curl, and Python.
The commands changed temperature and context size. The Python client used a
system prompt, and the benchmark compared repeatable runs with measured timing.

What surprised me

The first surprise was the model size. Phi-4 has 14.7 billion parameters, but
its quantized model layer is 9.1 GB instead of the roughly 29.4 GB needed for
16-bit weights alone.

The bigger surprise was the cold start. Generation speed stayed near 23.5
tokens/sec, but the first token took about 6 seconds when Ollama had to load the
model. Once warm, it arrived in about 0.15 to 0.20 seconds.

Changing context capacity from 2,048 to 4,096 did not improve this short
request. A larger limit gives the KV cache room for more tokens. It does not make
an 18-token prompt larger or automatically faster.

Mistakes and troubleshooting

I first called the output of ollama show a manifest. That was wrong. Model
metadata is a readable summary. The manifest is the JSON index that points to
the model, template, license, and parameter layers.

I also tried to estimate TTFT by adding load_duration and
prompt_eval_duration from a non-streaming response. That is useful server
timing, but it is not client-observed TTFT. The fixed benchmark streams the
response and timestamps the first non-empty text chunk.

One smaller issue came from Ollama's terminal-aware CLI. Its one-shot response
was not visible through plain SSH capture, so I used ssh -tt to allocate a
pseudo-terminal. The Python and curl clients did not need that workaround.

Production implications

A production service should keep commonly used models warm. Otherwise, a user
can wait seconds for model loading before generation even starts. Streaming also
matters because it lets the user see the first token instead of waiting for the
whole reply.

Context size is a capacity and memory decision. Setting it to the model maximum
for every request can reserve more KV-cache memory than the workload needs. The
right setting comes from measured prompt and conversation lengths.

The system prompt is not an access-control system. If I expose Ollama beyond
localhost, I still need a gateway with authentication, authorization, TLS, rate
limits, request-size limits, and logging. Prompt instructions do not replace
those controls.

What I will learn next

Week 3 moves from running a packaged model to opening a Hugging Face repository
and inspecting its files. I will compare base and instruct models, read the
model card and license, then inspect configuration, tokenizer, chat-template,
and Safetensors files.⁷

Run it yourself

The public Week 2 lab contains the chat client, benchmark, scripts, pinned
requirements, raw results, and model-run record.⁸

Can llama.cpp reuse this Ollama model?

This is optional reading. It is useful if you want to understand whether a model
file belongs to one runtime or can move between compatible runtimes.

For this Phi-4 download, yes. Ollama reports details.format = "gguf", and the
independent file-header check agrees. llama.cpp loads local GGUF files, so it
can use this same 9,053,114,464-byte model blob without downloading the weights
again.⁵ The content matters, not the missing .gguf extension on Ollama's
content-addressed filename.

A file header alone does not prove another runtime can load the model and
generate text, so I installed a pinned CUDA build of llama.cpp and pointed it
at the exact model blob named by Ollama's manifest digest. It loaded the file
and generated a reply:

Check	Result
Model blob	same file, Ollama manifest digest `fd7b...df20`
`llama.cpp` build	pinned commit `571d0d54`
Device used	CUDA0 (NVIDIA GB10)
GPU layers offloaded	41/41
Generated reply	`Shared model works.`

That is a concrete yes. The full investigation records the failed first check,
CUDA toolkit discovery, pinned ARM64 build, scripts, commands, raw output, and
the apparent terminal hang.²

One limit remains. The model blob is only one layer of the Ollama package. The
manifest also lists a prompt template, license, and default parameters as
separate layers. llama.cpp does not read those Ollama layers just because it
can read the model blob. This Phi-4 GGUF carries its own tokenizer and chat
template, but I still set the seed, temperature, context, and GPU layers
explicitly. Same weights do not guarantee the same output from two runtimes.

This result is specific to this Phi-4 file and this pinned llama.cpp commit.
For another Ollama download, I would repeat the test because an older runtime
may not support a newer model architecture.

Official project docs for the alternatives named here:
llama.cpp at https://github.com/ggml-org/llama.cpp, vLLM at
https://docs.vllm.ai, NVIDIA TensorRT-LLM at
https://nvidia.github.io/TensorRT-LLM/, and Hugging Face Transformers at
https://huggingface.co/docs/transformers/index ↩
Full Ollama-to-llama.cpp investigation, including the pinned
CUDA build and raw command output:
https://github.com/dramasamy/from-api-to-gpu/blob/main/week-02-first-local-model/ollama-model-in-llamacpp.md ↩
Ollama API reference:
https://docs.ollama.com/api/introduction ↩
tiktoken, OpenAI's tokenizer library, lists the standard vocabularies
and which model each one belongs to. See the model-to-encoding table at
https://github.com/openai/tiktoken ↩
GGUF is the model file format from the llama.cpp project. It stores the
weights plus metadata such as the tokenizer and chat template. See
https://github.com/ggml-org/ggml/blob/master/docs/gguf.md ↩
Microsoft Phi-4 model card, including its chat input format:
https://huggingface.co/microsoft/phi-4#input-formats ↩
Week 3 roadmap:
https://github.com/dramasamy/from-api-to-gpu/blob/main/roadmap/week-03.md ↩
Week 2 companion lab:
https://github.com/dramasamy/from-api-to-gpu/tree/main/week-02-first-local-model ↩

From API to GPU, Week 1: Understanding Your Local GPU Environment

Dinesh Kumar Ramasamy — Sat, 11 Jul 2026 20:57:28 +0000

Phase 1 of 8: Comfortable running local models. Week 1 of 32.

I've used AI through APIs to ship features: POST a prompt, get tokens back,
move on. I have never once deployed a model myself. No PyTorch, no GPU memory
math, no idea what actually happens between my HTTP request and the text that
comes back. This series is me closing that gap on purpose, one week at a time,
on an NVIDIA DGX Spark.

I'm a software engineer and technical program manager. I'm comfortable with
Linux, Python, Docker, Kubernetes, and APIs. I'm a complete beginner at machine
learning. So Week 1 is deliberately unglamorous: before running any model, I
want to know the machine: what CPU and GPU it has, how its memory works, and
what the NVIDIA software stack underneath is actually made of. Every claim below
is backed by a real command and its real output, so you can run the same thing
on your own box and compare.

About this series

From API to GPU is a 32-week journey from AI-API consumer to
local LLM systems architect: running, optimizing, and eventually
fine-tuning models on local hardware, documenting each week as a hands-on lab
plus a blog post. The full week-by-week plan lives in the roadmap¹, and
every week's runnable code lands in the companion GitHub repo². If you have
a similar machine, you can follow along and reproduce every result.

The plan runs in eight phases, with a parallel CUDA track starting around week 5:

Phase	Weeks	Focus
1	1–4	Comfortable running local models
2	5–8	Enough ML to understand inference
3	9–13	Transformers and terminology
4	14–16	Quantization and model formats
5	17–20	Inference engineering
6	21–24	Production model services
7	25–28	RAG and application integration
8	29–32	Fine-tuning

The goal of this first post is narrow on purpose: stand up and understand
the environment. By the end you'll be able to inventory a DGX Spark, read what
nvidia-smi tells you, explain how its unified memory differs from a normal
GPU, untangle the NVIDIA driver/CUDA-runtime/toolkit layers, and prove the GPU
is usable from PyTorch with a measured CPU-vs-GPU speedup. No model yet; that's
week 2. This is the foundation everything else builds on.

All the commands in this post are packaged as a runnable script in the companion
repo under week-01-environment/³. Clone it, set up an SSH alias spark
that reaches your DGX Spark, and run ./inventory.sh to reproduce everything
here. The repo holds the commands and scripts; this post holds the explanations,
so neither repeats the other.

Here's the whole inventory script, so you can see exactly what it runs without
cloning anything. It just wraps each command in an SSH call to the Spark and
prints a labelled section:

#!/usr/bin/env bash
# Week 1: DGX Spark machine inventory. Prereq: an SSH alias `spark`.
set -euo pipefail
SPARK_HOST="${SPARK_HOST:-spark}"

run() { echo "=== {% katex inline %}1 ==="; ssh "{% endkatex %}SPARK_HOST" "$2" 2>&1 || true; echo; }

run "uname"      'uname -a'
run "os-release" 'cat /etc/os-release'
run "lscpu"      'lscpu'
run "memory"     'free -h'
run "storage"    'lsblk'
run "gpu"        'nvidia-smi'
run "cuda-nvcc"  'nvcc --version || /usr/local/cuda/bin/nvcc --version'
run "cuda-dirs"  'ls -d /usr/local/cuda*'
run "python"     'python3 --version'
run "docker"     'docker version'

The rest of this post walks through the interesting parts of that output one
section at a time.

Where I started: two machines, and a rule I broke immediately

The setup is two machines. A MacBook Pro is the control machine, for writing,
editing, and opening SSH sessions. An NVIDIA DGX Spark is the workhorse, where
every model and every GPU command actually runs. I reach it over SSH as spark.

A theme for this whole series is that I verify facts with a command instead of
assuming them, even the obvious ones. So rather than start by stating what the
machines are, I'll show them. First the control machine:

# On the control MacBook
uname -m && sysctl -n machdep.cpu.brand_string

x86_64
Intel(R) Core(TM) i5-1038NG7 CPU @ 2.00GHz

An Intel x86_64 Mac. This matters more than it looks, because it's a
different architecture from the Spark, so I want it on the record, not assumed.
Now the Spark:

ssh spark 'uname -a'

Linux spark-66b9 6.17.0-1026-nvidia ... aarch64 aarch64 aarch64 GNU/Linux

The Spark is aarch64: ARM64, the same CPU family as phones and Apple
Silicon, but a different architecture from the Intel Mac.

Machine	Role	Architecture	Verified by
MacBook Pro	control / authoring	x86_64 (Intel i5)	`uname -m`
DGX Spark	model + GPU work	aarch64 (ARM64)	`uname -a`

This is not trivia. Because the two machines are different architectures, a
Python wheel or Docker image built for the Intel Mac will not necessarily run on
the ARM64 Spark. That is exactly why all the real work in this series happens
over ssh spark, on the box itself, and why "it works on my laptop" means
nothing here.

CPU versus GPU, and why models love the GPU

The Spark's CPU is a 20-core ARM chip:

ssh spark 'lscpu'

Architecture: aarch64
CPU(s):       20
Model name:   Cortex-X925   (10 performance cores)
Model name:   Cortex-A725   (10 efficiency cores)

Twenty cores sounds like a lot, but for running a model the CPU is mostly a
traffic director: it runs the OS, your Python, and the data loading. The heavy
lifting happens on the GPU. Here's what finally made that click for me.

A neural-network layer boils down to one operation repeated endlessly: multiply
a big grid of numbers (the model's weights, $W$ ) by a list of numbers (the
input, $x$ ) and add them up, a matrix multiplication. One output value is:

y_{i} = j \sum W_{ij} x_{j}

A model with billions of parameters does billions of these multiply-adds for a
single token. The magic property is that they're independent: computing
$y_{1}$ doesn't need $y_{2}$ . In terms I already know, that's embarrassingly
parallel, like the map phase of a MapReduce where no shard waits on another.

That's the whole reason a GPU wins:

Aspect	CPU (the 20 ARM cores)	GPU (the NVIDIA GB10)
Parallel workers	a few strong cores	thousands of small cores
Good at	branching logic, one-at-a-time	the same math on huge data at once
Analogy	a few expert chefs	a stadium of line cooks

A CPU is a handful of very smart workers doing tasks in sequence. A GPU is
thousands of simpler workers all doing the identical multiply-add on different
numbers simultaneously. Since a model is nothing but that identical operation
repeated, the GPU is the right tool.

One caveat I'm carrying forward: those cores are useless if you can't feed
them numbers fast enough, so memory bandwidth, not raw compute, usually
limits how fast a model runs. I'll test the CPU-vs-GPU speed difference directly
with a matmul benchmark once PyTorch is installed; my prediction is a 10x–50x
GPU speedup, with a slow first GPU run due to one-time warmup.

Reading `nvidia-smi`: my new `top` for the GPU

nvidia-smi is the command I'll run every day from now on. It's the GPU
equivalent of top or docker stats. Here's the real output, lightly trimmed:

ssh spark 'nvidia-smi'

NVIDIA-SMI 580.159.03   Driver Version: 580.159.03   CUDA Version: 13.0
GPU 0: NVIDIA GB10   Persistence-M: On
Temp  Perf  Pwr:Usage/Cap   Memory-Usage    GPU-Util  Compute M.
35C   P8    4W / N/A        Not Supported   0%        Default

Field by field, and why each one will matter later:

Field	Value	Meaning
Driver Version	580.159.03	kernel driver talking to the GPU
CUDA Version	13.0	max CUDA the driver supports
GPU name	NVIDIA GB10	the device (Grace-Blackwell)
Temp	35C	die temperature (heat → throttling)
Perf	P8	clock state, P0 = max … P8 = idle
Pwr:Usage/Cap	4W / N/A	current vs max power draw
Memory-Usage	Not Supported	would show VRAM used/total (see below)
GPU-Util	0%	% of last sample the GPU was busy

The bottom of the output also has a process table listing every PID holding the
GPU. Right now that's just Xorg, GNOME, and Firefox using it for the desktop.
That's my first stop whenever I hit "out of memory": find the offender, like
lsof on a stuck port.

For Week 1 I'm using nvidia-smi purely as an inventory tool: what GPU,
what driver, what max CUDA, who's using it. The deeper use (streaming monitors,
reading utilization and memory bandwidth to decide if a workload is
compute-bound or memory-bound) is a profiling skill I'm deliberately saving for
the CUDA track around Week 5, so I don't tangle the two learning tracks.

The surprise: unified memory, and a blank that isn't a bug

The one field that stopped me was Memory-Usage: Not Supported. On a normal PC
with a discrete GPU, that column is how you answer "did my model fit? how much
VRAM is left?" On the Spark it's blank, and that's not a bug; it's the whole
point of the machine.

A traditional GPU has its own separate memory (VRAM), physically distinct from
system RAM. The DGX Spark's GB10 is a Grace-Blackwell superchip that fuses the
ARM CPU and the GPU onto one package and gives them one shared memory pool:

ssh spark 'free -h'

               total   used   free   shared  buff/cache   available
Mem:           121Gi   6.2Gi   67Gi     36Mi        48Gi        115Gi
Swap:           15Gi   152Ki   15Gi

Traditional discrete GPU	DGX Spark (GB10)
System RAM (e.g. 64 GB) + separate VRAM (e.g. 24 GB)	one shared 121 GiB pool
Copy data RAM → VRAM over PCIe	CPU and GPU read the same memory
"Will it fit?" limited by VRAM (24 GB)	limited by total RAM (121 GB)

This is unified memory. nvidia-smi reports "Not Supported" for GPU memory
because there is no separate VRAM to report: the GPU's memory is the system's
121 GiB. In infra terms, a discrete GPU pays a "copy tax" moving weights across
the PCIe bus, like shuffling data between two services with separate caches;
unified memory removes that hop, like two services sharing one in-memory cache.

The practical consequence for me: on the Spark, the ceiling on model size isn't
a stingy 24 GB of VRAM; it's 121 GB. But I have to track model memory
differently, via free -h or PyTorch's own counters, not the nvidia-smi
memory column. The trade-off (which I'll measure later) is that shared memory
usually has lower peak bandwidth than a top-end discrete card's dedicated VRAM,
so the Spark trades some raw speed for the ability to fit much larger models.

Driver vs CUDA runtime vs CUDA toolkit

The most confusing part of the NVIDIA stack for a newcomer is that "CUDA" isn't
one thing: it's three separate layers, installed and versioned independently.
Mapping each to infrastructure I already understand finally made it stick:

Layer	What it is	Infra analogy	Who needs it
NVIDIA driver	kernel module that talks to the GPU	a device driver	everyone using the GPU
CUDA runtime (libcudart)	shared libs an app calls to run GPU work	the `.so` libs you link	anyone running GPU programs
CUDA toolkit	the `nvcc` compiler, headers, profilers	gcc + headers + build tools	only people compiling CUDA

The insight that unblocked me: you can run GPU code without the toolkit.
PyTorch ships its own copy of the CUDA runtime inside its wheel. So to run
models I need the driver (system-level) plus a CUDA-enabled PyTorch (which
brings its own runtime). I do not need nvcc; that's only for compiling
custom CUDA kernels, a much-later CUDA-track activity.

With that lens, two clues from the inventory make sense. First, the header line
CUDA Version: 13.0 is the maximum CUDA the driver supports, a ceiling, not
what's installed. That's the number that matters this week: when I install
PyTorch, I must pick a CUDA build ≤ 13.0 so the driver can run it.

Second, the alarming-looking one:

ssh spark 'nvcc --version'

bash: nvcc: command not found

This looks broken but isn't. It only means the toolkit's compiler isn't on my
PATH. The driver and runtime clearly work; nvidia-smi talks to the GPU. And
the toolkit is physically installed:

ssh spark 'ls -d /usr/local/cuda*'

/usr/local/cuda  /usr/local/cuda-13  /usr/local/cuda-13.0

So nvcc exists; it's just at /usr/local/cuda/bin/nvcc, not on PATH. Rather
than assert that, I confirmed it by calling the full path:

ssh spark '/usr/local/cuda/bin/nvcc --version'

Cuda compilation tools, release 13.0, V13.0.88

The toolkit is version 13.0.88, matching the driver's CUDA 13.0 ceiling, a
healthy, consistent stack. Nothing is broken; command not found was a PATH
issue, not a missing install. If I ever want nvcc on PATH, it's one line:
export PATH=/usr/local/cuda/bin:$PATH. But to run models, I never need it.

Installing PyTorch, the right way: a virtual environment

PyTorch is the Python library I'll use to talk to the GPU. Before installing it,
one habit worth keeping: never install into the system Python. I use a
virtual environment (venv): an isolated per-project Python with its own
packages, exactly like a per-service dependency sandbox so one project's
libraries can't break another's. On the Spark that's why the earlier
python3 -c "import torch" failed: the system Python genuinely has no torch, and
I want to keep it that way.

If you're following along, first check whether you already have PyTorch, since
many setups ship with it:

ssh spark 'python3 -c "import torch, sys; print(torch.__version__)" \
  || echo "no torch in this Python"'

I didn't, so I made a clean venv and installed torch into it:

ssh spark 'python3 -m venv ~/venvs/w1'
ssh spark '~/venvs/w1/bin/python -m pip install --upgrade pip'
ssh spark '~/venvs/w1/bin/python -m pip install torch numpy'

The install is packaged as setup.sh in the companion repo, which just runs the
three steps above against requirements.txt:

#!/usr/bin/env bash
set -euo pipefail
VENV="{% katex inline %}{VENV:-{% endkatex %}HOME/venvs/w1}"
HERE="{% katex inline %}(cd "{% endkatex %}(dirname "$0")" && pwd)"
python3 -m venv "$VENV"
"$VENV/bin/python" -m pip install --upgrade pip
"{% katex inline %}VENV/bin/python" -m pip install -r "{% endkatex %}HERE/requirements.txt"

The interesting part is what pip pulled in. Remember the stack tops out at CUDA
13.0, and without me specifying any special index, PyPI served an ARM64 +
CUDA 13 build automatically:

Successfully installed torch-2.13.0 nvidia-cuda-runtime-13.0.96
  nvidia-cudnn-cu13-9.20.0.48 nvidia-cublas-13.1.1.3 ... (aarch64 wheels)

This is the payoff of the "PyTorch ships its own CUDA runtime" point from
earlier: I did not install the CUDA toolkit or touch nvcc. Torch brought
its own CUDA 13 runtime libraries (libcudart, cuDNN, cuBLAS) as ordinary
Python wheels, matched to the ARM64 architecture and the driver's CUDA 13
ceiling. The driver was the only piece I needed pre-installed.

Proving the GPU actually works from Python

Now the moment this whole week builds to: does Python see the GPU? The
validation script (validate_gpu.py in the repo) is deliberately tiny:

import torch

print("PyTorch:", torch.__version__)
print("CUDA available:", torch.cuda.is_available())
print("CUDA version (torch):", torch.version.cuda)

if torch.cuda.is_available():
    print("Device:", torch.cuda.get_device_name(0))
    print("Capability:", torch.cuda.get_device_capability(0))
    x = torch.rand(3, 3, device="cuda")   # put a real tensor on the GPU
    print("Tensor on:", x.device)

Run it with the venv's Python:

ssh spark '~/venvs/w1/bin/python validate_gpu.py'

PyTorch: 2.13.0+cu130
CUDA available: True
CUDA version (torch): 13.0
Device: NVIDIA GB10
Capability: (12, 1)
Tensor on: cuda:0

Every line here is worth reading. CUDA available: True means PyTorch found a
usable GPU through the driver. 2.13.0+cu130 confirms it's a CUDA 13 build.
Device: NVIDIA GB10 is our chip. Capability: (12, 1) is the GPU's compute
capability, NVIDIA's versioning for GPU features; 12.x is the Blackwell
generation. And Tensor on: cuda:0 is the real proof: the tensor physically
lives in GPU memory, not on the CPU. That's the difference between "installed"
and "actually usable."

Testing the prediction: CPU vs GPU

Earlier I predicted the GPU would beat the CPU by 10x-50x on a large matrix
multiply. Time to measure instead of hand-wave. The benchmark
(benchmark.py) multiplies two 4096x4096 matrices 20 times on each device and
averages. Two details make the GPU timing honest: a warmup (the first GPU
call pays a one-time kernel-load cost, so I run a few throwaway iterations
first), and torch.cuda.synchronize() before stopping the clock (CUDA launches
kernels asynchronously, so without a sync I'd be timing queueing, not
computing):

def bench(a, b, sync=False):
    if sync: torch.cuda.synchronize()
    t0 = time.perf_counter()
    for _ in range(ITERS):
        c = a @ b
    if sync: torch.cuda.synchronize()
    return (time.perf_counter() - t0) / ITERS

cpu_s = bench(a, b)                       # on CPU
ag, bg = a.cuda(), b.cuda()
for _ in range(3): _ = ag @ bg            # warmup
gpu_s = bench(ag, bg, sync=True)          # on GPU

That is the core idea; the full benchmark.py also times the very first (cold)
GPU call and runs a TF32 pass, which is where the extra numbers below come from.

ssh spark '~/venvs/w1/bin/python benchmark.py'

Matrix: 4096x4096 float32, iters: 20
CPU        172.57 ms   0.80 TFLOP/s
GPU cold     135.73 ms   (first call: one-time kernel load)
GPU FP32       7.67 ms   17.9 TFLOP/s   22.5x faster than CPU
GPU TF32       3.42 ms   40.1 TFLOP/s   50.4x faster than CPU

A single 4096x4096 float32 matmul is about 137 billion floating-point
operations. The 20-core ARM CPU works through it in ~173 ms (~0.80 TFLOP/s); the
GB10 does it in ~7.7 ms (~17.9 TFLOP/s), a 22.5x speedup, inside my predicted
10x-50x range. That gap is why GPUs exist: the same math, done thousands at a
time instead of a few at a time.

Two details in that output are worth calling out. First, the cold GPU call
took 135 ms, nearly as long as the whole CPU run, then every warm call took 7.7
ms. That is the one-time kernel-load cost I predicted; it is exactly why the
benchmark warms up before timing. Second, I measured a TF32 number too:
40 TFLOP/s, a 50x speedup. TF32 is a lower-precision mode NVIDIA GPUs use on
tensor cores; PyTorch leaves it off for matmul by default, so the honest FP32
number is the 22.5x one. TF32 and other reduced-precision formats are a whole
topic for later weeks, but it is worth seeing early that precision is a dial you
can trade for speed.

One caveat so the number is not oversold: this matmul is compute-bound (it
keeps the cores busy), which is why the GPU looks so good. When a model generates
text token by token, it is usually memory-bandwidth-bound instead, and the
speedups are smaller. That distinction is a big theme later. (All these values
are in results.json in the repo; yours will differ by machine.)

What surprised me

A GPU with no VRAM number is a feature, not a fault. Unified memory reframes "will it fit?" from ~24 GB to 121 GB.
nvcc: command not found is not an error state; the CUDA that matters for running models lives inside PyTorch, not in the toolkit. Installing torch pulled its own CUDA 13 runtime as plain wheels; I never touched the toolkit.
Verifying every fact with a command, even the "obvious" ones, already caught assumptions I would otherwise have carried into later weeks.
The GPU's 22.5x matmul win was real and measurable on day one, not a slide.

What I'll do next

Week 1 is done: I know the machine, I understand the NVIDIA stack, and I've proven
the GPU is usable from Python with a real speedup. Week 2 leaves inventory behind
and runs an actual language model with Ollama, a model runtime with a local HTTP
API, where I'll start measuring the things that matter for serving: tokens per
second and time to first token.

32-week roadmap:
https://github.com/dramasamy/from-api-to-gpu/tree/main/roadmap ↩
Companion code repository:
https://github.com/dramasamy/from-api-to-gpu ↩
Week 1 lab:
https://github.com/dramasamy/from-api-to-gpu/tree/main/week-01-environment ↩

DEV Community: Dinesh Kumar Ramasamy

From API to GPU, Week 4: What Model Parameters Actually Cost in Memory

What a parameter actually is

Precision: how many bytes each number takes

GB versus GiB, before the numbers start

The weight-memory formula

Checking the formula against a real file

Why the running model needs more than its weights

Results

What surprised me

Mistakes and troubleshooting

Production implications

What I will learn next

Run it yourself

Optional depth: why Q4 uses more than four bits on disk

From API to GPU, Week 3: Reading a Hugging Face Model Repository

The model card page is the front door

The four names I kept mixing up

Base model versus instruct model

The license is the first thing to check

The Files and versions tab

config.json is the architecture contract

generation_config.json supplies defaults, not hard limits

The tokenizer is several files working together

Safetensors files are the checkpoint

How do I know it is really an instruct model?

The Qwen chat template, connected to Week 2

Results

What surprised me

Mistakes and troubleshooting

Production implications

What I will learn next

Run it yourself: the API way in the lab

From API to GPU, Week 2: What Actually Happens Behind the API

Concept 1: a model is not a runtime

Concept 2: the API is just HTTP over a local process

Concept 3: inference has two measured phases

Concept 4: temperature, or how random the model is allowed to be

Concept 5: chat is a list of role-tagged messages

First, chat through the CLI

Next, inspect messages through the API

Finally, implement the conversation in Python

Concept 6: measure cold starts and verify context allocation

What did this experiment prove?

The token questions, answered

Wait, how do I even know what each token ID means?

And how does Ollama know which tokenizer to use?

Results

What surprised me

Mistakes and troubleshooting

Production implications

What I will learn next

Run it yourself

Can llama.cpp reuse this Ollama model?

From API to GPU, Week 1: Understanding Your Local GPU Environment

About this series

Where I started: two machines, and a rule I broke immediately

CPU versus GPU, and why models love the GPU

Reading nvidia-smi: my new top for the GPU

The surprise: unified memory, and a blank that isn't a bug

Driver vs CUDA runtime vs CUDA toolkit

Installing PyTorch, the right way: a virtual environment

Proving the GPU actually works from Python

Testing the prediction: CPU vs GPU

What surprised me

What I'll do next

`config.json` is the architecture contract

`generation_config.json` supplies defaults, not hard limits

Reading `nvidia-smi`: my new `top` for the GPU