Dinesh Kumar Ramasamy

Posted on Jul 20

From API to GPU, Week 4: What Model Parameters Actually Cost in Memory

#ai #llm #gpu #machinelearning

Phase 1 of 8: Comfortable running local models. Week 4 of 32.

People throw around "7B", "14B", and "70B" like everyone knows what they mean.
For a long time I nodded along without really knowing. This week I make those
numbers concrete. By the end I can look at a model size and a precision and say
how much memory the weights need, then explain why the running model needs more.

This is the last week of Phase 1. In Week 1 I learned the machine, in Week 2 I
ran a model through Ollama, and in Week 3 I read a model's files on Hugging Face.
Now I connect the parameter count I saw in Week 3 to real memory on the DGX
Spark.

The calculator part runs anywhere with Python. The memory measurements run on
the Spark over ssh spark, the same setup as earlier weeks.

What a parameter actually is

A parameter is a single learned number inside the model. Training adjusts
billions of these numbers until the model predicts text well. When a model is
called "3B", it has about 3 billion of these numbers.

Wait, is a parameter the same thing as a token? No. A token is a piece of
text that enters or leaves the model. Tokens are the request and response data;
parameters are the learned numbers stored in the model files.

	Token	Parameter
What it is	a piece of input or generated text	a learned number
Where it comes from	the prompt or model response	training
What happens during inference	input tokens arrive and output tokens are added	values stay fixed
What its count affects here	context and KV-cache use	weight memory

They are connected, though. The tokenizer turns each token into an ID. The
model uses learned parameters to turn those IDs into internal values, process
them, and predict the next token. A 3B model therefore has about 3 billion
parameters, not 3 billion tokens. The same parameters are reused for every
token the model processes.

Parameter is the umbrella term. Two kinds sit under it:

A weight multiplies an input value. Most parameters are weights.
A bias is another learned value added after the multiply. Models usually have far fewer bias entries than weight entries.

The reported parameter count includes both. So when I say "weight memory" in this
post, I mean the storage for all parameters, weights and biases together. The
count is what drives the memory math.

A few more words show up when you look inside a model, so here they are in plain
terms:

A tensor is a block of numbers. A single number is a scalar, a list is a vector, a grid is a matrix, and a tensor is the general name for any of these.
A matrix is a two-dimensional grid of numbers, the most common shape for a block of weights.
A layer is one processing stage. A model stacks many layers, and each one holds its own tensors.
A shape is the size of a tensor along each dimension, like 2048 by 2048.

I do not need the internals of these this week. Week 5 works with tensors
directly in PyTorch, and Week 11 explains layers. Here they are just the words
for "the numbers we are about to measure".

In Week 3 I read the parameter count from the Hugging Face API. Here I pin the
API request to the same commit used for the weight index later in this post:

$ MODEL=Qwen/Qwen2.5-3B-Instruct
$ REV=aa8e72537993ba99e69dfaafa59ed015b17504d1
$ curl -s "https://huggingface.co/api/models/$MODEL/revision/$REV" | jq '{
    revision: .sha,
    parameters: .safetensors.total,
    precision_groups: .safetensors.parameters
}'
{
  "revision": "aa8e72537993ba99e69dfaafa59ed015b17504d1",
  "parameters": 3085938688,
  "precision_groups": {
    "BF16": 3085938688
  }
}

So this model has 3,085,938,688 parameters, and every one of them is stored in
the BF16 format. That second fact is the other half of the memory question.

Precision: how many bytes each number takes

A parameter is a number, and a number needs storage. Precision is the format
used to store each one. The format decides how many bits, and therefore how many
bytes, every parameter takes.

The short forms are worth spelling out. FP means floating point, a format for
fractional numbers. BF16 means bfloat16, a 16-bit floating-point format. INT
means integer. You saw torch_dtype: bfloat16 in the Week 3 config. Here is what
the common formats cost per parameter:

Precision	Bits	Bytes
FP32	32	4
FP16	16	2
BF16	16	2
INT8	8	1
INT4	4	0.5

FP32 is the 32-bit baseline, using 4 bytes per number. FP16 and BF16 are both
16-bit formats, so both use 2 bytes. INT8 and INT4 are the low-bit formats
produced by quantization, which is a whole phase later in this series (weeks 14
to 16). INT4's 0.5 byte is an average: two 4-bit values pack into one byte. For
now the only thing that matters is the bytes column.

GB versus GiB, before the numbers start

Memory numbers come in two units, and mixing them causes confusion, so I define
them once up front:

GB is decimal: divide bytes by 1,000,000,000.
GiB is binary: divide bytes by 1024 three times (1024 x 1024 x 1024).

The calculator prints both. Ollama later prints its own rounded GB labels, so
those are approximate runtime reports, not byte-exact comparisons.

The weight-memory formula

Put the two facts together and the weight memory is simple multiplication:

\text{weight bytes} = \text{parameter count} \times \frac{\text{bits per parameter}}{8}

Dividing by 8 converts bits to bytes. That is the whole formula. The Week 4
deliverable is a small calculator that applies it, so I do not redo the
arithmetic by hand each time.

Here is the complete script:

#!/usr/bin/env python3
"""Estimate model weight memory from a parameter count and a precision.

The math is exact for a single uniform precision:

    weight_bytes = parameter_count * bits_per_parameter / 8

This is the tensor payload only. Real runtime memory is larger because of the
KV cache, activations, and framework overhead, and it cannot be derived from the
parameter count. The blog measures that runtime memory separately.
"""

from __future__ import annotations

import argparse

BITS_PER_PARAMETER = {
    "fp32": 32,
    "fp16": 16,
    "bf16": 16,
    "int8": 8,
    "int4": 4,
}

GB = 1_000_000_000
GIB = 1024 ** 3


def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser(description=__doc__)
    group = parser.add_mutually_exclusive_group(required=True)
    group.add_argument(
        "--parameters",
        type=float,
        help="Parameter count in billions, e.g. 70 or 3.2",
    )
    group.add_argument(
        "--count",
        type=int,
        help="Exact parameter count, e.g. 3085938688",
    )
    parser.add_argument(
        "--precision",
        choices=sorted(BITS_PER_PARAMETER),
        required=True,
    )
    return parser.parse_args()


def weight_bytes(parameter_count: int, precision: str) -> float:
    return parameter_count * BITS_PER_PARAMETER[precision] / 8


def main() -> None:
    args = parse_args()
    if args.count is not None:
        parameter_count = args.count
    else:
        parameter_count = round(args.parameters * 1_000_000_000)

    raw_bytes = weight_bytes(parameter_count, args.precision)

    print(f"parameter_count    {parameter_count:,}")
    print(f"precision          {args.precision}")
    print(f"bits_per_parameter {BITS_PER_PARAMETER[args.precision]}")
    print(f"weight_bytes       {raw_bytes:,.0f}")
    print(f"weight_gb          {raw_bytes / GB:.2f} GB (decimal, / 1e9)")
    print(f"weight_gib         {raw_bytes / GIB:.2f} GiB (binary, / 1024^3)")


if __name__ == "__main__":
    main()

I run the roadmap example first: a 70-billion-parameter model at INT4.

$ python3 public/week-04-parameters-memory/model_memory.py \
  --parameters 70 --precision int4
parameter_count    70,000,000,000
precision          int4
bits_per_parameter 4
weight_bytes       35,000,000,000
weight_gb          35.00 GB (decimal, / 1e9)
weight_gib         32.60 GiB (binary, / 1024^3)

70 billion parameters at half a byte each is 35 GB of weights. The same 70B
model at BF16 would be four times that, about 140 GB. That is roughly 130 GiB,
already more than the Spark's 121 GiB pool before any runtime memory. This is
why big models are so often quantized before anyone tries to run them locally.

Checking the formula against a real file

A formula is only trustworthy if it matches reality. I ran the calculator with
the exact Qwen count and its real BF16 precision:

$ python3 public/week-04-parameters-memory/model_memory.py \
  --count 3085938688 --precision bf16
parameter_count    3,085,938,688
precision          bf16
bits_per_parameter 16
weight_bytes       6,171,877,376
weight_gb          6.17 GB (decimal, / 1e9)
weight_gib         5.75 GiB (binary, / 1024^3)

The calculator says 6,171,877,376 bytes. Now the model's own Safetensors index,
read directly with the pinned revision from Week 3:

$ MODEL=Qwen/Qwen2.5-3B-Instruct
$ REV=aa8e72537993ba99e69dfaafa59ed015b17504d1
$ curl -sL "https://huggingface.co/$MODEL/raw/$REV/model.safetensors.index.json" \
    | jq '.metadata.total_size'
6171877376

Identical. The formula is not a guess. For a uniform precision it lands exactly
on the tensor-data bytes the index reports. That total is the parameter payload,
not the whole file with its headers, but for the weight math it is the number I
want.

Why the running model needs more than its weights

Everything so far is weight storage: the model sitting on disk. A model doing
work needs more memory than that. The extra memory has three main parts:

Activation memory is the temporary numbers created while processing a request. They come and go as the model runs.
KV-cache memory stores saved intermediate data from earlier tokens, so the model does not recompute them for every new token. This grows with context length. The KV cache is a Week 18 topic; here I only need to see that it exists and costs memory.
Framework overhead is the runtime's own working memory and buffers.

The exact-file check used BF16 Qwen. This runtime experiment switches to
Ollama's quantized llama3.2:3b, the model already available from Week 2. Its
2.0 GB package, 2.6 GB load, and 4.1 GB load belong to one comparison; they
should not be compared with Qwen's 6.17 GB BF16 weights.

Ollama reports the loaded size of a model, so I can compare it with the 2.0 GB
package size. I expect the 4,096-token load to exceed 2.0 GB because the runtime
needs memory beyond the package. I expect the 16,384-token load to be larger
again because it reserves more KV-cache capacity. ollama ps reports the total,
so this experiment can show the change but cannot assign an exact number of
bytes to the KV cache.

I connect to the Spark once, then run each check directly:

$ ssh spark
$ ollama list | awk 'NR == 1 || $1 == "llama3.2:3b"'
NAME                       ID              SIZE      MODIFIED
llama3.2:3b                a80c4f17acd5    2.0 GB    3 days ago

Next I unload any existing copy, send one short request with a 4,096-token
context, and inspect the loaded model. The API's HTTP 200 confirms that the
otherwise silent request completed:

$ ollama stop llama3.2:3b >/dev/null 2>&1 || true
$ curl -fsS -o /dev/null -w 'HTTP %{http_code}\n' \
    http://localhost:11434/api/generate \
    -d '{"model":"llama3.2:3b","prompt":"hi","stream":false,
    "options":{"num_ctx":4096,"num_predict":1}}'
HTTP 200
$ ollama ps
NAME           ID              SIZE      PROCESSOR    CONTEXT    UNTIL
llama3.2:3b    a80c4f17acd5    2.6 GB    100% GPU     4096       4 minutes from now

I repeat the same request with a 16,384-token context:

$ ollama stop llama3.2:3b >/dev/null 2>&1 || true
$ curl -fsS -o /dev/null -w 'HTTP %{http_code}\n' \
    http://localhost:11434/api/generate \
    -d '{"model":"llama3.2:3b","prompt":"hi","stream":false,
    "options":{"num_ctx":16384,"num_predict":1}}'
HTTP 200
$ ollama ps
NAME           ID              SIZE      PROCESSOR    CONTEXT    UNTIL
llama3.2:3b    a80c4f17acd5    4.1 GB    100% GPU     16384      4 minutes from now

Ollama lists the package as 2.0 GB. Loaded with a 4,096-token context, it reports
2.6 GB. Loaded with a 16,384-token context, it reports 4.1 GB. The model did not
change. Only the configured context capacity changed, and the reported loaded
size grew by 1.5 GB. The KV cache grows with context, so it is the expected main
cause, but ollama ps reports one total and does not split KV cache, activations,
and framework buffers.

This is the practical lesson: weight memory is the floor, not the total. Runtime
memory is larger, and context length is one of the settings that pushes it up.

Results

Here is Week 4 in one place:

Item	Verified value
Formula	parameters × bits / 8
70B at INT4	35.00 GB of weights
Qwen2.5-3B-Instruct at BF16	6,171,877,376 bytes
Formula versus Safetensors index	exact match
Loaded at 4,096 context	2.6 GB
Loaded at 16,384 context	4.1 GB
Spark unified memory	121 GiB

What surprised me

The formula matching the Safetensors index to the exact byte was satisfying.
Weight memory really is just multiplication once you know the count and the
precision.

The context effect was the most useful. I knew the KV cache existed from Week 2,
but seeing a 2.0 GB package report 4.1 GB of loaded memory just by raising
context made it concrete.

Mistakes and troubleshooting

I also had to be careful about GB versus GiB. The calculator prints both, and
Ollama prints its own rounded GB labels, so I keep the units visible to avoid
comparing the wrong things.

Production implications

Weight memory sets the floor for whether a model fits. For capacity planning I
would:

Estimate weights from the count and precision, then confirm against the real file for quantized models.
Measure total runtime memory with the real prompt lengths and workload, since the KV cache and activations depend on how the model is actually used, not on the parameter count.
Treat the Spark's 121 GiB unified pool as shared by the model, its runtime, and everything else on the box.

The formula tells me if the weights fit. The runtime measurement tells me if the
working model fits. Both matter before scheduling hardware.

What I will learn next

Week 5 moves into PyTorch tensors directly. I will allocate tensors on the CPU
and GPU, compare FP32, FP16, and BF16, and watch allocated memory change. That
turns this week's byte math into something I can measure inside a running Python
process.¹

Weeks 14 to 16 cover quantization properly, which is where the Q4_K_M mixed
format and its real bit cost get explained in full.

Run it yourself

The public Week 4 lab has the calculator, direct runtime commands, captured
results, observations, and troubleshooting notes.² The
quantized-file note holds the optional Q4_K_M storage investigation with its
full commands and output.³

Optional depth: why Q4 uses more than four bits on disk

The clean formula assumes every parameter uses the same number of bits. A
quantized file can mix several tensor formats and store extra scale data, so a
model labeled "4-bit" may use more than 4 bits per parameter on disk.

I checked Ollama's llama3.2:3b package. Its API reports 3,212,749,888
parameters and Q4_K_M quantization.⁴ A clean INT4 calculation predicts
1,606,374,944 bytes, but the model blob is 2,019,377,376 bytes. That works out
to about 5.03 on-disk bits per parameter. I also matched the blob's byte size
and SHA-256 digest to its manifest record, so this is the exact file Ollama
references, not only a filename that looks right.

This was the opposite of the uniform BF16 result. I expected a "4-bit" model to
cost 4 bits per parameter, but this file is about 26 percent larger than the
clean estimate. The fix was to use the manifest and real blob instead of treating
the quantization label as an exact file-size promise.

The companion note keeps the full manifest-to-blob chain and tensor-type
counts.³ For Week 4, the rule is enough: use the formula for a
first estimate, then read the real file size for a quantized model.