Eight model repos, added up: the file list and the model disagree

I wanted one number: how big is this model. Not a parameter count, not a formula — the bytes that land on disk when you pull it down.

So I did the dumbest possible thing. I asked eight model repos on Hugging Face for their file list, added up every file that looks like weights, and compared that sum with the single file I would actually load. One request each, no token:

import json, urllib.request

url = "https://huggingface.co/api/models/openai-community/gpt2?blobs=true"
d = json.load(urllib.request.urlopen(url))
for s in d["siblings"]:
    print(s["rfilename"], s["size"], (s.get("lfs") or {}).get("sha256"))

Seven of the eight repos came back with a number between 2.0× and 9.7× the bytes I would download.

repo	weight files	of those, in a subdirectory	sum of every weight file	one loadable file	ratio
sentence-transformers/all-MiniLM-L6-v2	14	11	884,718,327	90,868,376	9.74×
openai-community/gpt2	7	3	4,054,448,846	548,105,171	7.40×
HuggingFaceTB/SmolLM2-135M-Instruct	10	8	1,966,318,064	269,060,552	7.31×
google-bert/bert-base-uncased	6	1	2,918,975,670	440,449,768	6.63×
distilbert/distilbert-base-uncased	4	0	1,167,291,991	267,954,768	4.36×
facebook/opt-125m	3	0	751,734,738	250,540,281	3.00×
EleutherAI/pythia-70m	2	0	332,078,951	166,029,852	2.00×
Qwen/Qwen2.5-0.5B-Instruct	1	0	988,097,824	988,097,824	1.00×

That last row is the control, and it is the one that tells you what the other seven rows are saying. Qwen2.5-0.5B-Instruct ships exactly one weight file, so its file list and its model are the same object and the ratio is 1. The others are not wrong about their own contents; they just contain more than one copy of the same weights, in more than one container.

One model, several containers

gpt2 is 137,022,720 parameters, which the Hub reports as safetensors.total. Its repo holds seven weight files:

onnx/decoder_model_merged.onnx        655,189,339
onnx/decoder_with_past_model.onnx     653,672,649
onnx/decoder_model.onnx               653,665,842
pytorch_model.bin                     548,118,077
model.safetensors                     548,105,171
tf_model.h5                           497,933,648
flax_model.msgpack                    497,764,120

Four of them are the same 137M weights written by four different libraries — PyTorch, safetensors, TensorFlow, Flax. The three .onnx files are graph exports for different decoding paths. Load one, and you download 523 MiB. Sum the list, and you get 3.78 GiB, which is 7.4× the file you would load, and none of that extra is more model.

Nobody would do that on purpose. But "sum the files that look like weights" is exactly what a crawler, a cost estimate, or a "repo size" badge does, and the result is internally consistent — it adds up, it is just not the model.

One blob, several names

Two of the eight repos list the same bytes under more than one name. Not similar bytes — the same lfs.sha256:

sentence-transformers/all-MiniLM-L6-v2
  23,026,053 bytes  ->  onnx/model_qint8_arm64.onnx
                        onnx/model_qint8_avx512.onnx
                        onnx/model_qint8_avx512_vnni.onnx

HuggingFaceTB/SmolLM2-135M-Instruct
  137,147,981 bytes ->  onnx/model_quantized.onnx
                        onnx/model_uint8.onnx

Three names, one object. Summing the file list counts that 23 MB three times; deduplicating by hash brings all-MiniLM-L6-v2 from 884,718,327 down to 838,666,221, and the difference is 46,052,106 — exactly two extra copies. Same arithmetic on SmolLM2-135M-Instruct: 1,966,318,064 minus 1,829,170,083 is 137,147,981, exactly the one duplicated pair.

So a file list is not a set. If you are counting storage, it is a multiset and you have to collapse it by hash first. If you are counting what a user downloads, it is not even that, because they will only ever take one of those formats.

No safetensors at all: null is not zero

facebook/opt-125m contains no .safetensors file. Three weight containers, all of them other formats. Ask the API for its parameter count and you get:

safetensors: null
safetensors.total: None
safetensors.parameters: {}

Not zero — absent. Which means a pipeline that does total = d["safetensors"]["total"] or 0 will record a 125-million-parameter model as having zero parameters, and nothing downstream will complain. That is not a hypothetical formatting problem; it is the same shape as reading a missing field as an empty result, and the number it produces is a perfectly ordinary-looking 0.

If you need a parameter count for a repo that ships only .bin, .h5 or .msgpack, you have to open the files. No field will answer for you.

Why my first number was wrong

The duplicate-blob section above is the second version of this article. The first version said something else: that none of the eight repos stored the same bytes twice.

I wrote the check to read siblings[i]["lfs"]["oid"]. There is no oid key there. The real field is lfs.sha256. My extractor returned None for every file, the deduplication map came back empty, and the script printed:

oids_in_weight_files=0  oids_under_two_or_more_names=0

for all eight repos. Read on its own, that line is a clean bill of health. What caught it was the line directly above it, which said every one of those files was LFS-backed:

onnx/decoder_model_merged.onnx    655189339  lfs=True

You cannot have seven LFS files and zero content hashes. The two readings contradicted each other, and the contradiction was the bug report.

The general shape is worth more than the field name: a lookup into a structure you have not actually printed will return "nothing" for every input, and "nothing" is the same answer you would get if the thing genuinely was not there. I got lucky because a second column disagreed. Usually nothing disagrees, and you publish the zero.

What I would do instead of adding up files

• For "what will this cost me to download" — pick the one file you would load, and quote its size. model.safetensors if it exists, otherwise the largest single .bin. The file list is not the answer to this question.
• For "how many parameters" — safetensors.total is the Hub's own count over the safetensors files, and where it exists it agreed with the per-format arithmetic I checked it against. Where it does not exist it is null, and null is a different word than 0.
• For "how much space does this repo occupy" — collapse by lfs.sha256 first. Two names, one blob, and the blob is charged once.
• If you filter the file list by anything, filter by both name and location. all-MiniLM-L6-v2 keeps 11 of its 14 weight files in subdirectories; a filter that only looks at root-level filenames sees 3 of 14 and reports a repo that is three times smaller than itself.

None of this needs a token, a script library, or a download. It is one request per repo and some arithmetic, and the arithmetic is the part that catches you.

What I did not measure

Whether a second copy of the weights in a subdirectory inflates safetensors.total the way it inflates a naive file sum. None of the eight repos I opened keeps two .safetensors files, so I had nothing to test it on, and I am not going to guess at the Hub's behaviour from its output on a different shape. If you know a repo that ships model.safetensors and a copy of it under a subdirectory, that is the experiment, and it is one request.

Eight repos is also not a survey. It is eight repos, chosen because I already had them open, and the ratios above are what those eight contain.