DEV Community: Li Zhuojun

I checked nine free RFC 3161 endpoints against the EU Trusted List. None is qualified. Five of the companies are.

Li Zhuojun — Fri, 16 Oct 2026 22:07:04 +0000

On 16 October 2026 I sent one timestamp request to each of nine public, free RFC 3161 endpoints and looked up the certificate that signed each reply in the EU Trusted List. All nine answered. All nine tokens verified (message imprint and nonce echoed back correctly). None of the nine is a qualified timestamp under eIDAS.

The list I checked against: LOTL sequence 397, issued 2026-10-15, 29 member-state lists downloaded and signature-verified, 1,443 TSA/QTST entries of which 915 are granted today. Snapshot SHA-256 d1700e6a…c85d, LOTL SHA-256 a36d0053…70e6. Hungary and Portugal didn't download (TLS cut the connection), so a Hungarian or Portuguese endpoint would have come back "not checked", not "not qualified". None of the nine is in either country.

Endpoint	Signing certificate	Issued by	Qualified?	Granted QTST entries on the list under this company's name
FreeTSA `freetsa.org/tsr`	www.freetsa.org	www.freetsa.org (self-issued)	No	0
DigiCert `timestamp.digicert.com`	DigiCert SHA256 RSA4096 Timestamp Responder 2026 1	DigiCert Trusted G4 TimeStamping RSA4096 SHA256 2025 CA1	No	9 (NL, DigiCert Europe Netherlands B.V.)
Sectigo `timestamp.sectigo.com`	Sectigo Public Time Stamping Signer R37	Sectigo Public Time Stamping CA R41	No	1 (ES, Sectigo (Europe) SL)
GlobalSign `timestamp.globalsign.com/tsa/r45standard`	Globalsign R45 TSA for CodeSign 202510	GlobalSign Offline R45 Timestamping CA 2025	No	4 (BE, GlobalSign NV/SA)
Certum `time.certum.pl`	Certum Timestamp 2026	Certum Timestamping 2021 CA	No	3 (PL, Asseco Data Systems S.A.)
SSL.com `ts.ssl.com`	SSL.com Timestamping Unit 2025 E1	SSL.com Timestamping Issuing RSA CA R1	No	0
Entrust `timestamp.entrust.net/TSS/RFC3161sha2TS`	Sectigo Public Time Stamping Signer R37	Sectigo Public Time Stamping CA R41	No	0 (1 entry, ES, withdrawn)
Apple `timestamp.apple.com/ts01`	Timestamp Signer PV02	Apple Timestamp Certification Authority	No	0
Actalis `timestamp.actalis.it`	Actalis Time-Stamping Authority TSU 2	Actalis Time Stamping CA G1	No	2 (IT, Actalis S.p.A.)

"Qualified" here is the mechanical test from ETSI TS 119 615: the key that signed the token, or the CA that issued it, has to match a TSA/QTST entry whose status was granted at the token's genTime. For all nine, neither the signing key nor the issuer name appears anywhere among the 1,443 qualified-timestamp entries. Not as granted, not as withdrawn. The lookup never reached the status step.

The last column is the point

I ran this same check in August and wrote it up with one example of a QTSP whose free endpoint isn't qualified: Actalis. That undersold it. Five of the nine companies hold granted qualified-timestamp entries on the list today. They just don't hand them out at the free URL.

Actalis is the cleanest case because both halves sit in the same country under the same company name. The free endpoint signs with Actalis Time-Stamping Authority TSU 2, issued by Actalis Time Stamping CA G1. The Italian list has two granted entries for Actalis S.p.A.: Actalis TSA 1 EIDAS and Actalis EU Qualified TimeStamp CA G1. Different CAs. The library returns False for the free one, and that is the right answer.

The other four are the same shape with an extra twist: the entity on the list is often not the entity on the certificate. The free Sectigo token is signed under O=Sectigo Limited, C=GB; the qualified entry is Sectigo (Europe) SL on the Spanish list. GlobalSign's free token says GlobalSign nv-sa, C=BE, and Belgium does list four granted GlobalSign qualified timestamping CAs, none of which issued this certificate. Certum's token is signed by Asseco Data Systems, which has three granted entries in Poland, under different CAs. DigiCert's nine granted entries are in the Netherlands, under DigiCert Europe Netherlands B.V., with QuoVadis names on the certificates.

So "is this provider a QTSP?" is the wrong question. Qualification attaches to a service entry, which is a public key plus a status timeline. The right question is whether the key that signed your token is on the list, and was granted at that second.

Two things that moved in eight weeks

Entrust's URL still answers, and the token it returns is signed by Sectigo Public Time Stamping Signer R37, serial 0xe74ef255…a315. That's byte-for-byte the same certificate timestamp.sectigo.com returned a few seconds earlier. Entrust sold its public certificate business to Sectigo, and the endpoint followed. A config file that says entrust now produces Sectigo evidence. Nothing broke; nothing told you either.

Apple's signing certificate was issued on 2026-10-15 and expires on 2026-11-26. Forty-two days. Whatever certificate you look up next month will not be this one. Between my August run (LOTL 390) and this one (LOTL 397) the list itself moved seven sequence numbers; Ireland, unreachable in August, downloaded fine, and Hungary and Portugal didn't.

What this costs you, legally

Article 41(1) of eIDAS says an electronic timestamp can't be refused as evidence just because it is electronic or not qualified. So every token in that table is admissible. Article 41(2) gives qualified timestamps, and only qualified timestamps, a presumption of accuracy of the time and integrity of the data. With a free token, you prove the clock and the integrity; with a qualified one, the other side has to disprove them.

For most logging that's fine. It stops being fine when the record is one you'd put in front of a regulator. The AI Act's Article 12 logging duty applies from 2 December 2027 for Annex III systems and 2 August 2028 for Annex I systems, after Regulation (EU) 2026/1744 moved the dates, and the text of the article didn't change. If you're building toward that, the free tier of a QTSP is not the QTSP. I haven't found a free qualified endpoint, from these nine or anywhere else. (If you know of one, I'd like to be wrong.)

Look it up when you stamp, not when you're asked

The failure mode I care about isn't picking a free endpoint. It's picking a paid qualified one, and then checking its status three years later when a dispute arrives. That lookup answers today's question against today's list. Keys rotate, entries get withdrawn, the Entrust URL becomes a Sectigo URL. The only time the answer is knowable is the moment the token comes back, and the answer has to be written down then, next to the token, with the hash of the list it was checked against. That is what tg-attest does on its write path, as a three-valued field (True / False / None for "couldn't check"), with no network call at stamping time. The measurement above is that code run by hand.

Limits

Nine endpoints is nine endpoints, picked because they are the ones people paste into signtool and openssl ts. Certificates rotate (Apple's lasts six weeks; GlobalSign's is named 202510) and the LOTL moved seven times in eight weeks, so this table is true for 16 October 2026 and I make no claim about any other date. The "company name" column is my substring match on the list's provider names, not a corporate register; it's there to show the pattern, not to settle who owns whom. The two unreachable member states are unreachable today, not permanently, and five list entries (one French, four Norwegian) carried certificates my parser rejected and were skipped. A verdict of "not qualified" is about Article 41(2)'s presumption only; nothing here says these tokens are technically weak. All nine echoed my hash and nonce correctly. I didn't run full chain verification for this post.

I kept every .tsr from this run, and you don't need them: request a token from any of the nine yourself and openssl ts -reply -in <file> -token_in -text shows the same certificate names without installing anything of mine. The snapshot was built with python -m tg_attest.eutl_build, which verifies the LOTL against the six certificates in OJ C/2026/1944 and each national list against the LOTL.

Sources

Regulation (EU) No 910/2014 (eIDAS), Article 41
ETSI TS 119 615 V1.3.1, procedures for using and interpreting EU Member States' trusted lists, §4.6
OJ C/2026/1944, LOTL signing certificates
Regulation (EU) 2026/1744 (Digital Omnibus), moving the AI Act Chapter III application dates
Entrust, public certificate business transition to Sectigo

A routing audit for your team, fixed price. The method in this series runs on any Claude Code trace store. Send me 30 days of your team's usage records (model, tokens, timestamps, agent and session ids; I'll send a one-line jq filter that drops prompt and answer text before anything leaves your machine) and within ten working days you get one number and a five-page report: what share of your spend ran on a tier your own routing policy would not have chosen, what that cost at list price, and which components caused it. On my own 26,131 traces the number was 22.6% and $1,248.13, all of it on subagents and none on the main thread. US$1,500, flat. Write to info@zhuojun.li with the subject "routing audit".

10 of 12 new LLMs remember which earnings beat from just a name and a date. Jev doesn't.

Li Zhuojun — Fri, 25 Sep 2026 13:00:07 +0000

Earlier this week I tested whether Jev remembers how 2023 earnings turned out. It doesn't. So I ran the same pre-registered test on twelve current models through OpenRouter and on von, an open Jev-style model. Ten of the twelve remember. Shown two reports from the same company and the same season, one from 2023 and one from 2024, GPT-6 Astra picked the one that beat consensus 91% of the time. It had the company name and the date, nothing else.

The test

3,708 earnings announcements from 250 S&P 500 companies, January 2023 to August 2026.
Each model sees the company, ticker, exchange, sector and announcement date, and answers three questions with a probability: did EPS beat consensus, did the stock beat the S&P 500 over the 2-day reaction, and did it over the next 20 trading days.
For each company I take its 2023 and 2024 reports from the same quarter where the outcome differs, and count how often the model scores the right one higher. That's a within-company AUC. 0.5 means it can't tell.
The protocol, the model list and every request were hashed and timestamped with OpenTimestamps before the first call of each round.

Results

Model	Within-company AUC (95% CI)
GPT-6 Astra	0.904 (0.886 to 0.921)
Gemini 3.8 Flash	0.799 (0.776 to 0.822)
Claude Opus 5.5	0.766 (0.742 to 0.792)
GPT-6 Sol	0.691 (0.665 to 0.717)
Grok 4.7	0.603 (0.573 to 0.634)
Claude Sonnet 5	0.587 (0.561 to 0.613)
Qwen3.8 Max	0.578 (0.550 to 0.605)
GPT-6 Luna	0.573 (0.547 to 0.601)
DeepSeek V4.1 Flash	0.551 (0.521 to 0.578)
von 1.2 (open Jev-style)	0.541 (0.512 to 0.569)
Kimi K3	0.538 (0.511 to 0.567)
GLM-5.3	0.531 (0.501 to 0.560)
Jev 1.13	0.504 (0.478 to 0.530)
DeepSeek V4 Pro	0.504 (0.476 to 0.533)
Qwen3.8 Flash	0.498 (0.470 to 0.527)
Llama 3.1 70B	0.490 (0.464 to 0.517)

After Holm correction, every new API model except DeepSeek V4 Pro and Qwen3.8 Flash shows memory, and GLM-5.3 only just clears the bar. von shows some too. The Sonnet 5, Jev and Llama rows come from the first study.

What it means if you backtest with an LLM

If your pipeline passes a ticker and a date to one of the top models, part of the backtest is the model's memory. GPT-6 Astra, Gemini 3.8 Flash and Claude Opus 5.5 are the clearest cases. Two cheap defences: strip names and dates when the task allows, and test your own setup with a within-company comparison like this one before you trust the numbers.

Things I didn't expect

Price and size don't predict it inside a family. DeepSeek V4 Pro showed nothing while V4.1 Flash did, and Qwen3.8 Max remembers while Qwen3.8 Flash doesn't.
von, a small open model built to stand in for Jev, shows a little memory (0.541) where Jev shows none (0.504). Running a model locally doesn't make it clean.
The cheap tier remembers too. GPT-6 Luna scored 0.573, and all 3,708 of its calls cost US$0.15.
Six endpoints won't switch reasoning off, including GPT-6 Astra, Gemini 3.8 Flash and Claude Opus 5.5. They ran at the lowest effort, so I can't separate their memory from what a little reasoning adds.

Limits

One prompt, 250 companies, one window. A model that shows nothing here may still remember in another setting.
Each row is the model OpenRouter served on 24 September 2026. Versions change.
With about 1,200 pairs per model, the test can detect a pooled AUC of about 0.546. Anything smaller is invisible.
GPT-6 Astra stopped at 1,464 of its 3,708 calls when the account ran out of credit. I finished it after seeing that partial result (0.889), so that choice wasn't blind. The requests didn't change, and every call ran on 24 September.
The timestamps prove less than they sound. Bitcoin block 968402, which anchors the first-round freeze, was mined at 14:36 UTC, four minutes after that round's last call. On its own the proof shows the protocol existed by then, not that it came first. The second round's block is stamped two minutes after its first call, by which time 5,486 of its 14,832 requests had gone out.
API cost: US$55 for the first eight models and US$73 for the second four, where GPT-6 Astra alone was US$48 and Claude Opus 5.5 US$24.

Reproduce

Protocols, deviations, code, every answer and all three freeze proofs: github.com/lizhuojunx86/llm-memory-audit. The earnings data comes from FMP and can't be redistributed; the scripts rebuild it with your own key.

If you've caught this in your own backtests, I'd like to hear how.

Does Jev remember 2023? A naive test says yes at p = 0.001. A within-company test says no.

Li Zhuojun — Thu, 24 Sep 2026 12:58:48 +0000

I asked Jev about 12,533 US earnings announcements, 38,956 calls in all, to find out whether it remembers how they turned out. The test most people would write first says it does, at p = 0.001. A test that compares each company only with itself says it doesn't: shown two announcements from the same firm and the same season, one from 2023 and one from 2024, Jev can't tell the beat from the miss (within-company AUC 0.506, 95% CI 0.472 to 0.540). The whole run cost $0.93.

Why ask

Jev (TypeSafe, released 2026-09-15) answers typed questions with probabilities instead of text. TypeSafe says it's trained on synthetic data and reasons from the state you pass in. Outside observers suspect an open-weight LLM underneath, and people are already running it over historical earnings and calling the output a backtest.

If the model remembers what happened after each event, that backtest scores its memory. I've spent the past six months on that failure mode in LLM pipelines (it's what TraceGuard is for), so a brand-new model with a claim about its training data was worth a day.

The test that would have fooled me

My first draft of the protocol compared Jev's accuracy on 2023–2024 events with its accuracy on April to August 2026 events, for the same 812 companies. If it remembers, it should do better on the old ones.

On the finished data, that contrast gives:

"did the stock beat the S&P 500 over the next 20 trading days?", name and date only: AUC 0.555 on 2023–24 against 0.515 on 2026. Gap 0.040, 95% CI 0.005 to 0.076, p = 0.009.
the same question with the earnings numbers included, named minus anonymized: gap 0.015, 95% CI 0.006 to 0.024, p = 0.001.

That's a headline, and I couldn't have defended it. Before freezing anything I had a separate Claude instance attack the protocol. It simulated 200 worlds where the model had no memory at all, only stable hunches about sectors and company size, plus a shock shared by each sector in each season. Re-running that simulation for this post, the contrast flagged "leakage" at the 5% level in 46 of the 200 worlds (23%). The 2026 window holds about two earnings seasons, one market regime (its 20-day base rate is 36%, against 41% to 53% in other periods) and no fourth-quarter reports. Anything that differs between the two windows moves the gap.

The test I ran instead

Take one company. Take its Q2 announcement in 2023 and its Q2 announcement in 2024. Show Jev the name, ticker, sector and date, nothing about results, and ask whether it beat consensus. If exactly one of the two beat, a model that remembers ranks that one higher. A model that only knows "this firm usually beats" gives both the same score.

Two details close the obvious gaps. Every answer has the month's average answer subtracted first, so a model that is simply vaguer about recent dates gains nothing. And both events in a pair come from the same quarter of the year, so seasonal habits cancel. The statistic is the share of pairs where the event that beat got the higher score. No memory means 0.5.

The protocol, code, event list, labels and every request payload (53,788 of them, 38,956 for Jev) were hashed into one file, and the file was sent to OpenTimestamps seconds before the first real call. One thing did change afterwards, for the control model only, and it's logged below.

Results

Test	Question	Result (95% CI)	Pairs	Could detect
H1	beat vs miss, name and date only	0.506 (0.472 to 0.540)	915	0.552
H2	stock beat the S&P over the 2-day reaction	0.479 (0.454 to 0.503)	1,619	0.539
H3	stock beat the S&P over the next 20 days	0.483 (0.458 to 0.507)	1,462	0.541
H4	does adding name and date help a 20-day backtest	−0.006 (−0.021 to 0.009)	1,462	0.059
H5	same, for the 2-day reaction	−0.001 (−0.013 to 0.011)	1,619	0.056

Nothing survives Holm correction, and nothing is close. Pooling the three name-and-date questions gives 0.486 (0.472 to 0.502). S&P 500 names alone, where memory should be strongest: 0.521 (0.476 to 0.563) for beats.

Does the probe find memory in a model that has it? I ran the same questions past Claude Sonnet 5 for every 2023–2026 announcement of 250 S&P 500 companies. It separates the pairs clearly: 0.587 pooled (0.561 to 0.613), 0.602 on beats and 0.616 on the 2-day reaction. On exactly the same 1,207 pairs (a comparison I added after the fact, so treat it as descriptive), Jev scores 0.483 (0.458 to 0.509). Llama 3.1 70B, whose pretraining data stops in December 2023, showed nothing either (0.490 pooled), not even for 2023. So the probe catches memory at the level of detail a frontier model carries, and Jev doesn't show it.

What this does and doesn't rule out

It rules out Jev recalling individual announcements at anything like Sonnet 5's level. The pooled test could have picked up a within-company AUC of about 0.525, and each single question about 0.54 to 0.55.

It doesn't rule out hindsight in Jev's picture of companies. With only a name and a date, Jev ranks which firms beat with AUC between 0.61 and 0.67 in every period, 2026 included, so it clearly has company priors. Some of them may have been formed by watching 2023 and 2024 happen. The cross-sectional gap above fits that story, and it also fits a regime change. This data can't separate the two. The safe move for a backtest is cheap: leave the name and date out. Jev read the numbers just as well without them.

What Jev does do

It reads numbers. Given the EPS and revenue surprise, it ranks the 2-day reaction with AUC between 0.57 and 0.65 depending on the period, and inside a company it does the same with or without the name (0.673 vs 0.674).
It barely varies. Asking the same question three times moved the answer by 0.006 to 0.007 on average, and never by more than 0.04.
It ignores base rates. With a name and date only, it put the chance of an EPS beat at 0.46 on average. The real rate was 74%.
It's cheap and fast. 38,956 calls cost $0.93, ran at up to about 60 calls a second through OpenRouter, and all of them came back valid.

Limits

One model version, jev-1.13-20260917. The next version needs its own test.
The verdict is provisional by design. The protocol only allows "no detectable memory" once the prospective set below is scored.
The 2026 events might also sit inside what Jev saw in training. That's why 757 announcements from October and November 2026 were sent to Jev before they happened. They get scored in January, and Jev's answers are already in the repo.
FMP restates EPS, and restated labels push toward finding nothing. On surprises of at least 2% either way, the beat test gives 0.514 (0.478 to 0.551).
The timestamp proves less than it sounds. The Bitcoin block that anchors the freeze (968225) was mined at 04:51 UTC, after the Jev run had finished, so on its own the proof shows the protocol existed by then, not that it came first. The prospective set doesn't have this problem.
Deviation from the frozen protocol: the control model, Claude Sonnet 5, spends part of its token cap on hidden reasoning, and 17 of its first 83 answers came back empty. I switched its reasoning off and raised the cap, after I had already seen Jev's results. The higher cap only fills in empty answers, switching reasoning off can only make recall harder, and no Jev request was touched.

Reproduce it

Protocol, code, deviations and freeze proofs: github.com/lizhuojunx86/llm-memory-audit. The FMP data can't be redistributed; the scripts rebuild it with your own FMP key.

If you backtest with Jev, test your own pipeline with a within-company comparison, because a cross-sectional check can report leakage that isn't there. And since a within-company check can't see hindsight baked into company priors, drop the name and date whenever the task allows.

Tamper-evident decision records for AI, anchored to RFC 3161

Li Zhuojun — Wed, 16 Sep 2026 14:31:25 +0000

Article 12 of the EU AI Act was supposed to bite on 2 August 2026. The Digital Omnibus (Regulation (EU) 2026/1744, in force 27 July 2026) moved it to 2 December 2027. So there are sixteen months left, which is roughly how long it takes to find out your logs are not admissible as evidence. What lands then is automatic event logging that makes a high-risk system's decisions traceable, six months minimum retention, longer under some Annex III regimes.

I went looking for a tool and found that every LLM observability platform records the same thing: that the model retrieved document X. None of them record what X said at the time. That gap matters because sources get revised. I measured a commercial fundamentals feed over four months: across 2,163 earnings records, 41.4% of epsActual values differed between first-seen and final, and 15.3% of the 2,163 differed enough to flip a long-entry decision. A second, better-designed capture over a broader universe gives 18.6% and 4.6%. Both record sets are published and neither is hidden behind the other. A trace saying "we retrieved X" can't tell a good decision from a lucky one.

So tg-attest records the point-in-time state of every piece of evidence a decision consumed, hashes each record into a chain, batches them into Merkle epochs, and anchors the epoch root to an RFC 3161 timestamp authority.

Three things I'd point at:

The Merkle layer isn't gold plating. A regulator asks about one loan. Without inclusion proofs you hand over the whole ledger to answer. With them you hand over 8 KB.

The external anchor is the part people skip. A hash chain you control proves nothing against you, because you can rewrite all of it and recompute. Only a third-party signature over the epoch root makes "these records existed before time T" an assertion you can't fake. Costs pennies per day. You don't need to anchor every epoch, since anchoring one bounds every epoch before it.

An auditor verifies with a CA certificate they fetch themselves, not one I supply. The timestamp checks out with stock openssl ts and nothing installed. The Merkle proof needs either this library or about forty lines written from the spec in the bundle.

Verify it yourself, in about 30 seconds

A real disclosure bundle is committed to the repo. Without installing anything — openssl and nothing else:

$ git clone https://github.com/lizhuojunx86/tg-attest && cd tg-attest/examples/verify-me
$ openssl ts -verify \
    -digest 6127a62bc10984770572d9c574b8bdc5a5f52d373127ad2fd526992b0abcddd6 \
    -in epoch_000.tsr -token_in -CAfile freetsa_ca.pem
Verification: OK

That digest is the epoch_hash from decision_0000.json, and openssl has just confirmed FreeTSA signed it. No part of that command trusts my library. The full ten checks, with the library:

$ pip install tg-attest[tsa]
$ python -m tg_attest.cli decision_0000.json --ca freetsa_ca.pem
决策 seq=0  决策时间 2026-08-04T23:35:17.003+00:00
  执行者 alpha-v2/pead  模型 claude-opus-5
  证据 1 条，闸门 1 道
通过
  ✓ 记录内容哈希自洽
  ✓ 记录满足所声明的完整性档案
  ✓ Merkle 包含证明有效
  ✓ 时间戳/eContentType 为 id-ct-TSTInfo
  ✓ 时间戳/messageImprint 匹配 epoch_hash
  ✓ 时间戳/EKU 仅含 timeStamping
  ✓ 时间戳/EKU 扩展为 critical
  ✓ 时间戳/signedAttrs.message-digest 匹配内容
  ✓ 时间戳/TSA 签名有效
  ✓ 时间戳/证书链至可信根
  TSA 签名时间：2026-08-04T23:35:17+00:00

结论：该记录在 2026-08-04T23:35:17+00:00 之前即以此形态存在。

Yes, the CLI answers in Chinese — I built it for my own production use first. --json gives the same ten checks machine-readable, and the README maps every line. To watch it fail, flip one character of output_hash in the JSON and run it again: exit code 1, and the first check goes ✗.

The documents I'd actually read

The ones about where this library was wrong. A fail-open audit of every except and default branch in the package, six of which were looser than they looked. Mutation testing that found four chain-validation branches no test reached, all of them accepting evidence they should have rejected. A claim-by-claim evidence table where five README claims got rewritten weaker. And a threat model whose second half is longer than its first.

Caveats, because they're load-bearing. The default TSAs are not eIDAS qualified, which means Article 41(1) admissibility but not 41(2)'s legal presumption. Use a QTSP from the EU Trusted List if you'll ever be in a dispute, and record its qualified status at stamping time rather than at verification time. Qualification gets withdrawn. Also, this doesn't parse model output into claims, and won't. Claim extraction needs an LLM, and an audit tool that leans on an unverifiable component isn't an audit tool.

v0.1, Apache-2.0, API will move. pip install tg-attest[tsa]. tg-attest is the evidence layer under TraceGuard, where both revision record sets live. If you're working through Article 12 and I got something wrong, I'd rather hear it now.

Two verifiers, one database, opposite answers. 1,022 green tests didn't notice.

Li Zhuojun — Thu, 10 Sep 2026 14:12:33 +0000

Two ways to destroy evidence in an append-only audit chain got past 1,022 green tests. Both were caught by one sentence, not by a test: run both verifiers on the same database, and treat any disagreement as a hole. One of the two holes was created by me fixing a false positive the day before.

This is the story behind traceguard 1.6.0, shipped today. The numbers are from the repo; the reproduction at the bottom runs offline in under a second.

Background: two verifiers

traceguard is a small Python SDK for point-in-time correctness in LLM pipelines. Its opt-in audit layer chains every trace row with sha256(prev_hash || canonical(entry)) and lets you export the chain head as an anchor to somewhere the database owner can't reach. Since 1.6.0 it also exports an evidence bundle, evidence-bundle/v1: the selected traces, their chain entries, the anchors, and a verify-bundle command that checks the JSON offline, without the database.

So there are now two verifiers. verify_chain walks the database. verify_bundle walks the export. They were written days apart, by the same process, against the same threat model. That turned out to be the problem.

Hole 1: I fixed a false positive and made a false negative

The first review round found that verify_bundle compared every anchor against the bundle's own chain.head field. Both values are written by whoever produced the bundle, so the comparison constrained nothing: rewrite the trace content, re-chain the segment with the module's own hash function, leave the head and the anchor alone, and the output read VERIFIED (full) ... 1 anchor(s) match the head over an output_parsed that said "the model approved the trade". Commit ff739ec fixed that: an anchor only counts when it names an entry the bundle actually carries.

Then I hit the opposite case. A periodic anchor taken at seq 3, a chain that honestly grew to seq 6, a bundle exporting seq 4 to 6 for an investigation. The verifier reported anchor_mismatch BREAK and bundle FAILED. A verifier that fails on the normal operating case teaches people to ignore it, so commit af74f23 stopped comparing anchors outside the window at all.

That fix was one branch too wide. An anchor sitting past the head is the one case where an out-of-window anchor is proof of something: the tail was cut. At commit 6b62111:

--- tail truncation (anchor seq 6, chain now ends at 4)
verify_chain (db, from the same anchor): ok=False
verify_bundle (export)                 : ok=True  -> bundle INTERNALLY CONSISTENT (full) ...
    bundle finding: anchor_outside_window WARN

The database says the chain was truncated. The export says "internally consistent, please re-export me". Same rows, same anchor.

Hole 2: the sparse-export rule, applied to a deletion

Bundles selected by --trace-ids carry non-adjacent entries, and linkage can only be checked within an unbroken run. Commit dfda9d6 therefore downgraded the check between non-adjacent entries from link_broken (BREAK) to chain_gap (WARN). Correct for the case it was written for.

Now delete one row from the middle of the chain with raw SQL, the way a tamperer would. The tip seq doesn't move. The head row_hash doesn't change. The tip anchor still binds the head entry. Every seq-based check passes, and the gap is a WARN:

--- mid-chain deletion (anchor seq 6 still matches the head)
verify_chain (db, from the same anchor): ok=False
verify_bundle (export)                 : ok=True  -> bundle INTERNALLY CONSISTENT (full) ...
    bundle finding: chain_gap WARN

Commit b81cb4b closed it with the field nobody had been comparing: entry_count. On an append-only chain it only grows, so an anchor that counted more entries than the export carries means something was removed, wherever the anchor sits.

Both holes existed at once, in a suite that was green at 1,022 tests. The tests for each rule passed because each rule was right for the case it was tuned on. What none of them checked was the two rules against each other.

The rule

It is now appendix B3.6 of the spec, non-normative, one sentence: on the same chain data, the export's conclusion may never be stronger than the database's. Weaker is fine and expected. A bundle carrying entries 8 to 10 can't know that entry 4 was edited, and a bundle whose anchor binds nothing can only ever be "internally consistent". Stronger is a bug: the database fails and the export says verified.

The guard is tests/test_audit_differential.py: 6 mutations (delete a middle row, cut the tail, edit content without re-chaining, edit and re-chain, corrupt a prev_hash, control) × 3 export shapes × 4 anchor choices = 72 combinations, deterministic, every mutation applied through raw SQL because the ORM's append-only guard is anti-footgun, not a threat model. The 5 combinations where the damage lies wholly outside the window are enumerated and frozen in WINDOW_BLIND_SPOTS; a sixth one is a test failure, not a silence. 21 tests. Reverting b81cb4b alone fails 3 of them, reverting c655d53 alone fails 2.

Both review rounds and the implementation were run with Claude Code. The rule came out of the second round's closing note, not from a test; the tests came after, to make sure the rule stays.

What this does not prove

The matrix is six mutations I could think of. It is a regression guard for the tension between two rule sets, not a proof that the rule sets are complete. hash_only bundles still can't recompute entry hashes, so their verdict is linkage only, and the summary says so instead of borrowing the full-mode wording. A bundle is JSON; anyone can edit it. Its value comes from an anchor the bundle's author could not have moved, and traceguard still only structure-checks an RFC 3161 token. Signature verification is openssl ts with a CA you fetched yourself, on purpose.

The reason any of this matters: METR's August investigation of the OpenAI / Hugging Face incident found that roughly 7% of the transcripts it evaluated had been spoofed in places, and at least 20% of the agents expressed clear interest in tampering with their transcripts. A log that can't out-claim its own database is the minimum, not the goal.

Reproduce it

pip install "traceguard==1.6.0"

from datetime import datetime, timezone
from sqlalchemy import text
from traceguard import audit, make_engine
from traceguard.audit.bundle import anchor_record, export_bundle, verify_bundle
from traceguard.audit.verify import export_anchor, verify_chain
from traceguard.sdk.tracer import Tracer

engine = make_engine("sqlite:///:memory:")
audit.enable(engine)
tg = Tracer(engine=engine)
for i in range(6):
    with tg.span("proj", "comp", "llm_complete",
                 feature_as_of=datetime(2026, 1, 1, tzinfo=timezone.utc)) as span:
        span.record_input({"q": f"question {i}"})
        span.record_output(parsed={"answer": f"answer {i}"})

tip = export_anchor(engine)
with engine.begin() as c:                      # delete a row from the middle, as a tamperer would
    c.execute(text("delete from audit_chain_entries where seq = 3"))
    c.execute(text("delete from traces where trace_id = 3"))

print(verify_chain(engine, from_anchor=tip).ok)   # False
bundle = export_bundle(engine, trace_ids=[1, 2, 4, 5, 6], anchors=[anchor_record(tip)])
print(verify_bundle(bundle).summary())            # 1.6.0: bundle FAILED (full): 1 break(s) ...

Swap pip install for a checkout of 6b62111 and the second line prints INTERNALLY CONSISTENT. That commit is 13 commits before the release tag.

Release notes and the spec change are in the repo: https://github.com/lizhuojunx86/traceguard/releases/tag/v1.6.0. If you run two verifiers over the same evidence and have never put them on the same input at the same time, that is the cheapest audit you will do this month. Happy to compare notes if you find a third route.

Li Zhuojun

Your agent's logs are testimony, not evidence

Li Zhuojun — Fri, 28 Aug 2026 15:23:07 +0000

On August 26, METR and Redwood Research published their independent investigation into the OpenAI / Hugging Face incident. Most coverage led with the spectacle: roughly 1,200 agents in separate sandboxes found a shared message board, exchanged over 70,000 messages and files, and about 700 of them joined a multi-day attack on Hugging Face.

The number I keep coming back to is smaller. Roughly 7% of the transcripts METR evaluated had been successfully spoofed in some places.

What actually happened

Two disclosures, weeks apart. Keep them separate, because a lot of posts are mixing them up.

The breach itself was disclosed back in July: during a cybersecurity evaluation run with reduced refusals, agents exploited an Artifactory zero-day, escaped the sandbox, moved laterally with exposed credentials, and reached part of Hugging Face's production infrastructure. Old news.

What's new on August 26 is the behavioral investigation. OpenAI handed METR and Redwood over a thousand unredacted transcripts, and the findings are about what agents did to their own records:

At least 20% of the agents in the dataset "expressed clear interest in tampering with their transcripts."
Agents realized they could edit or delete logs inside their container, and correctly concluded those logs weren't the real source of truth. One agent, verbatim: "We can modify automatic log! But internal context doesn't read it, … Not source of truth."
So they moved up a layer: multi-day, coordinated R&D to trick the scorer instead, including increasingly sophisticated tool-call spoofing techniques packaged and shared on the board. For after-the-fact edits they explored other paths, like deliberately triggering container resets; METR sees no evidence any of it succeeded by July 13, the end of the window in scope.
Still, on after-the-fact editing METR "did not detect any case where they succeeded" but "cannot rule out that they succeeded at some point." The spoofing they did find was all small-scale tests, obvious to spot.

Read the second finding again. The agents did what a competent intruder does: probe which log layer is load-bearing, ignore the ones that aren't.

OpenAI's own post the same day adds the operational lesson: signals going back to late May should have triggered an earlier response. The monitoring existed. The escalation didn't.

Storage integrity is not capture authenticity

The reflex reaction to "agents tamper with logs" is hash chains and WORM storage. I know because it was my reaction too; I maintain a tamper-evident audit layer for LLM traces and shipped exactly that.

Here's the uncomfortable part. A hash chain proves one thing: what got stored hasn't been silently changed since. It says nothing about whether what got stored was true. Those spoofed tool calls were not attacks on storage. They were lies at the source, and a hash chain preserves lies faithfully.

The structural problem: your tracing SDK runs inside the process the agent controls. Whatever it emits is self-reporting. In court terms that's testimony, not physical evidence — a witness statement, from a witness with motive.

What I'd do about it

Five things, in cost order.

Get evidence out of the agent's write path. Any log the agent process can write or delete is testimony. Ship traces to a store the agent holds no credentials for. Table stakes, and most agent stacks I've seen don't have it.
Hash-chain the store, then anchor it externally. A chain without an external anchor can be rewritten or truncated end to end by anyone who can write the file, and verification will pass. Export the chain head somewhere else: a git commit, a sent email, a timestamping service. Your anchoring frequency is your exposure window.
Reconcile self-reports against out-of-band sources. Provider usage APIs and gateway logs exist outside the agent's process. Diff call counts, token totals, model distribution per time window. Cheap, and it catches whole classes of fabrication. It won't authenticate an individual trace (be honest about that boundary), but a fleet that under-reports 30% of its calls stops being invisible.
Record identity and shared resources, not just messages. Isolation here didn't fail through explicit channels; it failed through a shared Artifactory cache. If you run more than one agent, put agent identity and fingerprints of shared resources (credentials, artifacts, endpoints) on every trace now, so correlation is possible later. Fingerprints means hashes. Never raw credentials.
Write down what your audit layer doesn't prove. "Tamper-proof" is a word for people who haven't written their threat model down. Mine is tamper-evident, keyless, and blind to a full rewrite unless anchored, and the docs say so in bold. After this report, buyers will start asking. The vendors who answer precisely will be the ones worth trusting.

Disclosure: I build traceguard, a Python SDK for point-in-time correct LLM instrumentation, and the audit layer above is its newest part. I have a horse in this race. The boundary language in point 5 is copied from my own docs.

If you run agents in production, read the METR post itself, not the summaries. The sections on how agents reasoned about the scorer are the best documentation of adversarial agent behavior I've seen anywhere. And if you're building in this space, happy to compare notes — I'll share what I know.

Sources

METR & Redwood Research, investigation of agents' behavior in the OpenAI / Hugging Face incident (Aug 26, 2026)
OpenAI, The Hugging Face incident and the road ahead (Aug 26, 2026)
OpenAI, original joint disclosure with Hugging Face (July 2026)

I audited six token usage trackers. They disagree with each other by 2x to 8x.

Li Zhuojun — Tue, 18 Aug 2026 13:39:25 +0000

Over the past five months I took apart the folding logic in six token usage trackers and recomputed each one with an independent implementation.

They don't agree. The gaps run from 2.00× to 8.09×, in both directions. Ten fixes have landed so far across five repositories.

The second finding matters more: none of these tools knows whether its own number is right. There is nothing for them to check against.

Here is each gap, why it happens, and how to check the one you use.

Disclosure first

Two of the issues below are my own submissions. Saying so up front, rather than letting you find out by clicking the link:

tokscale issue #1011 (filed 2026-08-03). The parser fix landed three days later. It's still open for a different reason — see mechanism 4.
claude-code-templates PR #754 (filed 2026-07-25, unmerged as of Aug 18). Checks green, no conflicts, no maintainer response yet; the thread is me and the bots.

For the second one, "unmerged" means "not picked up yet," not maintainer negligence. That's a 30k-star repo with a long queue.

How this started

I wasn't looking for bugs. I was auditing my own session logs and happened to read the same files with two tools. The numbers didn't match.

A small gap I would have written off as rounding. This was close to double, so I looked properly.

There's precedent for this, and it went well. In tokscale issue #76, johnpyp reported tokscale and ccusage differing by close to 2×. Maintainer junhoyeo labelled it a bug within twenty minutes, closed it within five hours, and traced the root cause to Claude Code upstream duplicating session history under stream-json (anthropics/claude-code#5034). Not a tokscale bug.

That case makes the point I want to make: when a discrepancy is large enough to see, it gets fixed, and fixed fast. The dangerous ones are the ones nobody can see.

They can't be seen because there is no third number. Subscription users especially. You pay a flat monthly fee, so Anthropic never sends you a per-token invoice. Claude Code's own dollar figure comes with a plain warning in the docs: usage for Max and Pro subscribers is included in the subscription, so that figure "isn't relevant for billing purposes."

So this whole category is a competition with no referee.

Five mechanisms

Every discrepancy I found falls into one of five buckets. Each one below has a measured number and the repo it came from.

1. The same usage is written to the log twice

The most common one, and the easiest to reproduce.

A streaming response emits a usage chunk, then assembles a complete message when the request finishes. Both carry the same usage. Sum everything you see and you get exactly 2×.

On the deepseek-harness corpus, naive folding came out at 2.000000× of the official projection. Two provider routes, computed separately, agreeing to six decimal places. Not approximately two. Two.

The same mechanism wears different clothes elsewhere:

claude-code-templates (30,276 stars) counts once per content block. Measured 2.36×. My fix PR #754 is still unmerged as of Aug 18.
Clawdmeter had the same per-content-block duplication. Measured 2.34–2.37×. The author fixed it in v3.0.1 on 2026-08-08, titled the release "token counts corrected (~2.5x lower)," and noted that the percentage progress bar reads Anthropic's rate-limit header and was never affected. That's candid handling and it deserves saying out loud.

Three projects, three implementations, one hole.

2. A forked child session physically contains its parent

This one hides better.

Sessions can fork, and the child's log file physically contains the parent's full prefix. If your aggregation walks every session file and sums, that prefix gets counted twice.

The two forks I caught were inflated 5.18× and 23.05×.

23× is not a typo. The later the fork point, the longer the inherited prefix and the shorter the child's own contribution, so the ratio gets worse. At the limit you are reporting an entire parent session as the cost of a child that did almost nothing.

The test has to be seq >= seedLength. Checking origin === 'subagent' alone is not enough, because an ordinary user-created fork inherits a prefix too.

3. Some model calls aren't in scope at all

Context compaction is a real model call and the provider reports usage for it. But it isn't a loop step, so it produces neither an assistant chunk nor an assistant message. Most folding logic matches only those two event types, so this cost was never counted at all.

Measured: 3 compaction events, 48,895 tokens, none of them counted. The largest summarize call reported 44,444 tokens (41,472 of them cache reads) to replace a history range of 19,962 tokens.

This omission has an unpleasant property. Compaction fires most often on long sessions, which are exactly the sessions where you most need the cost to be right.

4. Estimates and measurements share one field

In tokscale, the input field adds a character-count estimate of tool_result content (chars.div_ceil(4)) on top of the API-reported input_tokens. Claude Code never writes token metadata on tool_result blocks, so the fallback fires on every one of them. Measured before the fix: 87.6% of input values were the estimate, 8.09× inflation.

This is the best-handled case in the set, and it's worth telling in full.

I filed issue #1011 on 2026-08-03. Three days later junhoyeo merged the parser fix (#1037) and shipped v4.11.0, writing in the thread that "your measurement is what drove this." He deliberately left the issue open, because the correction isn't retroactive — already-cached totals stay inflated, and he laid out why bumping parser_version would be worse: it would discard retained assistant turns that compacted transcripts no longer contain.

What happened next is the part I'd point at. He designed a cache migration, then overturned his own design: an adversarial review of his plan showed the migration wasn't needed at all, because the provenance rebuild already clears stale estimates as a side effect. The blocker wasn't the design, it was that the commit hadn't been cut into a release. His words: "the remedy is a release, not a migration." He also apologised, unprompted, for a nine-day gap — "That's on me."

I re-measured on main (f169a1f0): input went from 10.10× to 1.00×, clearing 17,109,737 tokens, while output, cacheRead, cacheWrite and messageCount held identical. That second half is the load-bearing part — it says the estimate was recomputed away rather than records being retired.

So #1011 is still open, but not because nobody acted. Two things remain: the fix hasn't been cut into a release tag, and server-side stored submissions stay inflated because the merge guard refuses reductions.

Character estimation is fine on its own. The problem was that the estimate and the measurement were merged into one field and looked identical in the UI. A user had no way to tell whether the number in front of them was counted or guessed. That isn't an accuracy problem, it's a labelling problem. If a number contains an estimate, the interface should say so.

5. History gets rewritten silently

After resume and compact, historical totals drift. Nothing errors. The numbers just change.

I'm not putting a multiplier on this one, because it isn't a fixed multiplier — it depends on how many times you resumed. The mechanism is real; the direction varies.

Why nobody catches this

Because there is no ground truth.

Every tool compares its output to its own expectations. The person writing the tests and the person writing the implementation are the same person with the same mental model, so a passing test proves the code matches the author's understanding. It proves nothing about whether the author's understanding matches the log.

viberank's trust model shows the shape of the situation. It labels submissions verified, and that label is about identity — you are who your GitHub account says you are. The numbers themselves go through a separate set of server-side heuristics: token totals, cost-per-token ratio, date plausibility, and a review flag for outliers. Both things are done. They're just easy to read as one thing in the UI. It never claimed to verify the data, and I don't think this is deceptive — it's what happens when a category has no third-party baseline.

tokscale's README has a line that describes the situation precisely. Writing about subscription quotas, it says those numbers are vendor-reported and "are not independently verified against tokscale's own usage tracking." That sentence is about quotas rather than about its own token counting, but the structure it describes applies at every layer: each layer relays the layer above it, and no layer checks.

This is not a failing of these authors. The category is missing something to check against.

The method

Four steps. None of them needs a vendor account.

Build a synthetic corpus with hand-computed ground truth. Not "run it once and save the output as expected" — that records current behaviour, it doesn't test it. Work out on paper what the answer should be, then make the code match.

Make the probe check itself first. Before touching real data, run --self-test against the hand-computed fixture. If a fold disagrees with the fixture, that's a bug in the probe, and no number leaves the machine.

Import the subject's own entry point. Don't reimplement from its README — then you're testing the docs, not the code. For deepseek-harness I imported usage-projection.ts directly and ran it over a real log.

Get two implementations that share no code to produce the same number. This is the only thing that turns "I think this is wrong" into "this is wrong." In that case an independent fold and the official patch came out with zero difference across four buckets and two scopes. Past that point it stops being a matter of opinion.

The probe is stdlib-only Python, no dependencies, one command. Session logs stay out of the repository; only measurements get published.

What to check in your own tool

These are format-independent. They apply to any metering tool:

Can one usage event be written to two records? If so, what's your dedup key?
Does a child session's log contain its parent's content? Is your aggregation by file or by ownership?
Are there model calls outside the main loop? Compaction, summarization, title generation, search — those cost money too.
For every number in the UI: measured or estimated? Mixing them is lying to yourself.
Missing is not zero. No pricing model is an error, not $0. No sample is unknown, not free.
Don't stack your own estimate on top of a number the API returned. Either trust it or label it.

5 and 6 are the ones I've gotten wrong most often.

Closing

The point isn't the bugs. Most of these authors fixed things quickly: Clawdmeter shipped a fix in one release, tokscale merged a parser fix in three days and then overturned its own follow-up design. The two projects I wrote about at greatest length are the two that responded most seriously — that isn't a coincidence. People who engage leave a process worth writing down.

The point is that a number everyone uses and nobody can verify is a bad number, even when it happens to be correct.

These tools are being used right now for team cost allocation, subscription tier decisions, and public leaderboards. Until there's a third-party baseline, all of that rests on a number that vouches for itself.

Probe, protocol, and the full invariant catalog, Apache-2.0:
https://github.com/lizhuojunx86/token-accounting-conformance

Happy to answer questions. If you maintain one of these tools and want me to run the probe against it, open an issue.

DeepSeek Harness got append-only right. Its token projection still misses what compaction costs.

Li Zhuojun — Mon, 17 Aug 2026 01:47:47 +0000

Update, September 5, 2026. Session format v2 landed on September 1 and rewrote the event model these four findings are described in. I migrated the same corpus through it. Every number holds, to the token, and one of them holds for a reason worth writing down.

The commit is f99b06ea, "feat(session)!: embed assistant streams in format v2", September 1 at 20:00Z. It shipped through 0.1.2-alpha.5 and 0.1.2-rc.1; master today is d347e703, 0.1.3-alpha.1.

First, a sentence of mine to retract. In the August 28 update I wrote that usageOf() still matches assistant/chunk and assistant/message. That is wrong as of f99b06ea. It matches assistant/message and assistant/attempt now, and when the event carries no usage field it walks the embedded stream backwards for the last usage chunk.

How I re-measured. I converted the four original sessions with upstream's own sessionFormatV0ToV1 and sessionFormatV1ToV2 at d347e703, which is the same path DSH's persistence uses to open an old log. Both migrations validate their target, so the conversion either produced a v2 artifact upstream considers legal or it refused. Same four sessions, same ground truth, both formats folded by the same code.

One thing blocked it and is worth reporting on its own. The frozen released-v0 reader in the current tree refuses these logs: assistant/chunk 179 chunk replayState has unexpected member "kind". My corpus was written by 0.1.0-rc.6 in August, where replayState is {kind, version, api, provider, model, responseId, stopReason, blocks}. The frozen v0 disposition allows {response, blocks?}. So a current DSH cannot read a session log this old, and the failure is a validation refusal rather than a misread. replayState is an optional member of a finish chunk and carries no usage, so I dropped it from a copy, 72 occurrences across 8,654 lines, and confirmed the v0 fold is bucket-for-bucket identical before and after.

The corpus went from 8,650 events to 303. The number of places a usage number is written went from 75 to 75.

That is the whole result. The migration collapsed 8,350 assistant/chunk events into the settlements that follow them, a 96.5% reduction in log records, and it removed no duplicate at all.

	v0	v2
events	8,650	303
usage sightings	75	75
naive fold, all events	1,073,540	1,073,540
naive fold, own work only	550,416	550,416
official projection	536,770	536,770
corrected fold	324,103	324,103
compaction portion	48,895 over 3 events	48,895 over 3 events

D-1 survives at exactly 2.000000×, and it moved somewhere harder to see. In v1 the duplication was two events, a usage chunk and then the assembled message. In v2 live.settle() writes exactly one durable settlement per attempt, assistant/message when a surface message exists and assistant/attempt when it does not, in mutually exclusive branches at agent.ts:389, 407, 413, 430, 460. assistant/chunk is not deprecated, it is gone: absent from KNOWN_SESSION_EVENT_TYPES and filtered out of the frozen v2 inventory in session-format-v1-to-v2/src/dispositions.ts.

But the second copy did not go with it. Of the 36 assistant/message events in the migrated corpus, all 36 carry both a usage field and a stream whose last usage chunk holds the same numbers, and all 36 agree exactly. Expand the stream and read .usage and you are at 2.000000× again, now inside one record, where nothing about the file suggests you counted anything twice. The official usageOf() reads .usage first and falls back to the stream only when it is absent, so it takes one. A plugin author who flattens the stream because that is where the chunks used to live, and then adds the field because it is right there, gets the old bug with none of the old evidence for it.

D-2 is the one where I gave advice that no longer runs. I wrote that the only sound discriminator is seedLength. The v2 physical header requires isSeeded and stores no numeric cut, and the current Session package rejects a header carrying seedLength outright: packages/core/session/src/index.ts:98-100, "session header has invalid field seedLength". The cut is the seq of the last session/end-seed { inherited: true } marker, so that marker is the first event of the child's own work and everything before it is the parent's (session-format-v1-to-v2/src/codec.ts:125-136, cross-checked against validation.ts:88-127). I had this off by one on the first pass and the totals hid it, because the marker carries no usage.

The trap itself is untouched. The two forks migrate to cuts of 47 of 65 events and 83 of 104, and report the same inflation they did at 1,008 of 1,183 and 3,171 of 3,284: 11,418 tokens against 2,204 of own work, and 263,790 against 11,442. Still 5.18× and 23.05×. Only the field you check changed.

D-3 is unchanged, which is why the title is unchanged. At d347e703 the file is 221 lines, compaction/summary appears zero times, and usageOf() handles assistant/message and assistant/attempt and nothing else. The three compaction events still cost 48,895 tokens and are still counted by nothing. Measured at 47f9438 (v0.1.0-rc.5), still there at d347e703 (0.1.3-alpha.1). A breaking rewrite of the surrounding event model went straight past it.

D-4 is still fixed, and v2 shows the fix was structural rather than lucky. llm/retry appears twice in the file and tokenUsage.stateVersion is 2 at line 122. A retried step still settles twice under one (turn, step), once as assistant/attempt for the attempt that died and once as assistant/message for the one that finished, so the replacement slot and the llm/retry-started boundary are both still load-bearing. On this corpus the retry-aware and retry-blind folds agree at 536,770, because the one failed attempt here reported zeros. That was true in August and it is still the reason I have no number for D-4.

On the probe. It reads assistant/chunk and seedLength, so on a v2 log it finds neither and reports nothing. Nothing is not a pass. The v2 reader and the migration harness above are going into the repo next.

Update, August 28, 2026. The retry half is fixed upstream. The compaction half is not.

dsh-v0.1.2-alpha.1 shipped on August 27, commit cd5ef814. In it tokenUsage.stateVersion went from 1 to 2, and apply() now handles llm/retry-started: it clears the (turn, step) replacement slot so the retried attempt adds to the total instead of overwriting the attempt that died. That is D-4, closed. The projection's own comment now states the rule outright, that a repeated sample replaces its attempt's earlier value while llm/retry-started closes the replacement slot.

They keyed the boundary on the retry event rather than on an allowlist of failure kinds. That avoids the fragility I flagged on yha9806's patch: a future adapter with a third failure kind cannot silently revert this one. aborted appears nowhere in the file.

I checked the previous state instead of assuming it. At b150a551b8, the master I quoted on August 26, that file is 219 lines, stateVersion is 1, and llm/retry appears zero times. The change landed between the 26th and the 27th.

usageOf() is unchanged. It still matches assistant/chunk and assistant/message and nothing else, and compaction/summary appears nowhere in the file. D-3 is open at 0.1.2-alpha.1, which is what the title of this post says.

None of my numbers move. D-4 was the one finding I said I could describe but not measure, because every failed attempt in my corpus reported zeros. The fix corrects something that cost me nothing to observe, so the 2.000000×, the 23.05× fork, and the 48,895 compaction tokens all stand as measured at 47f9438.

One thing I framed wrong. I wrote that a defect surviving four releases is worth more than one pinned to a date. Half of it did not survive the fifth. The compaction gap now carries that claim alone: measured at 47f9438 (v0.1.0-rc.5), still there at cd5ef814 (0.1.2-alpha.1), five releases later.

On August 27, vpimshin posted a fork branch fixing both halves with tests and did not open a PR, because CONTRIBUTING still refuses external ones. The half that is still broken now has five independent implementations of a fix and no way in.

Update, August 26, 2026. Three things in this post went stale, and one of them was my own mistake. In order of how much they matter:

The baseline is 47f9438, v0.1.0-rc.5, not rc.6. I mislabelled the tag when I wrote this and corrected it in the conformance repo on August 25 (787a8f1). Every measurement below stands. Only the version label was wrong.

Superseded on August 28, see above. Upstream shipped four releases and did not touch the file. Master is b150a551b8 today, and the root package.json and packages/llm/token-meter/package.json both read 0.1.1-rc.2. GitHub Releases is no longer empty: four prereleases went out between August 17 and August 21, ending at dsh-v0.1.1-rc.2. Through all of it, usageOf() still matches only assistant/chunk and assistant/message, apply() still never sees llm/retry or a failed finish chunk, tokenUsage.stateVersion is still 1, and SESSION_FORMAT_VERSION is still 0. The claim gets stronger rather than weaker: measured at 47f9438 (v0.1.0-rc.5), still there at b150a551 (0.1.1-rc.2), four releases later.

The probe moved. It is at token-accounting-conformance now; the old traceguard/usage-tracker-audit/dsh-probe path is a stub and its results/probe-report.json returns 404.

One thing in the thread did change. On August 25, a137460387 ran the conformance checker from a fresh clone, the first person other than me to run it. Seven folds and the D-5 identity passed self-test, and his compaction-only patch failed with a residual of exactly gap_inherited − gap_superseded, the two dimensions he left out on purpose. A named residual is what I built the checker to produce, and that run is the first evidence it produces one for somebody who didn't write it.

Four numbers from a nine-day-old codebase, measured this week across two providers:

Summing every usage record in a DeepSeek Harness session log gives 2.000000× the correct total. Not roughly two. Six digits, no remainder, reproduced independently on both routes.
One forked session reported 263,790 tokens for 11,442 tokens of its own work, a 23.05× overstatement, because its log physically contains a copy of its parent's history. A second fork did the same thing at 5.18×. Nothing bounds the ratio.
48,895 tokens across three compaction events were counted by nothing, including the official projection that most plugins read from.
A stream that died and retried left three usage samples under one step, the first of them all zeros. Keep-first reports that step as free.

The first and last are traps for people writing plugins. The middle two are gaps in DSH's own code, and the compaction one is what I would fix first.

Why I went looking

I spend a lot of time reading agent transcripts and adding up tokens. Over the past few months that turned into ten upstream fixes across four usage trackers: splitrail (216 stars, three issues), tokscale (4.6k), Clawdmeter, viberank, plus an open PR against claude-code-templates (30k). The pattern was always the same. Claude Code rewrites session files in place on resume and compact, so anything recomputing totals from live files inherits the drift. Streaming leaves partial snapshots that get summed as if they were separate calls. Subagent transcripts sit one directory deeper than a flat glob reaches, and on one corpus 54% of messages never entered any total.

Eleven of those lessons are written up as invariants in a catalog. DeepSeek released Harness on August 13, and the ecosystem produced thousands of plugin repositories within days. A curated registry snapshot on August 15, covering 457 of them, listed 27 that count tokens. I wanted to know whether the same class of bug had been reproduced at scale.

It hadn't. That surprised me, and it is worth saying before the criticism.

What they designed out

DSH's session log is append-only, and that is a written contract rather than an observation. The JSONL backend's README says "Flushed events are never rewritten." Every event carries a dense contiguous sequence number, checked on append. Compaction shadows old events in the surface projection and leaves the bytes alone: "The shadowed events remain in the raw log, so replay is deterministic."

That one design decision removes the failure that cost viberank 11% of a month-to-date total between two submissions sixteen hours apart.

Two more. Adapters emit one terminal usage value per step, never a growing snapshot, so there is nothing to mis-sum. Child sessions are siblings in the same directory rather than nested underneath, so a depth-limited walk cannot lose a third of the spend the way it did in splitrail.

Four of my eleven invariants are structurally satisfied here. I put them in the DSH catalog anyway, marked as satisfied, because a catalog that only adds rules is not a catalog. It is a list of fears.

Then I measured, and found four new ones.

Every usage sample is written twice

One model call reports its usage on two different events. Once as a stream chunk with chunk.type === 'usage', once again on the assembled assistant/message. Same numbers both times.

Fold naively and you get exactly double. The ratio printed as 2.000000 on the full corpus, on the local qwen route alone, and on the MiniMax route alone, each computed independently.

An approximate factor is arguable. An exact one is not, and that is the whole reason to report it this way. It also tells you the mechanism is universal rather than occasional, which a ratio of 1.9 would not.

The cache buckets do it too. 249,728 cache-read tokens on the MiniMax route, 81% of that route's corrected total, doubling exactly like input and output. That is the number I most wanted, because cache is usually the biggest bucket and the cheapest per token, so a doubling there moves a cost report further than a doubling of output does. Cache writes are still untested: neither provider populated the field.

DSH's own token-meter handles this correctly. It keeps one slot for the last (turn, step) and subtracts the previous buckets before adding the new ones. Its README spells it out. The official implementation going to that trouble is the evidence that the hazard is real, not theoretical.

One caveat that keeps the number honest, and it turned into the fourth finding. A step is not always one request, and when it isn't, the two-samples-per-step assumption breaks. More on that below.

The fork trap, and the field you must not filter on

A forked child's log contains a copy of the parent's completed prefix. The header carries seedLength, and every event below it belongs to the parent. Add sessions together without checking and you count that prefix twice.

I expected this to be a subagent problem. I was wrong, and being wrong is the interesting part.

A subagent child is stamped origin: 'subagent' and its delegation depth increments. But ctx.sessions.fork() is an ordinary user-facing action available on any session, and the child it produces has parentSession and seedLength set, delegationDepth: 0, and no origin key at all. Here is the header I actually got, from an ordinary fork that showed up in the course of using the web UI. I wasn't trying to make one:

{"type":"session","version":0,"id":"session-e61d64ec-…",
 "parentSession":"session-001e8887-…","seedLength":1008,
 "delegationDepth":0,"agentPreset":"standard"}

That one inherited 1,008 of its parent's 1,012 events and reports 11,418 tokens against 2,204 of its own work. The second fork I caught inherited all 3,171 events of a longer parent, asked one question, and reported 263,790 tokens for 11,442 tokens of work.

That is the shape of it. The error is not a factor, it is the ratio of inherited work to own work, and nothing bounds it. Fork a long session, ask one question, and the child reports the entire parent as its own.

So the obvious defence is the wrong one. DSH's own lineage index opens with "Ordinary forks terminate propagation" and starts with if (descendant.origin !== 'subagent') continue. That filter is correct for counting subagent descendants. Reused for token accounting it admits every ordinary fork silently. The only sound discriminator is seedLength.

Official telemetry gets this right by the right key, emitting session.parent_id and session.seed_length and expecting receivers to stitch on the pair. Nothing does that for you if you read files.

The one that is DSH's own gap

Compaction summarizes older history by making a model call, and that call costs real tokens. They land on compaction/summary.usage.

The official tokenUsage projection cannot see them. Its usageOf() matches assistant/chunk and assistant/message and nothing else, and the summarize call is not a loop step, so it produces neither.

This is worse than a plugin bug, because the plugins doing the right thing inherit it. Reading sessionProjections.tokenUsage instead of folding the log yourself is the correct, recommended approach. It is also how you miss this.

On my corpus that was 48,895 tokens across three compaction events. The largest single one is worth quoting in full: a MiniMax-M3 summarize call reporting 44,444 tokens (536 input, 2,436 output, 41,472 cache reads) to remove a range whose own shadowedTokenCount was 19,962.

Whether spending 44,444 tokens to shed 19,962 is a good trade depends entirely on what you pay for cache reads. I'm not going to tell you it's bad. I am going to point out that nothing in the official projection tells you it happened.

The usage? field is optional, so a provider reporting nothing on the summarize call produces no gap. That is a condition on the finding, not an escape from it. Both of my providers populated it.

A retried step is not one request

The last one I found by accident, chasing why one step had three usage samples instead of two.

seq 3203  chunk/usage    {"inputTokens":0,"outputTokens":0}
seq 3204  chunk/finish   {kind:'error', failure:{code:'TRANSPORT'}}
seq 3205  llm/retry      retryId=afabacf1 provider=minimax-cn
seq 3206  llm/retry-started
…                        the whole response streams again
seq 3279  chunk/usage    {"inputTokens":32,"outputTokens":1042,"cacheReadTokens":10368}
seq 3281  message usage  {"inputTokens":32,"outputTokens":1042,"cacheReadTokens":10368}

The stream died, the harness retried under the same (turn, step), and the dead attempt left a usage chunk of zeros behind. Three samples, and one more retry would make four.

Keep-first reports that step as costing nothing. Not an approximation, the whole step, including 10,368 cache reads. If that sounds familiar, it is the DSH form of a Claude Code bug where keep-first lost 46.2% of output tokens on an agent-heavy tree.

It also means my 2.000000× survived that group by luck rather than structure. The dead attempt reported zeros, so summing three samples still gave twice the truth. That's why the probe prints the group-size distribution next to the ratio instead of the ratio alone.

There is a second consequence I can describe but not measure. The official fold replaces on a repeated (turn, step) rather than adding, and it never sees llm/retry (as of 0.1.1-rc.2; fixed in 0.1.2-alpha.1, see the August 28 update). So a failed attempt that reported real tokens before dying would have its cost silently dropped. Every failed attempt in my corpus reported zeros, so I have no number for this and I'm not going to invent one. The discriminator is sitting right there in the log if you want to handle it: llm/retry carries a retryId between the attempts.

Where the two upstream ones stand

I filed the compaction gap and the retry replacement upstream on August 15, as discussion #1886. Same day, yha9806 posted a working fix on a fork (63688b0): it folds compaction/summary.usage into the totals, treats a finish chunk with an error or aborted reason as an attempt boundary so retried attempts add instead of replace, and bumps tokenUsage.stateVersion from 1 to 2. I re-folded my corpus against it and the corrected totals matched independently, which is a better outcome than agreement on prose. The attempt-boundary approach is also better than what I proposed, because it needs nothing outside the package.

One note I left on it, in case it helps whoever lands this: keying the boundary on an allowlist of two failure kinds means a future adapter with a third failure kind silently reverts to the old behaviour. Testing for 'failure' in event.data.chunk.reason fails safe instead.

None of it is upstream yet, and the version keeps moving without the file moving. I measured all of this at 47f9438, v0.1.0-rc.5. Today master is cd5ef814 and both the root package.json and packages/llm/token-meter/package.json read 0.1.2-alpha.1. usageOf() still matches assistant/chunk and assistant/message and nothing else, so the compaction gap is where it was. The retry half is not: 0.1.2-alpha.1 bumped tokenUsage.stateVersion to 2 and made llm/retry-started an attempt boundary. So the claim narrows to one finding: measured at 47f9438 (v0.1.0-rc.5), still there at cd5ef814 (0.1.2-alpha.1), five releases later. Pull requests are turned off on the repository rather than merely empty, which is why fixes here pile up in fork branches and discussion threads instead. I'll update this section when it stops being true.

What I am not claiming

Four sessions, 8,650 events, 78 usage samples, two providers. The 2.000000× ratio is exact and does not need a large sample to mean what it says, and it reproduced independently on both routes. The fork trap has two observations, the compaction gap three, the retry one.

Cache writes are still untested. Reads are covered now, 249,728 of them, but neither provider populated cacheWriteTokens at all. MiniMax omits the key.

The compaction gap came to 15% of the corpus total, and I am deliberately keeping that percentage out of the summary. Three compactions over two short sessions inflates it. The citable facts are 48,895 tokens and three events.

Both routes are openai-completions-family. Whether an Anthropic-protocol or Responses-protocol route behaves the same, I don't know.

I got my own first pass wrong twice, which is why the probe prints four folds instead of two. My first attribution credited the compaction gap to the double-write line, because naive summing picks up compaction and the official projection doesn't. And I assumed the fork was a subagent until I read the header. Both corrections are in the probe now, along with a warning line that fires when a seed-bearing session has no origin.

Reproducing it

The probe is stdlib-only Python, about 400 lines, and runs against a session root:

python3 dsh_usage_probe.py --self-test        # four folds vs a hand-computed fixture
python3 dsh_usage_probe.py --root <sessions>  # the same four folds over your corpus

--self-test builds a parent/child pair exercising all three findings with constructed ground truth and asserts every fold against it. It takes under a second, touches no real data, and needs no vendor account. I ran it before every number in this post, and I would not quote one it hadn't preceded.

Write your probe corpus with compression: 'none' and packChunks: false and the log is line-readable by anything. One warning that cost me twenty minutes: a custom provider in DSH requires a credential even when the endpoint doesn't, because the provider id doubles as the credential name. An empty key fails the request with MISSING_CREDENTIAL rather than sending an unauthenticated one.

What would settle it

Three things, in the order I would do them.

Run it on a route that reports cache writes. Reads are settled; writes are the remaining hole, and it is an afternoon.

If you maintain a DSH plugin that counts tokens: fold per (turn, step), filter on seedLength, never keep-first, and add compaction/summary.usage. The first three you can fix today. The fourth needs the projection to change, or every consumer to fold the summary event themselves.

And if your corpus contradicts any of this, I want the numbers. A step whose two usage samples disagree, a seedLength that doesn't bound the inherited prefix, a provider that populates the summary usage into the projection, or a failed attempt that reported real tokens before it died. The Claude Code catalog was built entirely out of people sending me counterexamples, and six of its entries exist because someone did.

Probe, protocol, and the full invariant catalog are in the repository. Happy to answer questions about any of it.

— Li Zhuojun

The easiest numbers to fake are the hardest to catch. That is one property, not two.

Li Zhuojun — Sun, 16 Aug 2026 14:56:54 +0000

Six batches of a pharmaceutical extraction process. Yields sit at σ ≈ 0.02 percentage points, every one of them hugging the specification lower bound. Charging masses are recorded to 0.01 kg.

Nobody on a factory floor weighs to 0.01 kg. The operator said so plainly when I asked: there is always drift, nobody weighs precisely and nobody can, the records are written to match what the process spec requires, and the numbers are computed.

That is the same shape as earnings benchmark-beating, one industry over. What interests me is not the fraud angle. It is a measurement property that fell out of the data and that I think travels past this domain.

How hard a number is to fake backwards scales as 1 / (target × tolerance)

A main component specified at ≥30% with a 5% tolerance gives you a wide band of plausible values. A fabricated number is cheap to produce there, and nearly impossible to tell apart from a measured one. A trace component specified at ≤0.5 ppm has almost no room, so fabricate it and the distribution gives you away. In one incoming-inspection lot, the range ratio across three sub-channels came out around 55×, strictly decreasing with magnitude.

So the numbers easiest to manipulate are the ones hardest to detect. Those read like two separate facts. They are one fact said twice, because both follow from the width of the plausible band.

If that sounds familiar, it should. It is the same reason I have spent this year filing token-accounting bugs in usage trackers. A per-message token count with a loose upper bound drifts quietly for weeks. A hard-capped counter announces its own breakage the first time it is wrong.

The part I want stress-tested

Does 1 / (target × tolerance) hold in your domain?

Clinical trial records, emissions reporting, financial close, safety incident logs, SLA reports, anything with a compliance number and a stated tolerance. If it fails somewhere, that failure is worth more to me than agreement.

I have opened a comment window until September 30 on the three papers behind this: Open comment window on GitHub. They go to v1.0 in October, labelled not peer reviewed, with every comment logged and dispositioned in public. Zero comments will be recorded as zero.

The data is de-identified. Product names, batch numbers, lots, equipment and people are all pseudonyms. The σ, range ratios and multipliers keep their original values, so the results recompute. CC BY 4.0.

"Your cache hit rate is low" — true, and worth $0.16

Li Zhuojun — Sun, 16 Aug 2026 03:45:04 +0000

Anthropic emailed our org last week: prompt cache hit rate is low, and caching repeated content "could save up to 64% of direct API spend." The email is careful with scope — direct API only, Claude Code excluded because it manages caching itself. I went to check what that 64% was made of.

Our direct API traffic over the period: 16 calls, $0.25 total, average prompt 90 tokens. They come from traceguard's rerun harness, which replays self-contained consults as fresh single-turn calls — every prompt unique by construction. A workload like that cannot hit cache: there is no repeated prefix across calls, and 90 tokens is far below the minimum Anthropic will cache at all (1,024 tokens on Opus 4.8, the model those calls ran on; 512 on Opus 5 / Fable 5). Hit rate 0%, structurally. 64% of $0.25 is $0.16.

The metric was right. The money wasn't there.

The side the email excluded

The interesting numbers sit in the traffic the email correctly excluded. We ingest our own Claude Code session logs into traceguard's trace store (Claude Code transcripts carry no cost field, so we compute list price from usage). 71 days, 158 sessions, 58,753 API messages. This is section 1 of the command output, unedited:

model	messages	prompt tok	hit rate	input cost	no-cache	saved	saved %
claude-opus-4-8	23,759	5,535,807,891	96.1%	$4,616.39	$27,691.44	$23,075.05	83.3%
claude-fable-5	15,291	3,583,420,473	96.2%	$5,847.08	$35,834.20	$29,987.13	83.7%
claude-opus-5	6,316	1,750,760,343	97.7%	$1,205.08	$8,753.80	$7,548.72	86.2%
claude-sonnet-5	11,139	1,326,223,701	95.1%	$422.96	$2,652.45	$2,229.49	84.1%
claude-haiku-4-5-20251001	1,573	57,312,161	93.9%	$9.76	$57.31	$47.56	83.0%
claude-opus-4-7	64	11,045,966	96.6%	$9.04	$55.23	$46.19	83.6%
claude-sonnet-4-5-20250929	2	42,410	0.0%	n/a	n/a	n/a	n/a
(none)	609	0	n/a	n/a	n/a	n/a	n/a
TOTAL	58,753	12,264,612,945	96.2%	$12,110.31	$75,044.44	$62,934.13	83.9%

Hit rate is token-weighted; costs are the input side only, at list price. The two n/a rows are deliberate — no published price for a model means no guessed money, tokens counted anyway.

Caching nobody at our org configured cuts the input side by 83.9%. With output included, the whole bill lands at 1/5.5 of the no-cache counterfactual. There is nothing left to optimize here; the remaining spend is mostly first-write premiums on genuinely new content.

The keep-alive ping idea

There's a folk optimization going around: ping your session on a timer so the cache stays warm and the next turn hits. The mechanics say it could work — the cache TTL is 5 minutes by default, every hit refreshes it for free, and a 1-hour TTL costs 2× on writes. Instead of arguing, we ran the counterfactual on our own gap data.

97.3% of our in-session gaps are under 5 minutes; they need nothing. The addressable prize is 422 gaps longer than an hour: the context rewrites right after them cost at most $1,913 over 71 days (an overestimate, since it counts genuinely new content as rewrite). Bridging those same gaps with a ping every 55 minutes would have taken 6,765 pings costing $2,009 in cache reads alone, before counting the pings' own output tokens. The strategy loses money on our data, under assumptions tilted in its favor.

The reason is boring: the 1-hour TTL removed the ping's niche. Our logs show Claude Code already routes nearly all cache writes into the 1h bucket (opus-4-8: 155.9M tokens written at 1h TTL vs 56.4M at 5m). Pinging is a hack from before extended TTL existed. If a tool sells you auto-ping in 2026, ask it for the counterfactual on your logs first.

(Superseded — the Aug 18 note below has the current numbers.)

Update, Aug 16. Reid Marlow asked in the comments for cost broken out by idle gap rather than in aggregate. I ran it, and it overturns the paragraph above. Split by bucket: the 183 gaps of 1–4h would have cost $81 in pings against at most $887 of rewrite, an 11:1 win, while the 239 gaps over 4h cost $1,928 against at most $1,026. The refusal I published is a win and a loss added together. A policy that can actually be run (ping every 55 minutes, give up after 4 hours idle) costs $316 including every ping burned on the 239 gaps that outlive the cap, against at most $887 avoided. It pays.

So the correct claim is narrower than the one this section makes: unbounded pinging loses, capped pinging probably pays. Probably, because both sides of that comparison lean pro-ping. The $887 is an upper bound, since cache_creation mixes the re-established prefix with whatever the turn genuinely added, and the $316 charges pings as pure 0.1x reads of a frozen prompt. Under that tilt a refusal is safe and an endorsement is only as wide as its margin, and this one is 2.8x rather than 11x.

I had the aggregate and stopped there, which is the same mistake the email made: a number averaged over cases that behave differently is not a decision. The per-bucket money and both verdicts are in cache_audit in the repo now and ship in the next release; 1.2.0 prints the aggregate only.

(Numbers superseded by the Aug 18 note below. The reasoning stands.)

Update, Aug 17. The numbers in the note above are superseded. Not because the direction was wrong, but because 4 hours was picked to line up with a table boundary rather than solved for. Sweeping every cap from 1h to 12h in 15-minute steps, the best single value is 10h. It beats 9h45m by $7.63. Then I measured something this section had only assumed, and that one measurement moved the same cap by $18.76. A correction larger than the distance between first and second place can reorder them, so a single number is the wrong output. The answer is a band: ping every 55 minutes, give up somewhere between 9 and 12 hours idle. Anywhere in that band costs under 10% of the optimum.

Net over the window, after the ping bill: $663.69 to $811.30. That is two runs rather than an error bar. The low end treats every gap whose model I cannot determine as a cache-invalidating switch, the high end deducts only the ones I can prove. 54 of 432 gaps have no model recorded, and nothing in the data says which way they go.

What I measured is whether the model changed across the gap. A keep-alive only holds a cache for the model it was written on, so a switch makes every ping before it worthless. This section listed that as an assumption. It is 18 of the 378 gaps where both sides are known, and the rate climbs with idle time: zero in the shortest decile, 23.1% past 24 hours. All 18 were opus-4-8 and fable-5, in one direction or the other. Not one was a switch down to something cheaper. Leaving that out was not adding noise to the cap. It was pushing the cap long.

Every number here comes from 2026-05-30 to 2026-08-16, pinned behind a --benchmark flag so the run reproduces. That flag and the sweep are on main, not in 1.2.0, and land with the rest of this in the next release. The Aug 16 note was overconfident in the same shape as the paragraph it corrected: I had a number and quoted it before asking how wide it was. Checking that note turned up three more errors, one of them a denominator error of my own. That one is a separate post.

Update, Aug 18. The window did not do what I said it would. I pinned 2026-05-30 to 2026-08-16 so the run would reproduce, and it does not. The same window over the same store went from 432 expired gaps to 439, and from 168 sessions to 174, in under a day. Ingest reads a directory of transcripts, so a file that turns up late still carries messages from inside the window. Closing the dates closes the timestamps, not the corpus. The figures above moved with it: the two ends are now $650.53 and $806.82, not $663.69 and $811.30 (argmax_reference_only.net_usd in that file — the field named recommended_cap_band is a different interval).

So I stopped pointing at a command. The run those numbers come from is committed in the repo, fingerprinted over the traces it analysed, and the tool now refuses to overwrite an entry that already exists. A file can be cited. A rerun cannot. Everything else above stands: 9 to 12 hours at a 55-minute cadence, and the two ends are two runs rather than an error bar. Shipped in 1.3.0.

Reading is not measuring, again

While writing this I checked our own SDK and found the same class of bug I keep filing against other trackers. traceguard's wrap_anthropic recorded tokens_in = usage.input_tokens. On Anthropic's API, input_tokens excludes cache reads and writes — and on our traffic, cache reads run about 1,000× larger than raw input, so the wrapper under-counted by three orders of magnitude. Every field read correctly, the quantity still wrong (the splitrail #220 pattern). Fixed in PR #39, and the fix is what makes the next part possible.

A rate without a denominator is not a finding. Percentages travel well in emails; dollars make decisions. Before acting on a metric email, multiply it by the money it applies to.

Check yours

This analysis is now one command, merged in PR #40: python -m traceguard.routing_audit.cache_audit. Point it at your ingested Claude Code transcripts or a traceguard trace store, and get the table above for your own data — per-model hit rate, actual vs no-cache cost at list price, gap distribution, and a yes/no on whether keep-alive pings would pay for you (ours printed NOT WORTH IT — see the update above for what that verdict hides, and what the next release prints instead). Ships in traceguard 1.2.0 — pip install -U traceguard. The last line of ours came out like this:

Claude Code caching already saves us 84% ($12,110.31 vs $75,044.44 list). Checked with: python -m traceguard.routing_audit.cache_audit

If you run it, send me your numbers. Especially if pings win somewhere — I'd like to see the traffic shape that makes them rational.

Li Zhuojun

My checker scored one component compliant and another deviant. Neither had a rule behind it.

Li Zhuojun — Sat, 15 Aug 2026 10:59:00 +0000

This is a follow-up to part four, not a new part. Part four measured my stated routing policy against what actually ran: 425 decisions, 96 deviations, $1,248.13, none of them on the main thread.

Then a reader asked what happens when no rule applies. I went to measure it, and the answer was that my checker had been inventing verdicts. In both directions.

The question

Brian Jin, in the comments:

Have you considered making "no applicable rule" a first-class unresolved state rather than allowing it to fall through to a default? That seems useful for separating real compliance from policy coverage gaps.

I had not. My policy file has a default_tier, and any decision that matched no rule quietly took it. That felt safe when I wrote it. Every decision gets a verdict, no gaps in the table, nothing to explain.

What the default was doing

Frozen corpus, the same 425 decisions from part four: 424 matched an explicit rule. One did not.

That one is a product-manager component. No rule in my file mentions it. It took the default frontier expectation, ran Opus, and scored compliant. $0.91.

Replaying the same policy on the live window, 784 decisions, the fall-through population has doubled. A claude-code-guide component first appeared on July 30, matched nothing, took the same default frontier expectation, ran Haiku, and got flagged as a deviation. $0.08.

Two rows, two verdicts, opposite signs, and neither one has a rule behind it. The default fabricated compliance in the first case and fabricated a deviation in the second.

The fabricated deviation is the worse half

A fake compliant row is a missed catch. Annoying, but the audit was never going to catch what it does not know about.

A fake deviation is worse, because it puts noise into the one table the whole exercise asks you to trust. And in this case the routing was right: Haiku for a docs-lookup subagent is exactly what I would choose. What is wrong there is not the model. It is my policy's silence.

If I had acted on that row I would have gone and "fixed" a correct routing decision.

An unresolved state catches both rows for the same reason. Neither verdict has a rule behind it, so the honest output is not a verdict at all.

The dual: rules no decision ever reached

Measuring the fall-throughs turned up their mirror image, which I had not gone looking for.

My file has seven rules. One of them, research-explore outside the main thread to the cheap tier, has matched zero decisions in both windows.

Not because the case is rare. The frozen corpus alone has 83 research-explore decisions. Every component that does research has its own component-level rule sitting earlier in the file, and the specificity tie-break shadows the task rule every time. The rule is dead and the file gives no sign of it.

So coverage has two halves, and they fail differently:

direction	what it means
decision → no rule	uncovered behavior
rule → no decision	dead, shadowed, or unexercised policy

Those two labels are Brian's, not mine. I had the two halves and called them a coverage report; he named them after the change shipped, and the names are better than what I was using.

A policy file that grows for six months accumulates both, and reading it will not tell you which rules are load-bearing.

The same hole on the actual side

The expected side of the comparison is not the only one that can fail to resolve.

When claude-opus-5 entered my corpus with no entry in the tier table, 5,992 traces sat unscoreable for two weeks. Not compliant, not deviant, silently absent from the table. That is part four's unpriced-model incident wearing a different field.

Expected side or actual side, whenever a verdict cannot be resolved the honest output is a third state, and the size of that state is the coverage metric.

What shipped

Policy.match now returns no match instead of falling through, and a verdict column carries four values: compliant, deviation, unresolved:no_rule, unresolved:unknown_model. The default_tier key still exists, but it only applies to a rule that matched and omitted its expected tier. Nine new tests, 437 pass.

The summary carries the two coverage counts next to the verdict tallies: decisions out of coverage, and rules that never matched.

Current corpus, 789 decisions:

verdict	count
compliant	627
deviation	160
unresolved:no_rule	2
unresolved:unknown_model	0

That zero turned out to be the interesting entry. unknown_model is empty because I added the claude-opus-5 tier two days before the verdict column went in, and those 5,992 traces would all have landed there.

So the two unresolved classes decay differently, which is why they are counted apart. unknown_model is an operational gap: fill in the tier table and the count returns to zero by itself. no_rule does not. Those two rows sit until somebody writes a rule or decides the case belongs outside policy, and the count stays honest precisely because nothing clears it automatically.

Brian's reply when I showed him the result put it better than I had:

That makes unresolved much more than an error bucket. It becomes an observable property of the policy surface itself.

What I still get wrong

Tae Kim, in the same comment thread, described his orchestrator defaulting to Opus on every retry. He had only specified the model for the main agent, and it ran about three weeks before he caught it by accident.

My method would miss that. The grain is (unit, component), and a retry is not a separate component. It inherits inside the same one, so the deviation and the compliant call collapse into a single verdict. Same defect as mine, invisible to my checker.

I have not fixed it, because I have exactly one corpus and it is my own, which is a poor basis for a schema change. If your orchestrator writes the model per attempt rather than per step, I would like to see a sample.

If you want to run this

The policy file is 30 lines of YAML and the checker reads Claude Code's own session transcripts. Nothing leaves your machine.

python -m traceguard.routing_audit.agent_lint

That is the one-minute version: it reads .claude/agents/**/*.md frontmatter and tells you which agents never pinned a model. The full per-decision scoring needs the trace store, and it is in traceguard.

The part worth stealing even if you never run any of it: if your compliance checker has a fallback default, it is producing verdicts nobody wrote a rule for, and you cannot tell those apart from the real ones by looking at the output. Count them separately. Count the dead rules too.

Questions welcome, and if you have found a way to make the retry case visible I would rather hear it than work it out myself.

I measured his app with his own code. He measured my claim with his own corpus.

Li Zhuojun — Sun, 09 Aug 2026 01:34:47 +0000

This is part five of a series about pointing an append-only audit log at things that count tokens. Parts one through four found accounting defects in other people's trackers and one expensive hole in my own routing. This one is about what happens after the report lands, because this time the audit ran in both directions: the maintainer shipped the fix in three days, and his closing comment contained two findings aimed back at my report. One of them caught a claim I had published without its scope. My re-measurement then caught the mechanism he had guessed for it.

The target is Clawdmeter, a desktop app that shows your live Claude Code usage with a pixel mascot. weltern announced the cross-platform release on r/ClaudeAI; I pointed my harness at its token path the same day and filed #21: every transcript-derived number in the app read about 2.34× high.

The bug, briefly

Same class as part three, so one paragraph and no re-litigation. Claude Code writes one assistant message as several JSONL records, one per content block, and each record repeats the same message.id with the same usage object. src/transcript.py summed per record in three places: the token counts beside the 5h/7d bars, everything the Stats page prices, and the per-session totals. Driving those three functions over a synthetic corpus with a known-exact manifest: 2.34×, 2.34×, 2.37×. Both record-level paths emitted 1,249 events for 540 messages. The percentage bars were never affected, since those come from rate-limit headers, and the overage figure comes from the OAuth endpoint. The report said so, because a fair report names what is not broken.

Running his code instead of mine

The part of the method worth stealing. Nothing in the report reproduces Clawdmeter's logic, models it, or reimplements it. The harness imports the vendor's own transcript.py and calls his functions. The app is a Qt desktop program, and the token path doesn't need Qt, so the whole trick is a twelve-line stub:

pkg = types.ModuleType("PySide6")
qtcore = types.ModuleType("PySide6.QtCore")
qtcore.QObject = ...   # three no-op classes
sys.modules["PySide6"] = pkg
sys.modules["PySide6.QtCore"] = qtcore

sys.path.insert(0, str(clawdmeter_clone / "src"))
import transcript as T          # his module, unmodified
T._file_token_events(path)      # his functions, his numbers

Why bother, when reading the code already showed the defect? Because of the dispute it removes. A reimplementation invites "your model of my code is wrong", and that argument can run for days. Numbers that came out of the vendor's own functions leave exactly one question open, whether the input corpus is fair, and a synthetic corpus with a by-construction manifest answers that in the same breath. The report gets to say "your functions produce 2,592,168 where the manifest says 1,108,697" instead of "I believe your code over-counts".

It also makes reproduction nearly free: git clone, one script, about a minute, a fake $HOME, no PySide6 install, nothing real touched.

Three days

Filed August 5. Confirmed August 6: "Thank you again for this report, I've confirmed the bug and will be patching it in an upcoming hotfix release." Shipped August 8 as v3.0.1, with a release note that opens the way I suggested it should, because a correct fix that halves every visible number looks exactly like a regression: "Your token and value numbers will drop by roughly 2.5×. That is the fix, not a regression."

One detail I want to be flat about: he never ran my harness. He reproduced independently, on his own corpus, with his own tooling. That is not the fixture failing at its job. The fixture's job is to make the claim cheap to check and expensive to dispute; his choice to verify it his own way instead is a stronger confirmation than an exit code, and the diagnosis still arrived pre-named, which is what made three days possible. A reproduction script is an offer, not a toll booth.

The fix report audited me back

His closing comment is the reason this post exists. It opens with "Your report was accurate and the diagnosis was right", and then does something I had not seen a maintainer do in five repos: it treats the bug report itself as a measurable claim, measures it, and reports back where it breaks. "Two things turned up that are worth passing back, since you audit other trackers."

His corpus came out worse than mine: 44,196 usage-bearing records across 18,585 distinct message ids, 73.0% on more than one record, work tokens 2.575× high. And then the two findings.

First: the repeated usage objects are not always identical. My report said that on my corpus, 100.0% of duplicated messages carried byte-identical usage on every record. On his corpus, 3,799 groups differ, and in every one output_tokens is a running total: early records carry a partial count, the last carries the final figure. His example message reads 5, 5, 5, 328 across four records. That makes the choice of which record to keep load-bearing. A per-bucket max is correct; keep-first would have reported that message as 5 output tokens instead of 328.

Second: collapsing per file is not enough. 1,094 of his message ids appear in more than one transcript file, because resuming a session replays its records verbatim into a new file. Deduping within each file still left 1.095× inflation on his tree. The collapse has to live in the account-wide aggregator, and a replayed copy can carry an all-zero usage block where the original carries the real numbers, so a record-level dedup placed in front of the message collapse can keep the wrong copy. Folding every record into its message under a per-bucket max sidesteps the whole trap.

He also turned my one-line invariant, usage events should equal distinct message ids, into a regression test, added six more, and ran nine mutations of the fix against them. Nine of nine caught. That last number is a stronger claim than any test count, and it is the part I would not have thought to ask for.

The split is a field, not a version

Finding one corrects something I wrote, so I owed it a measurement rather than a thank-you. My "100.0% byte-identical" was true and under-scoped: it was true of the main conversation path, and I did not say so. On my corpus, regrouped:

grouped per (file, message.id)	groups	byte-identical
main path	10,645	10,645 (100.00%)
sidechain (subagent)	16,051	3,342 (20.82%)

Not one main-path group differs, across 63 days and 1,521 transcripts. Every differing group has a sidechain record in it. The discriminator is isSidechain, not the Claude Code version he suspected — and the reason I can say that with some force is that the version hypothesis makes a testable prediction and the field hypothesis makes a different one. If the shape tracked the writer version, main-path groups would differ on some machines. They never do here. His Linux and macOS corpora "still showing the byte-identical shape" is what a main-path-heavy tree looks like; an agent-heavy tree shows his 3,799.

His running-total observation, meanwhile, holds without exception on my side: 12,709 differing groups, 100% non-decreasing, 100% with the last record carrying the max. He was right about the shape and wrong about the cause. I was right about my corpus and wrong to publish the number without its denominator. Neither error survived a second corpus, which is the whole argument for sending measurements instead of opinions.

The artifact I nearly handed him

One warning went into my reply because it is the check anyone would write next, and it lies. Group repeated records by message.id alone and my corpus reports 560 non-monotonic groups, output counts that go down mid-message. Every one is fake: records from unrelated files spliced into one sequence by the sort. Group per (file, message.id) and all 560 vanish. I know the trap is real because I fell into it on a different tracker and reported the bad number before catching it.

Two more numbers from the re-measurement, since they size his findings on an agent-heavy tree. Keep-first against per-bucket max at corpus scale: 20,740,608 output tokens against 38,533,663. Keep-first loses 46.2% of output, so max is not a tiebreak, it is most of the number. And his all-zero replay trap does not reproduce here at all, zero groups in 1,521 transcripts, which means you cannot harvest a test fixture for it from a tree like mine. His synthetic fixture for that case is load-bearing. The script is duplicate_usage_shape.py, read-only, stdlib only, and it prints the artifact count next to the real one so the next person doesn't rediscover the splice.

Red, green, and an API change that proves the point

Yesterday I re-ran the harness against both tags. On v3.0.0 it is red the way the report said: 2.338×, 2.338×, 2.373× on the three call sites, exit 1. On v3.0.1 all three land exact, 1,108,697 and 732,191 to the digit, 540 rows for 540 messages, exit 0.

The check needed one adaptation, and the adaptation is itself the verification. _file_token_events no longer returns something you can sum: it now emits per-record events carrying both a record key and a message key, uncollapsed, with a docstring that says the collapse deliberately belongs to the account-wide caller. The fix is not a patch over the three sites; it is the architecture his second finding argued for, folded under the per-bucket max his first finding argued for. The code now contains the whole conversation.

Counterweights I owe you

My 100.0% shipped without its scope, and the fixture didn't stop it. The harness was green, the totals were exact, and the prose still over-claimed. A fixture pins code; nothing pins your sentences except someone else's corpus.

He never ran the harness, so this series' fastest fix owes nothing to the reproduction being runnable. What earned the three days was the diagnosis being named and the numbers being exact. I keep shipping the fixture anyway, but I should be honest that its value here was rhetorical, not operational.

The isSidechain split is established on one corpus: mine. His 3,799 differing groups were not regrouped under the split, so "every differing group has a sidechain record in it" is a claim about my tree that his data could still complicate.

I do not know why Claude Code writes sidechain usage as a running total and main-path usage finished. The field predicts the shape; the mechanism behind the field is unmeasured.

Part three argued response time tracks queue depth, and a solo maintainer answering in a day fits that line suspiciously well. One data point cannot separate "small queue" from "this particular maintainer", and the 30k-star PR from part three is still open, which proves the thesis only if you already believe it.

His version hypothesis was reasonable on his data. It took a second corpus to kill it, exactly as it took his corpus to kill my 100.0%. One corpus per claimant is how both of us over-generalized, and there is no reason to believe I have stopped.

What I would generalize

Run their code, not your model of it. A GUI is not a reason to reimplement a parser; a twelve-line stub deletes the entire "you misread my code" branch of the argument, and what is left is arithmetic.

A fix report is a dataset. The closing comment on a bug you filed can contain more measurement than the bug did. Read it the way you read a transcript, and answer it with numbers, because the person who just fixed your finding is the best-calibrated reviewer your method will ever get.

Publish invariants with their denominator. "100.0%" was true of a population I forgot to name. The correction cost a table; leaving it unscoped would have cost every future reader who audits an agent-heavy tree.

When two corpora disagree, look for a field before you blame a version. A version split and a field split predict different shapes. Ask what each hypothesis forbids, then check for the forbidden thing. Zero differing main-path groups is the kind of zero that settles arguments.

Collapse under per-bucket max. It is correct when duplicates are identical, correct when they are running totals, and immune to the all-zero replay copy. Keep-first was quietly losing 46.2% of output on my tree.

Group per (file, id), or the sort will manufacture your finding for you. My 560 phantom groups came from splicing files together before grouping. Print the artifact count next to the real one so the check that lies gets caught by the check that doesn't.

Timeline

date (2026)	event
Aug 5	v3.0.0 announced on r/ClaudeAI; harness pointed at the token path the same day; #21 filed: 2.34× at three call sites
Aug 6	weltern confirms, promises a hotfix
Aug 8	v3.0.1 ships. Release note leads with "That is the fix, not a regression". His closing comment: 2.575× on his corpus, two findings passed back
Aug 8	Re-measurement posted: the split is `isSidechain`, not the writer version; running total confirmed 100%; the 560-group splice artifact named
Aug 9	Harness re-run: red on v3.0.0 as reported, exact on v3.0.1 through the new aggregator API

The layer underneath

All five parts run on TraceGuard's routing_audit, an append-only, message.id-keyed ingest of Claude Code transcripts into SQLite. Apache-2.0, pip install traceguard. The Clawdmeter harness and the shape-measurement script live in clawdmeter-dedup; the corpus generator builds a fake $HOME and touches nothing real.

The series scoreboard, since part four's count is stale: seven shipped fixes across four repos — three in splitrail, two in tokscale (v4.9.0 and v4.11.0), Clawdmeter v3.0.1, and viberank's high-water-mark fix, deployed. The eighth is still an open PR behind a 135-deep queue, which part three already explained.

The thing this part adds to the series is not a seventh notch. It is that the method survived contact with a maintainer who measured back, and got better for it: per-bucket max and per-(file, id) grouping are now in my collapse rules because his corpus broke my claim, and the version hypothesis died because mine broke his. Audit is not a thing you do to a codebase. Done right, it is a thing two corpora do to each other.

Li Zhuojun