DEV Community: Li Zhuojun

The vendor documents this bug. A 30k-star repo shipped it anyway.

Li Zhuojun — Tue, 04 Aug 2026 13:03:18 +0000

This is part three of a series about pointing an append-only audit log at things that count tokens. Part one found that a single missing model line was half my AI agent overspend. Part two found two accounting bugs in splitrail, both of which undercounted — one hid 54% of my messages.

This one points the other way. The target is claude-code-templates — 30.1k stars, 3.3k forks, the largest Claude Code template repo — and its analytics over-counted my tokens by 2.35×.

The part that changed how I wanted to write this up: while drafting the report I went looking for whether the behaviour was documented anywhere. It is. Anthropic's own SDK documentation has a warning box about it, and ships a reference implementation of the fix. The bug was not hiding. Four separate mechanisms surrounded this code and none of them closed the loop, and the reason they didn't is the actual subject of this post.

The fix has now been open for ten days. Two AI reviewers approved it. No human has replied. More on that near the end, because the silence turned out to carry information too.

(Part four is the stated-vs-revealed routing analysis I promised at the end of part two. It got bumped. This was worth the detour.)

The bug

calculateRealTokenUsage() summed message.usage over every parsed record.

Claude Code writes a single assistant message as several JSONL records — one per content block (thinking / text / tool_use) — and every one of those records repeats the same message.id and a byte-identical usage object. So each message's tokens were counted once per block, and everything downstream inflated by the average block count: conversation.tokens, summary.totalTokens, the per-project rollups, and the Total Tokens figure on the analytics dashboard.

Measured against my frozen ~50-day corpus:

property	value
distinct assistant `message.id`s	8,123
ids appearing on exactly one record	2,533
ids appearing on more than one record	5,590 (68.8%)
of those, ids where every record's `usage` is byte-identical	5,590 (100.0%)
records-per-id histogram	`{1: 2533, 2: 1351, 3: 3553, 4: 415, 5: 58, 6: 138, 7: 24, 8: 33}`
records ÷ messages	≈2.36×

Three records per message is the mode — thinking, then text, then a tool call.
Two-thirds of my assistant messages were being counted more than once, and on
every single one of them the repeated usage was byte-identical, which is the
detail that makes the over-count a multiplication rather than a rounding error.

Then I drove the analyzer over a synthetic corpus with known-exact totals:

field	before	after	ground truth
input_tokens	27,678	11,447	11,447
output_tokens	2,632,674	1,115,321	1,115,321
cache_creation	19,609,374	8,130,148	8,130,148
cache_read	59,626,489	24,447,923	24,447,923
messagesWithUsage	1,308	540	540 (distinct ids)

The line that turns this from a hunch into a diagnosis: before the fix, every field equalled the per-record sum exactly. Not approximately. Exactness is what pins the cause to record-level summation rather than to anything in parsing, and it is why the report could name the defect instead of reporting a discrepancy.

There is also a check that needs no corpus, no fixture and no golden file, which is the part I'd want you to take away: messagesWithUsage should equal the number of distinct message.ids. It equalled the number of records. One assertion, available from day one, would have failed loudly.

It was in the docs the whole time

Here is what I found when I went to check whether this was known. From Anthropic's Agent SDK cost-tracking guide, in a callout box:

Parallel tool calls produce multiple assistant messages whose nested BetaMessage shares the same id and identical usage. Always deduplicate by ID to get accurate per-step token counts.

And in the prose above it:

When Claude uses multiple tools in one turn, all messages in that turn share the same ID, so deduplicate by ID to avoid double-counting.

The page ships a working seenIds implementation. Its explanatory diagram is captioned, in part, "Step 1 has four assistant messages sharing the same ID and usage (count once)." There is a public issue on the Claude Code repo with the symptom in its title — #6805, "Token Usage Statistics Duplicated in stream-json Mode Causing Massive Cost Inflation". And codeburn, another tracker in this space, documents its own global seenMsgIds set doing precisely what the SDK page recommends.

So this is not an obscure edge case. The SDK that produces the data tells consumers, in a warning box, to deduplicate by ID. A widely used consumer of that data summed per record instead, for as long as the function existed.

I want to be exact about what my audit log did and did not contribute here, because the distinction is the whole point and I would rather make it myself than have it made at me: an append-only log was not necessary to discover this. Reading the documentation would have done it. What the log contributed was three things the documentation cannot give you — whether this particular codebase had the defect, what it cost (2.36×, on 8,123 real messages), and proof that the fix was exact rather than merely different.

Documentation describes hazards. Measurement finds instances. Those are different verbs and the gap between them is where this bug lived.

What the two AI reviewers said

The PR went through two commercial AI code reviewers. Both cleared it. Neither mentioned the over-count.

Greptile posted a summary and a verdict:

Confidence Score: 5/5 — The PR appears safe to merge. No blocking failure remains.

It reviewed both commits, and left the only 👍 the PR has received.

cubic ran twice. Its first pass reported "AI review completed with 2 reviews. Found 2 issues across 3 files" — a maintainability note about the function being duplicated across two files, and this:

P3: Distinct records can still be merged when an id equals a generated positional key or another record's uuid, causing token undercounting for that conversation. Namespace persisted identifiers and use a non-colliding generated key for id-less records.

That is a fair catch and it deserves credit. It is a real hazard: ConversationAnalyzer.js:297 sets id: item.message.id || item.uuid || null, so a uuid can already arrive in the id field, and my fallback chain put id, uuid and a positional key in a single namespace. On real Claude Code data the probability is approximately zero. The cost to eliminate it was four lines — namespace each source (id: / uuid: / idx:) — plus a test that goes red before the change and green after. cubic's second pass dropped to "1 issue found across 3 files."

Now the observation, which I want to state as flatly as I can.

Both tools read this function closely enough to reason about a hypothetical key collision in a line I had written minutes earlier. Neither remarked that the function's output had been wrong by a factor of two since the day it was written.

That is not a miss. By the standard each was applying, both answers were correct. A diff-scoped reviewer answers "is this change safe and self-consistent?" The pre-existing error was not in the diff, and nothing in a review context contains an independent measurement of what the right number is.

The asymmetry is worth naming precisely, because it generalizes past these two products: cubic's finding was a hypothesis about the shape of code. The bug was a discrepancy between a number and reality. Only the second kind of claim requires evidence from outside the repository. A reviewer holding the entire codebase in context still would not catch this one, because the code is internally consistent — it does exactly what it says it does, and what it says is wrong. You cannot read your way to that conclusion. Somebody has to measure.

The hole nobody is looking at

While I was in there: this repo runs no test suite on pull requests. The only check on #754 is the AI reviewer. "Eight unit tests pass" is a sentence I had to type into a comment, because nothing in the pipeline can demonstrate it.

With 135 open pull requests and 86 open issues, that is capacity, not negligence. But it is exactly the condition under which a silent numerical defect lives indefinitely, and it reorders the priorities: two AI reviewers and no test CI is a weaker correctness story than no AI reviewers and one test that asserts a total against a known input. Reviewers are additive. They are not a ground truth.

Ten days later

As of publication, #754 has been open ten days. Every check is green, the branch merges cleanly, and the two AI-reviewer verdicts haven't moved since day one. Human words on the thread: zero. No label, no assignee, no milestone. The one open review question — which of the two duplicated copies of the function should survive — can only be answered by a maintainer, and hasn't been. In those same ten days the repo gained about 200 stars and its open-PR queue grew from 129 to 135.

The same ten days, elsewhere in this series:

splitrail, 216 stars. I filed #220, a streaming-snapshot over-count, with the inflation ratios predicted from my corpus before any fix existed. A fix matching the prediction on every field was merged within two hours. Third shipped fix in that repo for this series.
tokscale, 4.6k stars. My drift report #994 was fixed and released in v4.9.0. From the maintainer's closing note: "The measurement in this report is what made it actionable — the before/after table landing exactly on the drifted prediction pinned it to persistence rather than parsing, and saved us looking in the wrong place."
viberank. In an issue I filed about leaderboard drift, yoo-minho (he maintains the submission tool for a rival leaderboard, clauderank) confirmed the same class at a scale I can't generate: a month of roughly $15.5k list-price usage whose month-to-date total fell 11% between two submissions 16 hours apart. His words: "A cumulative month-to-date total went down, which shouldn't be possible." He then wrote the countermeasure up as usage-drift-log, a six-field append-only record, and traceguard now implements it verbatim. One page of spec, two independent implementations.

So the method is not slow, and maintainers are not indifferent. Response time tracked queue depth, nothing else: the 216-star repo answered in hours, the 4.6k-star repo in days, and the 30.1k-star repo has 135 pull requests ahead of mine.

Which corrected something I had wrong when I started this series. I treated the merge as the finish line. It isn't — a merge is a statement about maintainer bandwidth, and only the fixture is a statement about the code. It runs in a minute on anyone's machine, red on upstream main, green on the branch, and it means the same thing whether or not anyone at the repo ever reads it. If the Total Tokens figure on that dashboard matters to you, you don't need to trust me and you don't need to wait for the queue: ./run_check.sh settles it.

Counterweights I owe you

The documentation already said it. Worth repeating, because it is the strongest argument against the framing I originally wanted for this post.
Other trackers get this right. ccusage, opcode and ccseva key on messageId:requestId; codeburn keys on a global seenMsgIds set. On my corpus the two strategies are equivalent — distinct (message.id, requestId) pairs came out equal to distinct message.ids, so message.id alone was sufficient here. There is no "trackers are broken" story to sell.
Both signs occur. splitrail undercounted. This one over-counted. No directional rule.
A hypothesis of mine was false. After part two I expected the subagent-directory blind spot to be an industry-wide pattern. I read thirteen trackers from source. It wasn't — only one unmaintained project had it, and a third-party had already reported the same class of gap in tokscale eighteen days before I filed mine. Saying so is more useful than the story I was hoping for.
In tokscale, my proposed fix lost the argument, and should have. I suggested the append-only store this series runs on. The maintainer turned it down: retention that never deletes would resurrect sessions a user deliberately removed, so he scoped retention to dedup keys that stay stable across files instead. I tested his rule — deleting a transcript dropped exactly that file's contents, 40 messages, 78,770 output tokens, nothing resurrected — and conceded in the thread. The audit finds the discrepancy. It has no special authority over the fix.
cubic found something my method wouldn't have. A latent hazard in code that had not yet met hostile data leaves no trace in any measurement, because it hasn't happened yet. The two approaches cover different failure classes, and I'd run both.

What I'd generalize

Reading is not measuring. Documentation, stars, code review, and more code review are all mechanisms for reading. Stacking more of them does not asymptotically produce a measurement. If nothing in your pipeline compares an output to an independently derived number, that comparison is not happening, however many eyes are on the diff.

Assert an invariant, not a total. Totals need fixtures, corpora and maintenance. messagesWithUsage == distinct message.ids needs none of those and would have failed on the first commit. Nearly every accounting path has an invariant like this hiding in it; the work is noticing which one.

Exactness is the evidence. "Roughly double" is a shrug that invites a debate about whose numbers are right. "Exactly the per-record sum, to the digit, on every field" is a diagnosis that ends the debate before it starts.

Ship the mechanism with the report. Four upstream fixes across two projects so far, three in splitrail and one in tokscale, the fastest merged two hours after the report — all because a runnable red-green fixture arrived attached to the claim rather than after someone asked for one.

When the same hazard hits several tools, converge on a contract, not more reports. The rewrite-drift class now has a one-page spec with two independent implementations behind two different leaderboards. That does more than a third bug report would have.

When the vendor documents a hazard, treat it as a test case, not as trivia. Every warning box in an SDK's docs is a bug someone has already shipped. That is a free list of things to go measure.

Timeline

date (2026)	event
Jul 24	corpus reconciliation flags the ratio: 8,123 distinct message ids spread across ~2.36× as many records
Jul 25	PR #754 filed with fixture, fix and measurements
Jul 25	Greptile: 5/5, safe to merge. cubic: 2 issues found
Jul 25	`7793bba8` — namespaced dedup key, two added tests, answering cubic's P3
Jul 25	cubic re-review: 1 issue. All checks pass, no conflicts
Jul 31	splitrail merges the series' third fix there (#220 → #222) two hours after the report; yoo-minho confirms rewrite drift on a $15.5k month, −11% silent
Aug 1	his six-field usage-drift-log spec adopted verbatim into traceguard (PR #30) — second independent implementation
Aug 3	tokscale ships v4.9.0, closing #994 with credit to the measurement
Aug 4	this post. #754: checks green, two bot approvals, zero human replies, 135 open PRs in the queue

The layer underneath

All three parts of this series run on TraceGuard's routing_audit module — an append-only, message.id-keyed ingest of Claude Code transcripts into a SQLite trace store. It is a few hundred lines and a database file, and everything above is downstream of one design decision: never let the source rewrite history.

Since part two, that layer grew a public edge: after every scheduled ingest it appends the six-field usage-drift-log record (frozen at first sight, never recomputed), so the drift class yoo-minho measured shows up as a warning line instead of a silent change.

Stable totals are the substrate, not the product. The thing worth building on top is stated-vs-revealed routing analysis, priced per decision — which model you said you'd route to, which one actually ran, and what the difference cost. That's part four, and this time I mean it.

pip install traceguard — Apache-2.0. The reproduction harness for this post is self-contained, runs in about a minute, and touches no real ~/.claude data — it builds a synthetic $HOME and hands the upstream analyzer nothing but a directory: usage-tracker-audit/cct-dedup-check.

An append-only audit log caught two accounting bugs in a 216-star usage tracker

Li Zhuojun — Sat, 25 Jul 2026 02:17:08 +0000

Last month I wrote about auditing 34 days of multi-model Claude Code usage and finding that a single missing model line was half my overspend. The core of that piece was a boring engineering decision: ingest every assistant message from Claude Code's session files into an append-only log keyed by message.id, and never let the source rewrite history.

This is the follow-up I didn't plan to write. The same log just caught two accounting bugs in splitrail, a 216-star Rust usage tracker for agentic CLIs — and within twelve days of the first report, both were fixed in an official release: one through my PR, one through a maintainer-authored patch that adopted the report's analysis. Here's the play-by-play, with the numbers.

Bug 1: your history quietly rewrites itself

Claude Code rewrites session JSONL files in place when you resume or compact a conversation. Between two scans of the same file, five of my assistant messages vanished — usage that was already incurred, gone from the record. Any tracker that recomputes totals from live files inherits this drift: yesterday's numbers change while you sleep.

I filed it as #200. The maintainer confirmed the mechanism the same day — splitrail re-read live files, so messages removed by resume/compaction disappeared from historical totals — and shipped a fix in 3.6.0: a local SQLite history store that persists normalized usage and merges it with current session data before deduplication. Fast, clean work.

One wrinkle: he suggested validating by comparing 3.5.9 vs 3.6.0 totals, expecting "3.6.0 should be higher." That's subtly wrong, and the distinction matters for anyone testing this class of fix: on a cold start the two versions are equal — the history store has nothing to restore yet. The divergence only appears across drift events. So I built the test to create one.

The validation protocol

Everything ran against a frozen snapshot (APFS clone) of ~/.claude/projects, so both versions scanned identical bytes, with every invocation under an isolated $HOME — own config, own history store, no upload path. Then:

#	assertion	result
A	cold-start parity: fresh 3.5.9 == fresh 3.6.0 on identical input	✅ identical
B	3.5.9 across a simulated resume/compact rewrite	✅ drops by exactly the removed usage
C	3.6.0 across the same rewrite	✅ totals unchanged — history store restores
D	3.6.0 restart stability, 3 consecutive runs	✅ byte-identical output

The simulated rewrite removes the last five assistant message-groups from the largest transcript — the same shape as the original drift event. The whole thing is now a self-contained regression fixture (PR #208, since merged): it fails red on 3.5.9 and passes green on 3.6.0, which is exactly what you want a regression test to do.

The part I care most about: on the transcripts splitrail scans, 3.6.0 agreed with my append-only log token-exact — 18,548,947 output tokens on both sides across 13.5k messages. Two independent implementations, different languages, different dedup strategies, same number to the digit. (The tiny message-count delta was zero-usage api-error records splitrail intentionally skips.) When two systems reconcile exactly, every remaining discrepancy is a finding, not noise.

Bug 2: the discrepancy that was a finding

Because the log records where every message came from, I could classify all of them against the live tree:

class	files	messages	output tokens
live, main transcript (what splitrail scans)	76	13,704	18,548,947
live, `subagents/**` transcript	1,423	16,160	13,738,324
vanished (file exists, message.id gone)	0	0	0
deleted (session file gone)	874	11,821	14,238,006

Row two is the second bug. Claude Code writes subagent transcripts (Task tool: Explore, general-purpose, custom agents) under projects/<slug>/<sessionId>/subagents/, at directory depth ≥ 4. Splitrail's discovery hard-caps at depth 2 in three places (WalkDir…min_depth(2).max_depth(2), a components() == 2 path check, and the glob pattern). Those files are structurally invisible — 54% of my live messages, and roughly a third of the dollars, never entered any total.

The model mix makes the blind spot vivid: splitrail saw 22 of my 5,059 Sonnet messages and 0 of 1,566 Haiku messages. Those models run almost exclusively inside subagents. If you delegate heavily to cheap models — which is exactly what cost-conscious agent users do — your tracker undercounts the most, on the workflows you optimized hardest.

Filed as #207 with the layout, a two-line repro, and a suggested fix. Two days later it was closed — and not by my patch. The maintainer wrote the fix himself: #209 "Include Claude Code subagent transcripts" (+428/−100 across seven files), whose description restates the issue's analysis and adopts its central caution — simply lifting the depth cap isn't enough; subagent records have to flow through the same local_hash dedup semantics as main transcripts, or you trade an undercount for a double-count. Both fixes shipped together in splitrail 3.6.1, which hit Homebrew within a day. On my corpus, the newly visible class is row two of the table above: 16,160 messages and 13.7M output tokens finally entering the totals. A fresh 3.6.1-vs-log reconciliation of the live tree is the obvious next check — part 3 starts there.

What I'd generalize

Live mutable files are not an audit trail. If your source can rewrite history, recomputation is not accounting. Append-only ingest with a stable per-record identity (message.id, last-write-wins on partials) is cheap insurance — mine is a few hundred lines and a SQLite file.

Reconcile token-exact or you know nothing. "Close enough" totals hide entire bug classes. It was exact agreement on the scanned subset that turned the remaining gap into two nameable, fixable defects instead of a shrug.

Decompose the gap before you blame anyone. "Your numbers are lower than mine" is an accusation; "the gap is exactly deleted-files + an unscanned directory class, here's the table" is a bug report a maintainer can act on in hours. Both turnarounds here — #200→#204 in eight days, #207→#209 in two — happened because the mechanism arrived with the report.

Frozen snapshot + isolated $HOME is the whole trick for tracker A/B. Clone the data, pin HOME (and XDG_STATE_HOME/XDG_DATA_HOME — on Linux, state dirs honor XDG and will escape your sandbox; CodeRabbit's review caught that one in my own fixture, fair is fair), and two binaries scanning identical bytes become a controlled experiment.

Timeline

date (2026)	event
Jul 11	#200 filed — resume/compact rewrites session JSONL in place
Jul 19	#204 merged; 3.6.0 ships the SQLite history store
Jul 20	4-assertion A/B validation on a frozen corpus; token-exact reconciliation vs the audit log
Jul 21	#207 filed (subagent blind spot); PR #208 opened; CodeRabbit review round addressed same day
Jul 22	PR #208 merged
Jul 22–23	Maintainer authors and merges #209; #207 closed; 3.6.1 released, Homebrew bumped

Twelve days, two bugs, zero arguments about whose numbers were right — the log settled every question before it became a debate.

The layer underneath

All of this ran on TraceGuard's routing_audit module (v1.1.0, "audit evidence layer") — the append-only, message.id-keyed ingest of Claude Code transcripts into a SQLite trace store that part 1 was built on. Stable totals are the substrate; the point of the layer above is stated-vs-revealed routing analysis, priced per decision — which model you said you'd route to, which one actually ran, and what the difference cost. That's part 3, and my totals finally sit still long enough to write it. (Apparently not a niche itch, either — the splitrail maintainer starred TraceGuard during this exchange and said he'd be interested in the audit part.)

pip install traceguard — Apache-2.0. The regression fixture is ~600 lines of stdlib Python + bash, merged in PR #208; the subagent fix shipped in splitrail 3.6.1.

A missing `model:` line was half my AI agent overspend — auditing 34 days of multi-model Claude Code

Li Zhuojun — Sun, 05 Jul 2026 16:41:59 +0000

I'm a solo developer running 10+ projects with a multi-model agent setup: Opus 4.8 as the main thread, Fable 5 (2× Opus price) as a decision advisor, Sonnet/Haiku for mechanical subagent work. The question I couldn't answer: which tasks actually deserve the expensive tier, and where was I overpaying?

So I built a routing audit as an opt-in module of my tracing SDK, backfilled ~26,000 traces (34 days) from Claude Code's local session logs, priced them at official list prices, and diffed my stated routing policy against my revealed routing behavior. Everything below is list-price equivalent (I'm on a subscription; API-paying teams see these numbers for real).

Finding 0: your usage history is quietly rewriting itself

Claude Code's resume/compact rewrites session JSONL files. Between two ingest runs, 5 messages vanished from a source file. If your usage numbers come from re-reading live session files (as most usage dashboards do), they drift. An append-only ingest log fixed it — and gave an unexpected validation: my Fable traces reproduce June's export-control suspension window to the hour (1,599 traces on Jun 10, exactly zero from Jun 14–30, resuming Jul 1).

Takeaway: snapshot your telemetry into an immutable store, or you're auditing sand.

Finding 1: one config line was most of the waste

I wrote my routing policy down as a declarative YAML (frontier/mid/cheap tiers × agent role × task type), then diffed it against 425 observed routing decisions.

22.6% deviated from my own policy — $1,248 of list-price spend.
Root cause of the single biggest cluster ($657, 53% of deviation cost): subagents inherit the parent thread's model unless the agent definition pins one. When my main thread ran the advisor model, mechanical subagents silently ran on it too — 2× Opus price for grep-tier work. Verified 19/19 against parent sessions, zero counterexamples.
The fix is one model: line per agent file.

Bonus honesty: my first audit pass "found" 28.2% deviation. A third of that was my policy file being wrong, not my behavior (I'd written rules I never actually intended). The first thing a routing audit fixes is your policy statement.

Finding 2: counterfactuals cost pennies; protocols are the hard part

"Was the 2× advisor model worth it?" I replayed 12 real advisor consults on the cheaper tier. Predicted cost: $22.81. Actual: $0.21 — my estimator was 110× off, because replays are cold-start prompts without the original ~660k-token cached context. Quality counterfactuals are essentially free; nobody skips them because of cost.

Then I blind-judged the 12 pairs myself, and the protocol failed in three instructive ways: I recognized my own original conversations in 4/12 pairs (33% leak); my deterministic position assignment put 9/12 originals in slot B (so "B wins" was confounded with "original wins"); and originals had full context while replays didn't. Net result: the advisor-premium question is still open. Protocol v2: third-party judge, position-balanced, longer delay. I'm publishing the failure because an audit tool's dev log should look like this.

If you run agent fleets

Pin model: in every subagent definition. Inheritance is silent and expensive.
Snapshot usage logs immutably. Live session files rewrite themselves.
Write your routing policy down. The diff between stated policy and revealed routing is where the money hides — and half of what you'll find is that your policy was never real.

The module is contract-external, local-first (nothing leaves your machine), built on traceguard — a point-in-time-correct LLM instrumentation SDK originally built for quant pipelines, where "which model knew what, when" is an audit requirement, not a nice-to-have.

Ask: I'm doing a handful of free routing audits for teams running multi-model agent fleets (especially where quality is measurable — trading, data pipelines, eval-gated workflows), in exchange for anonymized learnings. DM or open an issue.

The `epsActual` That Wasn't: 15% of an LLM Backtest's Trades Were Decided on Data That Didn't Exist Yet

Li Zhuojun — Thu, 18 Jun 2026 11:16:59 +0000

We were backtesting an LLM-driven earnings signal against a field called epsActual — the kind of field everyone treats as ground truth. It isn't.

About 41.4% of those "actual" values were different from what the vendor had first reported. About 15.3% differed enough to flip a tradeable decision. When we re-ran the backtest using only the values that actually existed at each decision date, the strategy kept ~73% of its returns and ~82% of its Sharpe. The rest was look-ahead bias — and it rode in through a field whose name promised it was final.

This is a writeup of how we found it, how we measured it honestly, and the one-line invariant that turns it from a silent inflation into a loud test failure.

The setup

The signal is a post-earnings drift play: at each earnings print, an LLM scores the release and we take a position. To backtest it you replay history — for every past print, reconstruct what the model would have decided, then check what happened next.

That reconstruction needs one obviously-trustworthy input: what the earnings number actually was. Our vendor exposes exactly that, in a field named epsActual. "Actual." Final. Settled. You query a print from two years ago and get a number back. What could go wrong?

The invisible killer

Vendor "actuals" are not frozen at print time. They get backfilled, corrected, and restated — sometimes the next day, sometimes months later. Restatements, late filings, parser fixes, standardization passes: all of them quietly rewrite history. The value you query today for a 2023 print is not, in general, the value that was available the day after that print.

This is textbook look-ahead bias, and it's especially dangerous here because it doesn't look like leakage. Nobody fed the model future data on purpose. It rode in on a field everyone trusts — and "actual" is about the most trustworthy-sounding name a field can have. A backtest built on today's epsActual is quietly asking the model to react to numbers that, on the decision date, did not yet exist.

How we measured it honestly

You can't detect this from a single snapshot of the database — by definition the revision has already overwritten the original. So we built a forward-polling harness: poll the vendor on a schedule, snapshot every value we care about, and watch for changes over time. It had accumulated ~1,400 snapshots in the first day of polling.

The decision that mattered most:

Detect revisions by the value itself, not by the vendor's lastUpdated timestamp.

lastUpdated is unreliable — it doesn't reliably fire on silent backfills, and trusting it would have hidden exactly the revisions we were hunting. So change detection keys on the value-tuple: if any tracked field changes between two snapshots, that's a revision, regardless of what the metadata claims.

# Revision = the tracked value-tuple changed between snapshots,
# NOT "the vendor bumped lastUpdated".
def is_revision(prev_snapshot, curr_snapshot, tracked_fields):
    prev = tuple(prev_snapshot[f] for f in tracked_fields)
    curr = tuple(curr_snapshot[f] for f in tracked_fields)
    return prev != curr

To quantify the trading impact, we compared two backtests over a four-month point-in-time window: a naive one using today's revised epsActual, and an as-of one using only each value as first seen on (or before) the decision date.

What we found

41.4% of epsActual values (896/2163) differed between first-seen and final.
15.3% of cases (332/2163) differed enough to flip a tradeable decision — a sign change or a threshold crossing in the signal.
Over the four-month window, the as-of backtest retained ~73% of the naive backtest's returns and ~82% of its Sharpe. (The FINAL leg keeps drifting as the vendor keeps revising, so treat the ratio as more stable than the levels.)
Read inversely: roughly a quarter of the headline returns, and a fifth of the Sharpe, were look-ahead artifacts.

The encouraging half: most of the strategy survives honest data. The sobering half: a naive backtest overstated it by a wide margin, and a meaningful fraction of "winning" trades were decided on numbers that did not exist at decision time. A 15% decision-flip rate is not noise you can wave away.

Why this is structural, not a one-off

The natural reaction is "okay, we'll be careful with that field." That doesn't hold. The risk is reintroduced by every new feature, every new vendor, every rerun, every teammate who reaches for "the actual value." Carefulness is a property of a person on a good day; as-of correctness has to be a property of the pipeline.

So treat the question "could this value have been known at the decision time we're simulating?" as an invariant the code enforces and CI checks. A vendor "actual" is time-versioned reference data: it only becomes valid at the instant you first observed it. Use it to decide before that instant and you're using a value from the future.

That's exactly what the look-ahead invariant below checks — it requires valid_from <= feature_as_of:

from traceguard.validators.lookahead import validate_reference_timing

# The eps "actual" is time-versioned reference data: valid_from is when this
# specific value first existed (first-seen in our snapshots), feature_as_of is
# the decision moment we are simulating.
validate_reference_timing(
    valid_from=eps_first_seen,    # when this value actually existed
    feature_as_of=decision_date,  # the moment we're simulating
    kind="vendor_eps_actual",
)  # raises InvariantViolation if eps_first_seen > decision_date

When a value is used before its availability timestamp, the run fails loudly rather than silently inflating a Sharpe ratio.

Two kinds of look-ahead — don't conflate them

It's worth being precise about scope. There are two distinct kinds of look-ahead bias in LLM pipelines:

Training contamination — the model itself was pre-trained on the future you're predicting, so it "recalls" rather than reasons. That's a separate research problem (membership-inference tests, point-in-time LLMs, claim-level temporal verification), and it needs different tooling.
Harness / pipeline leakage — your code uses a value, prompt, or model that didn't exist at the simulated time. This story is entirely about this kind, and it's the kind a pipeline can be made to refuse structurally.

Both matter. They are not the same problem, and conflating them is how teams "fix" one and ship the other.

A checklist you can apply today

Treat every actual / final / reported vendor field as a moving target until you've proven otherwise with your own snapshots.
Detect revisions by value, not by the vendor's update timestamp.
Backtest on as-of (first-seen) data, and explicitly measure the gap against revised data. That gap is your look-ahead tax — quantify it instead of assuming it's zero.
Encode "known at decision time?" as a CI invariant, so the failure mode is a red test, not a flattering backtest.

Limitations

One vendor, one field, a four-month window. The exact percentages are dataset-specific and should not be read as universal constants — your numbers will differ. And again: this addresses harness leakage only, not whether the model itself has seen the future.

The validators and point-in-time instrumentation here are part of traceguard — an open-source Python library for point-in-time-correct LLM instrumentation: a model registry that refuses anachronistic picks, a git-tracked prompt registry, canonical input hashing, and look-ahead invariants you call in CI. It's not a dashboard — it exports OpenTelemetry spans into Langfuse / Phoenix, so it sits underneath your observability stack and keeps the timeline honest.

pip install traceguard

If you've been burned by a backtest that looked great and meant nothing, I'd genuinely like to hear how it happened — that's the failure mode this is built to catch.