DEV Community: Yusuke Shiki

Tracking Codex Astra costs took more than updating a price table

Yusuke Shiki — Sun, 06 Sep 2026 01:10:55 +0000

I updated agent-cost, a CLI I maintain, to version 0.1.1 because I wanted to measure Codex's GPT-6 Astra usage too. It reads local Claude Code and Codex CLI logs and reports token usage and estimated costs, not actual billed amounts.

Fast mode was where I ran into trouble.

After adding Astra to the price table, I tested it with synthetic logs. Usage marked as Fast was still being priced at the Standard rate. The multiplier was in the catalog, but the log reader wasn't recognizing the setting.

The fix involved both detecting the mode and deciding what to do when the logs didn't give me enough information.

What the synthetic tests showed

I used synthetic data with two million uncached input tokens, two million cached input tokens, and two million output tokens: six million tokens in total.

Here are the results for Standard, Fast, and unknown settings, using the rate catalog bundled with 0.1.1. This is a test fixture, not real usage or a list of current official prices.

Setting identified from the log	Estimated USD	Estimated credits	Unpriced tokens
Standard	122	3,050	0
Fast	305	7,625	0
Unknown	0*	0*	6,000,000

The zero in the unknown row means none of that usage could be priced. **It does not mean the usage was free.*

The same token counts produce an estimate of $122 at Standard rates and $305 at Fast rates. During development, the Fast case was incorrectly returning the Standard estimate.

The already-published 0.1.0 release didn't have Astra rates at all, so it treated Astra usage as unpriced. This was a problem found while implementing Astra support, not a report that 0.1.0 had shown users the wrong Fast amount.

The changes are in PR #8. The synthetic fixture and tests are public too. The table above summarizes test results; it isn't a copy of the CLI's output.

Does "not known to be Fast" mean Standard?

I checked the public source for Codex 0.153.4.

For this Astra support, agent-cost reads service_tier as well as the model name. It checks that the settings belong to the same session and match the target turn and model. An explicit default selects Standard; priority selects Fast.

But a setting can be missing. It can also conflict with the information recorded for a turn. In those cases, the mode isn't clear.

Treating everything that isn't recognized as Fast as Standard would still produce a number. It would also assign Standard prices to usage whose settings the reader simply couldn't establish.

Instead, usage stays unpriced when the reader can't determine which rate applies. Knowing Astra's rates doesn't establish whether a particular piece of usage was Standard or Fast.

This is based on the request settings recorded in the log. It doesn't confirm the processing mode actually used by the server or what was billed. The source references and attribution rules are in the implementation notes.

Keep the tokens even when the price is unknown

Dropping an unpriced row would hide the fact that the usage happened. Keeping only a zero amount would make it look free.

So the output carries pricing_status, which describes the pricing state, and unpriced_tokens, which counts tokens that couldn't be priced, separately from the amount.

Checking those fields alongside the total shows whether any usage was left out of the calculation. The unknown row above doesn't just say $0: it preserves all six million tokens as unpriced.

priced doesn't mean a confirmed charge either. It means the usage could be calculated using the selected rate catalog.

Which usage gets the new setting when Fast is switched on?

Settings can also change during a session.

Suppose a session starts in Standard, uses Fast for the next turn, then switches back to Standard. Pricing the entire session with its final setting would treat the Fast portion as Standard too.

When settings change between completed turns, the reader applies the new settings only to subsequent usage. The log contains cumulative token counts, so it takes the increase since the previous count and prices that increase using the turn's settings.

An update while a turn is still running leaves an ambiguous interval: the reader can't safely assign that usage to the old or new settings. Those tokens stay unpriced as well.

This behavior was added in 0.1.1 for logs containing Astra. It doesn't fully reconstruct missing history, concurrent updates, or alternative storage formats. The supported scope is limited to the reader implementation and the cases checked with synthetic data.

Trying it locally

With uvx available, you can run the published 0.1.1 package with the commands below.

Astra coverage in this version: the model ID must be exactly gpt-6-astra, and usage must be at or after September 5, 2026, 17:18:23 UTC (September 6, 02:18:23 JST). Earlier usage remains unpriced, even when Standard or Fast can be identified.

This is a catalog boundary based on when the rates were checked, not an official launch or tariff start time. This release also does not update every model's rates: known discrepancies in the GPT-5.6 family remain.

uvx --from coding-agent-cost==0.1.1 agent-cost doctor
uvx --from coding-agent-cost==0.1.1 agent-cost report

doctor checks things such as the configured input locations. report actually reads the local logs and aggregates usage. Running doctor alone doesn't verify that every log is valid.

The CLI doesn't send the logs anywhere. The uvx package runner may use the network to download the package. To see output before reading your own logs, use the README's synthetic output example. That example remains pinned to 0.1.0 as a record of an earlier verification.

For use from another program, there is JSON output and the measure/v1 interface. The output includes the rate catalog's version and SHA-256, so a result can be traced back to the catalog used to calculate it. The README covers installation requirements and the timezone-data caveat on Windows.

The additional Astra checks used synthetic data. I haven't validated compatibility against real Astra logs or measured a cost reduction from using the tool. API-specific tariffs and actual charges under individual contracts are outside this calculation. The 0.1.1 release notes record the scope and verification results at publication.

Where I landed

An unknown price isn't a particularly satisfying result when you ran the tool to find out the cost.

But assigning a Standard rate when the settings couldn't be read would hide that uncertainty from the person using the result. For this Astra update, I kept the tokens even when they couldn't be priced, so the output shows how much usage the calculation actually covers.

This is the English version of my article on Zenn.

Before Asking an AI Coding Agent to Fix a Bug, Verify the Bug Exists

Yusuke Shiki — Thu, 03 Sep 2026 02:25:58 +0000

Before asking an AI coding agent to “fix this bug,” there is one question worth asking first:

Does the bug actually exist?

An issue says it does. A review points it out. Someone reports seeing it in another environment.

Any of those is a good reason to investigate. None of them, by itself, proves that the problem currently exists exactly as described.

It may already be fixed. It may only happen in a particular version or environment. The reported behavior may even be expected.

As I have started relying more on AI coding agents, I have become more deliberate about checking this step.

A wrong premise can still produce a perfectly reasonable implementation

Suppose you give an agent this task:

Problem X is happening. Find the cause and fix it.

The agent can inspect the code, reason under the assumption that X exists, and produce a plausible fix.

That is not necessarily an AI capability problem.

If the input says “X is happening,” solving the problem under that premise is a reasonable thing to do.

The part I want to question happens one step earlier.

Bug report
 ↓
Fix

becomes:

Bug report
 ↓
Does the reported problem actually occur?
 ↓
Fix

Before implementation, verify the premise the implementation depends on.

I had a review finding that changed after I measured it

This happened publicly while I was working on PR #28 in my OSS project, spec-lane.

The PR added a gate that can execute an external verification command. During review, a strong claim came up around timeout behavior:

Even with SIGKILL, the CLI timeout itself may hang.

If I had accepted that statement as the premise, the next task would have been straightforward: implement a fix for a hanging timeout.

Instead, I measured it first.

Under the three conditions I tested, I could not reproduce a call exceeding the configured deadline.

That did not mean there was no problem.

I found a different boundary.

Even after the direct child process exits, a descendant process that still holds the inherited stdout/stderr pipes can affect how long spawnSync waits.

In one measurement, the direct child exited immediately while a grandchild remained alive for four seconds. The call took 3112 ms.

But when the grandchild remained alive for ten seconds, a two-second timeout returned in 2035 ms, and a 0.5-second timeout returned in 501 ms.

In the conditions I measured, the configured deadline still held.

So these were two different statements:

Original strong claim:
SIGKILL can allow the timeout itself to hang past its deadline.

What I could actually observe:
Descendants can influence waiting and timeout reporting within the deadline.

They sound similar, but they lead to different implementation work.

I did not add a new “hang fix.”

Instead, I changed the specification and tests to describe the boundary I could actually measure.

Measuring the premise changed the problem I was about to implement.

One important qualification: this incident was not automatically stopped by spec-lane's premise_evidence gate. It was a manual review-and-measurement decision made while developing the project.

Put “verify” between “reported” and “fix”

I now think about the flow like this:

Issue / review finding
          ↓
Verify the premise
          ↓
      Reproduced?
       ↙       ↘
     No         Yes
     ↓           ↓
Stop or        Spec
re-scope         ↓
              Implement

If the issue reproduces, proceed with the specification and implementation.

If it does not, do not force yourself—or an agent—to find a fix for the problem you expected to see.

Sometimes the right answer is to stop.

Sometimes, as in PR #28, the investigation reveals a different, measurable problem. In that case, change the scope to the problem that actually exists.

This extra step matters more as implementation becomes cheaper.

spec-lane can record that decision before implementation

I wanted this decision to exist in the development workflow rather than only in someone's memory.

In spec-lane 0.9.0, an intent can contain an optional premise_evidence record.

For example, when you decide that a change requires premise verification but fail to reproduce the reported problem:

premise_evidence:
  required: true
  method: live
  reproduced: false
  evidence: "Reported steps were executed against the current build, but the reported behavior was not observed."

Here, required: true records the decision that premise verification applies to this change.

reproduced: false records that the premise was not confirmed.

With that current record, spec-lane refuses the 1_intent -> 2_spec transition. A failed lane advance does not modify lane-state.json.

The purpose is not to have a CLI magically detect whether an AI is wrong.

The purpose is simpler.

Once the workflow has explicitly recorded “we could not confirm this problem,” do not silently continue into specification and implementation as if it had been confirmed.

Recorded evidence is not the same thing as truth

There is an important limit to this mechanism.

Suppose the file says:

reproduced: true

The CLI can verify that true was recorded.

It cannot verify that someone honestly reproduced the bug.

The same applies to:

method: live

The CLI can verify the field and its value. It cannot know whether somebody actually inspected a live system.

spec-lane also enforces a minimum length for the evidence text. That is a structural threshold, not a quality score.

A longer evidence string is not automatically better evidence.

The boundary I use is:

CLI proves recorded shape and transition behavior, not real-world truth.

The system can mechanically inspect the record and decide whether a transition is allowed.

The truth of the real-world observation remains outside that guarantee.

Missing premise evidence does not automatically block a lane

premise_evidence itself is optional.

If it is absent, the current CLI emits a warning rather than a hard error. A warning alone does not prevent the transition to the next phase.

That is deliberate.

The CLI cannot determine whether every change actually requires premise verification.

A bug already observed directly by a human is different from an unverified report. A feature addition may not have a “does this bug exist?” premise at all.

Making the CLI decide applicability on its own would introduce another form of false certainty.

So spec-lane starts enforcing after a human or agent has decided that premise verification applies and recorded the result.

It is a mechanical backstop for an explicit decision, not an oracle.

Faster implementation makes the question before implementation more important

AI coding agents have made the implementation step dramatically faster.

That makes the decision immediately before implementation more consequential.

What are we changing?

Why are we changing it?

Does the problem this change depends on actually occur?

If that premise is wrong, faster implementation only lets us move in the wrong direction faster.

So before handing an issue or review finding to an agent and saying “fix it,” I now want one more step:

Does this problem actually exist?

If it reproduces, proceed.

If it does not, stop.

If the investigation reveals a different problem, change the scope to what you can actually observe.

That is the role premise evidence plays in spec-lane: preserve that decision before implementation and provide a mechanical backstop against ignoring an explicitly failed premise.

Project: shiki-yusuke/spec-lane

Japanese version: Zenn

When you use AI coding agents, where in your workflow do you verify that the problem they are about to fix is real?

A Real Deployment Exposed Two Contract Defects Before I Froze the Contract

Yusuke Shiki — Wed, 26 Aug 2026 10:14:59 +0000

A green CI run does not prove that the artifact built from reviewed source matches what production later exposes.

The build job may have produced one directory while the deploy job selected another. A file may have been added after review. The live manifest and representative file bytes may match the digests sealed at build time—or they may not.

I built release-evidence and its v0 contract to connect those claims. It records source and build provenance, seals artifact digests, appends deployment events, and checks a bounded part of the live production target.

But I did not freeze the first reviewed version of the contract.

Deployment contracts are unusually good at lying while every synthetic fixture stays green. A
schema can be internally consistent and still describe a topology that does not exist. Two digest
fields can both look reasonable until an adapter has to decide which one to compute first.

Keeping the contract in draft until a real adapter exercised it exposed two defects.

The goal was not another “build succeeded” record

The first target was a static dashboard deployed through GitHub Pages.

The evidence chain needed to support a narrower statement:

reviewed source
  -> one built artifact
  -> that artifact deployed to production
  -> live manifest read back
     + representative file bytes spot-checked

The draft contract used a sealed release bundle and an append-only event ledger. The bundle held
the source commit and tree, toolchain evidence, and artifact digests. The ledger appended events
such as prepared, deployed, and verified; current state was derived by folding those events,
not by editing a mutable status row.

There were schemas, semantic checks, and accepted and rejected fixtures. The draft passed review.
The pull request still said that v0 would remain unfrozen until a real adapter had exercised it.

Defect 1: the contract expected a preview that did not exist

The first transition graph expected a preview environment before production.

This scheduled GitHub Pages deployment has no real preview tier. The workflow builds the static
dashboard and deploys it directly to the production Pages site.

I could have labeled a CI workspace “preview” and made the ledger satisfy the graph. That would
have been worse than an explicit gap: an evidence system would be changing the description of
reality to pass its own validator.

The contract changed instead.

A direct prepared -> deployed|production transition is now valid only when it carries
preview_skipped: true and a closed reason code. The same flag is rejected on the normal preview
path. The absence of a preview is an explicit property of the target, not a silently missing event.

GitHub Pages was not defective. The contract's assumption that every target had a preview was.

Defect 2: the manifest and bundle could not be finalized

The static site exposes a release-manifest.json containing a digest for each deployed path. A
read-back can fetch the manifest and compare a live file with the value it records.

The first design also made the release bundle carry the byte digest of that manifest, while the
manifest carried the digest of the bundle:

bundle
  contains sha256(release-manifest.json bytes)

release-manifest.json
  contains sha256(bundle)

The bundle required the finished manifest. The manifest required the finished bundle. Neither
value could be computed first.

The cycle looks obvious when written as two arrows. It was much less obvious while reviewing the
two record shapes separately. It surfaced when the real adapter had to generate files in an actual
order.

The deployed manifest now contains only its schema version and content map:

{
  "schema_version": "release-evidence/v0",
  "content": {
    "index.html": "sha256:..."
  }
}

It does not repeat the enclosing bundle digest. During production read-back, the adapter
canonicalizes the fetched content, computes its SHA-256 digest, compares that value with the
static-site artifact in the sealed bundle, and spot-checks live file bytes against the manifest.

Removing the mutual reference made the evidence flow one-directional.

I fixed both defects, then exercised production

On August 22, 2026, the corrected adapter ran in the real workflow.

Workflow run 32572501427
built the dashboard, deployed it to GitHub Pages, fetched the live manifest, compared its content
map, spot-checked a representative live file, and persisted the evidence ledger.

The ledger contains this sequence:

prepared
  -> deployed | production | preview_skipped=true
  -> verified | production

Recomputing the digest from the live manifest's canonicalized content produced the same value as
the artifact sealed in the bundle:

sha256:35e4b7ee5a17ca64f0851e54d83385f2c478b4d04e3210a2e08639fecd3b6d0c

Only after that receipt existed did the contract replace its synthetic dashboard fixture with the
real deployed bundle and freeze release-evidence/v0.

The order mattered:

review the draft
  -> let a real adapter break it
  -> repair the contract
  -> verify a production read-back
  -> freeze

Freezing first would have forced the adapter either to invent a preview or immediately violate the
contract I had just declared stable.

The second success verified the repeat-run link

After the first production exercise succeeded, I found another issue in the adapter. A later
release should derive previous_release_id from the last verified production event. The first
implementation still needed that repeat-run chain logic.

I keep this separate from the two contract defects. It did not change the transition model or the
digest definition. It was an adapter defect revealed by asking what the second release must do.

After that fix reached main, scheduled
run 32615972524
created a second production release from commit 6f0b5c8, which includes
PR #7.

The new bundle records the link explicitly:

release_id:
  agent-metrics-dashboard@32615972524-1

previous_release_id:
  agent-metrics-dashboard@32572501427-1

The first release was the preceding verified production release; its sealed bundle digest was
sha256:63392d1a635e68f670f07dcb717ad7e1437f8282be82a1dd8c0ab2c6a664c9e1.
The second release recorded prepared at 2026-08-23T03:41:48.031Z, deployed | production
at 2026-08-23T03:42:20.467Z, and verified | production at
2026-08-23T03:42:20.659Z.

Its ledger persisted the same three-event sequence under one sealed bundle digest:

prepared
  -> deployed | production | preview_skipped=true
  -> verified | production

The sealed bundle digest is
sha256:7dabd0a36be4308cb1c70387d1097dcb37c7fde3be3aee991f8287371bc661df.
Recomputing the static-site digest from the live manifest content produced
sha256:ac69214db9909b1f1d97bd1f806214034172e0dac780e1c1470434f158b70d49,
and the live index.html bytes matched the manifest entry
sha256:ce99db76efd55bace6b50bd9ba88caa76e1dbef841c76e87dcb07b7d88380c9e.

Auditing the persisted collection against the frozen v0 contract reported two bundles, six
events, and no problems. The repeat-run link is now observed in production. The order still
matters: the first success did not prove that future linkage until a second release exercised it.

Two later runs preserved the verified chain

Two subsequent scheduled runs created production releases from the same PR #7 merge commit.

Run	Release ID	Previous verified release	Sealed bundle digest
32687508604	`agent-metrics-dashboard@32687508604-1`	`agent-metrics-dashboard@32615972524-1`	`sha256:fd2ee5b9ef6d531b1c377257a60ced576a949964be186fd9b1c6337278b2a8a4`
32806015116	`agent-metrics-dashboard@32806015116-1`	`agent-metrics-dashboard@32687508604-1`	`sha256:94d9f3ee15e2238eb84dcae6dbe26bb4f69afd434ce8a1491be636e24d40b8e0`

The first follow-up chain recorded prepared at 2026-08-24T03:45:20.910Z, deployed | production at 2026-08-24T03:45:54.808Z, and verified | production at
2026-08-24T03:45:55.135Z. The second recorded the same sequence at
2026-08-25T03:40:22.482Z, 2026-08-25T03:40:53.785Z, and
2026-08-25T03:40:53.929Z. Within each chain, all three events carry the same sealed bundle
digest shown in the table. The Actions logs also record read-back verified against the
corresponding static-site artifact digest.

These runs add evidence that the previous_release_id chain and read-back continued to operate.
They do not prove that every future deployment will succeed, that every deployment is safe, or
that an external user adopted the tool.

Matching digests do not prove everything

The integrity level of release-evidence/v0 is digest_only.

There is no signature, SLSA, or SBOM claim. GitHub Actions credentials, pinned actions, the tool
checkout, and the production host remain inside the trust boundary.

A successful read-back shows that the live manifest's content map matched the static-site
artifact digest sealed in the bundle, and that a representative live file matched its manifest
entry at verification time.

It does not show that every production file was fetched and compared byte for byte, or that the
host cannot change afterward.

content integrity
!= semantic correctness

digest match
!= complete supply-chain security

successful read-back
!= future correctness

The v0 ledger assumes one writer. The first adapter is specific to a GitHub Pages/static-site
deployment. It is not a universal release database or an automatic promotion system.

Stating those limits does not weaken the receipt. It prevents a content-digest proof from silently
turning into a broader supply-chain claim.

Make reality answer before the contract becomes permanent

Schema review and synthetic fixtures were necessary. They remain valuable regression guards after
the fixes.

They were not enough to justify freezing a deployment contract.

Real targets have topology that diagrams omit. Artifacts have generation order that isolated
schemas hide. Production read-back crosses boundaries that a local build never touches.

The point was not to delay stability. It was to keep the contract breakable until the last cheap
moment to discover that it described something impossible.

The real adapter found a preview environment that did not exist and a digest cycle that could not
be computed. Both were contract problems, not cosmetic implementation details.

Before freezing a deployment-evidence schema, I now want one answer from the production path:
can an honest adapter actually satisfy this contract without inventing state?

For release-evidence/v0, asking that question before the freeze is what found the two defects.

AI-assistance disclosure

I used ChatGPT and Codex to help structure, edit, and translate this article. I reviewed the release IDs, digests, timestamps, implementation details, and claim boundaries against the recorded evidence before publication.

I Could Measure Claude and Codex Usage. I Still Couldn't Honestly Assign It to a Task.

Yusuke Shiki — Sat, 22 Aug 2026 21:20:35 +0000

Once you use Claude Code or Codex for real work, a total usage number stops being enough. You want to know which change consumed it.

I did not build agent-cost because I had missed the existing token and cost trackers. I knew about multi-agent reporting CLIs, local dashboards, and OpenTelemetry-style observability stacks. I had even built a similar view in Notion before.

The problem appeared when I tried to use that kind of reporting in an operational workflow. I needed agent logs to stay on the machine. I wanted a small runtime dependency surface, custom metrics I could audit, and a machine-readable result that another tool could consume. Most importantly, I needed session measurement and task attribution to remain two different claims.

I did not need another universal dashboard. I needed a boundary underneath the dashboard that could answer: is this number supported well enough to enter task accounting?

A measurement layer below the UI

Different tools optimize for different jobs. A broad CLI such as ccusage is useful when coverage across agents matters. Local interfaces such as token-tracker or AgentMeter are a better fit for visual exploration of projects, sessions, subagents, and tools. An OpenTelemetry stack is the natural choice for fleet-level metrics, logs, and traces.

Those are not inferior versions of agent-cost. They serve different use cases and trust models.

The layer I wanted looked like this:

local observations
  -> auditable normalized facts
  -> explicit pricing status
  -> caller-selected sessions
  -> task-attribution policy
  -> optional dashboard / Notion / spec-lane

agent-cost reads logs that Claude Code and Codex CLI have already written locally. It normalizes each usage event into a fact with a model, token kind, timestamp, and count. At runtime it makes no network calls and declares no Python runtime dependencies. Its price catalog has a version and SHA-256 digest, both carried into machine-readable output.

That “zero-network” claim is deliberately limited to runtime behavior. Installing from PyPI still means trusting a registry, installer, build backend, Python runtime, and operating system. The tool also needs access to the source logs. The design narrows runtime data egress and dependency surface; it does not make the supply chain disappear.

A session is observable. A task is another claim.

There is an attractive shortcut when building task-level cost reports:

Measure usage in a time window.
Find the issues or branches active during that window.
Apportion the total by working time or commit count.

This always produces numbers that add up. But the agreement comes from the allocation rule, not from an observation.

One session can cover several tasks. One task can span several sessions. A branch can stay unchanged while the operator investigates a different issue or reviews someone else's work. Elapsed time does not describe the computational weight of prompts and tool calls.

The invariant I wanted was:

Session usage is observable. Session-to-task attribution is a separate claim.

agent-cost measure accepts only session IDs selected by its caller:

agent-cost measure \
  --session-id <session-a> \
  --session-id <session-b> \
  --format json

It does not infer a task from a branch, pull request, or timestamp. A workflow that already owns the task-to-session binding passes the corresponding session set.

For example, the spec-lane adapter invokes agent-cost as a subprocess and checks the JSON, the measure/v1 protocol version, the schema, and forbidden personal dimensions. agent-cost does not learn what the task is. The caller that knows the task selects the sessions.

If a session crosses tasks and there is no defensible way to split it, I would rather leave that usage unattributed than manufacture a precise-looking allocation. Unknown is pending evidence, not zero.

The same rule applies to prices

agent-cost carries uncertainty instead of smoothing it away.

An unknown model is unpriced. A Claude cache write without a TTL breakdown is priced at the cheaper five-minute rate and labeled lower_bound. Codex logs do not expose cache-write tokens, so the tool does not invent a zero-valued cache-write row. Malformed events, unreadable files, and decreasing cumulative counters remain visible in data_quality.

“Fail closed” does not mean every imperfect input crashes the command. It means unsupported pricing or attribution does not quietly become a confirmed value downstream.

On August 23, 2026, I reran the published coding-agent-cost 0.1.0 package in temporary uvx directories. Its doctor command found the local sources and loaded catalog version 2026-07-29. The explicit unknown-model path still rejected a made-up model:

$ uvx --refresh --from coding-agent-cost \
    agent-cost rates show --model model-not-in-catalog
[unpriced] no rate entry for 'model-not-in-catalog'

A numeric zero next to an unpriced row is not a claim that the usage was free. Consumers must inspect pricing_status and unpriced_tokens, then choose a policy: exclude the value from a headline, stop the workflow, or supply a verified catalog.

The output field is estimated_cost_usd, not a bill. Allowances, contracts, credits, and batch usage are not fully recoverable from local logs. The number is a list-price estimate attached to observed tokens.

What the smaller boundary buys

There are intentional limits. agent-cost alone will not label a session as belonging to an issue. Local execution does not remove installation-time supply-chain risk or the need to trust local log access.

In exchange, each layer has a narrower claim:

Local logs support session usage facts.
A versioned catalog supports an estimated price.
Unsupported prices remain unpriced or lower_bound.
The caller owns task binding as separate evidence.
Ambiguous usage is not silently apportioned just to complete a total.

This is not an argument against dashboards. Use a dashboard when visual exploration is the job. Use OpenTelemetry when fleet observability is the job. Use a small accounting primitive when you need a subprocess contract and want measurement to remain separate from attribution policy.

Keeping an unknown visible is not a failure to measure. It is how the next layer avoids false confidence.

Start with the 60-second path in agent-cost. If you also need a workflow to own task attribution, see spec-lane.

Writing evidence-linked docs exposed two missing regression tests

Yusuke Shiki — Fri, 14 Aug 2026 23:15:18 +0000

This is a crosspost of the canonical version on GitHub.

1. Problem

I run agent-cost, a small open-source CLI that reads local Claude Code / Codex CLI usage logs and estimates token cost. I wanted documentation for it more trustworthy than a hand-written README: not prose claiming "the reader handles the TTL cache-write breakdown correctly," but claims that each point at the exact test or source line backing them, checked against real git history so the pointer can't silently go stale.

To do that I used a second tool I've been building, evidence-docs, to build a claim corpus against agent-cost: 17 individual observations grouped into 5 topics (pricing-catalog validation rules, the cache-write lower-bound behavior, unpriced-fact handling, decimal arithmetic for money, and the measure command's v1 contract). Each observation is one statement — a behavior, an invariant, a decision record — with a claim_kind, an epistemic_status, and one or more provenance entries naming the exact file, test, or spec section it's backed by, plus a content_digest of that source at a specific commit.

Building that corpus is what surfaced this post's actual finding. To write an honest provenance entry for a behavior claim, I had to point at the test that exercises it — reading the test, not just the source, for every claim. Doing that for the cache-write TTL breakdown logic and the pricing-status aggregation logic turned up two places where the claim I wanted to make ("the code does X in this case") was true by reading the code, but had no test asserting it. Both went into docs/claims/gaps.yaml as GAP-01 and GAP-02, and both became small, scoped pull requests that added regression tests with zero implementation changes.

2. Why it mattered

Neither gap was a live bug — that's the whole point of this post. agent-cost's existing test suite already covered the two extremes of each piece of logic; what was missing was the middle case, the one that's easy to reason yourself past while staring at the code.

GAP-01 (agent_cost/readers/claude.py, parse_session_facts): when a Claude Code usage event's cache_creation field is a dict, the reader computes leftover = cache_creation_input_tokens - (ephemeral_5m + ephemeral_1h) and emits it as a cache_write_unknown fact if positive. Existing tests covered a breakdown that exactly accounts for the total, and no breakdown at all — but not a breakdown dict that's present and partial, where the two counters sum to less than the total. That's a real, reachable path with no assertion on what it produces.
GAP-02 (agent_cost/aggregate.py, _STATUS_RANK / build_rows): a row's pricing_status should be the worst status among its facts, ordered unpriced (0) < lower_bound (1) < priced (2). The existing test only mixed an unpriced fact with a priced fact. No test built a row from a lower_bound fact (e.g. a cache_write_unknown token, priced as an explicit floor) mixed with a plain priced fact to confirm the row lands on lower_bound, not priced — the half of the ordering that isn't "obviously" covered by the unpriced case.

If either had silently broken in a later refactor — reordering _STATUS_RANK's values, or changing the leftover math — nothing in CI would have caught it. The cost-estimation output would have quietly started under- or over-reporting cost floors, without a single red test.

3. Why the obvious fix was insufficient

The obvious response to "I want good docs for this repo" is: write a good README, or write good docstrings, and be careful. I already had both — agent-cost's README has a "What this measures, and what it doesn't" section, and price_fact() has a docstring explaining the lower-bound design decision. Careful prose is necessary but not sufficient, for two reasons that showed up directly here:

Prose doesn't force you to check the thing it describes against a test. I could (and did, informally) describe the TTL-breakdown behavior accurately in a docstring without ever asking "is there a test where the breakdown dict is present but incomplete?" Writing prose doesn't create that question; writing a provenance entry that must name a specific test does.
A doc that "looks read" and a doc that's actually checked are indistinguishable from the outside. Nothing in a normal README tells a reader which sentences were verified by running something versus which were a best guess — every claim carries the same visual weight.

So the fix isn't "write better docs" in the abstract — it's structural: every behavior claim needs a machine-checkable pointer to what backs it, narrow enough that "no test covers this case" becomes visible while writing it, not something discovered later.

4. Invariant

evidence-docs enforces this as a schema-level invariant on every observation, not a style guideline. epistemic_status must be one of a fixed vocabulary (from execution_verified down to model_inference / single_source_observation / recorded_decision) — you have to say how strongly the claim was checked, not just assert it. provenance is a list of entries, each with a source_kind (source, test, spec, or review_memory), a repo-relative uri, a selector (which test/function/section), a content_digest, and the repo_commit the digest was taken against.

Neither field is optional, and both are validated structurally: unknown source_kind values are rejected outright, and content_digest is checked against the actual git blob at the declared commit, not the working tree. Two of agent-cost's own observations (OBS-004, the cache-write-TTL claim, and OBS-010, the pricing-status-ranking claim) exist because writing their provenance forced me to go find "the test that proves this," and in both cases the honest answer was "there isn't one for this specific sub-case" — which is what gaps.yaml records.

5. Design (evidence-docs, minimally)

An evidence-docs corpus is a directory (conventionally docs/claims/) with: id-registry.yaml (every topic_id/observation_id used anywhere must be pre-registered here, closing typo/duplicate ID bugs at generate time); topics/*.yaml and observations/*.yaml (one file per topic by convention, not enforced); and gaps.yaml, a ledger of gaps found (gap_found: true) or explicitly searched for and not found (gap_found: false), each tagged with a taxonomy — test_missing for both GAP-01 and GAP-02 here, independent_re_search_no_drift for a third entry, GAP-03, where I re-checked three README pricing claims against the current rates.json and found no drift, recorded as a negative result rather than omitted.

evidence-docs validate docs/claims --repo-commit <sha> runs full structural + provenance verification with no output written (CI-friendly); evidence-docs generate does the same and then deterministically writes a human-readable site/index.md and an AI-facing bundle/*.jsonl + manifest.json. evidence-docs context docs/claims --query '{"seeds": {"paths": [...]}, "token_budget": ...}' selects a relevant subset of claims from the bundle for a given set of source paths, for feeding into an LLM context window without shipping the whole corpus. --generated-at and --repo-commit are always explicit CLI arguments, never derived from datetime.now() or git rev-parse at run time, so the same corpus and arguments always produce byte-identical output.

6. Fail behavior

The parts of evidence-docs that made this finding possible are the parts designed to fail loudly on drift or forgery, not pass silently:

Digest verification targets the declared commit's git blob (git show <repo_commit>:<uri>, hashed with sha256), not the current worktree file. This closes a real bypass: checking only the worktree's current hash would still let someone edit the file, recompute the digest, and leave repo_commit on the old SHA. A worktree/declared-commit mismatch is a warning (repos move on after a snapshot); a mismatch against the declared commit's own blob is always fatal.
Unknown source_kind is a hard rejection, not a skip. An earlier design silently skipped unrecognized kinds — exactly what a forged or typo'd entry would want.
--repo-commit must equal every valid_at_commit and provenance[].repo_commit corpus-wide, with no exceptions, since a corpus is a snapshot of one commit.
Two independent path-traversal guards sit in front of the digest check, since git show never touches the filesystem and needs its own check separate from worktree path resolution.

None of this checks whether a statement is true (see Boundary, below) — but it does mean a fake or lazy provenance entry gets caught at validate time rather than trusted forever.

7. Verification: gap to merged test, with receipts

Both gaps followed the same path: recorded in gaps.yaml, then closed by a small PR that added a test and changed no implementation code.

GAP-01 → shiki-yusuke/agent-cost#2 ("test: cover the partial TTL-breakdown leftover branch in the Claude reader"), merged 2026-08-14. Adds test_cache_creation_partial_ttl_breakdown_leftover_is_unknown to tests/test_reader_claude.py: a cache_creation dict with ephemeral_5m_input_tokens=150 and ephemeral_1h_input_tokens=100 against a cache_creation_input_tokens total of 300, asserting the reader emits cache_write_5m=150, cache_write_1h=100, and cache_write_unknown=50 for the 50-token leftover. No implementation changes — the leftover math already behaved this way; the PR's own description says so.
GAP-02 → shiki-yusuke/agent-cost#3 ("test: pin the worst-status row aggregation order for pricing_status"), merged 2026-08-14. Adds four tests to tests/test_aggregate.py: test_build_rows_all_priced_facts_mark_row_priced (baseline), test_build_rows_mixed_priced_and_lower_bound_marks_row_lower_bound (the previously-untested half — a lower_bound fact plus a priced fact should mark the row lower_bound), test_build_rows_mixed_lower_bound_and_unpriced_marks_row_unpriced, and test_build_rows_worst_status_ranking_is_order_independent (same three facts constructed in three different orders, same resulting status). Again, no implementation changes.

Both PRs are public and merged; the diffs and CI runs are the actual evidence for this post, not my summary of them.

8. Boundary — what this does not show

To be precise about scope, since it's easy to over-read a post like this:

This is not a bug report. Both gaps were missing regression tests for code that already behaved correctly — nothing shipped broken, nothing was fixed except test coverage.
evidence-docs does not check whether a claim is true. Its own schema doc says so plainly: "the prose claim itself is never checked against the code... validation checks structure and provenance shape, not truth." What forced these two gaps into the open was my own process of trying to write an honest provenance pointer for each claim, not an automated truth-checker. A careless author could still write execution_verified next to a claim backed only by reading the source; the schema's negation_check field helps catch that but is optional and unenforced.
gaps.yaml completeness is not verified either. A corpus with zero recorded gaps is syntactically valid; whether the author actually looked for contradictions isn't something the tool can check. GAP-03 in this corpus (a re-check of three README pricing claims against rates.json that found no drift) exists because I chose to record a negative result, not because anything required it.
This is an n=1 experience report on one small repo, not an evaluation. No general effect size, no hit rate across repos, no claim that repeating this will surface N more gaps. It found two missing regression tests in one corpus-authoring pass over one repo I maintain. Trying the same process elsewhere might find zero, or something entirely different — the claim here is about what happened and how to reproduce the process, not what you should expect to get out of it.

9. Try it yourself

The minimal path that reproduces the "write a claim, discover you can't honestly back it" moment:

pip install evidence-docs

evidence-docs init docs/claims
# scaffolds topics/, observations/, id-registry.yaml, gaps.yaml, README.md

# ... author one observation for a behavior you believe is true, with a
#     provenance entry naming the exact test that proves it. If you can't
#     name one, that's the gap. ...

evidence-docs validate docs/claims --repo-commit "$(git rev-parse HEAD)"

validate will reject an unregistered ID, an unknown source_kind, or a digest that doesn't match the git blob at the commit you passed — all useful on their own — but the actual gap-finding step is upstream of the tool: it happens while you're trying to fill in the provenance field honestly, before you ever run validate.