DEV Community: Yusuke Shiki

The Cost Wasn't $0: My AI Coding Measurement Timed Out at 30 Seconds

Yusuke Shiki — Sun, 11 Oct 2026 18:04:38 +0000

I was recording usage and cost for AI coding sessions when I noticed something odd: one session had no cost entry at all.

My first assumption was that the measurement tool had failed. But when I ran agent-cost directly against the same Claude Code session, it completed successfully in 36.454 seconds. Through the spec-lane adapter, however, the same measurement failed after 30.019 seconds.

The reason was simple: the caller had a 30-second timeout.

What mattered more than the timeout itself was what happened afterward. The failed measurement did not produce a $0 record. It produced no measurement record at all.

That distinction became the main lesson from this incident:

Measurement failed != $0

Failing to measure something is not the same as measuring it as zero.

The same session succeeded directly and failed through the adapter

I first reran the session from Issue #42 directly with agent-cost:

agent-cost measure \
  --format json \
  --session-id <session-id>

The result:

exit: 0
wall: 36.454s
JSON parse: success
protocol: measure/v1

The measurement took longer than 30 seconds, but it completed normally.

I then passed the same session through an AgentCostTelemetryAdapter built from the pre-fix Git revision cca0338....

30.019s
TelemetryImportFailed
Command failed

This time it stopped around the 30-second mark.

For comparison:

Path	Result	Time
Direct `agent-cost measure`	Success	36.454s
Pre-fix adapter	Timeout	30.019s
Post-fix adapter	Success	35.410s

The pre-fix adapter had this default:

this.timeoutMs = opts.timeoutMs ?? 30_000;

So the underlying measurement could finish in roughly 36 seconds, while the caller terminated it after 30.

There was another problem: the error message was only Command failed. It did not say that a timeout had occurred or that the process had been stopped after 30 seconds.

From that message alone, it was difficult to distinguish a failure inside agent-cost from a timeout imposed by its caller.

No record is not a zero-dollar record

I wanted to know what a failed measurement left behind.

For this part, I used a separate controlled reproduction with a deliberately short timeout. This was not the same real session used in the timing comparison above.

When lane calibrate failed during measurement, the result was:

CalibrationObservation: 0
cost_ledger: 0

It did not write a record such as:

{
  "cost_usd": 0
}

There was simply no record.

The source code matched that behavior. If adapter.measure() fails, the command returns before generating the observation and before writing the cost ledger.

Running emit-metrics against the empty ledger produced:

built 0 record(s), coverage.status=no_data

Here, no_data does not mean "the measured cost was zero." It means there is no measured record available.

That matters once the numbers are aggregated. If ten failed measurements in a hundred tasks are silently converted to $0, the resulting average can look lower than the evidence actually supports.

Zero is a value. Missing measurement evidence is a different state.

The fix was more than changing 30 seconds to 180

PR #43 changed the default timeout from 30 seconds to 180 seconds:

DEFAULT_AGENT_COST_TIMEOUT_MS = 180_000

It also added a CLI override:

--agent-cost-timeout-ms <n>

The option is available for four commands:

calibrate
emit-metrics
next
usage-import

The accepted range is from 1ms through 3_600_000ms.

The timeout error was also made explicit. For example, a controlled 200ms timeout now reports:

timed out after 200 ms (killed with SIGTERM)

That makes it much easier to tell the difference between a measurement command failing on its own and the caller terminating it because of its deadline.

There is still a boundary here: 180 seconds is not a strict wall-clock guarantee. The implementation sends SIGTERM when the timeout is reached. A child process that does not terminate immediately after that signal can continue beyond the nominal timeout.

After the fix, a 38-second run persisted the measurement

Increasing the timeout is useful only if the measurement can actually finish and its evidence is saved.

I reran another historical session and lane snapshot using the v0.10.1 CLI with the default 180-second timeout.

exit: 0
wall: 38.01s

This time the process did more than return successfully:

CalibrationObservation: 1
cost_ledger: scope=lane
emit-metrics: coverage.status=complete

The observation was created, the cost ledger was written, and metrics coverage reached complete.

There is also a historical PR record reporting a successful run of roughly 39 seconds. I treat that separately: the raw log containing the exact historical wall time is no longer available, so that reported value is not the same evidence as the independently reproduced 38.01-second run.

Completing a measurement does not prove the number is correct

Another boundary became important while checking the fix.

What I verified here is that the measurement process can complete, return a protocol payload, and persist its result.

That does not prove all of the following:

that the estimated dollar amount is correct
that the pricing catalog is correct
that a session has been attributed to the correct task

In fact, the historical record and the new reproduction produced different token counts and estimated costs. They also used different agent-cost versions.

I did not investigate the cause of that difference as part of this timeout work.

Measurement completion, cost accuracy, and task attribution are separate claims. A successful run only establishes the first one.

`exit 0` can still mean measurement incomplete

There was one more edge case worth keeping.

calibrate exits with code 2 when measurement fails. usage-import, however, has a different contract.

In some measurement-failure cases it records:

measurement-incomplete
matched: false

and writes no ledger entry, while still returning exit 0.

So this is not a safe equivalence either:

exit 0 == measurement succeeded

The exit code, measurement state, and persisted evidence need to be interpreted separately.

Don't collapse unknown into zero

The immediate bug was a 30-second timeout. The more useful lesson for me was about state representation.

For measurement systems, these are not interchangeable:

0
unknown
failed
not measured

Collapsing all of them into zero makes aggregation easier. It also destroys information about what actually happened.

This applies beyond AI coding cost: test results, evaluation scores, usage metrics, and observability pipelines can all run into the same problem.

The rule I kept from Issue #42 is still the simplest description:

Measurement failed != $0

As more of an engineering workflow becomes automated, preserving the difference between "the value was zero" and "we never obtained a value" becomes more important, not less.

References

A note on the version reported in the original issue

Issue #42 records the historical runtime as:

lane 0.9.1, cca0338 on main

In Git, however, cca0338... is the peeled commit for the v0.10.0 tag, and packages/cli/package.json at that revision reports version 0.10.0.

The 30.019-second reproduction in this article was built from the cca0338... source revision.

I have not established why the historical executable reported 0.9.1, so I do not claim that the original incident occurred on the v0.10.0 CLI.

A YAML Comment Truncated an AI Coding Success Criterion — and the Gate Still Passed

Yusuke Shiki — Tue, 29 Sep 2026 01:37:56 +0000

When I hand implementation work to a coding agent, I want the definition of “done” to exist before the implementation does.

That usually means writing explicit success criteria, keeping them in a machine-readable form, and checking them again at a verification gate.

I do this in an open-source tool I maintain called spec-lane.

While dogfooding it, I ran into a small YAML detail with a much larger implication:

part of a success criterion was interpreted as a YAML comment before it ever reached the verification gate.

The more interesting part was that the gate still passed.

The bug was not in the string comparison itself. The information had already disappeared from the value being compared.

It started with one line of YAML

The minimal case reported in Issue #45 looked like this:

success:
  - ledger has exactly one PhaseGate row # include the negative case too

The intent described in the issue included the part after the #.

But this is an unquoted YAML plain scalar.

With yaml@2.9.0, it is parsed roughly as:

{
  "success": [
    "ledger has exactly one PhaseGate row"
  ]
}

So the two representations are different:

source:

ledger has exactly one PhaseGate row # include the negative case too


parsed value:

ledger has exactly one PhaseGate row

The original file has not been modified. The text after # is still physically present in the YAML file.

But it is a comment, so it is not part of the JavaScript value returned by yaml.parse().

That created a boundary I had not been checking closely enough:

what a human sees in the source
        ↓
      YAML parse
        ↓
what downstream code receives

The verification gate still passed

In spec-lane, the success criteria and the verification matrix are separate inputs.

The success criterion lives in intent.yaml.

The corresponding verification record lives in verification.yaml.

For the reproduction, the matrix contained the already-truncated form:

success_criteria_matrix:
  - criterion: ledger has exactly one PhaseGate row
    covered_by: test
    evidence: test/ledger.test.ts::records PhaseGate
    negation_test: no PhaseGate row fails

One important caveat: those evidence and negation_test values are declarations in the reproduction fixture.

For this reproduction, I did not create and execute the referenced ledger test and then prove that its behavior matched those strings.

The purpose here was to reproduce the gate behavior.

Reproducing the behavior before the fix

I built the CLI from the revision immediately before PR #48's fix and ran the reproduction in a temporary directory.

With the unquoted success criterion and the shortened matrix entry:

$ lane validate ...

intent.yaml is valid (phase=3_implement).

Exit code:

Then:

$ lane advance ... --phase 4_verify

Advanced ...: 3_implement -> 4_verify

Again:

exit code: 0

The verification phase was entered.

At that point, the two strings reaching the gate were effectively:

success:

ledger has exactly one PhaseGate row


matrix criterion:

ledger has exactly one PhaseGate row

They matched.

So the gate passed.

The important distinction is this:

the verification gate did not remove the text after #.

That had already happened at the earlier boundary:

YAML source
    ↓
parsed JS value

The gate was comparing the values it had been given, and those values were equal.

The relevant code paths are in:

packages/cli/src/intent-store.ts
packages/core/src/gate.ts

The full reproduction and investigation are documented in Issue #45:

https://github.com/shiki-yusuke/spec-lane/issues/45

Quoting the same criterion changed the result

I then changed only the success criterion so the full text became an explicit YAML string:

success:
  - "ledger has exactly one PhaseGate row # include the negative case too"

Now the parsed value retains everything:

success:

ledger has exactly one PhaseGate row # include the negative case too


matrix criterion:

ledger has exactly one PhaseGate row

With the same pre-fix CLI, both validate and advance now failed with exit code 3.

The gate reported that there was no matrix row corresponding to the full success criterion.

The phase remained 3_implement.

So the control case looked like this:

Success criterion	Matrix	validate	advance
Plain scalar; text after `#` is a comment	Short value	0	0
Quoted; full value is preserved	Short value	3	3

At least in this fixture, the result did not come from the gate failing to run.

The value reaching the gate changed.

That changed the outcome.

Schema-valid does not mean source-equivalent

The schema for this part of the success criteria is intentionally simple:

success: z.array(z.string()).min(1)

The shortened result:

ledger has exactly one PhaseGate row

is still a perfectly valid string.

So after parsing, these two source forms can result in the same value:

- ledger has exactly one PhaseGate row

and:

- ledger has exactly one PhaseGate row # include the negative case too

If a validator only receives the parsed value, it can no longer tell whether:

the author originally wrote the shorter criterion, or
additional source text became a YAML comment.

That distinction led me to separate several different claims that are easy to blur together:

the value has the correct shape
≠
the original source expression was preserved

And there are further boundaries after that:

the string was preserved
≠
the string correctly represents human intent

Likewise:

an evidence field contains a test name
≠
that test was actually executed and proved the claim

A schema validator can validate the structure it receives.

It cannot automatically recover information that disappeared before that boundary.

The v0.11.0 fix stops earlier

PR #48 did not change the success-criteria comparison itself.

Instead, the fix moved the check to the intent.yaml reading boundary.

The reason is straightforward: once only the parsed value remains, there is not enough information to distinguish the two source forms.

The implementation now roughly does this:

Walk the YAML AST and identify unquoted PLAIN scalars.
Use the scalar's source range to inspect the original text.
Check whether spaces or tabs immediately after the scalar are followed by # on the same line.
If such a candidate value appears in the parsed success criteria, reject the input before it reaches the gate.

The implementation is here:

https://github.com/shiki-yusuke/spec-lane/blob/64f2ca42c5929be0413e5f4f240897dffe56878b/packages/cli/src/intent-store.ts#L108-L164

The check deliberately does not rely only on the AST node's .comment property.

One reason is that certain anchor forms can associate a comment with a different node. The implementation therefore combines AST type and range information with the original source text.

With the v0.11.0 source, the same reproduction now behaves like this:

Success criterion	Matrix	validate	advance
Plain scalar + inline comment	Short value	2	2
Quoted full value	Full value	0	0

In the rejected case, the phase remains 3_implement.

If a literal # belongs in the success criterion, it can be made explicit by quoting the string:

success:
  - "ledger has exactly one PhaseGate row # include the negative case too"

The fix shipped in spec-lane v0.11.0:

https://github.com/shiki-yusuke/spec-lane/releases/tag/v0.11.0

PR #48 contains the implementation and regression tests:

https://github.com/shiki-yusuke/spec-lane/pull/48

This still does not mean the tool understands human intent

The fix is intentionally narrower than that.

It also has a known over-detection case.

Consider:

intent:
  business_goal: shared text # unrelated comment
  success:
    - "shared text"

Nothing was truncated from the quoted success criterion.

But the current check can still reject this document because another commented plain scalar has the same parsed value as the success criterion.

The implementation does not prove that the two occurrences share the same semantic origin.

That limitation is preserved in a regression test rather than hidden.

So I would not describe the fix as:

spec-lane now understands whether the author's full intent has been preserved.

It does not.

The design is closer to:

if the source contains a value that could have been silently shortened before it reaches a success criterion, stop instead of guessing.

That is a fail-closed choice.

It trades some false positives for avoiding this known silent-truncation path.

The part I now pay more attention to is the boundary

This incident changed how I think about verification in AI-assisted development.

It is tempting to think about the pipeline as:

specification
    ↓
implementation
    ↓
verification

But in practice there are more boundaries:

Human expression
what someone is trying to achieve
        ↓
Spec source
what was actually written
        ↓
Parsed representation
what the tool interpreted
        ↓
Verification
what the gate compared
        ↓
Execution evidence
what actually ran or was observed
        ↓
Acceptance
why the result was accepted

The failure in this post was mostly at:

Spec source
↓
Parsed representation

Everything after that can behave consistently and still verify less than a human reader thought had been written.

The same distinction appears elsewhere.

A test name written into a verification matrix is not the same thing as evidence that the test actually ran.

A command exiting successfully is not necessarily the same thing as the intended property being proved.

And a schema-valid specification is not necessarily the same thing as preserving every meaningful part of its source representation.

Machine-readable specifications are useful.

But once a workflow starts treating “the gate passed” as evidence of completion, I also want to know:

What exactly reached the gate, where did that value come from, and what transformations happened before it got there?

In this case, one small YAML line was enough to expose that boundary.

References

Issue #45 — reproduction and original report https://github.com/shiki-yusuke/spec-lane/issues/45
PR #48 — fix and regression tests https://github.com/shiki-yusuke/spec-lane/pull/48
spec-lane v0.11.0 https://github.com/shiki-yusuke/spec-lane/releases/tag/v0.11.0

AI-assisted disclosure: I used AI tools to help draft, translate, and edit this article. The reproduction, source inspection, and factual claims were checked against the linked issue, pull request, release, and the evidence collected during the investigation.

Tracking Codex Astra costs took more than updating a price table

Yusuke Shiki — Sun, 06 Sep 2026 01:10:55 +0000

I updated agent-cost, a CLI I maintain, to version 0.1.1 because I wanted to measure Codex's GPT-6 Astra usage too. It reads local Claude Code and Codex CLI logs and reports token usage and estimated costs, not actual billed amounts.

Fast mode was where I ran into trouble.

After adding Astra to the price table, I tested it with synthetic logs. Usage marked as Fast was still being priced at the Standard rate. The multiplier was in the catalog, but the log reader wasn't recognizing the setting.

The fix involved both detecting the mode and deciding what to do when the logs didn't give me enough information.

What the synthetic tests showed

I used synthetic data with two million uncached input tokens, two million cached input tokens, and two million output tokens: six million tokens in total.

Here are the results for Standard, Fast, and unknown settings, using the rate catalog bundled with 0.1.1. This is a test fixture, not real usage or a list of current official prices.

Setting identified from the log	Estimated USD	Estimated credits	Unpriced tokens
Standard	122	3,050	0
Fast	305	7,625	0
Unknown	0*	0*	6,000,000

The zero in the unknown row means none of that usage could be priced. **It does not mean the usage was free.*

The same token counts produce an estimate of $122 at Standard rates and $305 at Fast rates. During development, the Fast case was incorrectly returning the Standard estimate.

The already-published 0.1.0 release didn't have Astra rates at all, so it treated Astra usage as unpriced. This was a problem found while implementing Astra support, not a report that 0.1.0 had shown users the wrong Fast amount.

The changes are in PR #8. The synthetic fixture and tests are public too. The table above summarizes test results; it isn't a copy of the CLI's output.

Does "not known to be Fast" mean Standard?

I checked the public source for Codex 0.153.4.

For this Astra support, agent-cost reads service_tier as well as the model name. It checks that the settings belong to the same session and match the target turn and model. An explicit default selects Standard; priority selects Fast.

But a setting can be missing. It can also conflict with the information recorded for a turn. In those cases, the mode isn't clear.

Treating everything that isn't recognized as Fast as Standard would still produce a number. It would also assign Standard prices to usage whose settings the reader simply couldn't establish.

Instead, usage stays unpriced when the reader can't determine which rate applies. Knowing Astra's rates doesn't establish whether a particular piece of usage was Standard or Fast.

This is based on the request settings recorded in the log. It doesn't confirm the processing mode actually used by the server or what was billed. The source references and attribution rules are in the implementation notes.

Keep the tokens even when the price is unknown

Dropping an unpriced row would hide the fact that the usage happened. Keeping only a zero amount would make it look free.

So the output carries pricing_status, which describes the pricing state, and unpriced_tokens, which counts tokens that couldn't be priced, separately from the amount.

Checking those fields alongside the total shows whether any usage was left out of the calculation. The unknown row above doesn't just say $0: it preserves all six million tokens as unpriced.

priced doesn't mean a confirmed charge either. It means the usage could be calculated using the selected rate catalog.

Which usage gets the new setting when Fast is switched on?

Settings can also change during a session.

Suppose a session starts in Standard, uses Fast for the next turn, then switches back to Standard. Pricing the entire session with its final setting would treat the Fast portion as Standard too.

When settings change between completed turns, the reader applies the new settings only to subsequent usage. The log contains cumulative token counts, so it takes the increase since the previous count and prices that increase using the turn's settings.

An update while a turn is still running leaves an ambiguous interval: the reader can't safely assign that usage to the old or new settings. Those tokens stay unpriced as well.

This behavior was added in 0.1.1 for logs containing Astra. It doesn't fully reconstruct missing history, concurrent updates, or alternative storage formats. The supported scope is limited to the reader implementation and the cases checked with synthetic data.

Trying it locally

With uvx available, you can run the published 0.1.1 package with the commands below.

Astra coverage in this version: the model ID must be exactly gpt-6-astra, and usage must be at or after September 5, 2026, 17:18:23 UTC (September 6, 02:18:23 JST). Earlier usage remains unpriced, even when Standard or Fast can be identified.

This is a catalog boundary based on when the rates were checked, not an official launch or tariff start time. This release also does not update every model's rates: known discrepancies in the GPT-5.6 family remain.

uvx --from coding-agent-cost==0.1.1 agent-cost doctor
uvx --from coding-agent-cost==0.1.1 agent-cost report

doctor checks things such as the configured input locations. report actually reads the local logs and aggregates usage. Running doctor alone doesn't verify that every log is valid.

The CLI doesn't send the logs anywhere. The uvx package runner may use the network to download the package. To see output before reading your own logs, use the README's synthetic output example. That example remains pinned to 0.1.0 as a record of an earlier verification.

For use from another program, there is JSON output and the measure/v1 interface. The output includes the rate catalog's version and SHA-256, so a result can be traced back to the catalog used to calculate it. The README covers installation requirements and the timezone-data caveat on Windows.

The additional Astra checks used synthetic data. I haven't validated compatibility against real Astra logs or measured a cost reduction from using the tool. API-specific tariffs and actual charges under individual contracts are outside this calculation. The 0.1.1 release notes record the scope and verification results at publication.

Where I landed

An unknown price isn't a particularly satisfying result when you ran the tool to find out the cost.

But assigning a Standard rate when the settings couldn't be read would hide that uncertainty from the person using the result. For this Astra update, I kept the tokens even when they couldn't be priced, so the output shows how much usage the calculation actually covers.

This is the English version of my article on Zenn.

Before Asking an AI Coding Agent to Fix a Bug, Verify the Bug Exists

Yusuke Shiki — Thu, 03 Sep 2026 02:25:58 +0000

Before asking an AI coding agent to “fix this bug,” there is one question worth asking first:

Does the bug actually exist?

An issue says it does. A review points it out. Someone reports seeing it in another environment.

Any of those is a good reason to investigate. None of them, by itself, proves that the problem currently exists exactly as described.

It may already be fixed. It may only happen in a particular version or environment. The reported behavior may even be expected.

As I have started relying more on AI coding agents, I have become more deliberate about checking this step.

A wrong premise can still produce a perfectly reasonable implementation

Suppose you give an agent this task:

Problem X is happening. Find the cause and fix it.

The agent can inspect the code, reason under the assumption that X exists, and produce a plausible fix.

That is not necessarily an AI capability problem.

If the input says “X is happening,” solving the problem under that premise is a reasonable thing to do.

The part I want to question happens one step earlier.

Bug report
 ↓
Fix

becomes:

Bug report
 ↓
Does the reported problem actually occur?
 ↓
Fix

Before implementation, verify the premise the implementation depends on.

I had a review finding that changed after I measured it

This happened publicly while I was working on PR #28 in my OSS project, spec-lane.

The PR added a gate that can execute an external verification command. During review, a strong claim came up around timeout behavior:

Even with SIGKILL, the CLI timeout itself may hang.

If I had accepted that statement as the premise, the next task would have been straightforward: implement a fix for a hanging timeout.

Instead, I measured it first.

Under the three conditions I tested, I could not reproduce a call exceeding the configured deadline.

That did not mean there was no problem.

I found a different boundary.

Even after the direct child process exits, a descendant process that still holds the inherited stdout/stderr pipes can affect how long spawnSync waits.

In one measurement, the direct child exited immediately while a grandchild remained alive for four seconds. The call took 3112 ms.

But when the grandchild remained alive for ten seconds, a two-second timeout returned in 2035 ms, and a 0.5-second timeout returned in 501 ms.

In the conditions I measured, the configured deadline still held.

So these were two different statements:

Original strong claim:
SIGKILL can allow the timeout itself to hang past its deadline.

What I could actually observe:
Descendants can influence waiting and timeout reporting within the deadline.

They sound similar, but they lead to different implementation work.

I did not add a new “hang fix.”

Instead, I changed the specification and tests to describe the boundary I could actually measure.

Measuring the premise changed the problem I was about to implement.

One important qualification: this incident was not automatically stopped by spec-lane's premise_evidence gate. It was a manual review-and-measurement decision made while developing the project.

Put “verify” between “reported” and “fix”

I now think about the flow like this:

Issue / review finding
          ↓
Verify the premise
          ↓
      Reproduced?
       ↙       ↘
     No         Yes
     ↓           ↓
Stop or        Spec
re-scope         ↓
              Implement

If the issue reproduces, proceed with the specification and implementation.

If it does not, do not force yourself—or an agent—to find a fix for the problem you expected to see.

Sometimes the right answer is to stop.

Sometimes, as in PR #28, the investigation reveals a different, measurable problem. In that case, change the scope to the problem that actually exists.

This extra step matters more as implementation becomes cheaper.

spec-lane can record that decision before implementation

I wanted this decision to exist in the development workflow rather than only in someone's memory.

In spec-lane 0.9.0, an intent can contain an optional premise_evidence record.

For example, when you decide that a change requires premise verification but fail to reproduce the reported problem:

premise_evidence:
  required: true
  method: live
  reproduced: false
  evidence: "Reported steps were executed against the current build, but the reported behavior was not observed."

Here, required: true records the decision that premise verification applies to this change.

reproduced: false records that the premise was not confirmed.

With that current record, spec-lane refuses the 1_intent -> 2_spec transition. A failed lane advance does not modify lane-state.json.

The purpose is not to have a CLI magically detect whether an AI is wrong.

The purpose is simpler.

Once the workflow has explicitly recorded “we could not confirm this problem,” do not silently continue into specification and implementation as if it had been confirmed.

Recorded evidence is not the same thing as truth

There is an important limit to this mechanism.

Suppose the file says:

reproduced: true

The CLI can verify that true was recorded.

It cannot verify that someone honestly reproduced the bug.

The same applies to:

method: live

The CLI can verify the field and its value. It cannot know whether somebody actually inspected a live system.

spec-lane also enforces a minimum length for the evidence text. That is a structural threshold, not a quality score.

A longer evidence string is not automatically better evidence.

The boundary I use is:

CLI proves recorded shape and transition behavior, not real-world truth.

The system can mechanically inspect the record and decide whether a transition is allowed.

The truth of the real-world observation remains outside that guarantee.

Missing premise evidence does not automatically block a lane

premise_evidence itself is optional.

If it is absent, the current CLI emits a warning rather than a hard error. A warning alone does not prevent the transition to the next phase.

That is deliberate.

The CLI cannot determine whether every change actually requires premise verification.

A bug already observed directly by a human is different from an unverified report. A feature addition may not have a “does this bug exist?” premise at all.

Making the CLI decide applicability on its own would introduce another form of false certainty.

So spec-lane starts enforcing after a human or agent has decided that premise verification applies and recorded the result.

It is a mechanical backstop for an explicit decision, not an oracle.

Faster implementation makes the question before implementation more important

AI coding agents have made the implementation step dramatically faster.

That makes the decision immediately before implementation more consequential.

What are we changing?

Why are we changing it?

Does the problem this change depends on actually occur?

If that premise is wrong, faster implementation only lets us move in the wrong direction faster.

So before handing an issue or review finding to an agent and saying “fix it,” I now want one more step:

Does this problem actually exist?

If it reproduces, proceed.

If it does not, stop.

If the investigation reveals a different problem, change the scope to what you can actually observe.

That is the role premise evidence plays in spec-lane: preserve that decision before implementation and provide a mechanical backstop against ignoring an explicitly failed premise.

Project: shiki-yusuke/spec-lane

Japanese version: Zenn

When you use AI coding agents, where in your workflow do you verify that the problem they are about to fix is real?

A Real Deployment Exposed Two Contract Defects Before I Froze the Contract

Yusuke Shiki — Wed, 26 Aug 2026 10:14:59 +0000

A green CI run does not prove that the artifact built from reviewed source matches what production later exposes.

The build job may have produced one directory while the deploy job selected another. A file may have been added after review. The live manifest and representative file bytes may match the digests sealed at build time—or they may not.

I built release-evidence and its v0 contract to connect those claims. It records source and build provenance, seals artifact digests, appends deployment events, and checks a bounded part of the live production target.

But I did not freeze the first reviewed version of the contract.

Deployment contracts are unusually good at lying while every synthetic fixture stays green. A
schema can be internally consistent and still describe a topology that does not exist. Two digest
fields can both look reasonable until an adapter has to decide which one to compute first.

Keeping the contract in draft until a real adapter exercised it exposed two defects.

The goal was not another “build succeeded” record

The first target was a static dashboard deployed through GitHub Pages.

The evidence chain needed to support a narrower statement:

reviewed source
  -> one built artifact
  -> that artifact deployed to production
  -> live manifest read back
     + representative file bytes spot-checked

The draft contract used a sealed release bundle and an append-only event ledger. The bundle held
the source commit and tree, toolchain evidence, and artifact digests. The ledger appended events
such as prepared, deployed, and verified; current state was derived by folding those events,
not by editing a mutable status row.

There were schemas, semantic checks, and accepted and rejected fixtures. The draft passed review.
The pull request still said that v0 would remain unfrozen until a real adapter had exercised it.

Defect 1: the contract expected a preview that did not exist

The first transition graph expected a preview environment before production.

This scheduled GitHub Pages deployment has no real preview tier. The workflow builds the static
dashboard and deploys it directly to the production Pages site.

I could have labeled a CI workspace “preview” and made the ledger satisfy the graph. That would
have been worse than an explicit gap: an evidence system would be changing the description of
reality to pass its own validator.

The contract changed instead.

A direct prepared -> deployed|production transition is now valid only when it carries
preview_skipped: true and a closed reason code. The same flag is rejected on the normal preview
path. The absence of a preview is an explicit property of the target, not a silently missing event.

GitHub Pages was not defective. The contract's assumption that every target had a preview was.

Defect 2: the manifest and bundle could not be finalized

The static site exposes a release-manifest.json containing a digest for each deployed path. A
read-back can fetch the manifest and compare a live file with the value it records.

The first design also made the release bundle carry the byte digest of that manifest, while the
manifest carried the digest of the bundle:

bundle
  contains sha256(release-manifest.json bytes)

release-manifest.json
  contains sha256(bundle)

The bundle required the finished manifest. The manifest required the finished bundle. Neither
value could be computed first.

The cycle looks obvious when written as two arrows. It was much less obvious while reviewing the
two record shapes separately. It surfaced when the real adapter had to generate files in an actual
order.

The deployed manifest now contains only its schema version and content map:

{
  "schema_version": "release-evidence/v0",
  "content": {
    "index.html": "sha256:..."
  }
}

It does not repeat the enclosing bundle digest. During production read-back, the adapter
canonicalizes the fetched content, computes its SHA-256 digest, compares that value with the
static-site artifact in the sealed bundle, and spot-checks live file bytes against the manifest.

Removing the mutual reference made the evidence flow one-directional.

I fixed both defects, then exercised production

On August 22, 2026, the corrected adapter ran in the real workflow.

Workflow run 32572501427
built the dashboard, deployed it to GitHub Pages, fetched the live manifest, compared its content
map, spot-checked a representative live file, and persisted the evidence ledger.

The ledger contains this sequence:

prepared
  -> deployed | production | preview_skipped=true
  -> verified | production

Recomputing the digest from the live manifest's canonicalized content produced the same value as
the artifact sealed in the bundle:

sha256:35e4b7ee5a17ca64f0851e54d83385f2c478b4d04e3210a2e08639fecd3b6d0c

Only after that receipt existed did the contract replace its synthetic dashboard fixture with the
real deployed bundle and freeze release-evidence/v0.

The order mattered:

review the draft
  -> let a real adapter break it
  -> repair the contract
  -> verify a production read-back
  -> freeze

Freezing first would have forced the adapter either to invent a preview or immediately violate the
contract I had just declared stable.

The second success verified the repeat-run link

After the first production exercise succeeded, I found another issue in the adapter. A later
release should derive previous_release_id from the last verified production event. The first
implementation still needed that repeat-run chain logic.

I keep this separate from the two contract defects. It did not change the transition model or the
digest definition. It was an adapter defect revealed by asking what the second release must do.

After that fix reached main, scheduled
run 32615972524
created a second production release from commit 6f0b5c8, which includes
PR #7.

The new bundle records the link explicitly:

release_id:
  agent-metrics-dashboard@32615972524-1

previous_release_id:
  agent-metrics-dashboard@32572501427-1

The first release was the preceding verified production release; its sealed bundle digest was
sha256:63392d1a635e68f670f07dcb717ad7e1437f8282be82a1dd8c0ab2c6a664c9e1.
The second release recorded prepared at 2026-08-23T03:41:48.031Z, deployed | production
at 2026-08-23T03:42:20.467Z, and verified | production at
2026-08-23T03:42:20.659Z.

Its ledger persisted the same three-event sequence under one sealed bundle digest:

prepared
  -> deployed | production | preview_skipped=true
  -> verified | production

The sealed bundle digest is
sha256:7dabd0a36be4308cb1c70387d1097dcb37c7fde3be3aee991f8287371bc661df.
Recomputing the static-site digest from the live manifest content produced
sha256:ac69214db9909b1f1d97bd1f806214034172e0dac780e1c1470434f158b70d49,
and the live index.html bytes matched the manifest entry
sha256:ce99db76efd55bace6b50bd9ba88caa76e1dbef841c76e87dcb07b7d88380c9e.

Auditing the persisted collection against the frozen v0 contract reported two bundles, six
events, and no problems. The repeat-run link is now observed in production. The order still
matters: the first success did not prove that future linkage until a second release exercised it.

Two later runs preserved the verified chain

Two subsequent scheduled runs created production releases from the same PR #7 merge commit.

Run	Release ID	Previous verified release	Sealed bundle digest
32687508604	`agent-metrics-dashboard@32687508604-1`	`agent-metrics-dashboard@32615972524-1`	`sha256:fd2ee5b9ef6d531b1c377257a60ced576a949964be186fd9b1c6337278b2a8a4`
32806015116	`agent-metrics-dashboard@32806015116-1`	`agent-metrics-dashboard@32687508604-1`	`sha256:94d9f3ee15e2238eb84dcae6dbe26bb4f69afd434ce8a1491be636e24d40b8e0`

The first follow-up chain recorded prepared at 2026-08-24T03:45:20.910Z, deployed | production at 2026-08-24T03:45:54.808Z, and verified | production at
2026-08-24T03:45:55.135Z. The second recorded the same sequence at
2026-08-25T03:40:22.482Z, 2026-08-25T03:40:53.785Z, and
2026-08-25T03:40:53.929Z. Within each chain, all three events carry the same sealed bundle
digest shown in the table. The Actions logs also record read-back verified against the
corresponding static-site artifact digest.

These runs add evidence that the previous_release_id chain and read-back continued to operate.
They do not prove that every future deployment will succeed, that every deployment is safe, or
that an external user adopted the tool.

Matching digests do not prove everything

The integrity level of release-evidence/v0 is digest_only.

There is no signature, SLSA, or SBOM claim. GitHub Actions credentials, pinned actions, the tool
checkout, and the production host remain inside the trust boundary.

A successful read-back shows that the live manifest's content map matched the static-site
artifact digest sealed in the bundle, and that a representative live file matched its manifest
entry at verification time.

It does not show that every production file was fetched and compared byte for byte, or that the
host cannot change afterward.

content integrity
!= semantic correctness

digest match
!= complete supply-chain security

successful read-back
!= future correctness

The v0 ledger assumes one writer. The first adapter is specific to a GitHub Pages/static-site
deployment. It is not a universal release database or an automatic promotion system.

Stating those limits does not weaken the receipt. It prevents a content-digest proof from silently
turning into a broader supply-chain claim.

Make reality answer before the contract becomes permanent

Schema review and synthetic fixtures were necessary. They remain valuable regression guards after
the fixes.

They were not enough to justify freezing a deployment contract.

Real targets have topology that diagrams omit. Artifacts have generation order that isolated
schemas hide. Production read-back crosses boundaries that a local build never touches.

The point was not to delay stability. It was to keep the contract breakable until the last cheap
moment to discover that it described something impossible.

The real adapter found a preview environment that did not exist and a digest cycle that could not
be computed. Both were contract problems, not cosmetic implementation details.

Before freezing a deployment-evidence schema, I now want one answer from the production path:
can an honest adapter actually satisfy this contract without inventing state?

For release-evidence/v0, asking that question before the freeze is what found the two defects.

AI-assistance disclosure

I used ChatGPT and Codex to help structure, edit, and translate this article. I reviewed the release IDs, digests, timestamps, implementation details, and claim boundaries against the recorded evidence before publication.

I Could Measure Claude and Codex Usage. I Still Couldn't Honestly Assign It to a Task.

Yusuke Shiki — Sat, 22 Aug 2026 21:20:35 +0000

Once you use Claude Code or Codex for real work, a total usage number stops being enough. You want to know which change consumed it.

I did not build agent-cost because I had missed the existing token and cost trackers. I knew about multi-agent reporting CLIs, local dashboards, and OpenTelemetry-style observability stacks. I had even built a similar view in Notion before.

The problem appeared when I tried to use that kind of reporting in an operational workflow. I needed agent logs to stay on the machine. I wanted a small runtime dependency surface, custom metrics I could audit, and a machine-readable result that another tool could consume. Most importantly, I needed session measurement and task attribution to remain two different claims.

I did not need another universal dashboard. I needed a boundary underneath the dashboard that could answer: is this number supported well enough to enter task accounting?

A measurement layer below the UI

Different tools optimize for different jobs. A broad CLI such as ccusage is useful when coverage across agents matters. Local interfaces such as token-tracker or AgentMeter are a better fit for visual exploration of projects, sessions, subagents, and tools. An OpenTelemetry stack is the natural choice for fleet-level metrics, logs, and traces.

Those are not inferior versions of agent-cost. They serve different use cases and trust models.

The layer I wanted looked like this:

local observations
  -> auditable normalized facts
  -> explicit pricing status
  -> caller-selected sessions
  -> task-attribution policy
  -> optional dashboard / Notion / spec-lane

agent-cost reads logs that Claude Code and Codex CLI have already written locally. It normalizes each usage event into a fact with a model, token kind, timestamp, and count. At runtime it makes no network calls and declares no Python runtime dependencies. Its price catalog has a version and SHA-256 digest, both carried into machine-readable output.

That “zero-network” claim is deliberately limited to runtime behavior. Installing from PyPI still means trusting a registry, installer, build backend, Python runtime, and operating system. The tool also needs access to the source logs. The design narrows runtime data egress and dependency surface; it does not make the supply chain disappear.

A session is observable. A task is another claim.

There is an attractive shortcut when building task-level cost reports:

Measure usage in a time window.
Find the issues or branches active during that window.
Apportion the total by working time or commit count.

This always produces numbers that add up. But the agreement comes from the allocation rule, not from an observation.

One session can cover several tasks. One task can span several sessions. A branch can stay unchanged while the operator investigates a different issue or reviews someone else's work. Elapsed time does not describe the computational weight of prompts and tool calls.

The invariant I wanted was:

Session usage is observable. Session-to-task attribution is a separate claim.

agent-cost measure accepts only session IDs selected by its caller:

agent-cost measure \
  --session-id <session-a> \
  --session-id <session-b> \
  --format json

It does not infer a task from a branch, pull request, or timestamp. A workflow that already owns the task-to-session binding passes the corresponding session set.

For example, the spec-lane adapter invokes agent-cost as a subprocess and checks the JSON, the measure/v1 protocol version, the schema, and forbidden personal dimensions. agent-cost does not learn what the task is. The caller that knows the task selects the sessions.

If a session crosses tasks and there is no defensible way to split it, I would rather leave that usage unattributed than manufacture a precise-looking allocation. Unknown is pending evidence, not zero.

The same rule applies to prices

agent-cost carries uncertainty instead of smoothing it away.

An unknown model is unpriced. A Claude cache write without a TTL breakdown is priced at the cheaper five-minute rate and labeled lower_bound. Codex logs do not expose cache-write tokens, so the tool does not invent a zero-valued cache-write row. Malformed events, unreadable files, and decreasing cumulative counters remain visible in data_quality.

“Fail closed” does not mean every imperfect input crashes the command. It means unsupported pricing or attribution does not quietly become a confirmed value downstream.

On August 23, 2026, I reran the published coding-agent-cost 0.1.0 package in temporary uvx directories. Its doctor command found the local sources and loaded catalog version 2026-07-29. The explicit unknown-model path still rejected a made-up model:

$ uvx --refresh --from coding-agent-cost \
    agent-cost rates show --model model-not-in-catalog
[unpriced] no rate entry for 'model-not-in-catalog'

A numeric zero next to an unpriced row is not a claim that the usage was free. Consumers must inspect pricing_status and unpriced_tokens, then choose a policy: exclude the value from a headline, stop the workflow, or supply a verified catalog.

The output field is estimated_cost_usd, not a bill. Allowances, contracts, credits, and batch usage are not fully recoverable from local logs. The number is a list-price estimate attached to observed tokens.

What the smaller boundary buys

There are intentional limits. agent-cost alone will not label a session as belonging to an issue. Local execution does not remove installation-time supply-chain risk or the need to trust local log access.

In exchange, each layer has a narrower claim:

Local logs support session usage facts.
A versioned catalog supports an estimated price.
Unsupported prices remain unpriced or lower_bound.
The caller owns task binding as separate evidence.
Ambiguous usage is not silently apportioned just to complete a total.

This is not an argument against dashboards. Use a dashboard when visual exploration is the job. Use OpenTelemetry when fleet observability is the job. Use a small accounting primitive when you need a subprocess contract and want measurement to remain separate from attribution policy.

Keeping an unknown visible is not a failure to measure. It is how the next layer avoids false confidence.

Start with the 60-second path in agent-cost. If you also need a workflow to own task attribution, see spec-lane.

Writing evidence-linked docs exposed two missing regression tests

Yusuke Shiki — Fri, 14 Aug 2026 23:15:18 +0000

This is a crosspost of the canonical version on GitHub.

1. Problem

I run agent-cost, a small open-source CLI that reads local Claude Code / Codex CLI usage logs and estimates token cost. I wanted documentation for it more trustworthy than a hand-written README: not prose claiming "the reader handles the TTL cache-write breakdown correctly," but claims that each point at the exact test or source line backing them, checked against real git history so the pointer can't silently go stale.

To do that I used a second tool I've been building, evidence-docs, to build a claim corpus against agent-cost: 17 individual observations grouped into 5 topics (pricing-catalog validation rules, the cache-write lower-bound behavior, unpriced-fact handling, decimal arithmetic for money, and the measure command's v1 contract). Each observation is one statement — a behavior, an invariant, a decision record — with a claim_kind, an epistemic_status, and one or more provenance entries naming the exact file, test, or spec section it's backed by, plus a content_digest of that source at a specific commit.

Building that corpus is what surfaced this post's actual finding. To write an honest provenance entry for a behavior claim, I had to point at the test that exercises it — reading the test, not just the source, for every claim. Doing that for the cache-write TTL breakdown logic and the pricing-status aggregation logic turned up two places where the claim I wanted to make ("the code does X in this case") was true by reading the code, but had no test asserting it. Both went into docs/claims/gaps.yaml as GAP-01 and GAP-02, and both became small, scoped pull requests that added regression tests with zero implementation changes.

2. Why it mattered

Neither gap was a live bug — that's the whole point of this post. agent-cost's existing test suite already covered the two extremes of each piece of logic; what was missing was the middle case, the one that's easy to reason yourself past while staring at the code.

GAP-01 (agent_cost/readers/claude.py, parse_session_facts): when a Claude Code usage event's cache_creation field is a dict, the reader computes leftover = cache_creation_input_tokens - (ephemeral_5m + ephemeral_1h) and emits it as a cache_write_unknown fact if positive. Existing tests covered a breakdown that exactly accounts for the total, and no breakdown at all — but not a breakdown dict that's present and partial, where the two counters sum to less than the total. That's a real, reachable path with no assertion on what it produces.
GAP-02 (agent_cost/aggregate.py, _STATUS_RANK / build_rows): a row's pricing_status should be the worst status among its facts, ordered unpriced (0) < lower_bound (1) < priced (2). The existing test only mixed an unpriced fact with a priced fact. No test built a row from a lower_bound fact (e.g. a cache_write_unknown token, priced as an explicit floor) mixed with a plain priced fact to confirm the row lands on lower_bound, not priced — the half of the ordering that isn't "obviously" covered by the unpriced case.

If either had silently broken in a later refactor — reordering _STATUS_RANK's values, or changing the leftover math — nothing in CI would have caught it. The cost-estimation output would have quietly started under- or over-reporting cost floors, without a single red test.

3. Why the obvious fix was insufficient

The obvious response to "I want good docs for this repo" is: write a good README, or write good docstrings, and be careful. I already had both — agent-cost's README has a "What this measures, and what it doesn't" section, and price_fact() has a docstring explaining the lower-bound design decision. Careful prose is necessary but not sufficient, for two reasons that showed up directly here:

Prose doesn't force you to check the thing it describes against a test. I could (and did, informally) describe the TTL-breakdown behavior accurately in a docstring without ever asking "is there a test where the breakdown dict is present but incomplete?" Writing prose doesn't create that question; writing a provenance entry that must name a specific test does.
A doc that "looks read" and a doc that's actually checked are indistinguishable from the outside. Nothing in a normal README tells a reader which sentences were verified by running something versus which were a best guess — every claim carries the same visual weight.

So the fix isn't "write better docs" in the abstract — it's structural: every behavior claim needs a machine-checkable pointer to what backs it, narrow enough that "no test covers this case" becomes visible while writing it, not something discovered later.

4. Invariant

evidence-docs enforces this as a schema-level invariant on every observation, not a style guideline. epistemic_status must be one of a fixed vocabulary (from execution_verified down to model_inference / single_source_observation / recorded_decision) — you have to say how strongly the claim was checked, not just assert it. provenance is a list of entries, each with a source_kind (source, test, spec, or review_memory), a repo-relative uri, a selector (which test/function/section), a content_digest, and the repo_commit the digest was taken against.

Neither field is optional, and both are validated structurally: unknown source_kind values are rejected outright, and content_digest is checked against the actual git blob at the declared commit, not the working tree. Two of agent-cost's own observations (OBS-004, the cache-write-TTL claim, and OBS-010, the pricing-status-ranking claim) exist because writing their provenance forced me to go find "the test that proves this," and in both cases the honest answer was "there isn't one for this specific sub-case" — which is what gaps.yaml records.

5. Design (evidence-docs, minimally)

An evidence-docs corpus is a directory (conventionally docs/claims/) with: id-registry.yaml (every topic_id/observation_id used anywhere must be pre-registered here, closing typo/duplicate ID bugs at generate time); topics/*.yaml and observations/*.yaml (one file per topic by convention, not enforced); and gaps.yaml, a ledger of gaps found (gap_found: true) or explicitly searched for and not found (gap_found: false), each tagged with a taxonomy — test_missing for both GAP-01 and GAP-02 here, independent_re_search_no_drift for a third entry, GAP-03, where I re-checked three README pricing claims against the current rates.json and found no drift, recorded as a negative result rather than omitted.

evidence-docs validate docs/claims --repo-commit <sha> runs full structural + provenance verification with no output written (CI-friendly); evidence-docs generate does the same and then deterministically writes a human-readable site/index.md and an AI-facing bundle/*.jsonl + manifest.json. evidence-docs context docs/claims --query '{"seeds": {"paths": [...]}, "token_budget": ...}' selects a relevant subset of claims from the bundle for a given set of source paths, for feeding into an LLM context window without shipping the whole corpus. --generated-at and --repo-commit are always explicit CLI arguments, never derived from datetime.now() or git rev-parse at run time, so the same corpus and arguments always produce byte-identical output.

6. Fail behavior

The parts of evidence-docs that made this finding possible are the parts designed to fail loudly on drift or forgery, not pass silently:

Digest verification targets the declared commit's git blob (git show <repo_commit>:<uri>, hashed with sha256), not the current worktree file. This closes a real bypass: checking only the worktree's current hash would still let someone edit the file, recompute the digest, and leave repo_commit on the old SHA. A worktree/declared-commit mismatch is a warning (repos move on after a snapshot); a mismatch against the declared commit's own blob is always fatal.
Unknown source_kind is a hard rejection, not a skip. An earlier design silently skipped unrecognized kinds — exactly what a forged or typo'd entry would want.
--repo-commit must equal every valid_at_commit and provenance[].repo_commit corpus-wide, with no exceptions, since a corpus is a snapshot of one commit.
Two independent path-traversal guards sit in front of the digest check, since git show never touches the filesystem and needs its own check separate from worktree path resolution.

None of this checks whether a statement is true (see Boundary, below) — but it does mean a fake or lazy provenance entry gets caught at validate time rather than trusted forever.

7. Verification: gap to merged test, with receipts

Both gaps followed the same path: recorded in gaps.yaml, then closed by a small PR that added a test and changed no implementation code.

GAP-01 → shiki-yusuke/agent-cost#2 ("test: cover the partial TTL-breakdown leftover branch in the Claude reader"), merged 2026-08-14. Adds test_cache_creation_partial_ttl_breakdown_leftover_is_unknown to tests/test_reader_claude.py: a cache_creation dict with ephemeral_5m_input_tokens=150 and ephemeral_1h_input_tokens=100 against a cache_creation_input_tokens total of 300, asserting the reader emits cache_write_5m=150, cache_write_1h=100, and cache_write_unknown=50 for the 50-token leftover. No implementation changes — the leftover math already behaved this way; the PR's own description says so.
GAP-02 → shiki-yusuke/agent-cost#3 ("test: pin the worst-status row aggregation order for pricing_status"), merged 2026-08-14. Adds four tests to tests/test_aggregate.py: test_build_rows_all_priced_facts_mark_row_priced (baseline), test_build_rows_mixed_priced_and_lower_bound_marks_row_lower_bound (the previously-untested half — a lower_bound fact plus a priced fact should mark the row lower_bound), test_build_rows_mixed_lower_bound_and_unpriced_marks_row_unpriced, and test_build_rows_worst_status_ranking_is_order_independent (same three facts constructed in three different orders, same resulting status). Again, no implementation changes.

Both PRs are public and merged; the diffs and CI runs are the actual evidence for this post, not my summary of them.

8. Boundary — what this does not show

To be precise about scope, since it's easy to over-read a post like this:

This is not a bug report. Both gaps were missing regression tests for code that already behaved correctly — nothing shipped broken, nothing was fixed except test coverage.
evidence-docs does not check whether a claim is true. Its own schema doc says so plainly: "the prose claim itself is never checked against the code... validation checks structure and provenance shape, not truth." What forced these two gaps into the open was my own process of trying to write an honest provenance pointer for each claim, not an automated truth-checker. A careless author could still write execution_verified next to a claim backed only by reading the source; the schema's negation_check field helps catch that but is optional and unenforced.
gaps.yaml completeness is not verified either. A corpus with zero recorded gaps is syntactically valid; whether the author actually looked for contradictions isn't something the tool can check. GAP-03 in this corpus (a re-check of three README pricing claims against rates.json that found no drift) exists because I chose to record a negative result, not because anything required it.
This is an n=1 experience report on one small repo, not an evaluation. No general effect size, no hit rate across repos, no claim that repeating this will surface N more gaps. It found two missing regression tests in one corpus-authoring pass over one repo I maintain. Trying the same process elsewhere might find zero, or something entirely different — the claim here is about what happened and how to reproduce the process, not what you should expect to get out of it.

9. Try it yourself

The minimal path that reproduces the "write a claim, discover you can't honestly back it" moment:

pip install evidence-docs

evidence-docs init docs/claims
# scaffolds topics/, observations/, id-registry.yaml, gaps.yaml, README.md

# ... author one observation for a behavior you believe is true, with a
#     provenance entry naming the exact test that proves it. If you can't
#     name one, that's the gap. ...

evidence-docs validate docs/claims --repo-commit "$(git rev-parse HEAD)"

validate will reject an unregistered ID, an unknown source_kind, or a digest that doesn't match the git blob at the commit you passed — all useful on their own — but the actual gap-finding step is upstream of the tool: it happens while you're trying to fill in the provenance field honestly, before you ever run validate.

DEV Community: Yusuke Shiki

The Cost Wasn't $0: My AI Coding Measurement Timed Out at 30 Seconds

The same session succeeded directly and failed through the adapter

No record is not a zero-dollar record

The fix was more than changing 30 seconds to 180

After the fix, a 38-second run persisted the measurement

Completing a measurement does not prove the number is correct

exit 0 can still mean measurement incomplete

Don't collapse unknown into zero

References

A note on the version reported in the original issue

A YAML Comment Truncated an AI Coding Success Criterion — and the Gate Still Passed

It started with one line of YAML

The verification gate still passed

Reproducing the behavior before the fix

Quoting the same criterion changed the result

Schema-valid does not mean source-equivalent

The v0.11.0 fix stops earlier

This still does not mean the tool understands human intent

The part I now pay more attention to is the boundary

References

Tracking Codex Astra costs took more than updating a price table

What the synthetic tests showed

Does "not known to be Fast" mean Standard?

Keep the tokens even when the price is unknown

Which usage gets the new setting when Fast is switched on?

Trying it locally

Where I landed

Before Asking an AI Coding Agent to Fix a Bug, Verify the Bug Exists

A wrong premise can still produce a perfectly reasonable implementation

I had a review finding that changed after I measured it

Put “verify” between “reported” and “fix”

spec-lane can record that decision before implementation

Recorded evidence is not the same thing as truth

Missing premise evidence does not automatically block a lane

Faster implementation makes the question before implementation more important

A Real Deployment Exposed Two Contract Defects Before I Froze the Contract

The goal was not another “build succeeded” record

Defect 1: the contract expected a preview that did not exist

Defect 2: the manifest and bundle could not be finalized

I fixed both defects, then exercised production

The second success verified the repeat-run link

Two later runs preserved the verified chain

Matching digests do not prove everything

Make reality answer before the contract becomes permanent

I Could Measure Claude and Codex Usage. I Still Couldn't Honestly Assign It to a Task.

A measurement layer below the UI

A session is observable. A task is another claim.

The same rule applies to prices

What the smaller boundary buys

Writing evidence-linked docs exposed two missing regression tests

1. Problem

2. Why it mattered

3. Why the obvious fix was insufficient

4. Invariant

5. Design (evidence-docs, minimally)

6. Fail behavior

7. Verification: gap to merged test, with receipts

8. Boundary — what this does not show

9. Try it yourself

Links

`exit 0` can still mean measurement incomplete