A YAML Comment Truncated an AI Coding Success Criterion — and the Gate Still Passed

When I hand implementation work to a coding agent, I want the definition of “done” to exist before the implementation does.

That usually means writing explicit success criteria, keeping them in a machine-readable form, and checking them again at a verification gate.

I do this in an open-source tool I maintain called spec-lane.

While dogfooding it, I ran into a small YAML detail with a much larger implication:

part of a success criterion was interpreted as a YAML comment before it ever reached the verification gate.

The more interesting part was that the gate still passed.

The bug was not in the string comparison itself. The information had already disappeared from the value being compared.

It started with one line of YAML

The minimal case reported in Issue #45 looked like this:

success:
  - ledger has exactly one PhaseGate row # include the negative case too

The intent described in the issue included the part after the #.

But this is an unquoted YAML plain scalar.

With yaml@2.9.0, it is parsed roughly as:

{
  "success": [
    "ledger has exactly one PhaseGate row"
  ]
}

So the two representations are different:

source:

ledger has exactly one PhaseGate row # include the negative case too


parsed value:

ledger has exactly one PhaseGate row

The original file has not been modified. The text after # is still physically present in the YAML file.

But it is a comment, so it is not part of the JavaScript value returned by yaml.parse().

That created a boundary I had not been checking closely enough:

what a human sees in the source
        ↓
      YAML parse
        ↓
what downstream code receives

The verification gate still passed

In spec-lane, the success criteria and the verification matrix are separate inputs.

The success criterion lives in intent.yaml.

The corresponding verification record lives in verification.yaml.

For the reproduction, the matrix contained the already-truncated form:

success_criteria_matrix:
  - criterion: ledger has exactly one PhaseGate row
    covered_by: test
    evidence: test/ledger.test.ts::records PhaseGate
    negation_test: no PhaseGate row fails

One important caveat: those evidence and negation_test values are declarations in the reproduction fixture.

For this reproduction, I did not create and execute the referenced ledger test and then prove that its behavior matched those strings.

The purpose here was to reproduce the gate behavior.

Reproducing the behavior before the fix

I built the CLI from the revision immediately before PR #48's fix and ran the reproduction in a temporary directory.

With the unquoted success criterion and the shortened matrix entry:

$ lane validate ...

intent.yaml is valid (phase=3_implement).

Exit code:

0

Then:

$ lane advance ... --phase 4_verify

Advanced ...: 3_implement -> 4_verify

Again:

exit code: 0

The verification phase was entered.

At that point, the two strings reaching the gate were effectively:

success:

ledger has exactly one PhaseGate row


matrix criterion:

ledger has exactly one PhaseGate row

They matched.

So the gate passed.

The important distinction is this:

the verification gate did not remove the text after #.

That had already happened at the earlier boundary:

YAML source
    ↓
parsed JS value

The gate was comparing the values it had been given, and those values were equal.

The relevant code paths are in:

• packages/cli/src/intent-store.ts
• packages/core/src/gate.ts

The full reproduction and investigation are documented in Issue #45:

https://github.com/shiki-yusuke/spec-lane/issues/45

Quoting the same criterion changed the result

I then changed only the success criterion so the full text became an explicit YAML string:

success:
  - "ledger has exactly one PhaseGate row # include the negative case too"

Now the parsed value retains everything:

success:

ledger has exactly one PhaseGate row # include the negative case too


matrix criterion:

ledger has exactly one PhaseGate row

With the same pre-fix CLI, both validate and advance now failed with exit code 3.

The gate reported that there was no matrix row corresponding to the full success criterion.

The phase remained 3_implement.

So the control case looked like this:

Success criterion	Matrix	validate	advance
Plain scalar; text after # is a comment	Short value	0	0
Quoted; full value is preserved	Short value	3	3

At least in this fixture, the result did not come from the gate failing to run.

The value reaching the gate changed.

That changed the outcome.

Schema-valid does not mean source-equivalent

The schema for this part of the success criteria is intentionally simple:

success: z.array(z.string()).min(1)

The shortened result:

ledger has exactly one PhaseGate row

is still a perfectly valid string.

So after parsing, these two source forms can result in the same value:

- ledger has exactly one PhaseGate row

and:

- ledger has exactly one PhaseGate row # include the negative case too

If a validator only receives the parsed value, it can no longer tell whether:

1. the author originally wrote the shorter criterion, or
2. additional source text became a YAML comment.

That distinction led me to separate several different claims that are easy to blur together:

the value has the correct shape
≠
the original source expression was preserved

And there are further boundaries after that:

the string was preserved
≠
the string correctly represents human intent

Likewise:

an evidence field contains a test name
≠
that test was actually executed and proved the claim

A schema validator can validate the structure it receives.

It cannot automatically recover information that disappeared before that boundary.

The v0.11.0 fix stops earlier

PR #48 did not change the success-criteria comparison itself.

Instead, the fix moved the check to the intent.yaml reading boundary.

The reason is straightforward: once only the parsed value remains, there is not enough information to distinguish the two source forms.

The implementation now roughly does this:

1. Walk the YAML AST and identify unquoted PLAIN scalars.
2. Use the scalar's source range to inspect the original text.
3. Check whether spaces or tabs immediately after the scalar are followed by # on the same line.
4. If such a candidate value appears in the parsed success criteria, reject the input before it reaches the gate.

The implementation is here:

https://github.com/shiki-yusuke/spec-lane/blob/64f2ca42c5929be0413e5f4f240897dffe56878b/packages/cli/src/intent-store.ts#L108-L164

The check deliberately does not rely only on the AST node's .comment property.

One reason is that certain anchor forms can associate a comment with a different node. The implementation therefore combines AST type and range information with the original source text.

With the v0.11.0 source, the same reproduction now behaves like this:

Success criterion	Matrix	validate	advance
Plain scalar + inline comment	Short value	2	2
Quoted full value	Full value	0	0

In the rejected case, the phase remains 3_implement.

If a literal # belongs in the success criterion, it can be made explicit by quoting the string:

success:
  - "ledger has exactly one PhaseGate row # include the negative case too"

The fix shipped in spec-lane v0.11.0:

https://github.com/shiki-yusuke/spec-lane/releases/tag/v0.11.0

PR #48 contains the implementation and regression tests:

https://github.com/shiki-yusuke/spec-lane/pull/48

This still does not mean the tool understands human intent

The fix is intentionally narrower than that.

It also has a known over-detection case.

Consider:

intent:
  business_goal: shared text # unrelated comment
  success:
    - "shared text"

Nothing was truncated from the quoted success criterion.

But the current check can still reject this document because another commented plain scalar has the same parsed value as the success criterion.

The implementation does not prove that the two occurrences share the same semantic origin.

That limitation is preserved in a regression test rather than hidden.

So I would not describe the fix as:

spec-lane now understands whether the author's full intent has been preserved.

It does not.

The design is closer to:

if the source contains a value that could have been silently shortened before it reaches a success criterion, stop instead of guessing.

That is a fail-closed choice.

It trades some false positives for avoiding this known silent-truncation path.

The part I now pay more attention to is the boundary

This incident changed how I think about verification in AI-assisted development.

It is tempting to think about the pipeline as:

specification
    ↓
implementation
    ↓
verification

But in practice there are more boundaries:

Human expression
what someone is trying to achieve
        ↓
Spec source
what was actually written
        ↓
Parsed representation
what the tool interpreted
        ↓
Verification
what the gate compared
        ↓
Execution evidence
what actually ran or was observed
        ↓
Acceptance
why the result was accepted

The failure in this post was mostly at:

Spec source
↓
Parsed representation

Everything after that can behave consistently and still verify less than a human reader thought had been written.

The same distinction appears elsewhere.

A test name written into a verification matrix is not the same thing as evidence that the test actually ran.

A command exiting successfully is not necessarily the same thing as the intended property being proved.

And a schema-valid specification is not necessarily the same thing as preserving every meaningful part of its source representation.

Machine-readable specifications are useful.

But once a workflow starts treating “the gate passed” as evidence of completion, I also want to know:

What exactly reached the gate, where did that value come from, and what transformations happened before it got there?

In this case, one small YAML line was enough to expose that boundary.

References

• Issue #45 — reproduction and original report https://github.com/shiki-yusuke/spec-lane/issues/45
• PR #48 — fix and regression tests https://github.com/shiki-yusuke/spec-lane/pull/48
• spec-lane v0.11.0 https://github.com/shiki-yusuke/spec-lane/releases/tag/v0.11.0

---

AI-assisted disclosure: I used AI tools to help draft, translate, and edit this article. The reproduction, source inspection, and factual claims were checked against the linked issue, pull request, release, and the evidence collected during the investigation.