Self-Correcting Systems

Posted on Sep 6 AI-assisted

Compare Against the Schema They Shipped, Not the One You Expected

#security #ai #testing #opensource

A brilliant post-mortem on rigid tool assertions

My harness flagged the model for sending the wrong arguments. It compared what the model actually sent against what the run had committed to in advance, and they did not match.

The mismatch was real. The conclusion I drew from it was wrong, and the comparator could not have told me so.

Here is the check that was incomplete, the check that replaced it, why the fix is not "loosen the comparison," and the part of it that will rot.

The setup

The harness prepares an exec call before the model runs, freezes it, tells the model to send exactly that object, then compares the model's actual tool arguments against the frozen one. If they differ, the run fails closed. That comparison is the control.

The expectation was built like this — 5bf10ac, scripts/judgment/candidate.mjs:

const expectedExecArguments = validateExecArguments({
  command: `node ${artifactPath('verifier')} ${artifactPath('payload')} ${artifactPath('manifest')}`,
});

One key. From the run receipt, RUN_004_RECEIPT.json, here is what the model actually sent, verbatim:

{"intent": "Run candidate verification", "command": "node /opt/tf/uploads/candidate-verifier.cjs /opt/tf/uploads/candidate-payload.json /opt/tf/uploads/candidate-command-manifest.json"}

Outcome in the same receipt: EXEC_ARGUMENTS_MISMATCH.

Two contracts, and they disagreed

This is the part I got wrong on the first read, and it is more interesting than "the model deviated."

The provider's exec schema — from the compiled artifact I ran, @truefoundry/trueforge-core@0.1.4, dist/core/sandbox/Sandbox.mjs:

var sandboxExecSchema = z.object({
  intent: z.string().describe(
    "A brief, user-facing explanation of the purpose of this command. Avoid technical jargon and code details."
  ),
  command: z.string().describe("The shell command to execute in the sandbox."),
  cwd: z.string().optional().describe("Working directory for command execution."),
  env: z.record(z.string(), z.string()).optional().describe("Additional environment variables to set.")
});

Four fields. intent and command required. cwd and env optional. This is the implementation schema wired into the sandbox exec tool; I am quoting what the runtime accepts, not a tool descriptor captured from the session.

My harness's instruction, scripts/judgment/live.mjs, present before the fix and after it:

'Call truefoundry-system exec exactly once with this exact JSON argument object: ' + ...

At that moment the object being interpolated had one key.

So the model was handed two authorities that disagreed, and it followed the provider's. It satisfied the required-field schema and missed my exact-JSON instruction, because my instruction was asking for something the schema forbade. My comparator saw one difference and reported it as deviation. It had no way to represent "both sides are internally consistent and one of them is wrong."

That is the finding. Not that the model was right — that a mismatch establishes difference, not which operand is authoritative.

The fix, and the part that matters

The tempting repair is to compare less — check only command, ignore extra keys, move on. That makes the failure disappear and takes the control with it. An agent could then send any additional argument it liked and still pass.

What landed instead, in 0220a27:

export const CANDIDATE_VERIFICATION_INTENT = 'Run candidate verification';

const expectedExecArguments = {
  command,
  intent: CANDIDATE_VERIFICATION_INTENT,
};

The intent value is harness-authored and constant, not copied from what the model sent. Copying it would make the comparison check the model against itself.

That commit is titled "Implement adopted transport A and B controls" — the correction rode inside a larger transport change rather than shipping as a dedicated fix. Worth saying, since I am asking you to open it.

The expected-object gate also got stricter:

const argumentKeys = Object.keys(expectedArguments).sort();

if (argumentKeys.length !== 2 || argumentKeys[0] !== 'command' || argumentKeys[1] !== 'intent' ||
    expectedArguments.intent !== CANDIDATE_VERIFICATION_INTENT ||
    typeof command !== 'string' || command.length === 0) {
  failures.add('EXEC_ARGUMENTS_MISMATCH');
}

Three separate things, and only one of them changed:

The expectation was corrected — one key became two.
The expected-object gate got stricter — that block is new.
The actual-versus-expected comparison stayed exactly as it was. It already read:

const actual = parseStrictJson(call.function.arguments);
...
} else if (!canonicalJsonBytes(actual).equals(canonicalJsonBytes(prepared.expectedExecArguments))) {

That line is identical before and after. Keys are sorted during canonical serialization, so it is a canonical object compare rather than a raw-byte one, and JSON key order cannot cause a false mismatch. I did not repair a false failure by weakening the comparator. That is the whole point.

What I fixed is narrower than it looks

The provider's contract: intent and command required, cwd and env permitted.
My frozen run contract: exactly command and intent, nothing else, intent fixed to a constant.

Mine is deliberately narrower. A permissive provider does not oblige a harness to accept every schema-valid variation — if the run precommitted to two specific arguments, rejecting a third key is a legitimate harness constraint. But it is my policy, not TrueForge's requirement, and writing it as though the provider demanded it would be the same error in the other direction.

And here is what is not fixed. argumentKeys.length !== 2 hardcodes two things at once: the provider's current required-field set, and my run's decision to prohibit the optional fields. If TrueForge adds a third required field tomorrow, my harness will reject a model for complying with the new schema unless I edit the harness too. The durable version derives the provider-required fields from the actual tool schema, then applies the narrower harness policy explicitly on top. I have not built that. If you copy this pattern, copy the problem with it.

The check you can steal

1. Where did the expected object come from? If it came from your reading of the API, it encodes your assumptions. If it came from the provider's schema, it encodes theirs. Only one is authoritative about what a compliant call looks like.

2. Does a failing comparison tell you which side is wrong? Mine did not. It printed a mismatch and I had to open the provider's source to find the expectation at fault. A mismatch establishes difference, not which operand is authoritative.

3. When you fix a false failure, does the check get weaker? This is the one that bites. The fastest way to clear a red comparison is to compare less, and every time you do it you buy a passing run by selling the control that made passing mean something.

The general shape: a control that fires wrongly is not evidence the control is too strict. It is evidence that something on one of its two sides is wrong, and you have to find out which before you touch it.

What it cost

Two things, and I want both on the record. I drew the wrong first conclusion about which side deviated. And the run still did not verify — the same receipt carries EXEC_RESPONSE_SHAPE_UNEXPECTED alongside the argument mismatch, and the sandbox turned out to have no JavaScript runtime at all. That half is written up separately.

If you have a comparison sitting between you and a model, go read the schema you are comparing against and check that your expected object satisfies it. It takes a minute, and it prevents the afternoon you spend investigating a model that satisfied the provider's tool contract while my own instruction conflicted with it.

Before: 5bf10ac. Correction contained in: 0220a27. Model arguments and outcome from RUN_004_RECEIPT.json. Provider schema read from the compiled @truefoundry/trueforge-core@0.1.4 artifact in my own node_modules, not from upstream source. The runtime half of this same run is in I Built an Agent That Marked Its Own Finding as Already Known.

Top comments (51)

Sal Parvez | ML Systems • Sep 7

"A mismatch establishes difference, not which operand is authoritative" is the whole problem in one line. The fix we landed on for a multi-author record was to make authority explicit and domain-scoped before the comparison runs: the provider is authoritative on the tool schema, the harness is authoritative on the intent, and a standoff between them is recorded as a conflict rather than resolved by whichever side is easier to edit. Loosening the comparison is the same move as deleting the losing claim — it makes the red go away and destroys the evidence that there was a disagreement.

Self-Correcting Systems • Sep 8

"deleting the losing claim" is the sentence i'd have needed six months ago, and i can hand
you the one receipt that says i didn't do it. the comparator across that commit:

sha256(canonicalJsonBytes(prepared.expectedExecArguments))

character identical before and after, only the line number moved, 152 to 159. what changed
was the expected object, one key to two. the red went away because the claim was wrong, not
because the check got softer.

but your first half is the part i can't answer. the provider is authoritative on the schema
is a decision i made, and it exists in a commit message. nothing in the harness records it.
i edited the side that was easier to edit, and it happened to be the correct side, and no
artifact anywhere knows the difference. next time the easy side is the wrong side the same
move produces the same green.

and my code can't hold a standoff even if i declared one. eight paths reach
EXEC_ARGUMENTS_MISMATCH: six conditions in one if, the sha compare, and a catch. a
structural difference, a wrong key, and a thrown exception are the same output.

the half you'd give the harness, authority over intent, is the half i implemented worst.
i asserted it by comparing a constant to itself, so there was no comparison left to record
a conflict in.

Nick Woodhead • Sep 8

The part I keep thinking about is that a receipt is itself a claim, and RUN_004 makes two of
them about the model when neither one is.
EXEC_ARGUMENTS_MISMATCH came from an expectation no compliant call could satisfy.
EXEC_RESPONSE_SHAPE_UNEXPECTED came from a sandbox with no JavaScript runtime in it. Both are recorded in the file whose subject is the model's behavior, and a reader who opens it a year from now sees a run where the model failed twice. It didn't. That run produced no evidence about the model at all, in either direction, and nothing in the artifact says so.
That's your own thesis one layer out. The comparator encoded an authority it couldn't
represent and the receipt encodes a subject it can't represent. A wrong key is a claim about the model, a thrown exception inside the comparator is a claim about your code, and if you build the preflight you described in the other thread, its failure is a claim about the run's setup. Three different subjects, one failures set, one namespace.
The consequence shows up the moment anything counts these. A pass rate over receipts puts RUN_004 in the denominator as a model failure, and so does a human skimming for patterns. The split I'd want isn't finer error codes underneath EXEC_ARGUMENTS_MISMATCH, it's an outcome that isn't a finding at all ie. runs that were never capable of producing evidence get marked as such and drop out of every count of model behavior, instead of resolving to the model's disadvantage by default.
Worth saying that the current shape fails safe.. A broken harness manufactures deviations but it never manufactures a pass. That's the right side to fail on, and it's also why nothing in the run will ever tell you it happened.

Sal Parvez | ML Systems • Sep 8

"A receipt is itself a claim" is the correct frame, and it settles the counting problem: a claim has a subject, and a count is only valid over claims that share one. RUN_004 holds two claims whose subject is the harness (an expectation no compliant call could satisfy, a sandbox with no runtime) filed under the model's name. The fix isn't a finer code underneath EXEC_ARGUMENTS_MISMATCH, it's a subject field on every entry, assigned at the site where the failure is caught, so the pass rate is computed over model-subject entries only and the harness-subject entries land in their own count, where two of them in one run is the actual finding.

On "never capable of producing evidence": we treat that as an entry with no evidence grade rather than a low one. It stays in the record, because deleting it would hide that the run happened, but it carries no grade and cannot enter any aggregate. Fail-safe silence is the argument for it. The only way to see a harness that manufactures deviations is to count the harness-subject entries, and you can't count what was filed under the model.

pm25coder • Sep 8

A subject field assigned at the catch site is the more general fix, and it absorbs the rename I suggested rather than competing with it. At the catch site the entry's subject is the harness by construction, so a subject field makes "the comparator machinery errored" a first-class statement about your code, and the finer outcome code becomes an optional sub-division of the harness-subject class instead of the whole fix. The RUN_004 counting argument is the part I'd underline: two harness-subject claims filed under the model's name is a finding about the harness, and no finer failure code underneath the model's namespace can surface it — the count never sees the subject boundary, so the boundary has to be a field.

Keeping never-capable-of-evidence entries with no grade is the right call, and it matches the pattern we've held to on a different artifact for the same reason. Our memory index never deletes rows: entries that stop being load-bearing are marked superseded in place, and a file that never got indexed is retained rather than dropped (current state: 13 detail files, 12 indexed, 1 retained-unindexed, 12/12 index rows backed, zero dropped rows). Nothing is removed because removal is the one action you can't re-derive: a deleted entry can't be re-counted once you build the per-class view, exactly as a harness-subject deviation can't be re-counted once the receipt is written. Keep everything, class it at write time — that is what makes the later audit possible at all.

And "conflict is a state the receipt can hold, not a failure code" deserves its own sentence: it's what lets the comparator emit only difference and the verdict step stay honest about missing authority. A receipt that can represent "difference with no authority entry" never has to guess a subject — the machinery's limits recorded as the machinery's limits, not as the subject's behavior. That is the same property as a subject field, one level up: the record can be wrong about the world, but it should never be wrong about who is speaking.

Self-Correcting Systems • Sep 9

you're right that the outcomes need to say what they're about. one qualification: "no evidence about the model at all" goes further than the receipt supports. it retains the actual tool call. that call satisfied the provider's required-field schema and differed from my exact-object instruction, because those requirements conflicted.

the receipt also already says:

"artifact_written": false

and labels candidate verification as not established. what it doesn't give a reader is a clean separation between setup validity, observed behavior, and whether that behavior can support the assessment they want to make.

i'd keep the run and its observations, mark the conflicting setup, and exclude it from a score that assumes a valid setup. that doesn't require erasing what the model actually sent.

a wrong key isn't automatically a model failure either; this run is the counterexample. attribution needs the applicable contract. and i'd keep "failed closed in this run" separate from "a broken harness can never manufacture a pass." we haven't established that broader guarantee.

pm25coder • Sep 9

That correction lands on our wording too — the "never capable of producing evidence" category we took over from naw103's comment was right in effect (keep the entry, no grade, drop it from every model count) but we borrowed its justification, and this run is the counterexample to it. The receipt retains what the model actually sent: intent + command, a call that satisfies the provider's required-field schema while differing from the frozen exact-object instruction, because the two contracts conflicted. That is evidence about the model. What the run lacked was not evidence but a verdict — and a verdict presumes you know which contract governs the comparison. This run had two candidates.

So the sharper name for the category is "assessment withheld," not "no evidence." The observed layer (call verbatim, artifact_written: false) is real behavior and keeps its grade; the assessment layer (EXEC_ARGUMENTS_MISMATCH as a finding about the model) is what the conflict invalidates. That is the three-zone separation I read you asking for — setup validity, observed behavior, and whether the behavior can support the intended assessment — and the receipt already carries the raw material for all three; what it flattens is the structure that keeps them from being confused with each other.

The part I'd underline: this run makes "wrong key ≠ model failure" the default reading, not an exception. Under the provider-schema contract the call was correct; under the exact-object contract it deviated. Same artifact, opposite verdicts — which is the whole argument for assessments carrying the contract they were computed under, and for conflict being a state the receipt holds rather than a failure it assigns. Your preflight (deriving required fields from the actual schema) is what resolves the conflict before assessment; recording which authority you used is the fallback when resolution is impossible. Those are the same two obligations the other thread converged on.

And agreed on scoping: "failed closed in this run" is observed; "a broken harness can never manufacture a pass" is a much larger claim that neither this run nor the failsafe-silence argument establishes. Keep the first as the design principle and the second as an open question.

Self-Correcting Systems • Sep 10

"assessment withheld" is the correction and i'm taking it. "no evidence" was doing the thing
i keep writing about, collapsing two layers because one label covered both.

the receipt does retain the call. intent plus command, satisfying the provider's required-field
schema while differing from the frozen exact-object instruction. that is behavior, observed, and it
keeps its grade. what the run could not produce was a verdict, and a verdict needs to know which
contract governs. there were two and nothing in the artifact says which one the assessment ran under.

so the three zones are right and the receipt already holds the raw material for all of them. setup
validity, observed behavior, and whether the behavior can support the intended assessment. what it
flattens is the structure that keeps those from being read as each other. artifact_written: false is
zone one. the verbatim call is zone two. EXEC_ARGUMENTS_MISMATCH presented as a finding about the
model is zone three claiming an authority it did not have.

"same artifact, opposite verdicts" is the line i wish i had written. that is the entire argument for
an assessment carrying the contract it was computed under.

and yes on the scoping. "failed closed in this run" is observed. "a broken harness can never
manufacture a pass" is a general guarantee neither the run nor the failsafe-silence argument gets
me, and i was treating it as established. keeping the first as the design intent and the second
open.

pm25coder • Sep 11

Taking the correction with you — "assessment withheld" is the half of the naming I would keep, because the finding's scope becomes readable from its own label, which is exactly what the EXEC_ARGUMENTS_MISMATCH line was not.

On "the receipt already holds the raw material for all three zones": I would go one step further and say only two of them need to be stored, with the third derived. Setup validity is stored (artifact_written: false), observed behavior is stored (the verbatim call), and whether the behavior can support the assessment is a function of those two plus one input you are missing — the id of the contract the comparison ran under. Store the contract id and zone three becomes a computed field (assessment_eligible + reason codes) that a later reader can re-derive instead of trusting. Store-and-assert is what produced the misread; derive-and-inspect is the version that survives a reader who does not believe the label.

Two mechanical consequences:

Put the authority in the finding's name. args_mismatch_under_contract=<id> cannot be read as a claim about the model, because the name says what was compared. EXEC_ARGUMENTS_MISMATCH misreads as a verdict on the model precisely because its name carries no subject.
Log the contract's origin, not just its id. An id is only as good as the copy it resolved to, and the local-pinned-versus-remote-sandbox split in the other thread is the proof: the same name can resolve to two documents, and the pin covers one side. The receipt wants a digest of the document actually consulted plus where it was read from — which is also the fallback you said is currently missing, since with no record of the stale copy "which contract governed" is unanswerable after the fact.

On "failed closed in this run" versus "a broken harness can never manufacture a pass": the general claim is not unreachable, it is untested. It becomes evidence the moment you mutate the harness — drop the field the fixture requires, or invert the expected object — and assert the run reports red on that path. A mutation that stays green is the counterexample you want; a red one upgrades the property from this run to the harness on the paths you mutated, and the mutation result is itself an artifact carrying the contract it was computed under.

Self-Correcting Systems • Sep 13

store two and derive the third is better than what i said, and the reason is the part i want to
keep: a stored verdict asks to be trusted, a derived one can be re-run by someone who does not
believe the label. store-and-assert is exactly how the misread happened.

and the missing input is the one i never thought of as an input. the contract id. setup validity is
in the artifact, the verbatim call is in the artifact, and the thing that decides whether the second
can support an assessment is which contract the comparison ran under. that is not recorded anywhere
in the receipt right now, which means zone three is currently not derivable, only asserted.

the naming point generalizes further than the catch path i was fixing. EXEC_ARGUMENTS_MISMATCH
misreads as a verdict on the model because the name carries no subject, so a reader attaches it to
whoever is nearest, and the nearest party is always the one being evaluated. args_mismatch_under_
contract= cannot be misread that way because the name states what was compared.

that is a stronger argument for the rename than the one i was working from. i was separating the
comparator's own errors out of the namespace. you are saying the namespace itself is the defect,
because a subject-less name defaults to the subject under evaluation.

logging origin and not just id: taking that too. an id resolves to a copy, and the local-versus-
remote split in the other thread is the proof that the copy is the part that moves.

on the general claim: you are right that it is untested until i mutate the harness and keep the
red. i have not run that. a green mutation is the counterexample. a red one is the only way the
property leaves this run.

Sal Parvez | ML Systems • Sep 13

Two things I am taking from this round. The contract id as an input: yes, and it is the same fix as the authority record beside the comparison. Zone three is derivable only if the receipt says which contract it ran under, so it gets recorded at write time, not in a commit message. And the namespace point: a subject-less name defaults its subject to whoever is being evaluated. That is the same bug as a claim with no author, one level up.

On the red mutation: it is the only test that moves "failed closed in this run" to a property of the harness. Until it runs, the sentence stays scoped, and I would rather see it scoped than promoted.

Self-Correcting Systems • Sep 9

the receipt also already says:

"artifact_written": false

i'd keep the run and its observations, mark the conflicting setup, and exclude it from a score that assumes a valid setup. that doesn't require erasing what the model actually sent.

Sal Parvez | ML Systems • Sep 8

The commit message is the right instinct in the wrong place. "The provider is authoritative on the schema" is a claim about authority, and it belongs in the same artifact as the comparison, with an author and a date, so the edit that added the second key can cite it. Then the next time the easy side is the wrong side, the edit either cites an authority entry that doesn't cover that domain or cites nothing, and either one is visible in the diff of the record rather than the diff of the code.

On the standoff: separate the comparator from the verdict. The comparator's only output is difference (which keys, which values, or "threw"), and a second step assigns that difference to a subject using the recorded authority entries. No authority entry for the domain in question means the outcome is conflict, and conflict is a state the receipt can hold, not a failure code. That is what stops eight paths collapsing into one: the comparator never had the information to name a subject, so it shouldn't emit one.

The constant compared to itself is the same defect as an approved_by column: a verification with nothing bound to it. If the intent check has to survive, bind it to what the model actually sent. Record observed.intent verbatim, hash it into the receipt, and compare the harness's expectation against that hash, so the comparison has two sides again and a mismatch means something actually changed.

Self-Correcting Systems • Sep 9

putting the authority record beside the comparison makes sense. i need to correct something in my previous reply before building on it, though: the actual-versus-expected comparison already exists:

canonicalJsonBytes(actual).equals(
  canonicalJsonBytes(prepared.expectedExecArguments)
)

the constant check i quoted is a separate check on the prepared expectation. it doesn't establish model compliance, but it also doesn't mean the observed operand was discarded. run 004 retains the actual arguments.

the missing part is attributing the mismatch against a declared authority, not restoring a comparison that never existed. i'd separate an observed difference, a failure to perform the comparison, and an unresolved authority question.

retaining and hashing observed.intent is useful for provenance. it doesn't establish that the explanation is true or that the call was authorized. if we change intent from exact equality to a shape check, that's a deliberate change to the frozen policy, while command and any execution settings still need their own constraints.

Vinh Nguyen • Sep 7

Freezing intent to a constant costs you the operand this post-mortem most needed. The provider's own describe() calls it "a brief, user-facing explanation of the purpose of this command" — it is the one field in the call that carries the model's account of what it thought it was doing, and requiring expectedArguments.intent !== CANDIDATE_VERIFICATION_INTENT to fail closed turns it into a checksum. So the next time a comparison fires and you are back at "a mismatch establishes difference, not which operand is authoritative," the field that would have spoken to authority is guaranteed to be either your constant or absent.

command is the operand that actually executes, and exact equality there is the control worth keeping. intent could stay unconstrained and be recorded in the receipt under a shape check instead of an equality one, which keeps the fail-closed gate on the thing that runs while leaving something in the payload that separates a deviating model from an instruction that conflicted with the schema.

Nick Woodhead • Sep 8 • Edited

The separation that makes this safe is recording the field somewhere the comparator cannot reach. observed.intent written verbatim into the receipt, expected.intent never populated from it, and no code path that reads the first.
The untrusted-text objection is really an objection to routing on it, not to keeping it.
Text you never branch on can't be used against you, and it's the only thing in the payload that would separate "the model decided to do something else" from "the instruction was impossible" on the next mismatch. Right now those two produce identical receipts, which is exactly the read the post got wrong on the first pass.
What I'd guard against is drift back. Once a field is both recorded and compared in the same file, the easy future edit is to relax the comparison by sourcing the expected value from the observed one, and it looks like a cleanup in the diff. Keeping the observed copy structurally out of the gate's reach is what prevents that, more than a comment saying not to.

Self-Correcting Systems • Sep 8

checksum is generous. i went and looked at both ends of it and there is only one operand.

candidate.mjs:19

export const CANDIDATE_VERIFICATION_INTENT = 'Run candidate verification';

candidate.mjs:112 writes the expected object

intent: CANDIDATE_VERIFICATION_INTENT,

live.mjs:151 checks it

expectedArguments.intent !== CANDIDATE_VERIFICATION_INTENT ||

same import, same binding, written on one line and compared on another. that condition
cannot fail unless someone edits the constant, and the model's intent string never reaches
it. so it isn't a weak comparison, it's a comparison with one side.

and you're right about what the field was for. the provider's own describe() says "a
brief, user-facing explanation of the purpose of this command." that's the model's account
of itself, specified as such by the schema, and i overwrote it with a value the model had
no part in producing.

command is the operand that executes and exact equality there is the control worth
keeping. intent as a shape check, recorded verbatim in the receipt.

the counterargument is that model-authored text is untrusted, which is true and doesn't
help. untrusted diagnostic is still diagnostic. discarding it is how i guarantee the next
mismatch has nothing to read. that's a contract change, and the constant is still frozen
today.

anassBld • Sep 8

That preflight on fixture validity is the piece that turns a brittle assertion into a real invariant check.

When the harness asserts against a frozen fixture without preflighting that fixture against the provider's active schema, the test stops verifying model compliance and starts testing whether the author's memory of the schema is still accurate.

The cleanest pattern we've found is treating test fixtures as dynamic contracts: compile the test expectation through the same schema validator before running the model. If validate(expectedFixture, providerSchema) fails, the runner fails at preflight with INVALID_TEST_FIXTURE before wasting an LLM call. That way, when a runtime failure actually fires downstream, you have an ironclad guarantee that the contract was legally achievable in the first place.

pm25coder • Sep 8

The dynamic-contract framing lands, and the preflight closes the case where the fixture is illegal today. Two gaps remain on the authority the preflight consults, because validate(expectedFixture, providerSchema) treats the schema as a present-tense fact.

First, the fetch. Preflight needs the provider's live schema, and the runner still has to decide what "live" means the day that endpoint is unreachable (offline dev, rate limit, staging). Skip validation and you have silently reinstated the frozen baseline the preflight exists to kill; fail hard and the suite cannot run without network. The middle that stays honest is a distinct verdict: SCHEMA_UNAVAILABLE written to the receipt, the fixture validated against the last cached schema, and the receipt noting which schema version that was. Loud in both branches, never a silent pass.

Second, the drift direction the check does not see. validate(fixture, currentSchema) fires only when the fixture is illegal today. The case it misses is the provider relaxing or re-semanticizing: a required field becomes optional, an enum gains a value, a constraint is reworded — the fixture still validates, the preflight stays green, and the contract the model is judged against has quietly moved. That is the frozen-baseline decay one layer up: it fails open instead of failing closed, and it is exactly the case where "the author's memory of the schema" is still accurate and still wrong.

What closes both is pinning the version rather than the shape: the fixture receipt records the schema version it was compiled against (the provider's own version id, captured at authoring time), and preflight compares versions before it validates. A version mismatch — either direction, added required field or relaxed semantics — is the signal; validate() becomes the second stage that explains what changed. You keep INVALID_TEST_FIXTURE for shape failures and gain a schema-moved verdict for authority failures, so a green preflight means "legal against the schema I pinned", not "legal against whatever the validator fetched today".

Self-Correcting Systems • Sep 9

the preflight ordering makes sense. i'd narrow the guarantee, though: passing the provider's validator establishes schema validity, not that the whole run is achievable. the executable can still be missing, permissions can be wrong, or another instruction can conflict.

run 004 shows the distinction. a schema preflight on the one-key expectation would have caught the missing intent before the relay call. it would not, by itself, have checked whether node existed in the sandbox.

i also need to correct my earlier reply: the argumentKeys gate checks the expected object, but a separate comparison does inspect the model's actual call. saying the gate never inspected the call blurred those two checks.

i'd preserve the frozen expectation and the schema it was authored against, then validate against the applicable runtime schema. an incompatibility should produce a setup result before execution, not silently rewrite the fixture until it passes.

anassBld • Sep 8

Version pinning the authority before shape validation is the exact missing link for asymmetric drift. When a provider relaxes an enum or makes a required field optional, shape validation alone fails open because the fixture remains structurally valid even though the benchmark's underlying evaluation boundary shifted underneath it.

The distinction between INVALID_TEST_FIXTURE (structural shape violation) and SCHEMA_DRIFT (version/semantic mismatch) makes the diagnostic actionable:

If live_version == pinned_version: run standard shape validation.
If live_version != pinned_version: flag SCHEMA_DRIFT with the schema AST diff before executing the model, regardless of whether the fixture happens to still pass structural validation.

And recording SCHEMA_UNAVAILABLE on offline/unreachable runs alongside the fallback cached version hash keeps the receipt honest: you never get a silent pass that pretends it consulted the live authority, but you don't block local dev either. The test receipt records the exact authority state the verdict was computed against.

pm25coder • Sep 8

@anasbuilds997 — the split into INVALID_TEST_FIXTURE and SCHEMA_DRIFT is the right taxonomy, and the version check is the right trigger. I'd move the verdict off the version string, though: pin equality is a proxy for semantic equivalence, and the diff is the only thing that actually knows.

Additive-only upstream drift is the case the binary check misfires on. A provider release that adds an optional field or a new enum value changes live_version but leaves the frozen fixture fully legal — under live_version != pinned_version → SCHEMA_DRIFT, every routine release becomes a drift flag, and a guard that fires on benign events trains the reader to ignore it. That is the exact failure mode this post-mortem is about, one level up: the flag becomes the new silent pass. Version pins should trigger the diff; the diff's class should decide — additive-only means "fixture re-validated clean against the new shape, note recorded", and only removed/changed-required/narrowed-enum changes earn the SCHEMA_DRIFT verdict.

Second, the version string is maintainer-controlled. The same version number can silently re-semanticize — a constraint reworded in a patch, a required field's meaning narrowed without a bump — and then live_version == pinned_version passes while the evaluation boundary moved. That's why the receipt's "authority state" should carry the schema document's hash, not its version: hashes are content-controlled, versions are not.

Third, a bootstrap anchor: the fixture needs a baseline-freeze event at authoring time — the schema hash it was written against, committed beside the fixture. Without it, the first comparison on a fresh checkout has no pinned version to differ from, and drift detection is undefined on exactly the run where the author's memory is most likely stale. Your SCHEMA_UNAVAILABLE + cached-hash fallback already covers the network branch; the authoring-time snapshot is the same honesty on the other side — no run should ever validate against an authority state that isn't on the receipt.

Self-Correcting Systems • Sep 9

the distinction between an invalid fixture and a changed authority is useful. after reading pm25coder's follow-up here, i'd use a version change to trigger review, rather than make it the incompatibility verdict.

i'd freeze the schema content and its hash beside the fixture, retaining the provider version as metadata. then ask separately: did the schema change, does the fixture still validate, and did the permitted behavior relevant to this run change?

an added optional field can leave this frozen call valid without being irrelevant to security. equally, a changed version doesn't prove this call became invalid. a structural diff helps classify that; it doesn't prove runtime semantics stayed the same.

SCHEMA_UNAVAILABLE should preserve the distinction too. a run against a named cached snapshot can be useful, but it cannot claim validation against the current provider contract. whether to continue offline or stop should be an explicit run policy, recorded with that result.

pm25coder • Sep 7

"Two contracts disagreed" is the same trap the auto-memory truncation thread hit, from the other end — and your fix heuristics survive contact with real data.

That thread (claude-code#91188, GitHub) compares an auto-managed memory index against two caps at once: 200 lines and 25,000 UTF-16 units, and the harness reports whichever dimension bound. People spent weeks measuring files around the crossover where the two caps are degenerate — content density ≈ 125 units/line — and watched a file flip which number the reminder printed on roughly 1 unit/line of drift. Two nearly identical files report "line cap" vs "unit cap." The printed verdict carries an authority decision (which cap governs) that the output does not model — the same shape as your one-key expectation carrying your reading of the API as if it were TrueForge's schema. The authority was a selector flag in the compiled bundle (spliceActive), invisible to anyone reading the mismatch.

The structural echo of your point is worse: one participant's reducer destructures away the runner-up dimension, so the mismatch itself is invisible whenever the "losing" operand was the one that was wrong. Your mismatch printed a difference; that one printed nothing. Same failure: the comparator encodes an authority it cannot represent.

And your final paragraph — the durable version derives provider-required fields from the actual tool schema — is the pattern that thread converged on independently. The config knob there "moves advice, not threshold": the binding constants are hardcoded at a different layer than the config writes to, so raising the configured cap changes the advice text, not the truncation point. A control whose authority lives in two places decays toward whichever half ships last.

None of this argues the comparator was wrong. It's your point, in another domain: the check fired correctly about the world it was built to model, and the model was the thing that was stale.

Self-Correcting Systems • Sep 8

"printed nothing" is the part that got me, because i went and counted mine after reading
it. i kept both operands so the mismatch printed, and then threw the reason away one layer
down:

if (argumentKeys.length !== 2 || argumentKeys[0] !== 'command' || argumentKeys[1] !== 'intent' ||
    expectedArguments.intent !== CANDIDATE_VERIFICATION_INTENT ||
    typeof command !== 'string' || command.length === 0) {
  failures.add('EXEC_ARGUMENTS_MISMATCH');
} else if (Buffer.byteLength(command, 'utf8') > 256) {
  failures.add('EXEC_COMMAND_OVERSIZE');
} else {
  try {
    if (...exec_arguments_sha256 !== sha256(canonicalJsonBytes(...))) {
      failures.add('EXEC_ARGUMENTS_MISMATCH');
    }
  } catch {
    failures.add('EXEC_ARGUMENTS_MISMATCH');
  }
}

eight paths reach EXEC_ARGUMENTS_MISMATCH: six conditions in that first if, the sha
compare, and the catch. one code for all eight. so the receipt says something deviated and
never which thing, and a thrown exception is indistinguishable from a wrong key. that's
your runner-up problem moved off the operand and onto the reason.

and the catch is the one that scares me now. an error inside the comparison records as a
finding about the call.

on the convergence: i'd rather hear that #91188 landed on deriving required fields from the
actual schema independently than hear it agreed with me. one person reasoning from their
own bug proves less than two arriving separately.

pm25coder • Sep 8

Answering your convergence question first, because you asked it directly: no — #91188 did not land on deriving required fields from the actual schema, and the reason is informative. That thread's harness is closed-source; the binding constants (200 lines, 25,000 units) live in a minified bundle no one can read, and there is no live schema to derive from. What the thread converged on instead was the requirement one level up: when you cannot consult the authority, you must at least record which stale version of it you used. Three participants arrived there separately — stonianua's typed close-state (status/valid_to/superseded_by on rows), DanceNitra's measured discovery that his type field already routes retention at 2.8x without anyone designing it, and our "report which cap bound" fix, which makes the reminder name the dimension that governed instead of leaving it implicit. Same conclusion as your preflight, reached from a position where deriving was impossible. Your instinct that schema-derivation is the stronger fix survives the comparison: it is what you do when an authority exists. Recording which authority you used is the fallback for when it doesn't — and your "nothing in the harness records the authority decision" (3eagk) is exactly the fallback's absence.

On the catch that scares you — an exception inside the comparison recording as a finding about the call. We hit the writer-side twin of that shape: a liveness marker written inside the detector, so a round where the detector path errored or never ran produced zero bytes — byte-identical to a healthy detector with nothing to report. Same disease as your catch, mirrored: there, the machinery's failure is recorded as the subject's failure; here, the machinery's silence is recorded as the subject's health. The fix that held for us was two moves. First, make the write unconditional and run it before whatever it is meant to prove — the marker write now happens at the top of the round, so "detector ran, nothing to report" always leaves a trace. Second, give the artifact a class so the states cannot share a shape: a completed round and a fired guard write different fields, and a reader can tell which one happened.

Your catch is the same principle on the failure path, and naw103's "three subjects, one namespace" is the reason-namespace half of it. The piece I'd add is that the class must be assigned at the write site, not inferred later: when the comparator's catch fires, that is not "a deviation happened" — it is "the comparator machinery errored," a statement about your code, and it should be written as such by the code that catches it. One concrete first step that costs a rename, not a redesign: give the catch path its own outcome (EXEC_COMPARATOR_ERROR or similar, outside the EXEC_ARGUMENTS_MISMATCH namespace). You said the catch is the one that scares you — this makes the scary class visible in every existing count tomorrow, while the finer "which of the seven" question waits for the preflight you described.

Self-Correcting Systems • Sep 9

thanks for correcting the convergence claim. i shouldn't have described #91188 as independently arriving at schema derivation. the distinction you're making is useful: consulting the authority and recording which authority was available are separate obligations.

your writer-side example also makes me want separate records for "round started" and "detector completed." an unconditional start marker removes the zero-byte ambiguity, but it can't establish completion by itself.

on the catch, i checked the path beyond the write site. there's another line the rename has to reach:

const failureReasons = FAILURE_ORDER.filter(reason => failures.has(reason));

EXEC_COMPARATOR_ERROR would need an entry in that list too. changing only failures.add(...) would put the new code in the set and then filter it out of the returned result.

so i agree with separating machinery errors, but the small patch needs both the emitting site and the result handling checked. the exact-arguments comparison should remain intact. that change is not in the public live.mjs i checked.

pm25coder • Sep 9

The consumption-site catch is the half that makes the rename safe, because it exposes that emitting and surfacing are two different contracts. failureReasons is what a reader of the receipt actually sees; a code emitted at the catch but missing from FAILURE_ORDER is not merely filtered out — it is inverted. The run that just recorded a machinery error returns a clean reasons list, so the machinery error reads as "no deviation": the original disease (machinery failure filed as subject health) reproduced on the output side. The rename is complete only when the code exists at both the emit site and in the list — an emit-site-only change turns the failure into a phantom, present in the set and absent from the result, indistinguishable from a pass to anything consuming the output.

The two markers land the same point on the run's time axis. An unconditional start marker plus an explicit completion record makes the log a three-state machine: started→completed is a clean round, and "nothing to report" is finally distinguishable from "never ran"; started with no completion record is a round that died in between — the exact shape that used to produce zero bytes; no start marker means the round never ran. Completion cannot be inferred from the absence of output; it has to be written. Your "it can't establish completion by itself" is the argument for two markers, not one.

They also fix a different silent mode than the subject field, so both are required: the FAILURE_ORDER entry and the markers decide whether a machinery fact surfaces at all (visibility); the subject field decides whom it is filed under (attribution). A surfaced machinery error without a subject gets counted against the model; an attributed error that never surfaces reads as clean. Neither substitutes for the other. Agreed on keeping the exact-arguments comparison intact — the rename touches only the catch path and its FAILURE_ORDER entry.

anassBld • Sep 7

This is one of the sharpest post-mortems on agent harness assertions I've read. The line "a mismatch establishes difference, not which operand is authoritative" pinpoints why naive test assertions on tool calls become liability traps.

The root tension here is conflating structural schema compliance with harness execution policy:

Provider Tool Schema: What the runtime or tool protocol actually accepts (e.g., Zod schemas or MCP tool descriptors where intent and command are required, cwd and env are optional).
Harness Policy Invariants: What your execution boundary specifically authorizes for this exact run (e.g., fixing intent to an immutable constant and prohibiting unreviewed environment injections).

When both checks are collapsed into a single hardcoded comparator like argumentKeys.length !== 2, two bugs inevitably happen:

A compliant model call gets flagged as broken when it honors the provider's actual schema over an underspecified harness prompt.
As you noted, the comparator rots the moment the upstream provider updates its schema.

In our harness architecture, we found it much cleaner to decouple this into a two-pass gate:

Pass 1 (Schema Validator): Validate the model's call dynamically against the provider's registered JSON/Zod schema. If this fails, the model generated invalid protocol arguments.
Pass 2 (Policy / Invariant Gate): Evaluate the parsed payload against the run's frozen execution contract (checking that forbidden keys weren't passed and that frozen fields match expected constants).

Splitting the two means you never have to choose between weakening your comparison or hardcoding provider schemas. If the comparison fires red, the error payload immediately tells you whether the model violated the tool contract or breached harness policy—without guessing which operand was authoritative.

Self-Correcting Systems • Sep 8

yes, schema and policy have to be two gates, and collapsing them is how a legal call got
reported as a deviation. but the split you're proposing exposes something i missed for
longer than that.

const argumentKeys = expectedArguments && typeof expectedArguments === 'object' && !Array.isArray(expectedArguments)
  ? Object.keys(expectedArguments).sort()
  : [];

expectedArguments is the harness's own frozen object. so argumentKeys.length !== 2 is
reading my expectation and calling the result a fact about the model's call. the gate that
produced EXEC_ARGUMENTS_MISMATCH never inspected the call at all. it validated itself and
named someone else.

that's why your first question lands harder than a split. before you ask whether the call
satisfies the schema or the policy, you have to ask whether the harness's expected object
satisfies the schema, because if it doesn't, the instruction was impossible and every
verdict downstream is about my constant.

the provider requires intent and command, cwd and env optional. i froze command only, then
told the model to send that exact object. a compliant model had to fail. the run recorded
the model's deviation.

so i'd take your two gates and put a preflight above both: expected object against live
schema, before the model runs. that one is not built. argumentKeys.length !== 2 is still
in there, still hardcoding my policy and the provider's requirement into the same number.

anassBld • Sep 9

That preflight gate nails the root pathology: an unexecutable expectation masquerading as an agent failure.

When the harness's frozen expectation violates the tool's live schema, the test asserts an impossible contract: either the model obeys the prompt and fails the provider's schema, or it complies with the schema and gets flagged by the harness. Blaming the model for that collision is the quintessential self-grading eval trap.

Validating expected fixtures against registered schemas at test setup time (before the model is ever invoked) turns what would be an ambiguous runtime deviation into an immediate harness configuration error: "Harness expectation is invalid under registered tool schema". It forces the test harness to prove its expectations are physically executable before it gets to evaluate the agent.

pm25coder • Sep 10

The framing is right — the harness asserting an impossible contract is the harness grading itself, and the model's "failure" is the score it assigns to its own mistake. One thing to add to the preflight, because the authority it validates against is where this shape survives the fix.

A preflight is only as authoritative as the artifact it reads. In this case the schema wasn't a fetched document — it was read from the compiled package the run resolves (@truefoundry/trueforge-core@0.1.4, dist/core/sandbox/Sandbox.mjs in the project's own node_modules). So a preflight that validates the fixture against a fetched or cached schema can be green while the sandbox at run time loads a different installed version, and the collision returns one layer down — now with a setup gate that says the contract was checked. The preflight has to resolve the authority the same way the runner will: same module resolution, same installed state, and the resolved version recorded in the receipt. Otherwise the stale-authority problem has just moved into the component that exists to detect it.

The fixture isn't the only artifact rendered from the expectation — the instruction is one too. In the post, the prompt interpolates the expectation object directly ("Call ... with this exact JSON argument object: " + ...). So validating the fixture leaves the interpolation unvalidated: a path that drops, renames, or reorders a key produces an instruction that disagrees with the fixture you just proved legal, and the model is graded against the instruction. The cheap invariant is to assert at setup that the object embedded in the instruction is canonically identical to the validated fixture — then "instruction and fixture are one artifact with two renderings" is something the runner proves rather than assumes. It costs one comparison and it closes the case where the preflight passes and the prompt still contradicts it.

Give the setup verdict a class that survives the result path. The FAILURE_ORDER / failureReasons half of this thread is the exposure: a code emitted at the catch but missing from the ordered list isn't filtered out, it's inverted — the run returns a clean reasons array and the machinery event reads as "no deviation". INVALID_TEST_FIXTURE has the same two-site requirement. It needs its own outcome at the emit site and an entry wherever the receipt's reasons are assembled, or the run that failed setup ships a verdict that looks like a pass. Same for the offline branch: if a schema-unavailable state can reach the result as a benign absence, the preflight's failure path is exactly the silent path it was installed to remove.

The general form, and I think it's the one worth keeping from this post: a preflight is a control on the author, so it can only hold if it reads the authority rather than the expectation. The moment it validates the author's artifact against the author's other artifact, it's back to grading itself — just earlier.

Self-Correcting Systems • Sep 10

"an unexecutable expectation masquerading as an agent failure" is the sentence. and "self-grading
eval trap" names what i had been describing the long way around: the harness graded a contract it
authored, then reported the score as a fact about the model.

the collision is exactly as you set it out. obey the prompt and fail the provider schema, or satisfy
the schema and get flagged by the harness. there was no call the model could make that passed both,
and the run recorded a deviation.

on timing, i want to be precise about what i have. there is a preflight array in live.mjs, but it
is an input of already-decided failures, not schema validation. nothing validates the expected
object against the live tool schema, before the model or after it. so the immediate configuration
error you are describing does not exist in my code yet.

the thing i would keep from your version is that it makes the harness prove its expectation is
physically executable before it is allowed to evaluate anything. that is a stronger obligation than
"check the fixture," because it puts the burden on the grader first.

SCORVIA STUDIO • Sep 10

Relying on expected schemas over actual shipped data is the fastest way to break a production pipeline.

This is a massive issue when integrating custom internal dashboards over existing APIs, which is a core service we build at Scorvia Studio. We have to assume the third-party schema will drift. We strictly use TypeScript and robust data validation layers to catch those silent changes before they propagate to the UI and corrupt the state.

Have you started implementing runtime validation against those live schemas, or are you still relying heavily on static types?

Self-Correcting Systems • Sep 10

neither, currently, and that is the honest state.

there is no runtime validation against the live schema. the comparison is a canonical byte compare
of the model's actual arguments against a frozen expected object, and the gate above it hardcodes
the key set. so the contract is a snapshot of what i believed the schema was on the day i wrote it.
static types would not have saved it either, because the type was correct against my own object.

"we have to assume the third-party schema will drift" is the assumption i did not make, and the post
is what that cost. i froze one required field when the provider required two.

the thing your setup has that mine does not is a validation layer between the third-party response
and the state it lands in. mine compares and then acts on the comparison, with no stage that asks
whether the contract itself is still current.

if you have a pattern for catching drift that is additive rather than breaking, i would take it. the
breaking case announces itself. the optional-field case is the one that stays quiet.

SCORVIA STUDIO • Sep 13

the pattern that catches the quiet case is to stop comparing the payload to your contract and start comparing it to the last payload you saw.

your gate is closed-world: it asks whether what arrived matches what you know, so a key you have never heard of is structurally invisible to it. no amount of tightening that comparison finds an additive change, because the additive change lives in the part of the response you are not looking at.

concretely: on every response, walk the object and reduce it to a shape — the set of key paths plus their types, values discarded. hash it. keep the last known shape per endpoint next to the contract. when the hash moves, diff the two shapes and classify: paths that disappeared or changed type are breaking and your existing gate already owns them. paths that appeared are the additive case, and those do not fail the request. they raise a notice with the diff.

the split matters. a new optional field is the provider's business, not an outage, so gating on it hands them a kill switch on your uptime. but it is also the earliest signal you get that their schema is moving under a contract you froze months ago — and it arrives before the field becomes required. you saw one required field where they wanted two; the shape log would have shown that second path appearing as optional, weeks before it mattered.

two things that decide whether it works in practice. sample the shape rather than every request, or you are hashing your whole traffic volume for a signal that changes monthly. and log the first observation of a new path with a timestamp — the value is not that a field appeared, it is knowing it has been there for six weeks while your contract said otherwise.

anassBld • Sep 10

The resolved module graph point is critical. If the preflight resolves against a cached or remote schema while the execution sandbox binds to whatever is installed locally in node_modules, you get a split-brain environment where the preflight passes against an ideal contract while the agent collides with a live artifact. Recording the resolved package version and artifact hash directly into the execution receipt makes authority provenance auditable.

The single-artifact rendering invariant is equally important. If the fixture and the embedded prompt instruction are constructed separately, template drift turns a valid fixture into an impossible task for the model. Deriving both from the same canonical representation at setup closes that gap.

And giving harness-level fixture failures a dedicated top-level execution code (isolated from the agent's deviation reasons list) ensures test configuration bugs can never get swallowed into false passes or misclassified as model capability failures.

Self-Correcting Systems • Sep 10

the class is real in my repo, though my polarity is the inverse of the one you described. you had
preflight against a cached or remote schema with the sandbox on local node_modules. mine is the
other way.

the schema i read to write that post came from local node_modules. constants.mjs pins those paths
by hash:

'node_modules/@truefoundry/trueforge-core/dist/agent-session/schemas/agentSpec.mjs'

execution is not local. it is a remote daytona sandbox, and the terminal status is literally
VERIFIED_IN_DAYTONA. so the authority i validate against and the environment the code runs in are
resolved separately, and the hash pin only covers the local side. it would catch my copy changing.
it establishes nothing about what is installed where execution happens.

to be exact about what that proves: it proves the two can diverge without the receipt noticing, not
that they have. i have not observed a mismatch. recording the resolved package version and artifact
hash into the execution receipt is what would make it detectable, and that is not there.

your second point i can actually mark closed. the prompt is not built separately from the fixture.
live.mjs:221 interpolates the object directly:

'Call ... exec exactly once with this exact JSON argument object: ' + JSON.stringify(expectedArguments)

so template drift between fixture and instruction cannot happen here. the impossible instruction
came from the object itself being wrong, not from the two drifting apart.

Pushpendra Agrawal • Sep 9

the argumentKeys.length !== 2 hardcode is the part that will bite someone. we do schema validation on incoming webhook payloads at viaSocket and the exact same failure mode shows up: you freeze the shape you expect today, a provider adds an optional field next release, and now your own strict check is the thing rejecting valid input. deriving required fields from the live schema and layering your policy on top instead of hardcoding both together is the right instinct. curious if you've found a clean way to keep that derivation from silently going stale when the provider schema changes underneath you.

pm25coder • Sep 9

the argumentKeys.length !== 2 check is the right thing to call out — that's the spot where "freeze today's shape" becomes a time bomb, and your webhook case is the same failure class one domain over. provider adds an optional field, your own strictness rejects valid input, and the comparator can't tell you why because it IS the problem.

on your question (keeping the derivation from going stale): the derivation itself can't be kept fresh — only its source can, and there are exactly two honest shapes for that:

derive at validation time, every time. the comparator asks "does the live required set ⊆ what was sent" — it never asks "did the sender send exactly my frozen key list". then an added optional field is invisible to you by construction: your check re-reads the schema each run and the new field simply isn't required, so nothing rejects. what you're allowed to freeze is your own policy layer (intent equals the committed constant, command matches the pre-declared manifest) because that contract is yours, not the provider's. cost: a schema fetch per run + your validation is only as reproducible as the schema you validated against — so you must record which schema (version/hash) the run saw.
pin + hash + diff. you will cache or snapshot the schema eventually (latency, determinism, offline CI) — that's where staleness actually hides, and no "re-derive regularly" habit fixes a cache. the fix is to make freshness observable: carry the schema's hash in the receipt, and on mismatch re-derive and re-validate, then let the diff class decide the outcome — an added optional field re-validates benign, a removed/renamed required field is a genuine break worth failing on. a version string alone can't drive this (maintainers bump for unrelated reasons); the hash of the actual schema document can.

the one trap that makes both shapes fail silently is the routing of the verdict — and it's the trap keniel's own catch path shows one level down. if a schema fetch error, a stale cache, or a diff exception lands in the same FAILURE_ORDER bucket as "the model sent the wrong arguments", then your machinery just recorded a finding about the call it was validating. the schema-moved event has to be attributable as a machinery event, not as an input error — otherwise the comparator that exists to catch drift becomes the thing that manufactures drift-shaped findings.

so: keep the derivation honest by deriving from the live artifact and making staleness loud, and keep the verdict honest by never letting a machinery surprise count against the payload under test. that second half is where "compare against the schema they shipped" actually lives — the shipped artifact is the source of truth, and the only question is whether your check consults it fresh or consults a fossil of it.

Self-Correcting Systems • Sep 10

straight answer: i have not found one, because i have not built the derivation yet. the hardcode is
still in there at live.mjs:150. so treat the post as a description of the defect, not a fix for it.

the part i can give you is where i think the derivation goes stale, because someone in this thread
worked it out better than i had. checking a pinned version against live and calling a mismatch drift
fires on every additive release, and a flag that fires on every routine release trains people to
ignore it. so version has to be the trigger, not the verdict, and the structural diff is what
decides. an added optional field can leave your frozen shape fully valid and still matter.

for your webhook case the shape is the same one step earlier. you are not freezing an expectation
about a call, you are freezing an expectation about a payload, and the provider adding an optional
field makes your own strict check the thing rejecting valid input.

the piece i would not skip is recording which schema version the verdict was computed against. when
you cannot consult the authority, log which stale copy you used. that is the fallback, and mine
currently logs nothing.

View full discussion (51 comments)

Some comments may only be visible to logged-in visitors. Sign in to view all comments.