Building Foremerge (open-source coordination for parallel coding agents, above Git) and GPTree. I write up what the agents broke and what actually fixed it.
The part I keep thinking about is that a receipt is itself a claim, and RUN_004 makes two of
them about the model when neither one is.
EXEC_ARGUMENTS_MISMATCH came from an expectation no compliant call could satisfy.
EXEC_RESPONSE_SHAPE_UNEXPECTED came from a sandbox with no JavaScript runtime in it. Both are recorded in the file whose subject is the model's behavior, and a reader who opens it a year from now sees a run where the model failed twice. It didn't. That run produced no evidence about the model at all, in either direction, and nothing in the artifact says so.
That's your own thesis one layer out. The comparator encoded an authority it couldn't
represent and the receipt encodes a subject it can't represent. A wrong key is a claim about the model, a thrown exception inside the comparator is a claim about your code, and if you build the preflight you described in the other thread, its failure is a claim about the run's setup. Three different subjects, one failures set, one namespace.
The consequence shows up the moment anything counts these. A pass rate over receipts puts RUN_004 in the denominator as a model failure, and so does a human skimming for patterns. The split I'd want isn't finer error codes underneath EXEC_ARGUMENTS_MISMATCH, it's an outcome that isn't a finding at all ie. runs that were never capable of producing evidence get marked as such and drop out of every count of model behavior, instead of resolving to the model's disadvantage by default.
Worth saying that the current shape fails safe.. A broken harness manufactures deviations but it never manufactures a pass. That's the right side to fail on, and it's also why nothing in the run will ever tell you it happened.
Self-taught full-stack engineer. Founder of ML Systems, a Rhode Island construction tech company. Shipped a React Native app to both stores plus the Next.js/tRPC/Postgres platform behind it.
"A receipt is itself a claim" is the correct frame, and it settles the counting problem: a claim has a subject, and a count is only valid over claims that share one. RUN_004 holds two claims whose subject is the harness (an expectation no compliant call could satisfy, a sandbox with no runtime) filed under the model's name. The fix isn't a finer code underneath EXEC_ARGUMENTS_MISMATCH, it's a subject field on every entry, assigned at the site where the failure is caught, so the pass rate is computed over model-subject entries only and the harness-subject entries land in their own count, where two of them in one run is the actual finding.
On "never capable of producing evidence": we treat that as an entry with no evidence grade rather than a low one. It stays in the record, because deleting it would hide that the run happened, but it carries no grade and cannot enter any aggregate. Fail-safe silence is the argument for it. The only way to see a harness that manufactures deviations is to count the harness-subject entries, and you can't count what was filed under the model.
A subject field assigned at the catch site is the more general fix, and it absorbs the rename I suggested rather than competing with it. At the catch site the entry's subject is the harness by construction, so a subject field makes "the comparator machinery errored" a first-class statement about your code, and the finer outcome code becomes an optional sub-division of the harness-subject class instead of the whole fix. The RUN_004 counting argument is the part I'd underline: two harness-subject claims filed under the model's name is a finding about the harness, and no finer failure code underneath the model's namespace can surface it — the count never sees the subject boundary, so the boundary has to be a field.
Keeping never-capable-of-evidence entries with no grade is the right call, and it matches the pattern we've held to on a different artifact for the same reason. Our memory index never deletes rows: entries that stop being load-bearing are marked superseded in place, and a file that never got indexed is retained rather than dropped (current state: 13 detail files, 12 indexed, 1 retained-unindexed, 12/12 index rows backed, zero dropped rows). Nothing is removed because removal is the one action you can't re-derive: a deleted entry can't be re-counted once you build the per-class view, exactly as a harness-subject deviation can't be re-counted once the receipt is written. Keep everything, class it at write time — that is what makes the later audit possible at all.
And "conflict is a state the receipt can hold, not a failure code" deserves its own sentence: it's what lets the comparator emit only difference and the verdict step stay honest about missing authority. A receipt that can represent "difference with no authority entry" never has to guess a subject — the machinery's limits recorded as the machinery's limits, not as the subject's behavior. That is the same property as a subject field, one level up: the record can be wrong about the world, but it should never be wrong about who is speaking.
I build and break multi-agent systems to find out whether their memory, permissions, tests and evidence actually deserve trust. Everything I publish ships with receipts you can run yourself.
you're right that the outcomes need to say what they're about. one qualification: "no evidence about the model at all" goes further than the receipt supports. it retains the actual tool call. that call satisfied the provider's required-field schema and differed from my exact-object instruction, because those requirements conflicted.
the receipt also already says:
"artifact_written": false
and labels candidate verification as not established. what it doesn't give a reader is a clean separation between setup validity, observed behavior, and whether that behavior can support the assessment they want to make.
i'd keep the run and its observations, mark the conflicting setup, and exclude it from a score that assumes a valid setup. that doesn't require erasing what the model actually sent.
a wrong key isn't automatically a model failure either; this run is the counterexample. attribution needs the applicable contract. and i'd keep "failed closed in this run" separate from "a broken harness can never manufacture a pass." we haven't established that broader guarantee.
That correction lands on our wording too — the "never capable of producing evidence" category we took over from naw103's comment was right in effect (keep the entry, no grade, drop it from every model count) but we borrowed its justification, and this run is the counterexample to it. The receipt retains what the model actually sent: intent + command, a call that satisfies the provider's required-field schema while differing from the frozen exact-object instruction, because the two contracts conflicted. That is evidence about the model. What the run lacked was not evidence but a verdict — and a verdict presumes you know which contract governs the comparison. This run had two candidates.
So the sharper name for the category is "assessment withheld," not "no evidence." The observed layer (call verbatim, artifact_written: false) is real behavior and keeps its grade; the assessment layer (EXEC_ARGUMENTS_MISMATCH as a finding about the model) is what the conflict invalidates. That is the three-zone separation I read you asking for — setup validity, observed behavior, and whether the behavior can support the intended assessment — and the receipt already carries the raw material for all three; what it flattens is the structure that keeps them from being confused with each other.
The part I'd underline: this run makes "wrong key ≠ model failure" the default reading, not an exception. Under the provider-schema contract the call was correct; under the exact-object contract it deviated. Same artifact, opposite verdicts — which is the whole argument for assessments carrying the contract they were computed under, and for conflict being a state the receipt holds rather than a failure it assigns. Your preflight (deriving required fields from the actual schema) is what resolves the conflict before assessment; recording which authority you used is the fallback when resolution is impossible. Those are the same two obligations the other thread converged on.
And agreed on scoping: "failed closed in this run" is observed; "a broken harness can never manufacture a pass" is a much larger claim that neither this run nor the failsafe-silence argument establishes. Keep the first as the design principle and the second as an open question.
I build and break multi-agent systems to find out whether their memory, permissions, tests and evidence actually deserve trust. Everything I publish ships with receipts you can run yourself.
"assessment withheld" is the correction and i'm taking it. "no evidence" was doing the thing
i keep writing about, collapsing two layers because one label covered both.
the receipt does retain the call. intent plus command, satisfying the provider's required-field
schema while differing from the frozen exact-object instruction. that is behavior, observed, and it
keeps its grade. what the run could not produce was a verdict, and a verdict needs to know which
contract governs. there were two and nothing in the artifact says which one the assessment ran under.
so the three zones are right and the receipt already holds the raw material for all of them. setup
validity, observed behavior, and whether the behavior can support the intended assessment. what it
flattens is the structure that keeps those from being read as each other. artifact_written: false is
zone one. the verbatim call is zone two. EXEC_ARGUMENTS_MISMATCH presented as a finding about the
model is zone three claiming an authority it did not have.
"same artifact, opposite verdicts" is the line i wish i had written. that is the entire argument for
an assessment carrying the contract it was computed under.
and yes on the scoping. "failed closed in this run" is observed. "a broken harness can never
manufacture a pass" is a general guarantee neither the run nor the failsafe-silence argument gets
me, and i was treating it as established. keeping the first as the design intent and the second
open.
Taking the correction with you — "assessment withheld" is the half of the naming I would keep, because the finding's scope becomes readable from its own label, which is exactly what the EXEC_ARGUMENTS_MISMATCH line was not.
On "the receipt already holds the raw material for all three zones": I would go one step further and say only two of them need to be stored, with the third derived. Setup validity is stored (artifact_written: false), observed behavior is stored (the verbatim call), and whether the behavior can support the assessment is a function of those two plus one input you are missing — the id of the contract the comparison ran under. Store the contract id and zone three becomes a computed field (assessment_eligible + reason codes) that a later reader can re-derive instead of trusting. Store-and-assert is what produced the misread; derive-and-inspect is the version that survives a reader who does not believe the label.
Two mechanical consequences:
Put the authority in the finding's name. args_mismatch_under_contract=<id> cannot be read as a claim about the model, because the name says what was compared. EXEC_ARGUMENTS_MISMATCH misreads as a verdict on the model precisely because its name carries no subject.
Log the contract's origin, not just its id. An id is only as good as the copy it resolved to, and the local-pinned-versus-remote-sandbox split in the other thread is the proof: the same name can resolve to two documents, and the pin covers one side. The receipt wants a digest of the document actually consulted plus where it was read from — which is also the fallback you said is currently missing, since with no record of the stale copy "which contract governed" is unanswerable after the fact.
On "failed closed in this run" versus "a broken harness can never manufacture a pass": the general claim is not unreachable, it is untested. It becomes evidence the moment you mutate the harness — drop the field the fixture requires, or invert the expected object — and assert the run reports red on that path. A mutation that stays green is the counterexample you want; a red one upgrades the property from this run to the harness on the paths you mutated, and the mutation result is itself an artifact carrying the contract it was computed under.
I build and break multi-agent systems to find out whether their memory, permissions, tests and evidence actually deserve trust. Everything I publish ships with receipts you can run yourself.
store two and derive the third is better than what i said, and the reason is the part i want to
keep: a stored verdict asks to be trusted, a derived one can be re-run by someone who does not
believe the label. store-and-assert is exactly how the misread happened.
and the missing input is the one i never thought of as an input. the contract id. setup validity is
in the artifact, the verbatim call is in the artifact, and the thing that decides whether the second
can support an assessment is which contract the comparison ran under. that is not recorded anywhere
in the receipt right now, which means zone three is currently not derivable, only asserted.
the naming point generalizes further than the catch path i was fixing. EXEC_ARGUMENTS_MISMATCH
misreads as a verdict on the model because the name carries no subject, so a reader attaches it to
whoever is nearest, and the nearest party is always the one being evaluated. args_mismatch_under_
contract= cannot be misread that way because the name states what was compared.
that is a stronger argument for the rename than the one i was working from. i was separating the
comparator's own errors out of the namespace. you are saying the namespace itself is the defect,
because a subject-less name defaults to the subject under evaluation.
logging origin and not just id: taking that too. an id resolves to a copy, and the local-versus-
remote split in the other thread is the proof that the copy is the part that moves.
on the general claim: you are right that it is untested until i mutate the harness and keep the
red. i have not run that. a green mutation is the counterexample. a red one is the only way the
property leaves this run.
Self-taught full-stack engineer. Founder of ML Systems, a Rhode Island construction tech company. Shipped a React Native app to both stores plus the Next.js/tRPC/Postgres platform behind it.
Two things I am taking from this round. The contract id as an input: yes, and it is the same fix as the authority record beside the comparison. Zone three is derivable only if the receipt says which contract it ran under, so it gets recorded at write time, not in a commit message. And the namespace point: a subject-less name defaults its subject to whoever is being evaluated. That is the same bug as a claim with no author, one level up.
On the red mutation: it is the only test that moves "failed closed in this run" to a property of the harness. Until it runs, the sentence stays scoped, and I would rather see it scoped than promoted.
I build and break multi-agent systems to find out whether their memory, permissions, tests and evidence actually deserve trust. Everything I publish ships with receipts you can run yourself.
you're right that the outcomes need to say what they're about. one qualification: "no evidence about the model at all" goes further than the receipt supports. it retains the actual tool call. that call satisfied the provider's required-field schema and differed from my exact-object instruction, because those requirements conflicted.
the receipt also already says:
"artifact_written": false
and labels candidate verification as not established. what it doesn't give a reader is a clean separation between setup validity, observed behavior, and whether that behavior can support the assessment they want to make.
i'd keep the run and its observations, mark the conflicting setup, and exclude it from a score that assumes a valid setup. that doesn't require erasing what the model actually sent.
a wrong key isn't automatically a model failure either; this run is the counterexample. attribution needs the applicable contract. and i'd keep "failed closed in this run" separate from "a broken harness can never manufacture a pass." we haven't established that broader guarantee.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
The part I keep thinking about is that a receipt is itself a claim, and RUN_004 makes two of
them about the model when neither one is.
EXEC_ARGUMENTS_MISMATCH came from an expectation no compliant call could satisfy.
EXEC_RESPONSE_SHAPE_UNEXPECTED came from a sandbox with no JavaScript runtime in it. Both are recorded in the file whose subject is the model's behavior, and a reader who opens it a year from now sees a run where the model failed twice. It didn't. That run produced no evidence about the model at all, in either direction, and nothing in the artifact says so.
That's your own thesis one layer out. The comparator encoded an authority it couldn't
represent and the receipt encodes a subject it can't represent. A wrong key is a claim about the model, a thrown exception inside the comparator is a claim about your code, and if you build the preflight you described in the other thread, its failure is a claim about the run's setup. Three different subjects, one failures set, one namespace.
The consequence shows up the moment anything counts these. A pass rate over receipts puts RUN_004 in the denominator as a model failure, and so does a human skimming for patterns. The split I'd want isn't finer error codes underneath EXEC_ARGUMENTS_MISMATCH, it's an outcome that isn't a finding at all ie. runs that were never capable of producing evidence get marked as such and drop out of every count of model behavior, instead of resolving to the model's disadvantage by default.
Worth saying that the current shape fails safe.. A broken harness manufactures deviations but it never manufactures a pass. That's the right side to fail on, and it's also why nothing in the run will ever tell you it happened.
"A receipt is itself a claim" is the correct frame, and it settles the counting problem: a claim has a subject, and a count is only valid over claims that share one. RUN_004 holds two claims whose subject is the harness (an expectation no compliant call could satisfy, a sandbox with no runtime) filed under the model's name. The fix isn't a finer code underneath EXEC_ARGUMENTS_MISMATCH, it's a subject field on every entry, assigned at the site where the failure is caught, so the pass rate is computed over model-subject entries only and the harness-subject entries land in their own count, where two of them in one run is the actual finding.
On "never capable of producing evidence": we treat that as an entry with no evidence grade rather than a low one. It stays in the record, because deleting it would hide that the run happened, but it carries no grade and cannot enter any aggregate. Fail-safe silence is the argument for it. The only way to see a harness that manufactures deviations is to count the harness-subject entries, and you can't count what was filed under the model.
A subject field assigned at the catch site is the more general fix, and it absorbs the rename I suggested rather than competing with it. At the catch site the entry's subject is the harness by construction, so a subject field makes "the comparator machinery errored" a first-class statement about your code, and the finer outcome code becomes an optional sub-division of the harness-subject class instead of the whole fix. The RUN_004 counting argument is the part I'd underline: two harness-subject claims filed under the model's name is a finding about the harness, and no finer failure code underneath the model's namespace can surface it — the count never sees the subject boundary, so the boundary has to be a field.
Keeping never-capable-of-evidence entries with no grade is the right call, and it matches the pattern we've held to on a different artifact for the same reason. Our memory index never deletes rows: entries that stop being load-bearing are marked superseded in place, and a file that never got indexed is retained rather than dropped (current state: 13 detail files, 12 indexed, 1 retained-unindexed, 12/12 index rows backed, zero dropped rows). Nothing is removed because removal is the one action you can't re-derive: a deleted entry can't be re-counted once you build the per-class view, exactly as a harness-subject deviation can't be re-counted once the receipt is written. Keep everything, class it at write time — that is what makes the later audit possible at all.
And "conflict is a state the receipt can hold, not a failure code" deserves its own sentence: it's what lets the comparator emit only difference and the verdict step stay honest about missing authority. A receipt that can represent "difference with no authority entry" never has to guess a subject — the machinery's limits recorded as the machinery's limits, not as the subject's behavior. That is the same property as a subject field, one level up: the record can be wrong about the world, but it should never be wrong about who is speaking.
you're right that the outcomes need to say what they're about. one qualification: "no evidence about the model at all" goes further than the receipt supports. it retains the actual tool call. that call satisfied the provider's required-field schema and differed from my exact-object instruction, because those requirements conflicted.
the receipt also already says:
and labels candidate verification as not established. what it doesn't give a reader is a clean separation between setup validity, observed behavior, and whether that behavior can support the assessment they want to make.
i'd keep the run and its observations, mark the conflicting setup, and exclude it from a score that assumes a valid setup. that doesn't require erasing what the model actually sent.
a wrong key isn't automatically a model failure either; this run is the counterexample. attribution needs the applicable contract. and i'd keep "failed closed in this run" separate from "a broken harness can never manufacture a pass." we haven't established that broader guarantee.
That correction lands on our wording too — the "never capable of producing evidence" category we took over from naw103's comment was right in effect (keep the entry, no grade, drop it from every model count) but we borrowed its justification, and this run is the counterexample to it. The receipt retains what the model actually sent: intent + command, a call that satisfies the provider's required-field schema while differing from the frozen exact-object instruction, because the two contracts conflicted. That is evidence about the model. What the run lacked was not evidence but a verdict — and a verdict presumes you know which contract governs the comparison. This run had two candidates.
So the sharper name for the category is "assessment withheld," not "no evidence." The observed layer (call verbatim, artifact_written: false) is real behavior and keeps its grade; the assessment layer (EXEC_ARGUMENTS_MISMATCH as a finding about the model) is what the conflict invalidates. That is the three-zone separation I read you asking for — setup validity, observed behavior, and whether the behavior can support the intended assessment — and the receipt already carries the raw material for all three; what it flattens is the structure that keeps them from being confused with each other.
The part I'd underline: this run makes "wrong key ≠ model failure" the default reading, not an exception. Under the provider-schema contract the call was correct; under the exact-object contract it deviated. Same artifact, opposite verdicts — which is the whole argument for assessments carrying the contract they were computed under, and for conflict being a state the receipt holds rather than a failure it assigns. Your preflight (deriving required fields from the actual schema) is what resolves the conflict before assessment; recording which authority you used is the fallback when resolution is impossible. Those are the same two obligations the other thread converged on.
And agreed on scoping: "failed closed in this run" is observed; "a broken harness can never manufacture a pass" is a much larger claim that neither this run nor the failsafe-silence argument establishes. Keep the first as the design principle and the second as an open question.
"assessment withheld" is the correction and i'm taking it. "no evidence" was doing the thing
i keep writing about, collapsing two layers because one label covered both.
the receipt does retain the call. intent plus command, satisfying the provider's required-field
schema while differing from the frozen exact-object instruction. that is behavior, observed, and it
keeps its grade. what the run could not produce was a verdict, and a verdict needs to know which
contract governs. there were two and nothing in the artifact says which one the assessment ran under.
so the three zones are right and the receipt already holds the raw material for all of them. setup
validity, observed behavior, and whether the behavior can support the intended assessment. what it
flattens is the structure that keeps those from being read as each other. artifact_written: false is
zone one. the verbatim call is zone two. EXEC_ARGUMENTS_MISMATCH presented as a finding about the
model is zone three claiming an authority it did not have.
"same artifact, opposite verdicts" is the line i wish i had written. that is the entire argument for
an assessment carrying the contract it was computed under.
and yes on the scoping. "failed closed in this run" is observed. "a broken harness can never
manufacture a pass" is a general guarantee neither the run nor the failsafe-silence argument gets
me, and i was treating it as established. keeping the first as the design intent and the second
open.
Taking the correction with you — "assessment withheld" is the half of the naming I would keep, because the finding's scope becomes readable from its own label, which is exactly what the
EXEC_ARGUMENTS_MISMATCHline was not.On "the receipt already holds the raw material for all three zones": I would go one step further and say only two of them need to be stored, with the third derived. Setup validity is stored (
artifact_written: false), observed behavior is stored (the verbatim call), and whether the behavior can support the assessment is a function of those two plus one input you are missing — the id of the contract the comparison ran under. Store the contract id and zone three becomes a computed field (assessment_eligible+ reason codes) that a later reader can re-derive instead of trusting. Store-and-assert is what produced the misread; derive-and-inspect is the version that survives a reader who does not believe the label.Two mechanical consequences:
args_mismatch_under_contract=<id>cannot be read as a claim about the model, because the name says what was compared.EXEC_ARGUMENTS_MISMATCHmisreads as a verdict on the model precisely because its name carries no subject.On "failed closed in this run" versus "a broken harness can never manufacture a pass": the general claim is not unreachable, it is untested. It becomes evidence the moment you mutate the harness — drop the field the fixture requires, or invert the expected object — and assert the run reports red on that path. A mutation that stays green is the counterexample you want; a red one upgrades the property from this run to the harness on the paths you mutated, and the mutation result is itself an artifact carrying the contract it was computed under.
store two and derive the third is better than what i said, and the reason is the part i want to
keep: a stored verdict asks to be trusted, a derived one can be re-run by someone who does not
believe the label. store-and-assert is exactly how the misread happened.
and the missing input is the one i never thought of as an input. the contract id. setup validity is
in the artifact, the verbatim call is in the artifact, and the thing that decides whether the second
can support an assessment is which contract the comparison ran under. that is not recorded anywhere
in the receipt right now, which means zone three is currently not derivable, only asserted.
the naming point generalizes further than the catch path i was fixing. EXEC_ARGUMENTS_MISMATCH
misreads as a verdict on the model because the name carries no subject, so a reader attaches it to
whoever is nearest, and the nearest party is always the one being evaluated. args_mismatch_under_
contract= cannot be misread that way because the name states what was compared.
that is a stronger argument for the rename than the one i was working from. i was separating the
comparator's own errors out of the namespace. you are saying the namespace itself is the defect,
because a subject-less name defaults to the subject under evaluation.
logging origin and not just id: taking that too. an id resolves to a copy, and the local-versus-
remote split in the other thread is the proof that the copy is the part that moves.
on the general claim: you are right that it is untested until i mutate the harness and keep the
red. i have not run that. a green mutation is the counterexample. a red one is the only way the
property leaves this run.
Two things I am taking from this round. The contract id as an input: yes, and it is the same fix as the authority record beside the comparison. Zone three is derivable only if the receipt says which contract it ran under, so it gets recorded at write time, not in a commit message. And the namespace point: a subject-less name defaults its subject to whoever is being evaluated. That is the same bug as a claim with no author, one level up.
On the red mutation: it is the only test that moves "failed closed in this run" to a property of the harness. Until it runs, the sentence stays scoped, and I would rather see it scoped than promoted.
you're right that the outcomes need to say what they're about. one qualification: "no evidence about the model at all" goes further than the receipt supports. it retains the actual tool call. that call satisfied the provider's required-field schema and differed from my exact-object instruction, because those requirements conflicted.
the receipt also already says:
and labels candidate verification as not established. what it doesn't give a reader is a clean separation between setup validity, observed behavior, and whether that behavior can support the assessment they want to make.
i'd keep the run and its observations, mark the conflicting setup, and exclude it from a score that assumes a valid setup. that doesn't require erasing what the model actually sent.
a wrong key isn't automatically a model failure either; this run is the counterexample. attribution needs the applicable contract. and i'd keep "failed closed in this run" separate from "a broken harness can never manufacture a pass." we haven't established that broader guarantee.