DEV Community

Same Evidence, Different Judgment? A Small Experiment on Implied Conclusions and Evidence Ordering

GnomeMan4201 on October 31, 2026

GnomeMan4201 · Study: composed-inference-ordering-v1.1 · Collected October 31, 2026 UTC What I ran: Four synthetic evidence cases × three presenta...
Collapse
 
paullovelace profile image
Paul Lovelace •

The 19/36 rejected just for markdown fences around JSON is the kind of boring detail that actually matters — that's not a judgment problem, that's a parser problem eating half your sample. And your own caveat is the honest part: one response per condition can't prove ordering caused anything. If you reran it, would you fix the parser first, or keep it frozen to stay comparable?

Collapse
 
gnomeman4201 profile image
GnomeMan4201 •

I’d keep the original parser and results frozen for comparability, but fix the parser before running a new experiment.

I actually did a separate exploratory pass afterward that removed only the Markdown fences. That recovered the rejected responses without changing the original scoring. The explicit judge remained stable, but the open audit still showed one verdict difference.

I think the more important next step is repeated sampling under each condition. Otherwise, it’s still difficult to separate ordering effects from ordinary model variability.

Appreciate you pointing out the parser issue. It’s definitely something I’d handle differently in the next protocol.

Collapse
 
robertpaschal profile image
Odinaka Robert Nnamani •

The shared-host audit flip is the detail that stood out. Same evidence, different verdict, and you were clear about one sample per condition not proving cause.

Collapse
 
lin7c profile image
xcl •

The shared-host audit flip is difficult to separate from sampling variability because each task-condition was sampled once at the reported temperature of 1.0, with no replicate set. That limitation matters more than the ordering contrast itself, since the extract task also changed labels between baseline and evidence_reverse even though its paragraph input was identical in both conditions. The frozen parser result adds a second confound: 17 of 36 otherwise readable responses were rejected only for Markdown fences around the JSON, so format compliance and judgment quality need separate denominators. Would a follow-up with multiple replicates per condition and a predeclared format-normalized scorer be enough to distinguish a genuine paragraph-order effect from ordinary run-to-run variance?

Collapse
 
alexshev profile image
Alex Shev •

This is a useful reminder that evidence quality and evidence order are different variables. I’d be curious to see a follow-up that randomizes ordering while keeping source reliability labels fixed, then measures whether people calibrate confidence differently when a weak but vivid item appears first. That would separate a retrieval problem from a presentation problem.

Collapse
 
gnomeman4201 profile image
GnomeMan4201 •

That’s an angle I hadn’t really considered, Alex. Especially whether putting weaker but more vivid evidence first changes how confident someone feels about a conclusion.

Worth keeping keep in mind and i appreciate you pointing it out

If you’ve got a specific way you’d test that, I’d be interested to hear it. I know there’s plenty I haven’t considered, and that’s part of why I wanted to put this research out there. I’m always open to someone challenging the approach or pointing out something I’ve missed.

The research and protocol are on GitHub if you want to take a look:

github.com/GnomeMan4201/composed-i...

Collapse
 
deanlee profile image
Dean Lee •

The divergence between the explicit judge and the open audit is the cleanest data point in the run. When a prompt asks the model to isolate an implied claim and then evaluate it, the extraction step effectively primes the context with the very narrative it is supposed to test. In joint hypothesis testing, generating the claim first gives it an implicit prior advantage over neutral alternatives like routine shared hosting. The seventeen parser rejections from code fences are also a familiar operational tax. Brittle schema gates regularly wipe out half the sample before any semantic disagreement gets scored.

Collapse
 
gnomeman4201 profile image
GnomeMan4201 • • Edited

I wanted to take a closer look at your points before replying, particularly the possibility of the evaluation process influencing the conclusion it’s supposed to assess.

I went back through the v1.1 implementation, frozen prompts, and raw responses.

One distinction worth clarifying is that extraction and explicit judgment were separate stateless requests. The judge received a predefined candidate conclusion, not the extractor’s output. So there wasn’t a direct extraction to judgment feedback loop.

The combined audit is where your concern becomes more interesting. It identifies an implied claim and evaluates that same claim within a single request. I haven’t isolated whether that introduces a commitment or priming effect, and it’s a plausible alternative explanation that deserves its own controlled test.

I also checked the 17 parser rejections against the raw responses. All 17 were Markdown fenced JSON, rather than missing or otherwise malformed content. The strict parser rejected them as specified, and the separate exploratory analysis removed only the fences.

One additional limitation worth noting is that the formatting failures weren’t evenly distributed across conditions. That makes it important not to treat the exploratory results as equivalent to the original strict scoring.

What I’d like to test next is whether the audit’s judgment changes when the candidate claim is specified independently, rather than generated and evaluated within the same request.

I appreciate you digging into the methodology. Your comment pushed me to inspect the actual implementation rather than rely solely on the write up, which was worth doing.

Collapse
 
golflover2023 profile image
golflover •

The 17/36 parser rejections on Markdown fences are the most actionable number here: that is infrastructure, not model behavior. Strip fences (or constrain the decode grammar) and 47% of the rejections re-enter the evaluable pool - worth re-running the audit on the recovered set to see which condition effects survive. On the judge split: a free-form audit invites narrative completion, which is exactly the implied-conclusion failure you study - the arrangement invites coordination and the narrator obliges; scoring the invited conclusion as its own claim would test that directly. With n=1 per cell, how do you separate ordering effects from sampler noise? Multiple seeds at temperature 0 would give within-cell variance before attributing anything to presentation order.

Collapse
 
ontomato profile image
ontomato •

if two people can reorder the same pile and land on different totals, the pile was never locked to one claim.

we stopped treating retrieved chunks as the answer for that reason. every conclusion has to say which claim it used, and a teammate should be able to reopen the evidence for that claim without guessing the order you read it in. different judgment from the same stack is a definition problem, not a sorting one.

Collapse
 
jkming profile image
jkming •

The stability-vs-validity distinction is the part I'd underline. Judge agreement is the easiest thing to over-trust in an eval harness: two runs can be perfectly consistent and consistently wrong, because both inherit the same framing from the prompt. One thing that helped me was running claim extraction and support judgment as separate calls with fresh context, so the judge doesn't inherit the extractor's phrasing. It doesn't remove priming entirely since the extracted claim still carries framing, but at least the two failure modes show up separately in the logs. Curious whether your explicit-judge condition got the composed paragraph verbatim or a fresh extraction.