27 recorded LLM-judge runs, published as JSON

Decider answers one narrow question: given a task and two candidate answers, which one is stronger — and how confident is that call?

We publish the runs behind it as data rather than as marketing examples. Every entry is the verbatim output of a real run: the task, both candidate answers, the pick, the calibrated confidence, and the stated reason. Nothing is hand-written.

• The dataset: https://github.com/TuringCorp-net/poe-demo-public (examples.json, 27 runs)
• Browsable version, one page per run: https://api.turingcorp.net/platform/poe/examples
• What it is, how it is measured, what it costs: https://api.turingcorp.net/platform/poe

The part that is actually interesting: confidence

On raw pick rate a judge of this kind ties a plain single-model baseline — roughly 92.5% against 92.7% on JudgeBench. We do not claim otherwise.

What a single pass cannot give you is how close the call was. Ours is calibrated against outcomes: when Decider reports confidence of 90% or higher it is right 99.6% of the time, and that band is 45.6% of all judgments; in the 80–90% band it is right 94.0%.

That is the whole product idea: a pipeline can commit on a clear call and escalate on a close one, instead of stalling on two answers that both look fine.

Method and calibration are described in our preprints:

• Calibrated confidence — https://doi.org/10.6084/m9.figshare.33684823
• Multi-model collaboration on LiveBench/IFEval — https://doi.org/10.6084/m9.figshare.32824352
• The framework behind both — https://doi.org/10.6084/m9.figshare.30645869

Why publish the runs

Numbers in a README are unfalsifiable. A JSON file with every task, both options, the pick and the reason is checkable — including the runs we would rather not highlight. If you are building an evaluation pipeline, a routing decision or a quality gate, the file is the useful artifact, not the claim.

The 27 runs are free to read inside the app as well, without sending a message and without being charged.

Disclosure: JudgeBench results are our own runs on the official protocol, with failures and exclusions disclosed. Decider ties the single-model baseline on raw pick rate; the difference is the calibrated confidence value.