Most AI tools answer the question. The harder job is judging between two answers you already have: two drafts, two plans, two offers. You usually lean one way already — what you cannot do is price your own uncertainty.
That is the only thing we built: Decider, a judge that takes a question plus two candidate answers and returns the stronger one, a calibrated confidence value, and the reasoning for the call.
The number is the product, and it comes with a caveat
Here is the part most products would leave out. On raw pick rate, Decider ties a plain single-model baseline — roughly 92.5% against 92.7% on JudgeBench. We do not claim to be more accurate.
What a single pass cannot give you is how close the call was. Every judgment carries a confidence value calibrated against outcomes:
| Confidence reported | Share of judgments | Observed accuracy |
|---|---|---|
| ≥ 90% | 45.6% | 99.6% |
| 80–90% | 29.7% | 94.0% |
| 70–80% | 13.2% | 84.1% |
| < 70% | 10.5% | 67.7% |
JudgeBench, 614 judgments. These are our own runs on the official protocol; the runs that failed are disclosed rather than dropped, and the raw judgments are published.
Two outcomes, both useful
A high band is permission to commit and stop re-reading both options. A low band is not a failure — it is the answer you actually needed: the two are genuinely close, so take the one you already preferred and spend your time somewhere else. Terminating the comparison is the value.
Judge it on real work, not a demo script
We recorded 27 real runs across nine domains — writing, engineering, business, research, career, money, people, travel, everyday — and published each one end to end: the question, both candidate answers, the pick, the confidence and the full reasoning, verbatim. They are browsable as static pages:
https://api.turingcorp.net/platform/poe/examples
Nothing is charged for reading them, and the same set ships inside the app as free recorded runs.
Try it
Decider runs as an app on Poe: https://poe.com/TuringCorp-Decider — pay per decision, no subscription (list $0.50, launch offer $0.25). What it is, how it is measured and what it costs: https://api.turingcorp.net/platform/poe
If your pipeline needs a quality gate rather than a chat, there is also an OpenAI-compatible endpoint: https://api.turingcorp.net/
Disclosure: this post is written by the team that builds Decider.
Top comments (0)