Your agent picks one of two options. Can you test that choice?

There's a moment in almost every agent loop where two options are both defensible and one of them has to win.

Two migration plans. Two refactors. Two diagnoses. Two vendors. Two drafts.

You put both in the prompt, ask for a pick, and get back something confident. Then you ship it, and you have no idea whether that choice was well-founded or whether the model just happened to prefer the second thing it read.

I've been building around this problem for a while. Here's the part that took me too long to notice.

Prose judgements can't be tested

When the judgement lives inside a prompt, it isn't a component — it's a paragraph. You can't assert on it, you can't log it, you can't regression-test it, and you can't tell a confident pick from a coin flip. When it's wrong, you find out in production, and what you learn is "the agent chose B" — not how close the call was.

Worse, there's a well-known failure mode hiding there: a model asked to pick between A and B and report its own certainty will usually do both in the same breath. Ask it "which is better, and how sure are you?" and you get an answer plus a number that was generated to sound consistent with it. That number is not a measurement of anything.

Treat the decision as a tool call

The reframe that helped me: stop asking the agent to have an opinion and give it something to call. One tool, narrow contract:

decide(
  task:     "Pick a launch date",          // stated neutrally, no preferred answer
  option_a: "Ship now — <the case for it>",
  option_b: "Wait two weeks — <the case for it>"
)

and back:

better_option: "A" | "B"
confidence:    "83.3%"        // how far apart the two were judged
reason:        "<why>"

Two properties matter more than the tool itself:

It's a separate call, not an aside. The judgement is produced in its own request, so it isn't entangled with the text the agent was already committed to writing.

The confidence is a property of the comparison, not of the answer. It's the service's own reading of how far apart the two options were — the thing you actually need for a gate. That distinction is worth being pedantic about, because it changes what you can build on top:

• confidence >= your_threshold → take the pick automatically
• below it → escalate, gather more evidence, or ask a human

The threshold is yours. Nobody else can set it, because it depends on what a wrong call costs you.

What this looks like in practice

If you want to poke at it without writing a client, there's a public MCP server at https://mcp.turingcorp.net/mcp (discovery is open — tools/list works with no credentials):

curl -s -X POST https://mcp.turingcorp.net/mcp \
  -H 'content-type: application/json' \
  -H 'accept: application/json, text/event-stream' \
  -H 'mcp-protocol-version: 2026-07-28' \
  -H 'mcp-method: tools/list' \
  --data '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{"_meta":{"io.modelcontextprotocol/protocolVersion":"2026-07-28","io.modelcontextprotocol/clientCapabilities":{}}}}'

Calling it needs a credential (a 7-day pass, self-service), and the tool returns both a human-readable block and a machine-readable one so you can branch on a code rather than parse prose:

{"error":"invalid_credential","http_status":"401","action_url":"..."}

That last bit is a pattern I'd recommend regardless of whose judge you call: make failures branchable. If your agent can only tell "something went wrong" from a sentence, it can't decide to refresh a credential or top up a balance on its own.

The honest caveats

I'd rather you know these up front than discover them at 2am.

No idempotency key yet. The tool declares idempotentHint: false, which is the spec's way of saying: a client that times out and retries may be charged twice. If your agent retries aggressively, put your own guard in front of it. This is on the roadmap, not done.

No SLA. There's no availability commitment, and you shouldn't infer one.

It is not an autopilot. It returns a pick and a comparison distance. Whether you act on it, at what threshold, with what review — that's your policy, and it stays yours. For high-stakes or irreversible calls, apply your own review process.

The confidence is a reference, not a prediction. High confidence means the two options were judged clearly apart. It does not mean the choice will turn out well.

Why not just use a second prompt?

Fair question, and if your volumes are tiny and your stakes are low, a second prompt may be all you need.

What you get from pulling it out into a tool is the same thing you get from pulling anything out into a function: a name, a contract, a place to put a test, and a number you can plot. Whether that's worth a network hop is your call — it depends on whether you intend to gate on the result or just glance at it.

If you want to look at real recorded calls before deciding — actual tasks, both options, the pick, the confidence, and the reasoning — there are 27 of them, published as a dataset: TuringCorp/poe-decider-recorded-cases. They're recordings, not cherry-picked demos, and they include the low-confidence calls where the two options were close.

---

The server is net.turingcorp.mcp/decider in the official MCP registry. Published accuracy by confidence band, with the measurement protocol and failure disclosure, is at api.turingcorp.net — including how it was run rather than just the number.