DEV Community

Shivam Kumar
Shivam Kumar

Posted on

Stop Parsing LLM Essays: Build a Verdict Layer

#agents #ai #api #llm

Every agent you've ever shipped has a embarrassing secret in its control flow: somewhere, a regex is parsing a model's English back into a boolean. This week, the industry finally gave us the primitive that kills that hack.

On October 6, OpenAI's Decisions API went public beta — POST /v1/decisions, typed answers only: a probability, a choice, or a score. On October 9, Microsoft shipped Decision-1 in Foundry: a Qwen3.5-9B post-trained for single-pass decision scoring at $0.042 per million input tokens, output free. Same week, SGLang added a nightly /v1/decisions endpoint and vLLM merged /v1/systemone. The interface consensus is here: stop generating the decision, score it.

The duct tape we're all running

Be honest about your current stack. It looks like this:

response = llm.chat("Classify this lead as HOT, WARM, or JUNK. Reply with JSON only.")
label = json.loads(response.text)["label"]   # pray
Enter fullscreen mode Exit fullscreen mode

You pay output tokens for reasoning you discard. You wait on a full generation for a yes/no. And every few hundred calls, the model gets creative and your parser eats an exception at 2 AM.

JSON mode and structured outputs reduced the breakage, but the architecture was still wrong: you were using a text generator as a classifier because no served classifier existed. Now one does.

The verdict-layer pattern

The replacement pattern has three parts:

1. Typed decision call. Fixed options in, calibrated probabilities out. OpenAI's three shapes cover nearly all control flow: predicate (probability a condition holds — your if statement), choice (one option plus probabilities — your switch), score (position on ordered levels — your priority queue).

2. Threshold policy, not vibes. This is the part most tutorials skip. The number is useless without the policy around it:

score = decide("Is this support ticket urgent?", ticket.text)
# → {"urgent": 0.91, "normal": 0.09}

if score["urgent"] >= 0.85:
    page_oncall(ticket)
elif score["urgent"] >= 0.60:
    human_review_queue(ticket, score)   # score travels with the ticket
else:
    auto_resolve(ticket)
Enter fullscreen mode Exit fullscreen mode

3. Audit log. Every decision lands in a log with its input hash, scores, threshold outcome, and timestamp. When a decision looks wrong, you don't argue with a paragraph — you look at the number, the threshold, and the input. This is also what makes the pattern compliance-friendly: OpenAI ships the beta with Zero Data Retention support and HIPAA eligibility, because a scored, logged decision is auditable in a way prose never was.

What the vendors actually claim (read critically)

  • OpenAI Decisions API: public beta Oct 6, runs only on gpt-6-luna, $0.10/1M input tokens, zero output/cache charges. "About 10x faster than the Responses API" — measured against OpenAI's own API, no disclosed benchmark, no head-to-head accuracy published. Images must be inline base64; hosted URLs and file IDs are refused.
  • Microsoft Decision-1: announced Oct 9 by Achint Srivastava (VP, Office of the CTO), in Foundry and coming to OpenRouter. Post-trained Qwen3.5-9B, plans to rebase onto Microsoft AI and OpenAI models later. $0.042/1M input, output free. Claims: top accuracy on 36 benchmarks (~150k withheld questions), 4.5x faster than Quyet-1.0-Large, 35x faster than GPT-6 Sol at P50, only 1.3% decision flips across eight input perturbations, 5,250 safety requests across 11 benchmarks. All vendor-run. Trust, then verify with your own eval set.

Price comparison that's actually interesting: TypeSafe's Jev — the decision model one independent writeup called the product OpenAI was directly answering — lists the same $0.042/1M input. The category now has a price anchor, and it's cheap.

The migration checklist

  1. Grep for the duct tape. Find every prompt that ends with "reply with JSON only" or "respond with just the label." That's your migration list.
  2. Convert one classifier first. Pick the highest-volume, lowest-stakes one (triage, tagging, routing). Run the typed endpoint shadow-mode alongside your parser for a week and compare disagreement rates.
  3. Write the threshold policy down. Calibrated or not, a probability needs documented cutoffs and an escalation band. "Block below 0.85, human review 0.6–0.85" is a policy; a vibes-based cutoff is a bug.
  4. Log every verdict. Input hash, scores, outcome, timestamp. This is the cheapest reliability upgrade in the whole pattern.
  5. Grade your agents with it. Microsoft explicitly targets rubric-based grading of AI responses and agent actions. Your agent's output gets scored by a cheaper model before it reaches the user — the containment pattern, productized.

Why this matters beyond the API

I've run a decision stage in my own agent stack for a long time — THINK, ACT, then DECIDE: a small scoring model verdicts every consequential step, logged and thresholded. It was always the quiet stage and always the one that kept the system honest. This month the industry formalized it as a served primitive, with two vendors and both open stacks converging on the same interface in the same week.

When that happens, the interface is the story, not the leaderboard. Build the verdict layer. Your 2 AM self will thank you.

By Shivam Kumar — VisionQuantech. I run a decision engine (Jev) in production nightly for lead scoring, reply classification, and build prioritization. This week's launches formalized the stack I've been running.

Top comments (0)