Not every AI feature is a chat. A large share of production calls are a single small decision: is this message spam, which queue does this ticket belong to, does this review score 1 or 5, is this text safe to show. Treating those as a chat completion works, and it is slow, expensive and hard to test.
Ask a decision, not a question
A prompt like "classify this" returns prose you then have to parse. A decision endpoint returns one of the values you named:
- Yes/no, when there are two outcomes.
- A choice from a fixed list, when routing.
- A score, when you need a threshold you can move later.
Structured output is not a formatting preference. It is what makes the call testable: given this input, the answer is one of these five strings.
Constrain the output, then measure it
Build a small labelled set before you ship - a few hundred examples is plenty for a narrow task. Then read the confusion matrix, not the accuracy number. Two questions matter:
- Which category does it over-predict? That tells you where your prompt is ambiguous.
- What happens at the decision boundary? A 0.5 threshold on a scoring task is rarely the right one; pick the threshold from the cost of each error.
Calibration beats raw intelligence
For routing and moderation, a well-calibrated small model beats a stronger model with an unstable prompt. Keep the input format identical between training and production, pin the model version, and re-run the labelled set on every upgrade. If the answers move, you want to know before your users do.
Keep the decision cheap
Three habits shrink both latency and cost:
- Send only the text the decision needs. Long context is not accuracy.
- Cache the decision keyed by normalised input, and expire it.
- Fail loudly on unknown output. A malformed answer should raise, not silently become "no".
Put a human behind the irreversible ones
Automate the reversible decisions first: tagging, prioritising, drafting. Keep a review step for anything that deletes, bans or charges. The model's confidence on a single call is not evidence, and a queue with a human at the end is what lets you turn the automation up later.
Tooling
You do not need a full agent stack to try this: a playground such as Laya AI lets you ask a yes/no, choice or score question over short text and inspect the structured answer before wiring the same call into an API. Measure it on your own labelled examples first.
The rule
If the answer is one of five strings, use a decision call, test it with a confusion matrix, and keep the threshold in your code where you can change it.
Top comments (0)