OpenAI’s Decisions API became generally available this week. The interesting part is not simply that there is another API endpoint. It is the architectural distinction the release makes explicit:
Some AI tasks need generation. Many only need a bounded decision.
Asking the same general-purpose model to classify a request, select an action, draft a response, and approve the result may be convenient during prototyping. In production, however, those jobs have different requirements and failure modes.
A more testable AI workflow can be divided into four layers.
1. Deterministic rules for hard constraints
Some decisions should not depend on probabilistic inference.
Permissions, contractual restrictions, data-residency requirements, spending limits, and known security policies belong in normal application logic. A model can help identify relevant context, but it should not silently override a hard rule.
If a condition must always produce the same result, start with code.
2. Bounded models for repetitive judgments
Classification, prioritization, routing, and scoring are different from open-ended generation.
A bounded decision layer receives evidence and returns a constrained result: a probability, one choice from a known set, or a score against a rubric. This makes the output easier to validate and easier for the rest of the application to consume.
Typical examples include:
- Routing a support request to the right queue
- Scoring whether an agent action needs review
- Choosing which model should handle the next step
- Prioritizing documents for human inspection
This layer still needs evaluation. A probability is not automatically well calibrated, and a confidence threshold should be derived from representative application data rather than selected arbitrarily.
3. Generative models for open-ended work
Use generative models where the output space is genuinely open: drafting an explanation, synthesizing several documents, writing code, or planning a multi-step task.
The goal is not to avoid larger models. It is to reserve them for work where their broader capabilities provide value.
A simplified workflow might look like this:
if request violates a hard policy:
block or escalate
decision = classify_and_route(request)
if decision is low-confidence or high-impact:
send to human review
else:
use the selected model and workflow
4. Human review for consequential uncertainty
A human-review path should be part of the architecture, not an emergency patch added after deployment.
Review is especially important when an action is irreversible, affects another person, accesses sensitive data, or falls outside the examples used during evaluation.
Evaluate the whole route
When testing this architecture, do not measure only raw model accuracy. Track:
- Decision calibration
- End-to-end latency
- Cost per successfully completed workflow
- Escalation and fallback rates
- Performance drift as real inputs change
- Whether each decision can be audited
The emerging lesson is that “Which model should we use?” is becoming the wrong first question.
A better question is: What kind of inference does each step actually require?
References
AI assistance disclosure: This article was drafted with AI assistance. The author reviewed and edited the final version and verified its factual claims against the linked primary sources.
Top comments (0)