OpenAI just previewed an Ultrafast service tier for GPT-5.6 Sol that can run up to 14× faster than Standard processing.
That is a meaningful change for products where every second affects the interaction.
But it also creates a useful architecture question:
Does every AI request in your product actually need the fastest path?
Usually, no.
A voice agent responding during a live call has a very different latency requirement from a report that will be emailed five minutes later.
Treating both requests the same makes the system harder to reason about.
A cleaner approach is to give every AI workflow a latency budget before choosing how it should run.
Start with the user, not tokens per second
A latency budget is simply the amount of waiting the product can reasonably tolerate before the experience starts getting worse.
For example:
Live voice response
Expected wait: very low
User is actively listening
Compare that with:
Weekly account summary
Expected wait: minutes are acceptable
User does not need to watch it generate
Both workflows may use a strong model.
They do not need the same execution path.
Use three practical request classes
A useful starting point is to separate AI work into three lanes.
1. Live interaction
The user is waiting right now.
Examples:
- voice calls
- live support
- checkout assistance
- interactive copilots
- incident-response assistance
The latency budget here is tight because waiting changes the interaction itself.
2. Interactive work
The user is still present, but a few extra seconds may be acceptable.
Examples:
- document analysis
- research queries
- complex product questions
- multi-step tool calls
- coding assistance
The system should still feel responsive, but it has more room to trade speed against depth.
3. Background work
The result matters more than immediate response time.
Examples:
- report generation
- batch enrichment
- nightly analysis
- large ingestion jobs
- scheduled summaries
- asynchronous evaluation
These jobs should usually run outside the live request path.
Now the architecture has a useful decision surface.
Incoming AI request
↓
How long can the user reasonably wait?
↓
┌─────────────┬───────────────┬────────────────┐
│ Live │ Interactive │ Background │
│ interaction │ work │ work │
└─────────────┴───────────────┴────────────────┘
↓ ↓ ↓
Low-latency Balanced Async / queued
execution execution execution
Put the latency target into request metadata
Do not leave this decision buried inside random route handlers.
Represent it explicitly.
For example:
type LatencyClass =
| "realtime"
| "interactive"
| "background";
type AIRequestContext = {
latencyClass: LatencyClass;
userId: string;
feature: string;
};
A feature can then declare its requirement:
const context: AIRequestContext = {
latencyClass: "realtime",
userId,
feature: "voice_support"
};
Now the routing layer has something meaningful to work with.
Route by product requirement
A simplified router could look like this:
async function runAI(
input: ModelInput,
context: AIRequestContext
) {
switch (context.latencyClass) {
case "realtime":
return runLowLatencyPath(input);
case "interactive":
return runStandardInteractivePath(input);
case "background":
return enqueueBackgroundAIJob(input);
}
}
The implementation can change over time.
The product contract stays clear.
This is much easier than sprinkling provider-specific speed decisions across individual features.
Latency is bigger than model generation
Fast generation does not automatically create a fast product.
A request might still spend time in:
User request
↓
Authentication
↓
Database lookup
↓
Retrieval
↓
External API
↓
Model inference
↓
Another tool call
↓
Response rendering
If model inference drops from four seconds to one second but an external API still takes six seconds, the user will not experience a four-times-faster workflow.
So measure the whole path.
A useful trace might include:
type LatencyTrace = {
authMs: number;
retrievalMs: number;
toolsMs: number;
modelMs: number;
renderMs: number;
totalMs: number;
};
That tells you where faster inference can actually change the experience.
Set a deadline, not just a preference
For live workflows, it can help to define an actual deadline.
For example:
type ExecutionPolicy = {
latencyClass: LatencyClass;
deadlineMs?: number;
};
Then:
const policy: ExecutionPolicy = {
latencyClass: "realtime",
deadlineMs: 1800
};
The workflow can react if the deadline is at risk.
Perhaps it:
- skips a secondary enrichment call
- uses cached context
- returns a shorter first answer
- moves optional work into the background
- falls back to a faster path
Now latency becomes a product rule instead of a dashboard number.
Voice is a good example
Voice interactions expose delay very quickly.
A person asks something.
Silence follows.
The system retrieves context.
A model thinks.
A tool runs.
Then the answer begins.
Even small delays can stack up because the user is waiting through all of them. That makes voice a good candidate for a low-latency path. But the post-call summary does not have the same requirement.
A better architecture can split them:
LIVE CALL
User speaks
↓
Fast reasoning + required tools
↓
Response begins quickly
AFTER CALL
↓
Detailed summary
↓
CRM enrichment
↓
Quality checks
↓
Background processing
The same product uses different latency policies for different moments.
Do not make background work pretend to be realtime
There is also a product temptation to make everything feel instant.
That is not always useful.
A detailed report may need:
- multiple data sources
- verification
- several tool calls
- a longer model response
- document generation
If the user does not need to watch that happen, put it behind a task.
Request accepted
↓
Task created
↓
Work runs asynchronously
↓
User continues
↓
Result becomes available
That keeps your low-latency capacity focused on interactions where waiting actually changes the experience.
Cache what does not need to be recomputed
Latency-sensitive workflows should also ask:
What are we repeatedly doing that does not need to happen during the request?
Candidates may include:
- static system context
- tool metadata
- reusable retrieval results
- known account information
- configuration
- frequently requested reference material
Every unnecessary dependency inside the live path consumes latency budget.
The fastest model cannot compensate for a workflow that repeatedly performs avoidable work.
Track latency by feature
A single global P95 number is not enough.
Measure latency by product experience.
For example:
voice_support
P50: ...
P95: ...
checkout_assistant
P50: ...
P95: ...
document_report
P50: ...
P95: ...
nightly_enrichment
P50: ...
P95: ...
Now you can see which user journeys actually need optimization.
You can also see whether faster inference changes the total experience or just one component inside it.
OpenAI's Ultrafast preview makes this more relevant
OpenAI announced on August 13 that GPT-5.6 Sol Ultrafast can run up to 14× faster than Standard processing and produce up to 750 output tokens per second.
The company highlighted use cases such as:
- incident response
- customer support and voice
- commerce
- financial research
- interactive experimentation
Those examples have something in common:
the person or business process is sensitive to delay.
That is a much better selection rule than simply choosing the fastest execution path everywhere.
A practical latency-budget review
Before changing inference tiers, I would review each AI feature with five questions:
- Is a person actively waiting?
- What happens if the response takes five seconds longer?
- Which step currently consumes most of the total time?
- Can optional work move outside the live path?
- Would faster model inference materially change the experience?
If the answer to the last question is no, there may be a better place to optimize.
Final thought
Faster models create new product possibilities. They do not remove the need for architecture.
- Give each AI workflow a latency budget.
- Keep genuinely realtime interactions fast.
- Let background work stay background work.
- Measure the complete request path.
Then use higher-speed inference where the user can actually feel the difference.
Source
OpenAI
Previewing Ultrafast mode: GPT-5.6 Sol at up to 14X the speed
Top comments (0)