Getting an AI request to return successfully is not the same as making an AI product feel responsive.
In an interactive product, users notice the silence before they notice the final completion time.
They ask:
- Did the request start?
- Is the model thinking?
- Should I retry?
- Did the app freeze?
That is why time to first token, or TTFT, deserves its own metric in multi-model AI applications.
A faster completion can still feel slower
Imagine two models used in a support chatbot.
- Model A starts streaming in 700 ms and finishes in 8 seconds.
- Model B starts streaming in 4 seconds and finishes in 6 seconds.
Model B has a lower total completion time.
But it may still feel slower to the user because nothing happens for the first four seconds.
This distinction matters even more when an app uses multiple models for chat, RAG, coding agents, document analysis, automation, and background work.
A single average latency number cannot explain the real experience.
What TTFT actually measures
Time to first token is the time between sending a request and receiving the first visible streamed response.
It can include more than model generation:
- authentication and request validation
- queue time
- model routing
- retrieval
- provider connection time
- prompt prefill
- reasoning effort
- tool preparation
- fallback decisions
A slow TTFT does not always mean the selected model is slow.
The problem may be a large prompt, a cache miss, a slow retrieval step, a tool call that blocks streaming, or a fallback route.
That is why teams need request-level visibility.
Multi-model systems have multiple latency profiles
Different models can behave very differently in production.
A model may be excellent for batch reasoning but unsuitable for a live chat experience. Another may stream quickly but take longer to complete. A reasoning model may spend more time before producing a first token. A fallback model may protect availability while quietly worsening the user experience.
Do not ask:
What is our AI latency?
Ask:
What is the p95 time to first token for this workflow, this model, this route, and this prompt size?
That question leads to better decisions.
Log the right fields
For every AI request, record more than the HTTP status code.
json
{
"request_id": "req_123",
"workflow": "support_chat",
"model": "model-name",
"route": "primary",
"streaming": true,
"input_tokens": 12400,
"output_tokens": 860,
"ttft_ms": 920,
"total_latency_ms": 7400,
"cache_hit": true,
"tool_calls": 0,
"fallback_used": false,
"success": true
}
This makes it possible to answer useful questions:
Does TTFT rise when prompts become larger?
Which model route has the slowest p95 first token?
Are cache misses creating visible delays?
Does a fallback route improve availability but hurt UX?
Which workflows need streaming most?
Are retries increasing total wait time?
Different workflows need different targets
A customer-support chatbot needs a fast first visible response.
A RAG workflow may need fast retrieval plus clear streaming.
A coding agent may need an early progress event such as "Reading repository" or "Running tests," even if the full task takes longer.
A batch extraction job may not need streaming at all. Its priorities may be throughput, reliability, and cost.
A useful latency policy might look like this:
Workflow Primary metric
Chatbot Low TTFT
RAG answer Retrieval time + TTFT
Coding agent Early progress + successful completion
Background batch job Throughput + cost per completed task
Automation workflow Retry rate + reliable completion
The goal is not to make every model behave the same way.
The goal is to match the model and route to the user experience the workflow requires.
Measure percentiles, not only averages
Average latency hides bad experiences.
If nine requests return their first token in one second and one request takes 20 seconds, the average may look acceptable. The user waiting 20 seconds will disagree.
Track at least:
p50 TTFT
p95 TTFT
p99 TTFT
p95 total latency
timeout rate
retry rate
fallback rate
successful task completion rate
The p95 metrics usually reveal whether production traffic is becoming unhealthy.
Improve the experience before replacing the model
When first-token latency gets worse, changing models is not always the best first move.
Check whether:
unnecessary history is being sent
context caching is working
retrieval blocks streaming
a tool call happens before any progress is visible
the route has fallen back
reasoning effort is too high for the task
batch traffic competes with interactive traffic
Sometimes a smaller prompt, a better route, or an earlier progress event improves the product more than a model switch.
Final thought
Users do not experience an AI application as an API status code.
They experience waiting.
For multi-model products, time to first token is not only an infrastructure metric. It is a product metric.
Teams that track TTFT alongside total latency, cost, retries, routing, and successful task completion can build AI applications that feel faster and behave more reliably.
Top comments (0)