You cannot make a six-second model call feel instant, but you can make it feel short and honest: get the first token on screen inside a second, do the predictable work before the user asks for it, arrange the answer so the useful part arrives first, and show progress that corresponds to something real. Fake spinners and a "thinking..." label with no end in sight teach users that the feature is slow and shifty. Both are fixable, and most of the fixes are product decisions rather than infrastructure work.
Where does the time actually go?
Two numbers, measured separately. Time to first token is dominated by queueing at the provider and by prompt processing, which scales with input length. Total time is generation, output tokens multiplied by per-token speed, plus every tool call, and tool calls run one after another unless you deliberately make them run together.
A system prompt of six thousand tokens plus a big retrieved context adds a noticeable delay before anything appears, and every one of those tokens costs money too.
The most common surprise when we instrument a slow feature: the model was fine, and the time went to three retrieval calls done sequentially, or to a bloated prompt nobody had trimmed since the pilot.
What works
Seven fixes, roughly in the order I'd apply them. Most are product and code decisions rather than infrastructure spend, which is why they're worth trying before anyone proposes a faster model or a dedicated endpoint. The first two usually account for the bulk of the improvement a user can feel.
- Stream everything visible. A first token inside a second changes how the whole wait feels, even when the total is unchanged.
- Parallelize tool calls that don't depend on each other. Three sequential 800-millisecond lookups become one 800-millisecond wait.
- Prefetch on intent. If a user has typed a question and paused, start retrieval before they hit enter. Wrong guesses cost pennies; right guesses save seconds.
- Split the work by model. A small, fast model produces the first visible piece (a title or a one-line summary) while the larger model produces the detail.
- Order the output. Ask for the direct answer first and the reasoning after. The user reads the useful part while the rest streams.
- Trim the prompt. Long system prompts are the cheapest latency fix nobody makes, because trimming feels risky. That's what the eval set is for.
- Cap output length for tasks that don't need an essay.
What honest progress looks like
Honest progress shows real steps in words a user recognizes, and each label changes only when that step completes. If a task will genuinely take thirty seconds, say thirty seconds and offer to notify the user instead. Always give them a way to cancel. A progress indicator that corresponds to nothing is worse than no indicator at all.
"Searching your last 30 tickets," then "drafting reply." Those are labels a support agent can check against reality.
We learned the dishonest version the hard way. Early on we added a progress bar that animated to about 90 percent and then sat there until the response arrived. Actual latency didn't change. User feedback got noticeably worse, because a bar stuck at 90 percent feels broken in a way that a plain "working on it, about 20 seconds" doesn't. We ripped it out within two weeks.
| Loading state | What the user infers | Whether it survived |
|---|---|---|
| Bar animating to 90%, then stalling | "This thing is broken" | Removed after two weeks |
| Bare spinner with no end in sight | "This is always slow" | Replaced with named steps |
| Named steps that advance on completion | "It's working through my request" | Current default |
| "About 20 seconds, want a ping?" | "Fine, I'll do something else" | Used for anything over ~15s |
Why this matters commercially
Buyers judge AI features by feel, in the first minute of a demo and the first week of use. Slow and vague loses to fast and honest even when the slow one is more accurate, because latency is the one quality signal a buyer can read without doing the evaluation work themselves.
Anyone who has watched how buyers evaluate AI agents knows that latency reads as competence. At Shanti Infosoft we now put a time-to-first-token budget in the acceptance criteria for any user-facing AI feature, alongside the accuracy target, because one without the other doesn't ship. It goes into the scope on every generative AI build we take on.
What does your loading state tell the user right now, and is any of it true?
Sagar Jain is technical co-founder of Shanti Infosoft, a CMMI Level 5 company that has built software for 700+ businesses.
Top comments (0)