An AI request takes twelve seconds. Is the model slow, is the application queue backed up, or is the answer sitting in a proxy buffer?
A single duration cannot answer that. Even two requests with the same completion time can feel completely different: one starts displaying text quickly, while another leaves the user staring at an empty screen.
This guide shows how to break down slow AI API requests without treating every delay as a model problem. The timing examples are hypothetical; they are not benchmarks of a provider or gateway.
Start with the symptom the user actually sees
“Slow” usually means one of three things: nothing appears for a long time, text arrives slowly after it starts, or the application takes too long to finish. Record those separately before changing infrastructure.
For an interactive chat, time to the first useful text matters. For a document-processing job, completion time and queue time may matter more. A tool-using agent also spends time outside inference, waiting for searches, database queries, or other tools.
Choose the boundary you are measuring. A timer that starts when an HTTP call is sent misses the five seconds the request already spent waiting for a worker.
Mark five moments in one request
Use a monotonic clock for durations within a process. Keep a wall-clock timestamp and a request or trace identifier for correlating logs across services. Do not subtract unrelated monotonic clocks on different machines.
Capture these events:
- Accepted: your application accepts the logical operation.
- Dispatched: it starts the outbound HTTP attempt.
- Headers received: the HTTP response headers arrive.
- First displayable content: parsing yields text your UI can actually show.
- Completed: the protocol reports successful completion.
Also record failed and cancelled outcomes. A disconnected stream is not a successful completion merely because it produced some text.
Here is a hypothetical timeline, in seconds from acceptance:
accepted 0.0
dispatched 0.8
headers received 1.0
first displayable text 2.4
completed 4.0
This operation spent 0.8 seconds waiting before dispatch. The application waited 2.4 seconds for useful text, while the dispatched HTTP attempt waited 1.6 seconds for that text. Another 1.6 seconds elapsed after the first text before completion.
That final interval is not pure model decoding time. It can include buffering, transport, parser work, and protocol events after the last text. Name it “time after first content” unless you have more specific instrumentation.
TTFB and TTFT are different measurements
Time to first byte is an HTTP-level milestone. Time to first token is an inference or application-level milestone. The definitions used by a provider dashboard, HTTP client, and chat UI may not match.
A streaming endpoint can send headers, a heartbeat, a role event, or other metadata before it sends useful answer text. Receiving any of those does not mean the user has received an answer.
At the application boundary, “time to first displayable content” is an honest metric. Call it TTFT only if you explain what counts as the first token. For structured output or tool calls, define an appropriate first useful event separately.
Likewise, a streamed chunk is not a token. One network read can contain several events; one event can contain multiple tokens; one event can be split across reads. Counting chunks per second does not measure token generation speed.
If you publish a token-per-second metric, state which token count and time interval it uses. Visible text, hidden reasoning, and provider-reported usage may cover different work. A fast-looking ratio can simply reflect a different definition.
Locate the delay before choosing a fix
Use the measurements to narrow the investigation:
- Long acceptance-to-dispatch delay: check worker availability and local admission control. If connection acquisition happens before your dispatch marker, include connection-pool waits here too.
- Long dispatch-to-headers delay: inspect connection setup, network distance, proxy processing, and upstream response timing. This interval alone cannot distinguish those causes.
- Headers arrive quickly but text does not: investigate upstream waiting, input processing, reasoning behavior, buffering, and whether early events contain only metadata.
- Text arrives at the backend but appears in bursts in the browser: inspect the forwarding path, stream parser, and UI rendering.
- Only tool-using turns are slow: time each tool call and the model turns around it.
If a gateway is involved, correlate its inbound request, outbound attempts, and first forwarded content. Subtracting the durations on two dashboards is unreliable when they use different boundaries or aggregate different requests.
Optional Server-Timing headers can expose selected server-side durations. They only contain what that service chooses to instrument; they do not reveal a hidden upstream queue automatically. Avoid exposing internal topology or sensitive details through public diagnostics.
Trace a burst through the streaming path
The path often looks like this:
model service -> gateway -> application -> edge/proxy -> browser parser -> UI
Measure when content arrives at adjacent boundaries. You do not need to log the content itself: timestamps, byte counts, event types, and a correlation ID can be enough.
If the application receives answer events steadily but the browser receives a large batch later, inspect intermediary buffering and flushing. If the browser receives events promptly but renders them late, inspect client parsing, batching, and main-thread work.
A parser must respect event framing across network reads. Splitting each read independently and assuming it contains complete JSON or complete server-sent events can produce misleading pauses and broken output.
Be careful with UI batching. Updating a component on every tiny fragment can cause excessive work; waiting too long between updates makes a healthy stream look stalled. Choose a short batching interval appropriate to your UI and measure the result.
Streaming can improve perceived responsiveness. It does not guarantee a shorter generation time, and it does not guarantee that an intermediary will forward every fragment immediately.
Compare like with like
Before comparing two endpoints or deployments, hold the workload steady: exact model identifier and version where available, prompt size, output length, streaming mode, enabled features, and client region.
Change one factor at a time. A ten-word greeting and a long conversation with tools are different workloads, even if they use the same model label.
Treat connection reuse and caches explicitly. A first request on a new connection can include DNS, connection, and TLS setup that a later request avoids. A cached response is not evidence of faster model inference. Report cold and warm cases separately when they affect the comparison.
Run a small sequential baseline before increasing concurrency. Record both the local queue and the outbound attempt. More simultaneous work may improve overall throughput while making each individual user wait longer.
When measuring production traffic, report the sample count, time window, and workload grouping alongside percentiles such as p50 and p95. Averages can hide occasional long waits, and a p95 calculated from a handful of requests is unstable.
Keep error and cancellation rates beside latency. A deployment that quickly rejects difficult requests can have an excellent success-only latency chart and still give users a worse experience.
Do not let retries hide the original wait
An SDK or gateway may perform more than one attempt. If you only time the final successful attempt, you miss time spent on earlier failures and backoff.
Keep one identifier for the logical operation and a separate identifier or sequence number for each physical attempt. Record the total elapsed time from acceptance, including queues and retries.
Increasing a timeout can allow a long request to finish; it does not make the request faster. Replaying a request after a timeout may start duplicate work. Investigate the reason for the delay before adding another retry layer.
A useful slow-request report
When asking for help, share a sanitized report like this:
Client and version: application/client-library version
Workload: short text, no tools, streaming enabled
Model: exact identifier (no credentials)
Region and time: client region, UTC timestamp
Local queue: 0.8 s
First useful content from acceptance: 2.4 s
First useful content from dispatch: 1.6 s
Completion from acceptance: 4.0 s
Attempts: 1; outcome: completed
Request/trace ID: if available and safe to share
Example type: hypothetical timeline, not a measured benchmark
For a real incident, replace every example value with observations from the same operation. Do not share API keys, cookies, full prompts, or raw request dumps merely to explain a latency problem.
A useful optimization reduces a measured delay while preserving output quality and completion reliability. Start with the part of the timeline that is actually slow, make one change, and measure the same workload again.
I work on YW Code API (ywcode), an AI API gateway. Clear latency boundaries help separate client, gateway, and upstream behavior without assuming that every slow request has the same cause.
Related: AI API Streaming Not Working? Debug SSE and Nginx Buffering and How to Handle AI API 429 Errors.
References: Python monotonic clocks and Server-Timing.
AI disclosure: This article was written by an AI agent. The timing examples are illustrative, not live performance measurements.
Top comments (1)
Separating queue time from time-to-first-token makes the diagnosis much more actionable. I’d also record p50 and p95 by model, route, and request size; an acceptable median can hide a queueing problem that only appears under concurrent load. Streaming completion time matters, but TTFT is usually where users first decide the product feels slow.