You Were Standing in a Line You Could Not See

You type a question into ChatGPT and the dot starts pulsing. Sometimes the answer comes in a second. Sometimes you watch that dot for ten. And sometimes you get a message instead: too many requests, try again later.

Nothing was broken. You were standing in a line you could not see. This post draws the line, and it needs no numbers to do it, only three pictures.

The machine reads the whole list

The AI itself is a very long list of numbers. To produce an answer, the machine has to read that entire list, start to finish, because every part of the answer touches every part of the list. Reading the list is the slow part. The arithmetic for your one question is quick.

That is the whole foundation. If you hold on to one sentence from this post, hold that one: the expensive thing is not thinking about your question, it is re-reading the model to think about anything at all.

(For the readers who want the precise version: at low batch sizes, generation is memory-bandwidth bound. The weights stream from the card's memory on every step, so the read dominates and the per-request compute is small. Everything below is that fact wearing a costume.)

No line: the door slams instantly

Say the machine can hold sixteen questions in flight, and a hundred people ask at the same moment, and there is no queue in front of it.

Sixteen get in. The other eighty-four get an error immediately, before a single person has been answered. Not a slow no. An instant one, because with no line, overflow has nowhere to stand.

That is one of the messages you have seen. Not a wait: a door closing.

The line: refusals become waits

Put a queue in front of the machine and nobody is refused. Everyone waits their turn instead. This is the first honest thing to understand about queues: a queue does not make anything faster. It converts errors into waiting.

But one at a time, the machine reads the same list for every single person. A hundred people means a hundred readings of the very same numbers, and the last person in line waits through all of it, watching the dot pulse.

The bus: one reading, many riders

So AI services batch. Instead of one question per pass, the machine takes sixteen, reads the list once, and answers all sixteen together. A hundred people becomes seven readings instead of a hundred, and the last person's wait collapses.

Now look at the first person. On a quiet afternoon they would have gone alone and had their answer almost immediately. Instead the bus waited while fifteen strangers boarded. Every seat you add spreads one reading across more people, and makes the first passenger wait longer.

That is the real dial inside every AI service, and it has no right answer. Small batches favour the person at the front. Big batches favour the crowd and the bill. Someone chooses how long the bus waits at the stop, and that choice is part of every wait you have ever had in that chat box.

In code, the naive version of the machine's loop is almost embarrassingly small:

java
while (true) {
List batch = new ArrayList<>();
batch.add(queue.take()); // wait for at least one rider
queue.drainTo(batch, SEATS - 1); // take whoever else is already in line
Answer[] answers = model.forward(batch); // ONE reading of the list
deliverAll(batch, answers); // everyone gets off together
}

Real servers improve on this in one important way: they do not wait for a full bus or drive it empty. Seats turn over mid-ride, so the moment one answer finishes, the next question takes its seat without the wheels ever stopping. The industry calls that continuous batching. Same idea: one reading, many riders, just never standing still.

The door: no, said early

One problem is left. What if people arrive faster than the buses can leave? The line grows, and keeps growing, and an answer an hour late is worse than no answer at all.

So the service says no at the door, early, while the line is still short. That is what a rate limit is for. Too many requests is not the machine failing. It is the door protecting everyone already inside, including you, five minutes from now, when you are the one in the line.

How the door decides who gets in, and who is told to wait, is its own machinery, and it is the next episode.

Where you were standing

So the next time the dot pulses: you are in a line, you are waiting for a bus to fill, and if you got the message, the door said no early so the line would not grow forever.

Honesty, before you quote any of this in a design review: this is a picture, not a blueprint. Sixteen seats is a number chosen for the drawing; real capacity is a per-deployment setting that depends on the model, the card, and conversation lengths, and it is usually far higher. Nothing here is a measurement. The shape is the claim: one reading shared by many questions, a queue that turns refusals into waits, and a door that says no on purpose.

The video draws all of this in four minutes: