One paid route took 27 seconds at its committed 2048 token budget and came back HTTP 200 with zero text blocks.
- When one provider shipped a new model, the same prompts came back costing roughly 30 percent more tokens. We didn't benchmark the tokenizer. We read our own counts and raised the budget.
- Reading your prompt runs in parallel and is mostly a one time charge. The answer is serial, paid a token at a time, which is why prefix caching makes a follow up turn start faster.
- Sampling is a draw, not a maximum. The model commits to its first word before it knows its last, and every later word is conditioned on the earlier ones, wrong ones included.
Read the full article: What Happens When You Hit Send: The 200-Millisecond Life of a Prompt
This summary was written with AI assistance (Claude) from the full article at https://www.orbytlabs.ai/blog/life-of-a-prompt.
Top comments (0)