DEV Community

Cover image for Nothing between enter and the first token checks your answer
Justin Bartak for Orbyt Labs

Posted on Originally published at orbytlabs.ai AI-assisted

Nothing between enter and the first token checks your answer

One paid route took 27 seconds at its committed 2048 token budget and came back HTTP 200 with zero text blocks.

  • When one provider shipped a new model, the same prompts came back costing roughly 30 percent more tokens. We didn't benchmark the tokenizer. We read our own counts and raised the budget.
  • Reading your prompt runs in parallel and is mostly a one time charge. The answer is serial, paid a token at a time, which is why prefix caching makes a follow up turn start faster.
  • Sampling is a draw, not a maximum. The model commits to its first word before it knows its last, and every later word is conditioned on the earlier ones, wrong ones included.

Read the full article: What Happens When You Hit Send: The 200-Millisecond Life of a Prompt

This summary was written with AI assistance (Claude) from the full article at https://www.orbytlabs.ai/blog/life-of-a-prompt.

Top comments (0)