Originally published on AI Tech Connect.
The default that quietly doubles your bill Open any provider's quickstart and the first code sample you meet is a single synchronous call. You send a message, you block, you get a response. That shape is correct for a chat demo and it is the reason the demo feels magical. It is also the shape that ends up in production for every workload the team subsequently builds, because the client is already configured, the error handling is already written, and nobody stops to ask whether the request/response pattern actually fits the job. Six months later the picture inside a typical codebase looks like this. There is a chat endpoint, which genuinely needs to stream tokens to a person watching a cursor blink. There is a document enrichment job that runs on upload, synchronously, inside a request…
Top comments (0)