The Model Streamed Its Answer. My Free Server's Proxy Swallowed It Whole.

I built a chat UI that streamed tokens, and it felt magical on my laptop. The text appeared word by word, the cursor blinked, and the whole experience felt alive. Then I deployed the same code to a free server, and the magic turned into a spinner that sat there for thirty seconds. Long answers died at exactly sixty seconds, right before the final sentence, which was the cruelest part.

The model API was identical, the client code was identical, and the only variable was the server sitting in between. That asymmetry should have been my first clue, but I spent an hour blaming the model anyway. Why would the same endpoint stream perfectly on localhost and then buffer everything in production?

I hit this while prototyping on MonkeyCode's free server option with one of their free models. The debugging trail taught me more than the deployment ever did. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The fix turned out to be two separate problems hiding behind one symptom, and both are worth knowing.

Step one: measure before you blame

I added timestamps to the client and logged every event that arrived, and the numbers told the real story immediately. On localhost, the first token appeared in about two seconds, and subsequent tokens arrived every few hundred milliseconds. On the free server, the client received nothing for the entire generation time, and then the whole response landed in one burst. The model was clearly streaming, but something between the model and my browser was collecting every chunk and releasing it at the end.

That pattern has a name: response buffering. A reverse proxy in front of your app holds the response body until the handler finishes, which defeats the entire point of Server-Sent Events. The client is not slow, the model is not slow, and your code is not slow. The proxy is just being polite to the network and greedy with your latency.

Here is the small probe I used to prove it, measuring time-to-first-byte against total time:

const start = Date.now();
let firstByte = null;

const res = await fetch("/chat", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({ prompt: "Write a short story about a stubborn proxy." }),
});

const reader = res.body.getReader();
const decoder = new TextDecoder();

while (true) {
  const { done, value } = await reader.read();
  if (!firstByte && value) firstByte = Date.now();
  if (done) break;
}

const ttfb = firstByte - start;
const total = Date.now() - start;

console.log({ ttfb, total, buffered: ttfb > total - 1000 });

If buffered comes back true, your stream is not streaming. On my localhost the difference was about two seconds; on the free server it was the full generation time, which meant the proxy was holding everything.

Step two: check the headers, then check the proxy

The next move was to inspect the response headers, because proxies usually announce their behavior even when they do not announce their presence. My handler set Content-Type: text/event-stream, Cache-Control: no-cache, and Connection: keep-alive, all of which looked correct. The missing piece was X-Accel-Buffering: no, a header that tells Nginx-style proxies to pass chunks through as they arrive instead of collecting them.

Three things to check when a stream arrives all at once:

• The response headers, for X-Accel-Buffering: no or an equivalent opt-out
• The proxy logs, for buffering or timeout messages that hint at the real path
• The platform documentation, for hard execution limits that no header can bypass

Adding that one header changed everything for short responses:

app.post("/chat", async (req, res) => {
  res.setHeader("Content-Type", "text/event-stream");
  res.setHeader("Cache-Control", "no-cache");
  res.setHeader("Connection", "keep-alive");
  res.setHeader("X-Accel-Buffering", "no");

  // model.stream() is whatever streaming API your provider exposes
  const stream = await model.stream(req.body.prompt);

  for await (const chunk of stream) {
    res.write(`data: ${JSON.stringify(chunk)}\n\n`);
  }

  res.write("data: [DONE]\n\n");
  res.end();
});

Suddenly the tokens appeared on screen in real time, and I celebrated for about ten minutes. Then I asked the model a question that required a long answer, and the connection died at exactly sixty seconds again.

Step three: the hard timeout is a different beast

The buffering fix solved the delivery problem, but a second problem was hiding underneath it. The free server platform enforced a hard execution limit of around sixty seconds per request, and no header in the world can override that. When the handler was still streaming after the limit, the platform killed the process. The proxy closed the connection, and the client saw a network error instead of a complete answer.

Heartbeats can keep an idle connection alive, but they cannot extend a hard deadline. I added a comment ping every fifteen seconds just to rule out idle timeouts, and the failure still happened at sixty seconds on the dot. That consistency told me the platform was enforcing a limit, not the proxy being impatient.

The honest fix was to stop holding the connection open for the whole generation. I switched the long path to a job-based pattern: the server accepts the prompt, returns a job ID immediately, and the client polls for the result. The model finishes in the background, and the handler never approaches the timeout.

Decision table: pick the pattern that fits your host

Pattern	First token latency	Max response length	Survives buffering proxies	Best for
SSE streaming	Immediate	Bounded by platform timeout	Only with X-Accel-Buffering: no	Interactive chat, short answers
Polling with job ID	One poll interval	Effectively unlimited	Yes	Long generations, background work
WebSocket	Immediate	Bounded by connection lifetime	Needs upgrade support	Bidirectional real-time apps

I use SSE for short interactive answers and polling for anything that might exceed the platform's limit. The decision is not about which is cooler; it is about which one your host will actually let you run.

Limitations and who should skip this

The X-Accel-Buffering header is an Nginx convention, so other proxies may ignore it or use a different mechanism entirely. Some platforms strip unknown headers, and some enforce timeouts that are far shorter than sixty seconds, so always verify the actual behavior with the probe above. If your responses are under five seconds, skip streaming altogether and return plain JSON, because the complexity is not worth the perceived speed.

You should also skip this approach if you need bidirectional communication, because SSE is one-way by design and WebSocket is the better tool. If your platform kills long-running handlers no matter what, do not fight it — use the polling pattern and let the background job do the heavy lifting.

Run the probe on your own host before you trust any streaming UI, because the proxy you forgot about is the one that decides your latency. The real lesson is that streaming is a contract between your client, your server, and every proxy in between. One missing header broke the illusion of real-time, and one hard limit broke the connection entirely. Next time your stream feels slow, measure the time-to-first-byte before you blame the model, because the bottleneck is rarely where you think it is.