SSE Myths That Make Your Free Model API Feel Slower

You blame the free model server. I get it.

Fifty percent of the time, the server is fine. Your client code is eating the tokens.

Let's talk about streaming. Specifically, the myths around Server-Sent Events (SSE).

Myth One: SSE Is Just JSON Over WebSocket

I see this everywhere. People say, "Streaming needs websockets. It's two-way, right?"

No.

SSE is HTTP. The MIME type is text/event-stream. Websockets require a protocol upgrade. SSE works over plain HTTP.

One direction. Server to client. That's it.

For AI chat, that's exactly what you need. You send the prompt up. The model streams the response down.

Using WebSocket here adds overhead. You're managing: ws, onopen, onmessage. Why? HTTP works fine.

Myth Two: await res.json() Is Fine For Streaming

This is the biggest one.

When you use fetch, that data is a stream. Calling res.json() buffers the entire response before it resolves.

You are not streaming. You are waiting for the file to end.

The fix? Read the body stream chunk by chunk.

const resp = await fetch("/v1/chat/completions", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify(payload),
});

if (!resp.ok) throw new Error(`HTTP ${resp.status}`);

const reader = resp.body.getReader();
const decoder = new TextDecoder();
let buffer = "";

while (true) {
  const { value, done } = await reader.read();
  if (done) break;

  buffer += decoder.decode(value, { stream: true });
  const events = buffer.split("\n\n");

  // Last chunk might be incomplete. Keep it in the buffer.
  buffer = events.pop();

  for (const evt of events) {
    if (!evt.startsWith("data:")) continue;
    const raw = evt.slice(5).trim();
    if (raw === "[DONE]") continue;

    const json = JSON.parse(raw);
    const token = json.choices?.[0]?.delta?.content || "";
    process.stdout.write(token);
  }
}

Do you see the difference?

The first token appears on your screen. It doesn't wait for the last token.

Myth Three: Nginx Reverse Proxy and Edge Caching Buffers Are Some Mystery

Your proxy is probably buffering everything to disk before sending it.

Nginx has a feature called X-Accel-Buffering. FastAPI, Node, or your reverse proxy trusts it. If your proxy buffers chunks, your fellow devs.

Your low TTFB means nothing if the first byte you see hits a 500 ms boundary.

Try adding this header before streaming:

res.setHeader("X-Accel-Buffering", "no");
res.setHeader("Cache-Control", "no-cache");
res.setHeader("Connection", "keep-alive");

If you use FastAPI, you also need StreamingResponse:

from fastapi.responses import StreamingResponse

def stream_result(prompt: str):
    for token in call_free_model(prompt):
        yield token

@app.post("/v1/chat/completions")
def chat_route(payload: dict):
    return StreamingResponse(
        stream_result(payload["messages"]),
        media_type="text/event-stream",
        headers={"X-Accel-Buffering": "no"},
    )

If you skip the header, your model feels slow. It was never slow. Your infrastructure just said "hold on, I'll hold your beer, let me buffer first."

Myth Four: curl Shows The Real Behavior

I have an old habit. I run this:

curl -N https://api.example.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"free","stream":true}'

See the -N flag? That disables buffering.

A lot of people omit it. Then curl buffers. Then they tell me the API is slow.

Again: it's critical to separate the server's behavior from the client's behavior.

Here's a small executable script. It measures the time to the first token versus the time to the full response.

import time
import httpx
import statistics

def time_to_first_token(url, payload, runs=5):
    first = []
    total = []

    for _ in range(runs):
        start = time.monotonic()
        with httpx.stream("POST", url, json=payload, timeout=60) as resp:
            seen_first = False
            for line in resp.iter_lines():
                if line.startswith("data:") and not seen_first:
                    first.append(time.monotonic() - start)
                    seen_first = True
        total.append(time.monotonic() - start)

    return statistics.median(first), statistics.median(total)

# Example
url = "http://localhost:8000/v1/chat/completions"
payload = {"model": "my-free-model", "stream": True, "messages": [{"role": "user", "content": "hi"}]}

first_ms, total_ms = time_to_first_token(url, payload)
print(f"First token (median): {first_ms * 1000:.1f} ms")
print(f"Full output (median): {total_ms * 1000:.1f} ms")

If first_ms is in the 80-120 ms range but total_ms is 8 seconds, your server is actually streaming. The user perceives it as slow because the model spends 7 seconds generating tokens.

The MonkeyCode Proxy Check

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

When I run this against a free model endpoint, it helps to have the proxy in the loop. MonkeyCode's free server option lets you spin up a small proxy. That proxy can hide the model endpoint by design.

But if you set that proxy up, respect the streaming path. Don't let your proxy controller collect the whole payload and then forward it. That's a very common failure: your cloud function is invisible, but it kills the streaming.

I keep the proxy file simple:

service: proxy
build:
  command: python -m app

Again, don't add an intermediate layer that buffers. Your free model API becomes one giant lie. The endpoint responds in 200 ms, but your gateway shows 9 seconds.

Who Should Ignore This Advice

Not everyone needs to chase first-token latency.

If you're writing a CLI tool that processes a batch job, call res.json() honestly. It's easier to read. It's simpler to maintain.

If you're building a UI with a progress bar or chat bubble, you need streaming. You need first-token observability.

If you're building a debugging test suite, write the script I shared above. Keep it in your CI. Protect your proxy from buffer regressions.

The Correct Mental Model

Every SSE increment is a TCP packet.

Every buffer is a lie waiting to happen.

Your free model is not your car. It's a broker. If you send a response in the wrong way, the broker is the traffic jam.

Use -N in curl. Use getReader() in the browser. Use iter_lines() in Python. Force X-Accel-Buffering: no on your gateway.

Then you can complain about the model honestly—because sometimes it is genuinely slow.

But at least you'll have the numbers to prove it.