Streaming on a Free Model Server: The Probe I Run Before Trusting a Stream

A stream ends. No error. No warning. Just silence.

Is the answer complete? Or did the connection die?

You can't tell. Not without a probe. I built one. I ran it against MonkeyCode's free server. Here's what I learned.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Why Streaming Is a Trap

Streaming feels fast. The first token arrives in seconds. Users see progress.

But a stream is a promise. It says "more is coming." When it breaks, that promise dies mid-sentence.

Chat UIs need streaming. Batch jobs don't. Know which one you're building.

The Experiment

I designed a simple probe. Twenty runs. One fixed prompt. Same free model endpoint.

Each run records:

• Time to first token
• Total duration
• Chunk count
• Finish reason
• Interrupt flag

That's it. No fancy metrics. Just the signals that matter.

The Probe Script

Here's the harness. OpenAI-compatible client. Python. Copy it and run it.

import time
import json
from openai import OpenAI

client = OpenAI(
    base_url="https://your-endpoint.example/v1",
    api_key="your-key",
)

PROMPT = "Write a detailed 300-word explanation of TCP backpressure."

def run_probe(run_id: int):
    started = time.perf_counter()
    first_token_at = None
    chunks = []
    finish_reason = None

    stream = client.chat.completions.create(
        model="free-model",
        messages=[{"role": "user", "content": PROMPT}],
        stream=True,
    )

    for chunk in stream:
        if first_token_at is None:
            first_token_at = time.perf_counter() - started
        choice = chunk.choices[0]
        if choice.finish_reason:
            finish_reason = choice.finish_reason
        if choice.delta and choice.delta.content:
            chunks.append(choice.delta.content)

    total = time.perf_counter() - started
    return {
        "run": run_id,
        "ttft_s": round(first_token_at, 3),
        "total_s": round(total, 3),
        "chunks": len(chunks),
        "finish_reason": finish_reason,
    }

for i in range(20):
    try:
        print(json.dumps(run_probe(i)))
    except Exception as exc:
        print(json.dumps({"run": i, "error": str(exc)}))

Run it. Then look at the finish reasons. That's where the truth hides.

Reading the Results

A healthy run ends with finish_reason="stop". Anything else is a red flag.

• stop — complete. Trust it.
• length — truncated. The model hit a cap.
• None — silent death. The connection dropped.
• exception — loud death. At least you know.

My runs showed a pattern. Interrupts happened. Some streams died with no finish reason at all.

That's the worst case. No error. No reason. Just a half answer.

Detecting the Silent Drop

You can't catch a silent drop inside the loop. The loop ends. The stream is gone. You only have the finish reason.

So check it. Always.

if finish_reason != "stop":
    print(f"[WARN] run {run_id} incomplete: {finish_reason}")

One line. It turns an invisible failure into a visible one.

Retry, But With Context

A retry fixes network blips. But a naive retry repeats the whole prompt. You lose the partial text.

My fix: send the partial text back. Ask the model to continue.

def continue_stream(client, partial_text, messages):
    continued = messages + [
        {"role": "assistant", "content": partial_text},
        {"role": "user", "content": "Continue exactly where you stopped."},
    ]
    return client.chat.completions.create(
        model="free-model",
        messages=continued,
        stream=True,
    )

This worked in my tests. The model resumed mid-sentence. No duplicates. No lost context.

Don't retry forever. Three attempts. Then show a friendly error.

Buffer, Don't Blink

Token-by-token printing looks jittery. Users see flicker. The UI feels broken.

Buffer until a sentence boundary. Then flush.

buffer = ""
for chunk in stream:
    delta = chunk.choices[0].delta
    if delta and delta.content:
        buffer += delta.content
        if any(mark in buffer for mark in (". ", "! ", "? ", "\n")):
            print(buffer, end="", flush=True)
            buffer = ""
if buffer:
    print(buffer, end="", flush=True)

Smoother output. Fewer visual glitches. Same stream underneath.

The Decision Table

Here's how I decide what to do after a bad run.

Finish reason	Likely cause	Action
stop	None	Accept
length	Token cap	Retry with continuation
None	Dropped connection	Retry with backoff
exception	Network or server error	Retry with backoff, then fail

Free servers have no SLA. Plan for every row. Hope for the first one.

Limitations

This probe is not a benchmark. It measures one endpoint, one day, one network.

Free servers change. Rate limits shift. Your results will differ.

Also, streaming burns tokens fast. A long stream can eat thousands. Watch your usage.

Don't build mission-critical apps on this. Prototypes, demos, personal tools. That's the right scope.

Who Should Skip This

Skip streaming if you need guaranteed delivery. Batch mode is simpler.

Skip it for long document processing. One request, one complete response. No reconnects.

Skip it if you can't handle retries. Streaming without retries is gambling.

Run It Yourself

Grab your free server credentials. Run the probe. Log your finish reasons.

Share your numbers. Compare interrupt patterns. That's how we learn.

Free tiers exist for experiments. The probe is the shovel. Dig in.