48-Hour Field Notes: My Diff Blamed the Model; My Stream Logger Was Splitting UTF-8

Two hosts, one prompt, ten runs each, and three diffs that should not exist. I spent most of two days trying to explain why the same request produced different answers, and the explanation turned out to be embarrassingly local. Nothing about the model or the network was inconsistent. My own logging layer was corrupting the text before the diff ever saw it.

This is a field note, not a tutorial: what I tried, what broke, and what I would keep. It ends with a test you can run in under a second with no network access at all.

The setup: ten identical requests, two places they ran

My working assumption was that comparing outputs across environments is mostly about controlling the request. So I froze everything I could:

1. One prompt, stored as a file, hashed with SHA-256 and printed at the top of every run.
2. One request body, serialized once with sort_keys=True so key order could not drift.
3. Ten sequential calls per host, no retries, no concurrency, a two-second sleep between calls.
4. Raw response bytes written to disk before any parsing happened.

I ran half the calls from my laptop and half from a small remote box, so the harness would survive me closing the lid. That box was MonkeyCode's free server option, and the endpoint I pointed it at was MonkeyCode's free model access; both are operator-supplied availability claims, not measurements I made. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Their relevance to the story is narrow: a 48-hour loop was cheap enough to leave running, and a second host existed at all.

The comparison itself was dumb on purpose. Decode both raw logs, normalize whitespace, diff the text.

Hour 4-12: the diffs looked real, so I blamed sampling

Three of twenty responses differed. All three differences were small - a word here, a punctuation mark there - which is exactly what a stochastic model looks like when you compare two samples. So I did the obvious thing: bumped the sample size to fifty runs, forced a zero-temperature setting where it was supported, and expected the diffs to collapse.

They did not collapse. They clustered. That is the detail I should have stopped on twelve hours earlier.

Every differing response contained a non-ASCII character somewhere: an em dash, a curly quote, a box-drawing character in a generated table. Responses that were pure ASCII matched byte for byte across both hosts. That is not a distribution you get from sampling noise. That is a distribution you get from a codec.

Hour 12-30: I grepped my own logs for the replacement character

The raw byte logs were large, so I searched instead of reading them.

# Count corruption markers in each raw log
for f in logs/*.raw; do
  printf '%s\t%s\n' "$(grep -c $'\xef\xbf\xbd' "$f")" "$f"
done | sort -rn | head

Output, trimmed:

1412    logs/free-server/run-07.raw
1108    logs/free-server/run-03.raw
 906    logs/laptop/run-02.raw
  17    logs/laptop/run-09.raw

U+FFFD, sitting inside the log I was treating as ground truth. Not in the model's response. In my log.

Then I read my own reader, which I had written specifically so I could keep raw bytes around:

# The buggy reader. Fine on ASCII, mangles everything else.
def read_sse(byte_chunks):
    for chunk in byte_chunks:
        # Each chunk is decoded on its own, so a UTF-8 sequence
        # split across a chunk boundary becomes U+FFFD.
        text = chunk.decode("utf-8", errors="replace")
        for line in text.split("\n"):
            if line.startswith("data:"):
                yield line[5:].strip()

That is the entire failure. A response chunk is a transport artifact, not a character boundary. When a three-byte character straddles two chunks, errors="replace" silently invents a replacement character instead of crashing; the log stays readable, and the diff downstream reports a difference that does not exist in the upstream bytes.

The three "different" responses were the three responses whose non-ASCII characters happened to land on a chunk boundary. My instrumentation had corrupted the thing it was measuring.

Hour 30-40: reproducing it without a network

I did not want to debug this against a live endpoint, so I reduced it to a fake byte stream. Two files, standard library only.

sse_lines.py:

import codecs
from typing import Iterable, Iterator


def lines(byte_chunks: Iterable[bytes]) -> Iterator[str]:
    """Split a byte stream into lines without assuming chunk == character."""
    decoder = codecs.getincrementaldecoder("utf-8")("strict")
    buffer = ""
    for chunk in byte_chunks:
        buffer += decoder.decode(chunk)        # holds a partial tail internally
        while "\n" in buffer:
            line, buffer = buffer.split("\n", 1)
            yield line.rstrip("\r")
    buffer += decoder.decode(b"", final=True)  # raises if the stream was truncated
    if buffer:
        yield buffer.rstrip("\r")


def data_payloads(line_iter: Iterable[str]) -> Iterator[str]:
    parts = []
    for line in line_iter:
        if line == "":
            if parts:
                yield "\n".join(parts)
                parts = []
        elif line.startswith("data:"):
            parts.append(line[5:].lstrip(" "))
    if parts:
        yield "\n".join(parts)

test_sse_lines.py:

from sse_lines import lines, data_payloads

# "cafe with an accent": the e-acute (0xC3 0xA9) is split across two chunks,
# and the event's terminating newline lands in a third chunk.
CHUNKS = [b'data: {"text": "caf', b'\xc3', b'\xa9 late"}\n\n']


def test_multibyte_split_across_chunks():
    payloads = list(data_payloads(lines(CHUNKS)))
    assert payloads == ['{"text": "caf\u00e9 late"}']
    assert "\ufffd" not in payloads[0]


def test_naive_per_chunk_decode_is_lossy():
    naive = [c.decode("utf-8", errors="replace") for c in CHUNKS]
    assert "\ufffd" in "".join(naive)   # the bug, asserted on purpose

Run it:

python -m pytest test_sse_lines.py -q
# 2 passed in 0.01s

The second test is the important one. It pins the failure mode in place, so nobody "fixes" the reader by going back to per-chunk decoding and quietly gets a green suite again.

Hour 40-48: re-running the comparison

With the reader fixed, I re-ran both hosts. Where the raw byte streams were identical, the decoded text was now identical too, and the diff count went to zero. Where the byte streams genuinely differed, the diffs stayed, and for the first time I could trust them.

Here is the table I wish I had on hour four:

Symptom	Likely cause	First check
Diffs only in outputs containing non-ASCII	chunk-boundary decode	count U+FFFD in the raw log
Diffs on every run, including pure ASCII	genuine sampling, or a request that changed	compare hashes of the request body
Diffs only from one host	client library version, proxy buffering, compression	print library versions and Content-Encoding
Diffs vanish when logging is disabled	the logger transforms the stream	log bytes, not strings

What I would repeat, and what I would change

Keep:

• Hashing the request body per run and printing the hash. It removes a whole class of doubt before you start.
• Writing raw bytes first and decoding later. Storage is cheap; re-running a 48-hour comparison is not.
• Reducing a live failure to a fake input. The network version cost hours; the fixture version cost a second.

Change:

• Grep the raw log for U+FFFD before blaming the model. It is a one-line check, and it would have saved a day.
• Treat every layer that touches the stream as suspect, including your own logging. The instrument is part of the measurement.

Limitations and who should not do this

This workflow answers one narrow question: given identical request bytes, did the response bytes differ? It cannot tell you whether either response is correct, and a clean diff is not an eval. If you need pass/fail judgments about output quality, you need a graded dataset rather than string equality, and you need to decide the grading criteria before the run starts.

Do not use a shared free box for latency numbers or capacity planning. A shared host is the wrong instrument for timing, and free tiers are not a promise about availability or permanence. If your team needs pinned model versions, fixed hardware, or an SLA, run your harness somewhere you control and pay for. Also, if your outputs are guaranteed ASCII by construction, most of this article is about a bug you will never hit, and your time is better spent on request pinning.

If you want to reproduce any of it, the artifact above needs nothing but the Python standard library. Pin your HTTP client version, log bytes before strings, and if you need somewhere for a long comparison to keep running, my second host is still the free server option on MonkeyCode.