I built a small service that turns messy support notes into clean JSON: a summary, a sentiment score, and a category. It ran on a free server, called a free model endpoint, and for two days every row in the output table looked healthy. Then a user opened one of those rows and asked why the summary was empty and the sentiment was neutral.
Disclosure: This article was prepared as part of MonkeyCode's product outreach. I deployed the service on MonkeyCode's free server and used their free model endpoint for the calls, so the whole experiment cost me nothing but patience. What it cost me in debugging time is another story.
The symptom that looked like a model problem
The batch had two hundred notes, and thirty of them came back with empty summaries. There were no exceptions, no failed HTTP statuses, and no retries, because the code never knew anything was wrong. The log line for a bad row looked identical to the log line for a good one, which is the first sign that you are not looking at the real error.
# processing.py (before the fix)
def process(note: str) -> dict:
raw = call_free_model(note)
try:
data = json.loads(raw)
except json.JSONDecodeError:
data = {"summary": "", "sentiment": "neutral", "category": "unknown"}
logger.debug("parse failed, using defaults: %s", raw[-200:])
return data
See the trap? The except branch converts a truncated response into a plausible default object, and the evidence goes to a DEBUG log that production never writes. The free server also rotated its logs quickly, so by the time I looked, even that evidence was gone.
The first wrong hypothesis
My first instinct was to blame the model, because free models are variable and everyone has a story about a weird answer. I added a retry with a fresh prompt, redeployed, and watched the same thirty notes fail in the same way. That determinism was the clue I almost missed, because random model quality would have produced random failures.
Ask yourself: if the same input fails identically on every attempt, is the model really the problem? Usually the answer is no, and the real problem lives between the model and your parser. I wrote a script that sent notes of increasing length and recorded the response length, and the pattern was immediate.
Reproducing the cliff
Short notes produced complete JSON, and long notes produced responses that stopped at the same ceiling every time. It looked like someone had cut the reply with scissors, and the cut position depended on the output length, not on the input content. I ran the script three times to be sure, and the ceiling did not move.
# reproduce_truncation.py
import requests
API_URL = "https://your-free-model-endpoint.example/v1/complete"
def call_model(note: str) -> str:
prompt = (
'Return JSON only: {"summary": string, '
'"sentiment": "positive"|"neutral"|"negative", '
'"category": string}\n\nNote: ' + note
)
response = requests.post(API_URL, json={"prompt": prompt}, timeout=30)
return response.text
for size in range(500, 6001, 500):
note = "The customer says the invoice is wrong and the total looks off. " * (size // 55)
raw = call_model(note)
print(f"input={size:5d} output={len(raw):5d} tail={raw[-30:]!r}")
The output showed a hard ceiling: once the note passed a certain size, the response length stopped growing. Some replies ended mid-string, which made json.loads fail, and others ended right after a complete field, which made the parser succeed with missing keys. Both cases produced the same empty row, and neither case produced an error.
The root cause in one sentence
The free model endpoint enforced a maximum output length, my prompt asked for a long summary, and the gateway cut the reply at the limit. My code then did the worst possible thing: it converted that truncation into a clean, plausible, silent default. Here is what each failure mode looked like:
| What happened | What my code saw | What the user saw |
|---|---|---|
| Response cut mid-string |
JSONDecodeError → default object |
empty summary, neutral sentiment |
| Response cut after a complete field | valid JSON with missing keys | empty category, neutral sentiment |
| Full response | valid JSON, all keys present | correct row |
The fix: make truncation loud
The fix had three parts. First, treat any parse failure or missing key as a TruncatedOutput exception instead of a default. Second, log the raw tail at ERROR level so the evidence survives log rotation. Third, retry once with a prompt that asks for a shorter summary, then fail loudly if it happens again.
# processing.py (after the fix)
class TruncatedOutput(Exception):
pass
REQUIRED_KEYS = {"summary", "sentiment", "category"}
def parse_model_output(raw: str) -> dict:
stripped = raw.strip()
if stripped.startswith("```
"):
stripped = stripped.split("
```", 2)[1]
try:
data = json.loads(stripped)
except json.JSONDecodeError as exc:
raise TruncatedOutput(f"invalid JSON near: {raw[-120:]!r}") from exc
missing = REQUIRED_KEYS - data.keys()
if missing:
raise TruncatedOutput(f"missing keys {missing}; tail: {raw[-120:]!r}")
return data
def process(note: str) -> dict:
for attempt in range(2):
raw = call_free_model(note, short=attempt > 0)
try:
return parse_model_output(raw)
except TruncatedOutput as exc:
logger.error("attempt %d truncated: %s", attempt + 1, exc)
raise TruncatedOutput(f"gave up after 2 attempts for note {note[:40]!r}")
Notice that the second attempt passes short=True, which changes the prompt to ask for a one-sentence summary. That usually fits under the output cap, and if it still does not, the exception propagates to the caller instead of hiding in a default. The caller now has a choice: skip the row, park it in a dead-letter queue, or page a human.
The regression test that would have caught it
I added a test that feeds a deliberately long note and asserts that the function raises instead of returning defaults. It failed before the fix and passed after, which is exactly what a regression test should do. The test is boring on purpose, because boring tests catch the most embarrassing bugs.
# test_processing.py
import pytest
from processing import process, TruncatedOutput
def test_long_note_raises_instead_of_silent_default():
long_note = "The customer says the invoice is wrong. " * 200
with pytest.raises(TruncatedOutput):
process(long_note)
Run that against the old code and it fails, because the old code returned a default object. Run it against the new code and it passes, because truncation is now a first-class failure. That one test is worth more than any amount of staring at logs.
Limitations and who should not copy this
Be honest about the limits of this approach. If your gateway exposes a finish_reason field, read that first, because it is more reliable than guessing from JSON shape. If you need guaranteed complete structured output, use an endpoint with a JSON mode or a higher output cap, because a guard that raises loudly is still a failure you must handle. And if your downstream system cannot tolerate a failed row, this pattern only moves the problem: you still need a queue, a retry policy, and an alert.
Who should not use this? Anyone processing medical, legal, or financial text where a silent default could cause real harm. For those cases, a truncated response must block the pipeline, not just raise an exception that a worker catches and ignores.
The lesson
Free infrastructure does not fail politely. It fails at boundaries: memory limits, output caps, ephemeral disks, and timeouts, and every boundary looks like a product bug until you measure it. The next time a free model returns something that looks almost right, ask one question before you trust it: did the reply actually finish?
I ran the fixed version on MonkeyCode's free server for another week, and the empty rows disappeared. If you are about to build on a free model endpoint, add the truncation guard before you add the feature. It is a five-minute change, and it turns a silent lie into a loud, debuggable truth.
Top comments (0)