The JSON Was Valid, the Log Was Green, and the Fifth Article Vanished

The symptom: a digest with one article missing

Last Tuesday my daily digest service shipped four articles instead of five. The cron job on my free server reported success, the logs showed no exceptions, and the health check stayed green the entire time. The JSON response parsed without a complaint, and my schema validator approved every field it saw. Nothing in the system said data had gone missing, which is exactly why it took me two days to find the problem.

What the service actually does

The service itself is deliberately boring. Every morning a cron job pulls the five most recent links from a queue, sends them to a free model, and asks for a JSON array with a title and a one-sentence summary for each link. I run the whole thing on MonkeyCode's free server because the workload is tiny and I did not want to pay for a box that idles for twenty-three hours a day. The free model tier handles the summarization, and for two weeks the output was flawless. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The first two hypotheses went nowhere

Then a reader asked why Tuesday's digest only had four items. My first instinct was to blame the queue, because a free server had already eaten my data once before. I checked the queue table and found five rows consumed, which meant the model had received all five links. The job stored the raw model response, so I opened it and found perfectly valid JSON containing exactly four objects.

My second hypothesis was that the model simply skipped one. I re-ran the same prompt with the same five links by hand and got five objects back, which made the failure look random. Then I re-ran the actual job and got four again. The difference between my manual test and the real job turned out to be the clue, but I almost missed it because both runs looked identical on the surface.

The root cause was in the envelope, not the content

The response envelope included a finish_reason field, a standard part of most OpenAI-compatible APIs, and mine said "length". That means the model ran out of output tokens before it finished generating, so the content string was truncated mid-object. My parser threw an exception on the broken JSON, and then my own salvage code took over and hid the evidence.

Here is roughly what the stored response looked like:

{
  "finish_reason": "length",
  "content": "[{\"title\":\"A\",\"summary\":\"...\"},{\"title\":\"B\",\"summary\":\"...\"},{\"title\":\"E\",\"summary\":\"truncat"
}

The envelope is valid JSON, the content is not, and the finish_reason was screaming at me the whole time. The debugging technique that saved me was simple: inspect the full response object, not just the field you plan to consume.

The salvage function that betrayed me

Here is the exact salvage function that turned a recoverable error into silent data loss:

function parseItems(raw) {
  try {
    return JSON.parse(raw);
  } catch {
    // Salvage: grab any complete objects that look like items
    const matches = raw.match(/\{[^{}]*\}/g) || [];
    return matches
      .map((m) => {
        try {
          return JSON.parse(m);
        } catch {
          return null;
        }
      })
      .filter(Boolean);
  }
}

The regex /\{[^{}]*\}/g finds every complete object with no nested braces and silently drops anything malformed. The fifth object was truncated, so the regex skipped it, and my code built a beautiful four-item digest from a five-item request. The job logged success because the function returned a non-empty array, and the schema validator was happy because every object it saw was well-formed.

Why the manual test passed

Why did the manual test pass, then? Because the summaries in my hand-typed test happened to be shorter, so the whole response fit under the token ceiling. The real job failed because one summary ran long and pushed the output just past the limit. Free model tiers often have tighter output ceilings than paid ones, and I had never measured how close my prompt sat to that boundary.

The real lesson is that a manual test is not a reproduction. A reproduction has to use the exact stored input, the exact prompt, and the exact token budget; otherwise it proves nothing.

Four fixes, in order of importance

First, log the finish reason and the raw content length on every call. finish_reason: "length" is a loud alarm, but it only helps if someone actually reads it.

Second, validate completeness, not just syntax. If the prompt asks for five items, the code must require five items:

if (items.length !== EXPECTED_COUNT) {
  throw new Error(`Expected ${EXPECTED_COUNT} items, got ${items.length}`);
}

Third, delete the silent salvage path or make it scream. A fallback that quietly drops malformed data turns a recoverable error into silent data loss, which is strictly worse than a failed job.

Fourth, restructure the prompt so truncation is less likely. I split the work into one call per article, so each response is tiny and the ceiling never comes into play. A single-item call that truncates fails loudly instead of vanishing into a regex.

A reusable checklist for parsing model output

Here is the checklist I now apply to any code that parses model output:

• Log the raw response and the finish reason before parsing anything.
• Validate the count and required fields, not just the types.
• Treat salvage and repair code as a code smell; if it can drop data, it must log a warning.
• Measure how close your prompt sits to the output ceiling, then leave headroom.
• Reproduce failures with the exact stored input, not a fresh hand-typed prompt.
• Alert on any finish_reason other than stop; treat it as a warning at minimum.

Limitations and who should not use this

This approach has limits, and I want to be honest about them. Splitting one request into five costs more calls and can hit rate limits, so it is not free. If your workload genuinely needs one large structured response, you need a repair strategy that fails loudly and retries with a shorter prompt. The regex salvage trick also breaks on nested braces, which is another reason it should never be the quiet path.

Who should not use this pattern? Anyone building a pipeline where silent data loss is worse than a failed job, which is honestly most pipelines. A failed job wakes you up at night, but a successful job with missing data just looks normal forever.

The free tier is fine for experiments, but it rewards code that measures its limits instead of assuming them. I still run this digest on MonkeyCode's free server, but now the job fails loudly when the model under-delivers. A red alert about a missing article is annoying, and it is infinitely better than a green log that lies to me.