Empty Arrays Passed Validation, So the Bug Shipped

The model was fine. The validator was wrong. An empty list is a valid list, so nothing failed.

What follows is a debugging walkthrough of a failure pattern I keep meeting in LLM extraction pipelines. It is a reconstruction of the shape, not a war story with a dashboard. Every step is something you can replay on your own machine today.

The symptom: exit code 0, half the rows

The nightly job pulls line items out of inbound email. It ran on five containers. Two of them returned far fewer rows than the others.

No exception. No error log line. Exit code 0. The dashboard was green because the job measured completion, not content.

So the first question is not "why did the model fail?" The first question is "why did nothing notice?"

The wrong suspect

Is the model lazy? Probably not. Laziness is not a reproducible property. It is a story we tell when we cannot diff two inputs.

I ran the same email through the same prompt in a notebook and got three rows. Production got zero. Same model, same text, different result.

The difference was not the weights. It was the container around them.

Step 1 — Freeze the failing input

Before touching code, capture the request and response pair to disk. A log line is a rumor. A fixture is evidence.

# capture.py — one request, one response, one file
import hashlib
import json
import pathlib


def capture(prompt: str, response: dict, out_dir: str = "fixtures") -> str:
    p = pathlib.Path(out_dir)
    p.mkdir(exist_ok=True)
    key = hashlib.sha256(prompt.encode()).hexdigest()[:12]
    path = p / f"{key}.json"
    path.write_text(
        json.dumps({"prompt": prompt, "response": response}, indent=2),
        encoding="utf-8",
    )
    return str(path)

Two rules make fixtures useful. Record the model name and every sampling parameter inside the file. If you cannot pin those, you cannot replay, only re-guess.

Step 2 — Replay outside production

Now build a seam. One function, one job: take a prompt, return parsed items. Everything else is replaceable.

# replay.py — provider-neutral seam
import json
from typing import Any


def call_model(prompt: str) -> dict[str, Any]:
    # swap in your provider call here
    raise NotImplementedError


def replay(fixture_path: str) -> dict[str, int]:
    data = json.loads(open(fixture_path, encoding="utf-8").read())
    expected = len(data["response"].get("items", []))
    actual = len(call_model(data["prompt"]).get("items", []))
    return {"expected": expected, "actual": actual}

For my replay loop I route that seam to MonkeyCode's free model access, so re-running dozens of fixtures does not turn into a budgeting conversation. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The output is a count diff, not prose. {"expected": 3, "actual": 0} is a bug report you can act on.

Step 3 — Find where the exception vanished

The replay hit zero on the first fixture. Same input, same prompt, zero rows. That meant the failure was in my code, and the container was the clue.

Here is the retry wrapper that shipped.

def extract_with_retry(prompt, attempts=3):
    for _ in range(attempts):
        try:
            return call_model(prompt)
        except Exception:
            continue
    return []   # <- the bug lives on this line

Three failed attempts became an empty list. The empty list passed the schema, because items: list[Item] accepts zero elements. The job then reported success with nothing to show.

The retry did not hide an error. It manufactured an answer.

Step 4 — Make silence impossible

A schema validates shape. It does not validate meaning. Add an explicit failure type and let it escape.

class EmptyExtraction(RuntimeError):
    pass


def extract_with_retry(prompt, attempts=3):
    last_error = None
    for _ in range(attempts):
        try:
            items = call_model(prompt).get("items", [])
            if not items:
                raise EmptyExtraction("model returned zero items")
            return items
        except Exception as exc:
            last_error = exc
    raise EmptyExtraction(f"all {attempts} attempts empty") from last_error

Careful: zero items is sometimes the correct answer. Validate against the fixture's expected count, not against a hardcoded one. Then emit a counter, so the next silent drop shows up as a metric instead of a support ticket.

Step 5 — Run the harness somewhere disposable

The replay service has an odd shape. It is small, bursty, and idle most of the day. Nobody pages anyone when it sleeps.

That is exactly the shape where MonkeyCode's free server option fits, so I parked the harness there instead of on the production cluster.

# replay_service.py — illustrative, not a production deployment
import json

from fastapi import FastAPI

app = FastAPI()


@app.post("/replay")
def run_replay(body: dict) -> dict:
    data = json.loads(open(body["fixture"], encoding="utf-8").read())
    actual = len(call_model(data["prompt"]).get("items", []))
    expected = len(data["response"].get("items", []))
    return {"expected": expected, "actual": actual, "match": expected == actual}

# endpoints below are placeholders, not shipped defaults
curl -s -X POST "$REPLAY_URL/replay" \
  -H 'content-type: application/json' \
  -d '{"fixture":"fixtures/a1b2c3.json"}' | jq .

I keep the harness off the critical path on purpose. A debugging tool that can page you is a liability.

The techniques worth keeping

1. Freeze the input before you read the code. Evidence first, theories second.
2. Replay outside production. Same prompt, same parameters, different host.
3. Diff counts, not sentences. Numbers survive summarization.
4. Grep for every except that returns a default. Each one is a possible silence.
5. Convert the fix into a fixture. A bug without a test comes back.

Limitations and who should skip this

If your fixtures contain customer data, do not ship them to a third-party host. Redact first, or replay locally.

If you cannot pin the model name and parameters, your replay is a rerun, not a test. Treat the diff as a hint, not proof.

Free tiers are fine for a dev harness. They are not an SLA. Measure capacity yourself before you depend on it, and keep latency-sensitive benchmarking on hardware you control.

Finally, if zero results is a legitimate business outcome, do not blanket-ban empty lists. Add the invariant where emptiness is actually wrong.

The question to ask tomorrow

When a job exits 0, ask what it counted. Then freeze one input and replay it. That habit found this bug faster than any prompt rewrite.

If you want to try the harness shape without a bill, the free model access and free server option were enough for one fixture at a time.