Two of Ten Items Ran, and the Log Said Success: A Guide to Spotting Batch Silent Failures

Your batch agent processes 10 items. The logs show success. But only 2 actually ran. The other 8? They vanished, without error, without trace, without the system ever telling you something went wrong.

Why this happens

Batch processing is where silent failures breed. Here's the mechanism:

Your agent loops over an input list. It calls a model (or a chain of tools) for each item. If the model returns an empty response, HTTP 200, but zero output tokens, the agent has a choice: explicitly handle that empty case, or skip it and move to the next item.

Most codebases skip it. The loop continues. No exception is thrown. The run completes with a "success" status because the loop itself completed without crashing. From the outside, everything looks fine. From the inside, 80% of your work never happened.

The trap: success at the orchestration level does not equal success at the execution level.

How to spot it in your logs (without a dashboard)

You don't need monitoring infrastructure to catch this. You need reasoning plus your existing logs.

Step 1: Count what you expected vs. what ran

Start here:

Input: [item_1, item_2, item_3, item_4, item_5, item_6, item_7, item_8, item_9, item_10]
Expected executions: 10

Now grep your logs for the actual model calls, the real token-generating events. Look for patterns like:

• OpenAI API call or Anthropic API call or your model provider's log marker
• tokens_sent, output_tokens, or whatever your SDK/client logs
• Anything that shows "a model was asked and answered"

Count them. If you see 2 real model calls but 10 input items, you already have your problem.

Step 2: Trace the gap, which items were processed?

Extract the item identifiers from the successful model calls. For example:

Call 1: processed item_2 → 120 output_tokens
Call 2: processed item_7 → 98 output_tokens

Items processed: {item_2, item_7}
Items missing: {item_1, item_3, item_4, item_5, item_6, item_8, item_9, item_10}

Now look at the logs around each missing item. Search for:

• Did the loop reach item_1? (Look for log lines like "processing item_1" or similar.)
• If it reached the item, did it call the model?
• If it called the model, what was the response? Was it empty? Did it include output tokens?

You're looking for the place where the logic broke. Usually it's one of these:

Scenario A: The loop never reached the item. Your batch input was truncated, filtered, or partially failed before iteration even started. Less common, but check your batch construction code.

Scenario B: The loop reached the item, but skipped calling the model. Your code has a conditional: if item.valid(): call_model(), and several items failed validation silently. Or a fallback triggered without logging it visibly.

Scenario C: The loop called the model, but the model returned empty output. This is the silent failure. The model returned HTTP 200, but output_tokens == 0 or the response body was blank/whitespace. Your code saw the empty response and either returned it as-is or skipped it without alerting you.

Step 3: Check token math

For each item that did run, log the input and output tokens:

item_2 input: 450 tokens → output: 120 tokens (good, output exists)
item_7 input: 480 tokens → output: 0 tokens (empty response)

If an item has input but output equals 0, that's a silent failure. The model was called, it returned success, but it produced nothing.

Check your code: does it handle this case explicitly?

# Without explicit handling (the dangerous version):
for item in batch:
    response = model.call(item)
    results.append(response)  # if response is empty, it still gets appended as "nothing"

# With explicit handling (safer):
for item in batch:
    response = model.call(item)
    if not response or response.tokens == 0:
        log_alert(f"Empty response for {item}")
        results.append({"error": "empty_response", "item": item})
    else:
        results.append(response)

The first version swallows the failure. The second surfaces it.

Step 4: Reason backward to the cause

Once you've identified which items failed (and how), ask:

Why did the model return empty?

• Rate-limited? (Check for 429 errors in the logs, or long latency gaps.)
• Malformed input? (What was actually sent to the model for that item? Log it.)
• Model-specific quirk? (Some models return 200 with empty output on certain inputs; check your model's behavior.)
• Timeout or partial read? (Did the connection drop mid-response?)

Why didn't the code catch it?

• No explicit check for output_tokens == 0?
• No retry logic?
• The result was treated as valid because the API call didn't throw an exception?

Most of the time, the root cause is: your code assumes that HTTP 200 means success, and never checks whether actual output was produced.

What to do right now

1. Add explicit output validation. After every model call, check output_tokens > 0 (or whatever means "real output" for your use case). Log failures.
2. Count your batch completeness. Before shipping, compare input count to output count. If they don't match, you have a silent failure.
3. Log item-level decisions. Every time an item is skipped, processed, or returns empty, log it with the item ID. Then grep your logs later with full visibility.
4. Retry on empty. If an item returns empty output, retry it once or twice before giving up. Log the retry. Most transient failures clear on retry.

The deeper lesson

Batch processing is high-leverage and high-risk: process 100 items and one failure hides in plain sight. Process 1,000 and you won't notice until a customer complains or a metric suddenly drops.

The mechanic is always the same: success at the wrong layer. The orchestration layer (the loop) succeeds. The execution layer (the model call) fails silently. The mismatch never bubbles up because nobody's looking at both layers together.

This is why builders who catch these failures early don't rely on "the system will tell you if something's wrong." They instrument the intersection: input count, output count, per-item token accounting, and explicit checks for empty responses.

Once you reason through this mechanism, you can spot batch silent failures in your logs in minutes. It's not mysterious. It's just accounting.

---

Have you hit this in a batch job? What made you notice? Drop a comment, I'd like to hear what surfaced it.