My Batch Job Had a 100% Success Rate and a 4% Corruption Rate

Have you ever watched a batch job finish with a perfect success rate, only to find the data was quietly garbage? Last week I ran 2,000 prompts through a free model endpoint, and every request returned HTTP 200 with zero errors in the logs. Three days later, a data quality check found 84 rows that were truncated, empty, or duplicated. My pipeline had a 100% success rate and a 4.2% corruption rate at the same time.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The symptom: green logs, dirty data

The job itself was boring in the best way: read a prompt from a queue, send it to the model, and store the response in Postgres. I used MonkeyCode's free model access on the free server option because the workload was asynchronous, and I did not want to spend paid credits on a one-off enrichment pass. The run took about an hour, and the logs looked beautiful.

• All 2,000 requests returned HTTP 200.
• Zero timeout exceptions, zero connection errors, zero 429s.
• Average latency sat at 1.8 seconds, which felt perfectly reasonable.

I moved on to other work and forgot about the job entirely. Three days later, I ran a quality check on the table, and the numbers did not make sense. The check counted empty bodies, unparseable JSON, and duplicate responses, and it flagged 84 rows out of 2,000. That is a 4.2% corruption rate hiding behind a 100% success rate.

Why the status code lied to me

The core problem was that I treated HTTP 200 as a contract, when it is really just a receipt. A 200 response only means the server received the request and returned something; it says nothing about whether that something is complete, well-formed, or even related to the prompt I sent. I started reading through the corrupted rows, and each one told a different story about how a request can fail while still reporting success.

The free tier did not cause these failures, but it did make them more visible, because a free server under load is more likely to cut corners than a paid one with generous headroom. How many times have you checked response.status_code == 200 and then moved on without examining the body? That habit is exactly how 84 rows of garbage ended up in my database.

The three corruption patterns I actually found

I sorted the 84 bad rows into three buckets, and each bucket had a distinct signature that pointed to a different failure mode.

Pattern one: truncated JSON

The response body ended mid-string, usually right around the 1,024-character mark, which was a suspiciously round number that suggested an internal buffer limit. The server had clearly hit an internal limit and returned whatever it had buffered so far, complete with a content-length header that matched the truncated body. My code parsed the JSON, hit an exception, and the exception handler wrote the raw string to the database instead of failing the row.

# The bug: swallowing parse errors and storing garbage
try:
    data = json.loads(response.text)
    text = data["text"]
except json.JSONDecodeError:
    text = response.text  # <-- this is how garbage got in

Pattern two: empty bodies

About thirty rows had a 200 status with an empty string as the body, which meant no JSON, no whitespace, and nothing to parse at all. The server probably timed out internally and returned an empty response rather than an error, and my client happily accepted it as a valid answer.

Pattern three: duplicated content

Twelve rows contained the exact same response text as an earlier row, even though the prompts were completely different. The server was likely returning a cached or repeated generation, and nothing in the response headers indicated that the content was a duplicate. My deduplication logic did not exist, so every copy landed in the table without a complaint.

The validation layer that caught them all

The fix was not to make the client more aggressive or the retries smarter. The fix was to stop trusting the status code and validate every response before writing it to the database. I wrote a small function that applied five rules to each response, and any response that failed a rule was treated as a retryable failure.

import json
import hashlib

seen_hashes = set()

def validate_response(prompt: str, response_text: str) -> str | None:
    # Rule 1: reject empty bodies
    if not response_text.strip():
        return None

    # Rule 2: reject unparseable JSON
    try:
        data = json.loads(response_text)
    except json.JSONDecodeError:
        return None

    # Rule 3: reject missing required fields
    if "text" not in data or not data["text"].strip():
        return None

    # Rule 4: reject suspiciously short answers
    if len(data["text"]) < 20:
        return None

    # Rule 5: reject exact duplicates
    digest = hashlib.sha256(data["text"].encode()).hexdigest()
    if digest in seen_hashes:
        return None
    seen_hashes.add(digest)

    return data["text"]

The function returns None for anything suspicious, and the caller treats None as a retryable failure. I also added a retry loop around the whole thing, capped at three attempts. Any prompt that still failed after three tries went into a separate quarantine table for manual inspection.

def process_with_validation(prompt: str, max_attempts: int = 3) -> str:
    for attempt in range(max_attempts):
        response = client.chat(prompt)
        validated = validate_response(prompt, response.text)
        if validated is not None:
            return validated
        time.sleep(1 + attempt)
    raise ValidationError(f"prompt failed validation after {max_attempts} attempts")

After deploying this, the corruption rate dropped from 4.2% to zero, and the quarantine table gave me a clean list of prompts that genuinely needed attention. The retry loop handled the transient truncation cases, and the validation layer caught everything else before it could poison the dataset.

What this taught me about trusting APIs

The deeper lesson is that a status code is a transport-level detail, not a data-quality guarantee. I had built the entire pipeline around the assumption that 200 meant success. That assumption silently corrupted 84 rows of data that I almost used for downstream analysis. The validation layer cost me about an hour to write, and it saved me from making decisions based on data that was quietly wrong.

How many pipelines in your codebase check the status code and then trust the body without a second thought? If you are like me, the answer is too many. The fix is not more careful coding; the fix is a validation step that treats the response body as untrusted input.

Limitations and who should skip this approach

This validation layer is not a universal solution, and there are cases where it will cause more harm than good. If your model outputs free-form creative text with no fixed schema, rules like minimum length and required fields will reject perfectly valid responses. You should tune the thresholds to your actual distribution.

The duplicate check also assumes that identical responses are always wrong, which is true for my enrichment workload but false for many other use cases. If you are generating summaries of similar documents, identical output might be a legitimate result, and the hash check would incorrectly discard it.

And if you are just prototyping and the data does not feed anything important, a full validation layer is probably overkill. The cost of this approach is real: more code, more retries, and a quarantine workflow that needs human attention. For a throwaway experiment, you can afford to skip it; for anything that feeds a database you plan to trust, you cannot.

The free tier from MonkeyCode was adequate for this workload, and the validation layer turned a silently corrupt dataset into a clean one. The validation logic itself is the part I would reuse everywhere. Status codes are a starting point, not a conclusion, and the next time a batch job reports 100% success, I will check the actual rows before I believe it.