The Exit Code That Lied: Debugging a Silent Failure on a Free Server

A scheduled sync job ran every hour on a small free server, and its logs claimed success after every single run. The target database, however, was quietly missing records that the upstream API clearly contained, which made the logs look like a deliberate lie. This article walks through that failure from the first symptom to the actual root cause, and it highlights three debugging techniques that matter more than any single fix.

The job was deliberately simple in its design. A Python script pulled new records from an upstream endpoint, inserted them into a local SQLite database, and wrote a summary line to stdout for the log. Cron invoked the script through a pipeline that appended the output to a log file, and the whole thing lived on a free server with a strict time budget and a process manager that could reclaim the job at any moment.

The first sign of trouble appeared when a comparison query showed a gap between the upstream record count and the local one. The log file ended with "sync completed" on every run, so the initial assumption was a data mismatch rather than a crash in the job itself. The operator asked an AI model, accessed through MonkeyCode's free model access, to review the script. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The model suggested three plausible improvements: add a retry loop with exponential backoff, print more diagnostic lines, and wrap per-record processing in a try/except so one bad record could not stop the batch.

All three suggestions were reasonable in isolation, and all three made the failure harder to see in practice. The retry loop re-fetched the same broken record, the extra prints vanished into the void, and the try/except converted a crash into a silent skip. With AI-assisted debugging becoming a default workflow for many teams, the lesson here is that a model's plausible suggestion is still a hypothesis, not a verified root cause. This is the moment when the debugging retrospective really begins, because the fix was never going to come from more logging or more retries.

Technique 1: Audit the real exit code

The cron entry looked like this:

0 * * * * cd /srv/sync && python3 sync.py 2>&1 | tee -a sync.log

A pipeline returns the exit status of its last command, so the reported status always came from tee rather than from Python itself. The shell keeps the real statuses in the PIPESTATUS array, and reproducing the pipeline by hand exposed the lie immediately:

cd /srv/sync
python3 sync.py 2>&1 | tee -a sync.log
echo "pipeline exit: $?"                 # always 0
echo "python exit:   ${PIPESTATUS[0]}"   # 1

The Python process had been failing for days, and the pipeline had been reporting success the entire time. Adding set -o pipefail to the wrapper script would have caught this on the very first run.

Technique 2: Force unbuffered output

The next mystery was the missing diagnostic output from the script. Python buffers stdout when it is not attached to a terminal, so prints that happened just before a crash stayed in the buffer and disappeared when the process was killed. Running the script with python3 -u or setting PYTHONUNBUFFERED=1 revealed exactly where the job stopped, which turned out to be the first record that lacked an external_id field.

Technique 3: Build a minimal reproducer

The full script was too noisy for bisection, so the operator stripped it down to the smallest failing case: one upstream call, one insert, and one deliberate raise.

# repro.py
import sys, time

print("connecting to upstream...", flush=True)
time.sleep(1)
raise KeyError("missing field: external_id")

Running this tiny script under the same pipeline and the same unbuffered mode confirmed the root cause in under a minute. The broad except Exception that the AI review had added was swallowing the KeyError, logging a single "skipped" line, and moving on as if nothing had happened.

The root cause, in three layers

The failure was not one bug but three compounding layers that reinforced each other. The pipeline masked the exit code, stdout buffering hid the last prints, and the broad exception handler converted a real error into a silent skip. The AI-suggested retry loop made the situation worse because every retry re-fetched the same broken record, and the "skipped" log line looked like progress to anyone reading the output.

The fix

The corrected shell wrapper checks the real exit status and fails loudly instead of hiding it:

#!/usr/bin/env bash
set -euo pipefail
cd /srv/sync
python3 -u sync.py 2>&1 | tee -a sync.log

The corrected Python side logs the traceback and re-raises the error instead of swallowing it:

import logging

logging.basicConfig(
    filename="sync.log",
    level=logging.INFO,
    format="%(asctime)s %(levelname)s %(message)s",
)

def process(record: dict) -> None:
    try:
        insert(record)
    except KeyError as exc:
        logging.exception("record %s is missing a field", record.get("id"))
        raise

A heartbeat file gave the scheduler a way to detect stalls even when the process vanished without a trace:

#!/usr/bin/env bash
# heartbeat.sh — run from cron every five minutes
last=$(stat -c %Y /srv/sync/heartbeat 2>/dev/null || echo 0)
now=$(date +%s)
if (( now - last > 300 )); then
  echo "sync job heartbeat expired" >&2
  exit 1
fi

The job itself touches the heartbeat file after every batch of records, so a stalled or killed run becomes visible within five minutes instead of days.

The reusable checklist

• Check the real exit status of every pipeline with PIPESTATUS or set -o pipefail.
• Run Python with -u or PYTHONUNBUFFERED=1 whenever output feeds a log file.
• Never wrap a loop body in a bare except Exception that only logs a line; log the traceback and re-raise.
• Add a heartbeat file for long-running jobs on servers that can reclaim processes at any time.
• Make the job idempotent with a watermark so a re-run after a crash is safe.
• When an AI model suggests a fix, apply the smallest change first and confirm the failure is still visible before adding retries.

Limitations and who should skip this approach

The free server is an experiment environment, not a production platform, and the constraints that made this bug visible are the same constraints that make it unsuitable for critical workloads. Teams with strict compliance or availability requirements should keep their jobs on infrastructure they control. The 10-million-token free allowance and the free server option are useful for reproductions like this one, but limits and conditions can change, so checking the current documentation before building a workflow around them is the responsible move.

The debugging techniques themselves transfer everywhere, and they are the real takeaway from this retrospective. MonkeyCode is an open-source project that pairs free model access with a free server option, which makes it a practical place to run constrained-environment experiments like this one. If that kind of debugging is a regular part of your week, the project is worth a look; the free allowance makes it easy to test a hypothesis without opening a wallet.