Exit Code 137 Was the Only Clue: My Free Server Had a Memory Ceiling I Never Measured

Last week my batch processor died three times in one night, and every death looked identical: no traceback, no error log, no restart message. The process just vanished while the API kept answering, so my first instinct was to blame the code I had written. I drafted that code with a free model in MonkeyCode and deployed it to MonkeyCode's free server. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The real culprit turned out to be the kernel, not the code, and the only clue was an exit code I almost ignored.

The setup that looked fine

The job was simple: read a CSV export, transform every row, and write the results to a JSON file. The free model produced a clean, readable implementation, and locally it finished a 50 MB file without complaint. I ran it once, watched it finish, and scheduled it to run every hour.

# process.py — the version that died silently
import csv
import json

def process_all():
    with open('input.csv') as f:
        rows = list(csv.DictReader(f))      # every row in memory
    results = []
    for row in rows:
        results.append(transform(row))      # every result in memory
    with open('output.json', 'w') as f:
        json.dump(results, f)

The first two runs succeeded, and then the deaths started. The code had not changed, the input had not changed format, and the supervisor reported no restart. What had changed?

The symptom: gone without a trace

At 2:00 AM the process was running, and at 2:07 AM it was not. No Python exception, no supervisor alert, no core dump, and the journal showed nothing because there was nothing to show. The health endpoint stayed green because the API was a different process with a different memory profile. I checked the usual suspects in order: disk space, file permissions, CSV encoding, and the Python version, and all of them were fine.

Then I checked the one thing I normally skip: the exit status that the supervisor had recorded. It was 137.

The clue: exit code 137

That number is 128 plus 9, and signal 9 is SIGKILL. The kernel terminated the process, which means no Python handler ever ran, no finally block executed, and no log line could possibly have been written. That is why the silence was so complete.

# how I confirmed the exit code
systemctl show batch.service -p ExecMainStatus
# or, when running manually:
python process.py
echo $?   # prints 137

The convention is worth memorizing: exit codes above 128 mean the process was killed by a signal, and the signal number is the exit code minus 128. 137 means SIGKILL, 143 means SIGTERM, and 139 means SIGSEGV. Each one tells a different story, and 137 points straight at the kernel's out-of-memory killer.

The root cause: the OOM killer

Once I knew the signal, the next question was why the kernel decided to kill my process. On Linux, the OOM killer activates when the system runs out of memory, and it picks a victim using a heuristic called oom_score. My free server had a memory ceiling I had never measured, and my batch processor was climbing toward it with every row it loaded.

The first two runs survived because the CSV was small. The third run coincided with a larger export and a busier API, and the combined memory pressure crossed the ceiling. The kernel did not care that my code was correct; it cared that something had to die, and my process was the most expendable thing in the container.

Reproducing the kill on purpose

Reproducing an OOM kill is easy once you know what to measure. I wrote a tiny probe that allocates memory in a loop and watched where it died. The pattern was boringly predictable, which made it perfect for confirming the diagnosis.

# memory_probe.py — grow until the kernel pushes back
import time
chunks = []
while True:
    chunks.append(bytearray(10 * 1024 * 1024))  # 10 MB at a time
    time.sleep(0.1)

Run it with a monitor in another terminal and you will see the exact ceiling. Watch the RSS column, not the VSZ column; RSS is the physical memory the kernel counts against the limit, while VSZ includes virtual reservations that may never materialize.

ps -o pid,rss,vsz,cmd -p $(pgrep -f memory_probe.py)

The probe died at the same ceiling my batch processor had hit, which confirmed the diagnosis without any guessing. The RSS trend was a straight line upward, and the kill happened at the same boundary every time. That reproducibility turned a suspicion into a measurement.

The fix: stream instead of load

The fix was to stop loading the whole dataset into memory. The CSV reader already supports iteration, and the output can be written row by row, so peak memory drops from "size of everything" to "size of one row." Why hold a whole file in RAM when you only need one row at a time?

# process.py — the streaming version
import csv
import json

def process_all():
    with open('input.csv') as f, open('output.json', 'w') as out:
        reader = csv.DictReader(f)
        out.write('[')
        for i, row in enumerate(reader):
            if i:
                out.write(',')
            json.dump(transform(row), out)
        out.write(']')

Streaming fixed the batch processor, but the deeper fix was adding a memory budget to every job I deploy. I now ask three questions before scheduling anything: what is the largest input, how much memory does one row need, and what happens when both grow at once? The third question is the one that bit me, because the answer was "it dies."

A reusable workflow for silent deaths

• Check the exit code first: 137 means SIGKILL, 143 means SIGTERM, and anything below 128 means the process had a chance to handle the failure.
• Measure memory over time, not just at startup: sampling ps -o rss every few seconds reveals the trend that a single snapshot hides.
• Reproduce with a deliberate memory probe so you learn the platform's ceiling before your real data does.
• Look for unbounded lists, full-table loads, and missing pagination; those are the usual memory escalators.
• Set an explicit memory limit in the process manager so the failure becomes a clean error instead of a mystery.

Limitations and who should skip this

This workflow assumes you control the process and can read its exit status. If your platform hides exit codes or restarts processes transparently, you need a heartbeat that records memory usage instead. If your jobs are tiny and short, measuring memory is overkill. And if your workload genuinely needs more memory than the free tier offers, no amount of streaming will save you; the honest answer is a paid tier or a managed queue. The OOM killer is not a bug in your code; it is the platform enforcing a boundary you never measured.

The lesson

Looking back, the expensive mistake was not the memory usage; it was assuming that a missing traceback meant the code was innocent. Exit codes are the first witness at any silent death, and 137 has a specific story to tell. Next time your process vanishes, ask what killed it before you ask what went wrong in the code. If you have a silent-death story of your own, I would like to hear which clue cracked it for you.