I OOM'd on 100 Rows Because One File Was 590 MB

My validator got SIGKILL'd on 100 rows. Not 100 million — a hundred. And I'd already done the responsible thing and capped how many files it reads.

The cap bounded file count, not file size. One 590 MB parquet file defeated it, and 100 rows took down a 7 GB VM.

This is from validating large agent-trace corpora for agentwatch, part of agentic-suite — a suite of open-source, harness-agnostic security tools for AI agents. Agent traces are not like web request logs: a single run can produce megabytes of text, and when the payload is that large per row, file-count limits stop being memory bounds.

The setup that looked safe

The corpus: 4,974 parquet files totaling 7.5 GB. The Docker VM had 7 GB of RAM shared across eight containers (postgres, jaeger, otel-collector, api, analytics worker, web, recorder, verifier). Each file gets loaded fully into memory, so I set --max-files 5 to keep the working set bounded.

Five files. 100 rows. The process starts, reads the first file, and gets killed.

/ft/corpus.sh: line 35: 55 Killed python -m analytics.main validate --input ... --max-files 5

Not an error. Not a timeout. Killed — SIGKILL from the OOM killer. The Docker VM ran out of memory and the kernel reclaimed it by killing the process. On 100 rows.

The investigation — three wrong turns

First instinct: "the corpus is too big, we need to shard." So I sharded — process one dataset at a time instead of all 20 at once. Same Killed. Same OOM.

Second instinct: "we need fewer files." So I set --max-files 5. Still killed. Still OOM.

Third instinct: "we need even fewer files." --max-files 3. --max-files 2. --max-files 1. Still killed. One file is enough to OOM the VM.

At that point you stop turning dials and go look at the file.

What was in it

$ ls -lS data/traces/processed/Exgentic__agent-llm-traces-v2/ | head -5
-rw-r--r--  618475751  traces-0001.parquet   (590 MB)
-rw-r--r--  360507057  traces-0005.parquet   (344 MB)
-rw-r--r--  350227908  traces-0002.parquet   (334 MB)
-rw-r--r--  323410706  traces-0006.parquet   (308 MB)
-rw-r--r--  200731923  traces-0009.parquet   (191 MB)

590 MB. One file. How many rows in that 590 MB file?

import pyarrow.parquet as pq
t = pq.read_table("traces-0001.parquet", columns=["trace_id"])
print("rows:", t.num_rows)
# rows: 100

100 rows. 590 MB. 5.9 MB per row.

Each row carries a full conversation transcript as its payload. Not a summary, not a reference — the entire conversation text, embedded in the parquet row. 100 conversations, each averaging 5.9 MB of text. That's the nature of agent observability data: conversation traces are inherently large per row.

The size distribution across the dataset ranged from 300 KB to 590 MB — a 2,000× range. A file-count limit that works for the 300 KB files is useless for the 590 MB files.

Why --max-files didn't help

--max-files 5 limits the number of files the validator processes. It does not limit the size of each file. The first 5 files by name happen to include the two largest files in the dataset (590 MB and 344 MB). Even --max-files 1 would OOM on the 590 MB file.

Here's what happens inside the validator:

for pq_file in parquet_files:              # ← max_files limits this list
    table = pq.read_table(str(pq_file))    # ← loads the ENTIRE file into memory
    for row in table.to_pylist():          # ← converts to Python dicts (5-10x expansion)
        # ... build SpanNode ...

pq.read_table() loads the entire parquet file into memory as an Arrow table. Then .to_pylist() converts every row to a Python dict. Python dicts are memory-hungry — a 6 MB string in parquet becomes a 6 MB Python string, plus dict overhead, plus the SpanNode object, plus the attributes dict. The expansion factor is 5–10x.

So: 590 MB parquet → ~600 MB Arrow table → ~3–6 GB of Python objects. On a 7 GB VM with 8 other containers running, there's maybe 2–3 GB free. 3 GB of Python objects from one file is enough to OOM.

tip: The real memory bound is max_files × max_file_size, not max_files alone. Check your largest file, not just the file count. One giant file defeats any count you set.

The math is simple: when one file is 590 MB, max_files=1 gives you a 590 MB bound (→ 3 GB after expansion). When all files are 300 KB, max_files=5 gives you a 1.5 MB bound (→ 15 MB after expansion). The file-count limit is only useful when files are roughly uniform in size. When they vary by 2,000×, the limit is useless.

The fix: pre-split, don't stream

The textbook fix is to stream the parquet reader: use iter_batches() instead of read_table(). And that works for most cases. But the validator has a constraint: it groups spans by (trace_id, source_row_idx) within each file. A single trace's spans must be in memory together to build the span tree. With 100 rows per file and ~6 MB/row, even one row group is 600 MB. Streaming helps, but the grouping logic still needs one full row group in memory.

The simpler fix: pre-split the oversized files into single-row chunks. Each chunk is one row, ~6 MB. --max-files 5 now bounds memory to ~30 MB. No validator code changes needed.

def split_file(path, chunk_rows=1):
    pf = pq.ParquetFile(str(path))
    idx = 0
    for batch in pf.iter_batches(batch_size=chunk_rows):  # streaming read
        table = pa.Table.from_batches([batch], schema=pf.schema_arrow)
        out = path.parent / f"c{idx:05d}.parquet"
        pq.write_table(table, str(out))
        idx += 1
    path.unlink()  # remove the original giant file
    return idx

iter_batches(batch_size=1) reads one row at a time from the parquet file. It never loads the whole file into memory. Each row is written as a separate parquet file. The 590 MB file becomes 100 files of ~6 MB each.

The 13 oversized files (2.3 GB total) became 4,965 small files, all ≤ 50 MB. The corpus went from 4,974 files to 4,965 — the 13 originals were deleted and replaced by their chunks. --max-files 5 now bounds memory to ~250 MB. The validator completes, and FT-15c passes.

The gotcha inside the fix

First attempt: threshold 50 MB, chunk size 20 rows. The 590 MB file split into 5 chunks of 51, 75, 160, 163, and 169 MB. Four of the five chunks were still over 50 MB. I had to run the split script again on the chunks, which produced filenames like traces-0001-part0000-part0000-part0000-part0000.parquet — name too long for the filesystem.

The fix was chunk_rows=1 (one row per file) and a flat naming scheme (c00000.parquet, c00001.parquet, ...). One row per file guarantees each file is ≤ 6 MB. No re-splitting needed, no filename explosion.

Why this matters for agent observability

Agent traces are not like web request logs. A web request log is a few hundred bytes — method, path, status, latency. An agent trace is a full conversation transcript — every tool call, every LLM response, every planning step, every retry. A single agent run can produce megabytes of trace data. And when you store traces as parquet files (columnar, compressed, efficient for analytics), the per-row payload can be enormous.

Any dataset that records full conversation transcripts will have large per-row payloads. A file-count limit that works for the small files is useless for the giant ones. The lesson is not "always pre-split your files." The lesson is: know your file size distribution. If files vary by 2,000×, a file-count limit is not a memory bound. You need either a file-size limit, a streaming reader, or a pre-split step.

What I took away

File count ≠ file size. A file-count limit is only a memory bound when files are roughly uniform. When one file is 2,000× larger than the average, the limit is useless.

OOM on a small number of rows is a red flag for payload size, not row count. Check the largest row, not the row count.

pq.read_table() loads the whole file. Use iter_batches() for streaming reads — but check whether your downstream logic needs the full file in memory.

Pre-splitting is a legitimate fix. Sometimes the cheapest, safest change is to fix the data, not the code.

All of it shipped in agentwatch v0.1.0, with the split script checked in so the corpus is reproducible.

The useful check for any pipeline processing large-row payloads: what's the largest single file, and does the current memory bound survive it? max_files alone answers the wrong question when files vary by 2,000×.

References

• Release: agentwatch v0.1.0 on GitHub · on PyPI
• Suite: agentic-suite — the full set of agent-security tools
• Field test report (defect D-2, FT-15c)
• Split script: split-big-parquet.py
• Validator: validator.py