Chunk Size Is a Reliability Setting

A crash destroys the entire unit of work in flight. All of it. So the size of
that unit decides what a crash costs you. Throughput is the thing you were
thinking about. Blast radius is the thing that actually bites.

I learned this during a run on 2026-08-04.

What happened

I was running a fully offline document pipeline over 5,000 scanned PDFs. One
container per slice of the input. Four vCPU each, no network, read only
filesystem, no GPU. The pipeline splits its input into chunks and runs each
chunk in a fresh interpreter.

I set that chunk size to 1,000, which was the maximum.

Over the next four hours, three worker processes died with native faults. Not
clean exceptions. Signal level deaths: exit -5 and exit -11. They died at
roughly 50 percent, 50 percent, and 72 percent of their chunks.

Every one of those crashes threw away the whole chunk. About seven hours of
compute, gone, on a deadline.

How much work does one crash erase?

A fault lands at a random point in the unit. So on average you lose half the
unit each time.

At 1,000 cases per unit, a fault costs about 500 cases. At 250, the same fault
costs about 125. Cut the unit by four, cut the loss by four. That is the whole
idea, and it is boring, and it would have saved me most of the night.

The overhead of smaller units is real but tiny. You pay for a few more process
restarts. I measured the cost at a few percent. The insurance is total.

Why did parallelism make the failure worse?

My first instinct was speed. More containers, finish sooner.

That was wrong twice over. More containers means more processes that can die,
and it does nothing to reduce what each death costs. Worse, I sized them
against the host's specs instead of the container runtime's actual ceiling,
blew past the memory limit, and the OOM killer took out a healthy run that had
been going for 33 minutes.

Adding parallelism to long, uncheckpointed units is the worst of both.

What order should you use to make a long job recoverable?

1. Shorten the unit. Set it from your observed failure rate, not your throughput preference. If faults show up every two hours, a two hour unit is the wrong unit.
2. Make failure loud. I had an in process retry that silently restarted a crashed chunk with one worker instead of four. It hid the signal and turned a twelve minute problem into a four hour one. A fast, noisy exit is better.
3. Make resumption cheap. Durable checkpoint per completed unit, validated when you read it back, skipped on restart. Then a crash costs one unit and a container start.
4. Only then parallelise, bounded by the ceiling you actually measured.

After I made those changes, a full restart of the container runtime cost me
nothing but the one chunk in flight. Everything else resumed from disk.

How can you tell a machine fault from a code fault?

Some crashes are your bug. Some are the machine. Tell them apart before you
spend an hour tuning code that is fine.

It is probably the machine when:

• The same input succeeds on one run and dies on another.
• Two or more independent workers die identically on one host.
• The crash is memory corruption rather than an exception. Watch for SIGSEGV, SIGBUS, signal 139, or the one that gave it away for me: an unknown opcode error raised from inside a pure Python standard library module.

That last one is worth internalising. Python does not raise "unknown opcode" in
difflib because your code is wrong. It raises it because something wrote
garbage into the memory holding a compiled code object. No amount of
refactoring fixes that. Go test the hardware.

What did the repaired run measure?

For anyone sizing similar work: the pipeline ran at 3.2 seconds per PDF on four
vCPU, fully offline, no network calls and no GPU. Nothing in it phones home.
After moving to 250 case units, faults stopped costing me anything I could not
recover in about twelve minutes.

What should you change in your next long-running job?

Go look at your longest running job right now. Find how long one unit of work
takes between durable checkpoints. Then ask how long it has been since your
last unexplained failure.

If the unit is longer than the gap between failures, your unit is wrong, and
you are one crash away from finding out.

Accompanying prompt

What the prompt does: It turns a long-running local AI job into a checkpoint plan that limits the work lost when a worker dies.

Copy/paste this prompt:

Role:
You are reviewing a long-running local AI or offline document job.

Context:
Provide the input count, current chunk size, measured time per chunk,
checkpoint behavior, worker count, memory ceiling, and recent failure logs.

Task:
1. Estimate the work and time lost when one worker fails.
2. Recommend a chunk size tied to the observed failure interval.
3. List the durable checkpoint fields needed for safe resume.
4. Give one bounded parallelism test to run after recovery is cheap.

Output:
- Current blast radius.
- Recommended chunk and checkpoint plan.
- One next measurement.

Constraints:
- Keep measured values separate from estimates.
- Do not claim a hardware fault without repeated evidence.
- Do not increase parallelism before the resume path is tested.
<!-- blog-prompt-scope:2026-06-24 -->

Copy the block above.

Get the artifact-backed local AI lab notes by email: https://bmdpat.com/5090-reports

Related reading

• VRAM Fit Is Not Runtime Support
• My 5090 benchmark was missing the field I needed most

---

Get the local AI lab notes (benchmark rows, VRAM fit, quant choices, what runs on consumer GPUs), M-F only when there is something worth sending: https://bmdpat.com/newsletter?utm_source=blog_md&utm_medium=aeo&utm_campaign=chunk-size-is-a-reliability-setting-2026

---

Originally published on bmdpat.com. I run a one-person AI agent company and write about what actually works.

Want these in your inbox? Subscribe to the newsletter - no spam, unsubscribe anytime.