The 5-Minute Wall: Splitting Long n8n Runs Before the Executor Kills Them

This is part of my build-in-public series on running LLM workflows in n8n in production. Everything here is from real templates I sell and support — the mistakes are mine, the fixes are tested.

The batch that died at row 312

A nightly batch job looked simple on paper: read a list of items, run each one through an LLM step, write the results back. It ran fine for weeks, until the list grew. One night the execution died partway through — no error in the business logic, no provider outage, just a run that stopped and never resumed.

Restarting it from the beginning was the obvious fix, and it was the wrong one. The first 300 items ran again, re-sent outputs that had already gone out, and then the run died again, a little further along this time. Each restart made the duplicates worse and the progress less certain. The problem was not a bad item. The problem was a run that tried to do unbounded work in a single execution.

That is the 5-minute wall, in spirit if not in exact number: every execution environment has a limit — a timeout, a memory ceiling, a provider rate window, a host that restarts. A long run will eventually meet one of them. When it does, everything after the failure point is lost, and everything before it is in an unknown state unless you planned for the interruption.

No workflow should do unbounded work in one execution

The rule I now design to is blunt: one execution, one bounded unit of work. A workflow that loops over "however many rows exist today" has no bound, and its runtime grows with the data. The day the data grows past the limit, the workflow does not degrade gracefully — it dies mid-batch.

Bounding the work means deciding, up front, what one execution is allowed to do: a fixed number of items, a fixed time budget, or one item. If the input is bigger than that, the workflow does not try harder. It splits.

Split first, then checkpoint

Splitting is a two-part change, and the second part matters more than the first.

First, chunk the input. Instead of one execution processing 400 items, a scheduler hands out chunks of, say, 25 or 50 items, and each chunk runs as its own execution — often as a sub-workflow call, so the per-chunk logic stays in one place. A chunk that fails fails alone: it does not take the other 375 items down with it, and it can be retried on its own without re-running everything.

Second, checkpoint after every chunk. After a chunk completes, the workflow writes a progress record: which chunk finished, how many items it covered, and where the next chunk starts. That record is the difference between "restart and hope" and "resume." On a restart, the scheduler reads the progress record and starts from the next unprocessed chunk, not from the beginning. Without the checkpoint, chunking only makes the failures smaller — it does not make them recoverable.

The checkpoint store does not need to be fancy. A row in a database, a sheet, a small key-value record — anything that survives the death of the execution that wrote it. What it must not be is execution-local state: variables that live only inside the run are gone the moment the run dies, which is exactly when you need them.

Give each chunk its own failure boundary

Chunking also fixes a failure-isolation problem that a single long loop hides. In one long loop, a failure anywhere aborts everything after it, and the error handling has to decide — mid-loop, mid-state — what counts as done. With chunks, the boundary is clean: a chunk either completes and checkpoints, or it does not, and the next run picks up after the last completed chunk.

That is a different job from handling a bad row inside a loop, which I covered in the partial-failure piece: that was about one bad item not killing its neighbours. This is about time, not bad data. A chunk can contain only good items and still die, because the run simply ran out of wall-clock. The fix for a time failure is structural — smaller runs, recorded progress — not better error handling inside the same oversized run.

Two practical details make chunking work in n8n. Keep a short pause between chunks when the downstream provider has rate limits; a chunked job that fires all its chunks at once just moves the rate-limit failure from inside one run to across many of them. And make the chunk size a configuration value, not a hardcoded number — the right size changes as the per-item cost changes, especially when an LLM step is involved, and you will want to tune it without editing the workflow.

The resume test

Testing this is not a happy-path run. Kill a run mid-batch — stop it, let it time out, or simulate the failure — and then restart it, asserting three things: no completed chunk runs again, no chunk is skipped, and the final output count equals the input count exactly. If the restart re-sends outputs, your checkpoint is recording intent instead of completion — it is written before the work finishes, not after. If items go missing, a chunk is marking itself done before its side effects land.

The test I actually run is boring and specific: a batch of known size, a forced stop after a known chunk, a restart, and a count of outputs at the end. Anything less than an exact match means the resume logic is a story, not a mechanism.

The principle

A long run is a bet that nothing will interrupt it, and the longer the run, the worse the bet. Split the work into bounded chunks, record progress after each chunk in a store that outlives the execution, and let a restart resume from the last completed chunk instead of from the beginning. The workflow gets slower in a trivial way — a few extra executions, a few extra writes — and in return it stops turning one interruption into a pile of duplicates and a batch nobody can trust. When a run dies, and it will, the question should never be "what did we already do?" The checkpoint should already have answered it.

---

I'm building AI Automation Lab — production-ready n8n workflow templates. The free AI Hook Generator shows how I structure real workflows; my open-source templates live at github.com/zhp910318/n8n-ai-automation-lab.