DEV Community

Pirate Prentice
Pirate Prentice

Posted on

n8n Split in Batches Node: Process Large Datasets Without Crashing (Free JSON)

If you've ever tried to loop over 500 rows from Google Sheets or process a large API response in n8n, you've probably hit a wall: workflows slow down, timeouts fire, or your downstream node gets hammered with 500 simultaneous requests.

The Split in Batches node (also called the Loop Over Items node in newer n8n versions) solves this. It's the native way to process large datasets in controlled chunks — 10 items at a time, 50 at a time, whatever your downstream API can handle.

This guide covers how it works, when to use it, and a free workflow JSON you can import directly.


What the Split in Batches Node Does

The Split in Batches node takes a large list of items and divides it into smaller chunks, processing one chunk per loop iteration. It handles the loop state for you — you don't need to track indexes or write recursion logic.

Two outputs:

  • Loop — fires on each batch (connect your processing nodes here)
  • Done — fires once when all batches are processed (connect your cleanup/summary nodes here)

When to Use It

Scenario Why Split in Batches?
Sending emails to 500 contacts Avoids SMTP rate limits
Updating 1,000 CRM records Prevents API 429 errors
Processing rows from a big spreadsheet Controls memory usage
Calling an AI model per item Manages token concurrency
Downloading files one by one Avoids parallel connection limits

As a rule of thumb: if you're iterating over more than ~20 items that trigger external API calls, use Split in Batches.


How to Add It

  1. In your workflow canvas, click + to add a node
  2. Search for Split in Batches (or Loop Over Items)
  3. Set your Batch Size (start with 10 for API-heavy workflows, up to 100 for lightweight transforms)
  4. Connect the Loop output to your processing node
  5. Connect the Done output to whatever runs after all items are processed

The Core Pattern

[Data Source] → [Split in Batches] → (Loop) → [Process Item] → back to [Split in Batches]
                                   → (Done) → [Final Step]
Enter fullscreen mode Exit fullscreen mode

The key: the processing node's output feeds back into the Split in Batches node's input. n8n handles the iteration automatically — you just close the loop.


Batch Size: How to Choose

Batch Size Use When
1 Sequential processing required; each item depends on previous result
5–10 API calls with rate limits (OpenAI, Slack, external CRMs)
25–50 Moderate processing; light API calls
100+ Local transforms with no external calls

Tip: When in doubt, start at 10. You can always increase once you confirm the downstream system handles it without 429s.


Common Gotcha: The Loop Must Feed Back

The most common mistake is connecting the processing node's output to a new node instead of back to Split in Batches. If you don't close the loop, n8n processes only the first batch and stops.

Wrong:

[Split in Batches] → (Loop) → [HTTP Request] → [Set Node]
Enter fullscreen mode Exit fullscreen mode

Right:

[Split in Batches] → (Loop) → [HTTP Request] → [Set Node] → [Split in Batches]
Enter fullscreen mode Exit fullscreen mode

The arrow going back to Split in Batches is what makes the iteration work.


Adding a Wait Between Batches

If your API is strict about rate limits, you can add a Wait node inside the loop:

[Split in Batches] → (Loop) → [HTTP Request] → [Wait 1s] → [Split in Batches]
Enter fullscreen mode Exit fullscreen mode

Set the Wait node to a fixed interval (e.g., 1 second). This gives rate-limited APIs breathing room between batches.


Free Workflow JSON: Batch-Process Google Sheets Rows

This workflow reads rows from Google Sheets, processes them 10 at a time, and logs a summary when done. Import it directly into n8n.

{
  "name": "Batch Process Sheets Rows",
  "nodes": [
    {
      "parameters": {},
      "id": "a1b2c3d4-0001-0001-0001-000000000001",
      "name": "Manual Trigger",
      "type": "n8n-nodes-base.manualTrigger",
      "typeVersion": 1,
      "position": [240, 300]
    },
    {
      "parameters": {
        "resource": "sheet",
        "operation": "read",
        "documentId": { "__rl": true, "value": "YOUR_SHEET_ID", "mode": "id" },
        "sheetName": { "__rl": true, "value": "Sheet1", "mode": "name" }
      },
      "id": "a1b2c3d4-0002-0002-0002-000000000002",
      "name": "Google Sheets",
      "type": "n8n-nodes-base.googleSheets",
      "typeVersion": 4,
      "position": [460, 300]
    },
    {
      "parameters": {
        "batchSize": 10,
        "options": {}
      },
      "id": "a1b2c3d4-0003-0003-0003-000000000003",
      "name": "Split in Batches",
      "type": "n8n-nodes-base.splitInBatches",
      "typeVersion": 3,
      "position": [680, 300]
    },
    {
      "parameters": {
        "jsCode": "// Process each item\nfor (const item of $input.all()) {\n  item.json.processed = true;\n  item.json.processedAt = new Date().toISOString();\n}\nreturn $input.all();"
      },
      "id": "a1b2c3d4-0004-0004-0004-000000000004",
      "name": "Process Batch",
      "type": "n8n-nodes-base.code",
      "typeVersion": 2,
      "position": [900, 220]
    }
  ],
  "connections": {
    "Manual Trigger": { "main": [[{ "node": "Google Sheets", "type": "main", "index": 0 }]] },
    "Google Sheets": { "main": [[{ "node": "Split in Batches", "type": "main", "index": 0 }]] },
    "Split in Batches": { "main": [[{ "node": "Process Batch", "type": "main", "index": 0 }], []] },
    "Process Batch": { "main": [[{ "node": "Split in Batches", "type": "main", "index": 0 }]] }
  }
}
Enter fullscreen mode Exit fullscreen mode

To use: Replace YOUR_SHEET_ID with your actual Google Sheets ID and swap out the Code node logic for your actual processing.


Split in Batches vs. Other Approaches

Approach When to Use
Split in Batches Large datasets, rate-limited APIs, sequential processing
IF + SplitOut Simple branching, not iteration
Code node (loop) Complex custom logic that can't be expressed as a workflow
Execute Workflow Sub-workflow per item (more isolated but heavier overhead)

For most "process every item" use cases, Split in Batches is the right tool.


Quick Reference

  • Node name: Split in Batches (or Loop Over Items in n8n ≥ 1.x)
  • Key setting: Batch Size (default 10)
  • Output 1 (Loop): Current batch items — connect to processing nodes
  • Output 2 (Done): Fires once at end — connect to summary/cleanup
  • Critical: Processing output must loop back to this node's input
  • Rate limiting: Add a Wait node inside the loop

What's in the n8n Workflow Starter Pack?

The free JSON above is a simplified example. The n8n Workflow Starter Pack ($29) includes:

  • 10 production-ready workflows (lead capture, Stripe receipts, AI email drafts, Slack alerts, and more)
  • Batch-processing patterns with error handling baked in
  • Each workflow documented with setup steps and credential requirements
  • Lifetime access + free updates

Get the n8n Workflow Starter Pack ($29)


What batch size do you typically use? And have you hit the "loop doesn't loop" gotcha where you forgot to feed the output back? Drop it in the comments — it's one of those things that trips up almost everyone the first time.

Top comments (1)

Collapse
 
pirateprentice profile image
Pirate Prentice

What batch size do you run in production? I've found 10 is a safe default for most rate-limited APIs, but bump to 50+ for pure local transforms. Also curious if anyone's hit the classic gotcha: process output not looping back to Split in Batches, so only the first batch fires. Drop your setup below 👇