DEV Community

Cover image for Spark Ingest: How Do You Load Large Tables Without Driver OOM?
Sai Prudhvi Neelakantam
Sai Prudhvi Neelakantam

Posted on Originally published at geekindata.com AI-assisted

Spark Ingest: How Do You Load Large Tables Without Driver OOM?

I used AI tools to help draft and edit this post. I checked the facts and the wording, and the original lives at geekindata.com. The diagrams are on that page.

Orchestration, not a bigger VM

Key takeaways

  • Ban whole-entity driver InsertAll above documented row or byte thresholds.
  • Make batch size and concurrency explicit, tunable, and documented per entity.
  • Log per-batch id, rows, duration, and success or fail as structured logs.
  • Support entity and key-range filters so reruns stay finite.
  • Keep rebuild and incremental as separate modes with separate entry points.
  • Ship the pattern to prod only with batch-level observability and a runbook.

Whole-table InsertAll on one driver is a known failure mode for large tables. The notebook looks simple. The driver dies. Someone asks for a bigger cluster. The next reload dies later, with a larger bill.

Treat bulk reload as orchestration: batched parallel page ingest, progress monitoring, and entity filters for operable reruns. Separate one-time rebuild paths from incremental watermarks. Publish the pattern to prod only after batch-level observability exists. OOM here is a design smell in notebook API usage, not proof that "Fabric is flaky."

Named model: batch → progress → filter → observe

  • Batch: page or key-range writes with explicit size and concurrency; no whole-entity driver collect above threshold.
  • Progress: structured per-batch logs (id, rows, duration, success or fail) so "is it stuck?" becomes a number.
  • Filter: entity, partition, or key-range recovery without replaying the whole load.
  • Observe: batch-level metrics and a runbook are the ship gate before prod.
  • Contract: throughput for large tables comes from controlled batches and visibility, not from hoping the driver can materialize everything at once.

Named model with four stages: batch page writes, emit progress, filter reruns, and observe as the ship gate

Figure 1. Batch, progress, filter, observe. Bigger VMs are optional. Clear batch boundaries are not.

Context and constraint

A common pattern: full reloads work for small tables and collapse on the big ones. The API and Spark were fine. The driver was asked to hold a universe. Once we batched with progress and filters, reloads became dull operations instead of weekend heroes.

Parallelism without batch boundaries still OOMs; it just fails faster. Progress monitoring turns "is it stuck?" into a number. Entity filters let you rerun the failed slice without replaying the whole load. Incremental watermarks are not a rebuild strategy; keep the paths separate. Observability is a ship gate for the pattern, not a nice-to-have dashboard later.

Figure 2 is the Learn framing for copy work inside a pipeline. Whole-table InsertAll on one driver is an orchestration smell. Batch ingest belongs in an operable pipeline.

Adding a Copy data activity into a Microsoft Fabric pipeline

Figure 2. Whole-table InsertAll on one driver is an orchestration smell. Batch ingest belongs in an operable pipeline. Source: Microsoft Learn: copy data activity.

Figure 3 is the Learn framing for promoting a scheduled run. Batch with progress and entity filters, then promote the pattern like any other scheduled pipeline.

Scheduling a pipeline run in Microsoft Fabric

Figure 3. Batch with progress and entity filters, then promote the pattern like any other scheduled pipeline. Source: Microsoft Learn: pipeline overview.

Design

1. Ban whole-entity driver collects for large tables

If the pattern requires the driver to assemble the full entity before write, it will fail at scale. Design for page or key-range batches that write incrementally. Simplicity that only works on small tables is a trap. Figure 4 is the ownership split that keeps memory thin.

Anti-pattern whole-entity InsertAll leading to driver OOM versus operable page A B C writes with capped concurrency

Figure 4. Same API. Same Spark. Different ownership of memory. Ban whole-entity driver collects above documented thresholds.

2. Batch with explicit size and concurrency knobs

Choose batch sizes you can tune per entity. Cap concurrency so you do not DDoS the source or your own capacity. Document defaults and the dials operators may turn. Magic numbers buried in a cell become folklore under stress.

3. Emit progress per batch

Log batch id, row counts, duration, and success or fail. Operators need a trail when a long reload stops partway. Progress also feeds SLAs and capacity planning. Silent loops are not enterprise ingestion.

4. Entity filters for operable reruns

Support rerunning one partition, one entity, or one key range without truncating everything. Full restart as the only recovery path guarantees fatigue and shortcuts. Filters turn incidents into finite work.

5. Separate rebuild from incremental

Watermark-based incremental paths optimize for steady state. Rebuild paths optimize for correctness after schema or logic changes. Mixing them produces half-applied states that are hard to explain. Name the modes in the notebook and in the pipeline. Figure 5 is the split plus the ship gate.

Rebuild path, incremental path, and a ship gate of batch-level observability before production

Figure 5. Two modes. Two entry points. One ship gate: batch-level observability. Mixing modes produces half-applied states that are hard to explain.

6. Require batch-level observability before prod

Ship to production when you can answer which batch failed and why, not when a happy-path demo finished once. Metrics and structured logs are part of the ingest contract. Without them, every OOM becomes a platform complaint.

What good batch logging looks like

Each batch should leave a row or log line you could build a small table from: entity, filter, batch key, attempt, rows in, rows written, duration, error class. Aggregate for a run summary. On failure, operators replay from the failed batch key, not from zero, unless integrity requires a clean rebuild. If integrity requires clean rebuild, say so in the runbook. Ambiguous recovery is how two engineers choose opposite strategies at 2 a.m.

In a demo batched load of five batches, batch_03 failed on both attempts with a simulated schema mismatch. The other four loaded, and the run summary says where to replay from instead of leaving it to whoever is on call (Figure 6).

Per-batch log table from a demo batched load in a Fabric notebook: batch_01, batch_02, batch_04 and batch_05 loaded with rows in equal to rows written, batch_03 failed on attempts 1 and 2 with zero rows written and a SchemaMismatch error class, followed by the summary line Batches loaded: 4 of 5. Replay from: batch_03

Figure 6. One row per batch attempt: entity, batch key, attempt, rows in, rows written, duration and error class. The summary names the failed batch key, so the rerun replays batch_03, not the whole load.

Source courtesy is part of reliability

Source APIs have throttling realities. Batch design must respect them: backoff, concurrency caps, and clear error classes for throttle versus hard fail. Treating throttle as a generic Spark error trains people to enlarge clusters. Treating it as a first-class signal trains people to tune the orchestrator. Put the courtesy rules next to the batch knobs so "go faster" has a documented ceiling.

Thresholds belong in the ADR, not in memory

Define approximate row or byte thresholds where driver-collect is forbidden. Below the line, simple paths may remain for tiny entities. Above the line, batched orchestration is mandatory. Update thresholds when you learn; do not pretend every entity is the same size forever. The point of a number is to end the argument mid-incident about whether "this one" qualifies as large.

Common mistakes

Bigger VM as architecture.
Cost up, failure delayed.

Unbounded parallelism.
Source throttling and cluster chaos.

No progress.
Humans cancel jobs that were still working (or babysit forever).

Rerun-all-only recovery.
Engineers skip reloads; drift grows.

Incremental pretending to be rebuild.
Missing history, unclear watermarks.

Prod pattern without batch metrics.
Support cannot localize failure.

Use as-is or adapt

Use as-is

  • Large-table full loads use batched writes; driver-wide InsertAll is forbidden above documented thresholds.
  • Batch size and concurrency are explicit, tunable, and documented per entity.
  • Progress and per-batch outcomes are logged as structured logs.
  • Rerun filters exist for entity and key-range recovery.
  • Rebuild and incremental are separate modes with separate entry points.
  • Pattern reaches prod only with batch-level observability and a runbook.

Adapt before you use

Batch orchestration adds code and knobs. Driver-collect simplicity fails at large-table scale. Lower concurrency is slower and safer; tune with metrics, not pride. Separate rebuild paths duplicate some logic; duplicated modes beat corrupted modes. Observability costs a day or two up front and saves weeks of incident archaeology.

Try it yourself

The batched ingest replay demo in my public tech-demos repo loads five batches in a Fabric notebook, logs every attempt, and ends with 4 of 5 loaded and batch_03 to replay, so you can see the run log and the replay list for yourself.

Primary references: Fabric Spark job monitoring and notebook guidance on Microsoft Learn, plus source-system API throttling docs for your source connectors. Pair them with a runbook that names rebuild versus incremental. Start from copy data activity and pipeline overview.

Batch the work. Show progress. Filter reruns. Keep rebuild honest. Observe batches before you call the pattern production. Bigger VMs are optional. Clear batch boundaries are not.

Top comments (0)