The memory ceiling: why moving large datasets through application memory doesn't scale

Hook

Many data integration systems start the same way..
Read records from a source, transform them in application memory, and push them to a destination. It works beautifully, until the dataset grows, then you meet two old enemies:

• CPU pressure
• out-of-memory failures.

The pattern that doesn't scale

The naive architecture loads and transforms data inside the integration service's own process: read every row into memory, transform it in place, buffer the full result, then send it on. Simple to write — until volume grows.

The problems compound as volume grows:

• Heap pressure: Per-record overhead in memory is often several times the record's size on disk, holding large intermediate collections per task pushes usage into GC thrashing, then OOM.
• CPU cost: per-record object allocation and serialization dominate once you reach tens of millions of rows.
• Hard ceiling: dataset size becomes bounded by a single node's memory, not by the cluster.

A failure near the end throws away all the buffered work, with no partial progress.

Why "just add memory" fails

Vertical scaling buys time, not a solution. Doubling heap doubles the dataset you can process once, but the architecture is still single-node-bounded and still O(dataset) in memory. The ceiling moves; it doesn't disappear.

None of this is a flaw in the language — it's what naturally happens when an architecture designed for "this fits in memory" is used on data that no longer does.

The Reframing

The fix isn't a bigger machine (that only moves the ceiling, not removes it); it's rethinking the shape of the pipeline itself — which is where the next article picks up: moving from "read-everything-then-process" to a streaming model that keeps memory flat regardless of dataset size.

That's the subject of the next two articles:

1. Push transformation down into a lakehouse engine.
2. Stream the result to the destination in bounded memory.

Takeaway

If memory usage grows with dataset size, you don't have a scalability problem you can tune away — you have an architecture that needs to stop carrying the whole dataset at once.