Hook
The single highest-leverage change we made was to stop pulling raw data into the application to transform it, and instead push the transformation down to the engine that already stores the data.
Before: application-side processing
Every row crosses the network and lands in the service heap before any filtering or projection happens. You pay I/O, allocation, and GC for data you may immediately discard.
After: pushdown processing
What changes:
• Filtering and projection happen in the engine, so only the rows and columns you actually need ever leave storage.
• Columnar format (e.g. Parquet-style) means efficient encoding, compression, and column pruning.
• The integration service becomes a thin transport layer, not a compute bottleneck. Its memory no longer scales with dataset size.
The engineering nuances that matter
This isn't a free swap. The realistic write-up includes the trade-offs that make it credible:
• Schema alignment: the exported file schema must line up with what the destination expects, including awkward type edge cases (e.g. legacy timestamp encodings) that need explicit handling on read.
• File partitioning for parallelism: capping per-file size produces more files, which gives downstream readers more units of work to parallelize across — a deliberate tuning knob, not an accident.
• Bounded read batches: columnar readers can pull a whole row group (hundreds of MB) in one logical read; without a record-count cap on batch accumulation, you reintroduce the very OOM you were trying to remove.
The result, stated honestly
• Memory per worker is now bounded by batch size, not dataset size.
• Throughput scales with worker count and file/partition count.
• The previous hard ceiling on dataset size is removed — within the limits of the lakehouse and destination, not "to infinity."
Takeaway
Move the compute to the data. When transformation runs in the columnar engine, your integration tier stops being the bottleneck and starts being plumbing — which is exactly what it should be.
Top comments (0)