DEV Community

Itay Waisman
Itay Waisman

Posted on

LTAP: Unifying Storage Isn't the Same as Unifying Resource Utilization

The reaction to LTAP has mostly been about correctness: what happens to a transaction that commits in Postgres but hasn't finished converting into Delta format yet, and whether "immediately queryable" has an actual bound. Those are fair questions. But there's a second problem sitting underneath the first one that almost nobody in this conversation is asking about, and it's the one that shows up on your cloud bill, not in your consistency model.

One disclosure: I'm from Regatta, where we build RegattaDB around the same problem LTAP is addressing. What follows is a technical argument about resource utilization, not a claim about correctness, which the previous piece already covered.

The Question Storage Unification Doesn't Answer

Say you've solved the data-copy problem. Transactional and analytical workloads now read from the same physical bytes. Great. Now run them concurrently on the same cluster and ask a different question: what's actually happening to your buffer cache?

In almost every database architecture, including the one underneath Lakebase, each node manages its own buffer pool as a private, per-node resource. When a large analytical scan runs, it pulls cold data through that node's cache, and in doing so it evicts whatever transactional working set was sitting there. When the transactional workload picks back up, it has to re-warm the cache the analytical query just flushed. Unifying the storage layer doesn't change this. Postgres's buffer cache and Spark/Photon's execution memory are still two separate resource pools contending for the same physical RAM underneath, even when they're pointed at the same Delta files.

This is why conventional database deployments, even well-tuned ones, typically sustain only 20 to 40 percent hardware utilization. Not because the hardware is idle. Because most of it is reserved as headroom against exactly this kind of cache-eviction storm, the moment a heavy analytical query collides with transactional traffic.

What Sharing Actually Means

The fix isn't giving OLTP and OLAP the same storage. It's giving them the same buffer, cluster-wide, so that a scan on one node doesn't force cold reads on another.

RegattaDB shares peak buffer capacity across the entire cluster instead of walling it off per node. A node running a heavy scan can draw on buffer capacity sitting idle elsewhere in the cluster rather than evicting the transactional cache it's sitting next to. The result is sustained utilization around 85 percent, against the 25 percent baseline conventional architectures need as safety margin. In practical terms, that's the difference between needing 40 servers to safely run a mixed workload and running the same workload on 10.

That's not a benchmark trick. It's a direct consequence of not treating memory as a per-node private resource in a workload that was never going to stay confined to one node's access pattern.

Why This Gets Missed

It gets missed because "unified storage" is a diagram you can put on a slide, and "sustained cluster-wide buffer utilization under concurrent mixed load" is not. The first is a satisfying story about where bytes physically sit. The second requires someone to actually run both workloads at once, for long enough, and measure what the hardware was doing the whole time.

Nobody publishes that number by accident. It's genuinely harder to get right, and it's the number that determines whether "unified" data architecture actually saves you money, or just saves you an ETL job while your infrastructure spend stays exactly where it was.

The Question Worth Asking Instead

Before the storage diagram, ask the operations question: under a realistic mixed workload, at steady state, what percentage of provisioned compute and memory is actually doing useful work? If the honest answer is somewhere in the 20 to 40 percent range, unifying where the data lives hasn't touched the part of the bill that actually matters.

That number is measurable, publishable, and, as far as I can tell, still missing from this conversation.

Top comments (1)

Some comments may only be visible to logged-in visitors. Sign in to view all comments.