Data teams waste roughly 30% of their compute spend on recurring OPTIMIZE jobs that move data around just to keep queries from crawling. If you are still running ZORDER BY on every high-cardinality column in your schema, you are essentially paying a "legacy tax" to the cloud provider for the privilege of manually managing data distribution.
Why I chose this topic: I spent three weeks troubleshooting a 4-hour
OPTIMIZEjob in a healthcare pipeline that was failing due to OOM errors, only to realize the distribution was static while our query patterns had drifted. We need to stop treating data layout like a manual maintenance chore and start treating it like a dynamic resource.
The common industry advice—that Z-ordering is the gold standard for multi-dimensional filtering—is dangerously outdated.
Why the common approach falls short
The "Z-order everything" dogma assumes two things that are rarely true in modern production environments: that your query patterns are static, and that you have the time to babysit your partitioning strategy.
When you run OPTIMIZE table_name ZORDER BY (user_id, event_date), you are baking a specific search pattern into the physical layout of your Parquet files. If a business analyst decides to start filtering by region_id or device_type next month, your Z-order becomes a liability. Your query engine will perform a full table scan because the data is physically organized for a query that no one is running anymore.
Furthermore, Z-ordering is a global operation. To maintain the Z-curve, you have to rewrite the entire partition. If you have 50TB of data, you aren't just "optimizing"; you are thrashing your storage layer. Every time you trigger that command, you are incurring a massive I/O penalty, inflating your cloud storage bill, and potentially locking up your tables for hours.
Photo by Logan Voss on Unsplash
The case for Liquid Clustering
Liquid Clustering, introduced in Databricks Runtime 13.3 LTS, is a fundamental shift in how we handle data layout. Unlike Z-ordering, which is a rigid, global operation, Liquid Clustering is a native, incremental approach to data organization.
When you define a table with CLUSTER BY (user_id, event_date), you aren't creating a static map. You are giving the Delta Lake engine permission to reorganize the data as it lands. It is a declarative, not imperative, approach.
In a recent production migration of a healthcare claims dataset, we moved from a weekly Z-order job to Liquid Clustering. The Z-order job cost us ~$450 in compute per run and took 210 minutes. With Liquid Clustering, the background maintenance is handled by the engine, keeping the data performant without the massive, monolithic "all-at-once" rewrite. We saw a 40% reduction in our monthly compute bill because we stopped rewriting files that didn't need touching.
The syntax is cleaner, but the real magic is under the hood. You can change your clustering columns at any time without needing to rewrite the entire history of the table.
-- The old way: The "hope it stays relevant" approach
OPTIMIZE claims_data ZORDER BY (patient_id, claim_date);
-- The new way: The "let the engine handle it" approach
ALTER TABLE claims_data CLUSTER BY (patient_id, claim_date);
Once you run that ALTER statement, new data written to the table is automatically clustered. Old data is reorganized lazily. You stop fighting the engine and start letting it work for you.
Measuring the success of your layout
If you aren't measuring your data layout efficiency, you’re just guessing. Most engineers look at query duration and call it a day. That’s a vanity metric.
To actually measure if your clustering is working, you need to look at Data Skipping Stats. You can check this by running DESCRIBE DETAIL table_name. Look at the minValues and maxValues in your file metadata. If you are filtering by user_id and the min/max ranges in your files are massive, your clustering is failing.
Beyond that, use the EXPLAIN command in Spark. If you see DataSkipping enabled but your scan_files count is still high, it means your files are not pruned effectively. I prefer to pull these metrics into a simple dashboard using the delta.history() command. If I see the number of files read consistently exceeding the number of files that should contain the target data, I know it’s time to re-evaluate the clustering keys.
Don't just watch the clock. Watch the I/O. If your bytes_read is significantly higher than the size of the result set, your physical layout is actively sabotaging your costs.
The objections (and my answers)
The pushback I hear most often is: "Liquid Clustering is Databricks-specific; I want to keep my data portable."
Fair enough. If you are running an open-source Delta Lake stack on your own managed Kubernetes cluster without the Databricks optimization layer, Liquid Clustering isn't available to you. But be honest—if you are building a production-grade data platform in 2024, are you really doing it to avoid vendor lock-in, or are you doing it because you haven't calculated the cost of maintaining your own infrastructure? The complexity of manual Z-ordering at scale is a "hidden cost" that far outweighs the cost of the platform.
The other objection is "lack of control." Engineers hate giving up the manual ZORDER knob because they feel like they lose precision. But here’s the truth: your manual "precision" is just an optimization for the query you wrote yesterday. The data lifecycle is dynamic. Your tables should be, too. If you think you know better than an engine that can analyze petabytes of access patterns in real-time, you’re likely overestimating your intuition.
Lastly, some argue that Liquid Clustering is "magic" and therefore unpredictable. I’ll take "predictably good enough" over "unpredictably perfect but expensive to maintain" any day of the week.
Conclusion
Stop treating your data layout as a permanent architectural decision. It’s a transient optimization that should evolve with your business requirements.
If you are dealing with tables that grow daily and query patterns that change monthly, Z-ordering is a dead end. It forces you into a cycle of expensive, global rewrites that provide diminishing returns. Liquid Clustering represents the shift toward infrastructure that actually manages itself.
Start by auditing your OPTIMIZE jobs. If you see a job that takes more than an hour to run and touches the same columns every single time, swap it for a CLUSTER BY configuration. You’ll save money, you’ll stop babysitting your pipelines, and your query performance will be more resilient to the inevitable changes in how your users interact with your data.
The era of manual data maintenance is over. Stop paying for it.
Cover photo by Ian Talmacs on Unsplash.
Top comments (0)