DEV Community: joni sar

Why Apache Iceberg Needs a Table Management Platform

joni sar — Mon, 28 Sep 2026 11:28:05 +0000

Every Iceberg table accumulates structural debt from its first commit. Files fragment. Snapshots pile up. Manifests bloat. Orphan files sit on object storage that no snapshot references and no query will ever read, but your bill counts every byte. Query performance degrades gradually, then suddenly — and because the architecture is deliberately decoupled, no single engine or catalog owns the operational picture. Nobody is responsible for the health of the lake as a whole.

The format is not the problem. Iceberg ships the primitives: rewrite_data_files, expire_snapshots, remove_orphan_files, rewrite_manifests, compute_table_stats. These are fsck and defrag, not an operating system. They require external intelligence to decide which tables need attention right now, what parameters to use, in what order to run, how to avoid conflicts with the streaming writer committing every 30 seconds, and how to adapt when access patterns shift from one quarter to the next. The missing piece is not a tool. It is a layer.

At 20 tables, you solve that with a script. At 200, the scripts are a full-time job. At 2,000, the scripts need their own team — and that team is no longer building data pipelines.

A lakehouse control plane such as LakeOps exists to close this gap. It sits above the stack you already have — your catalogs (Glue, Polaris, Nessie, Gravitino, Lakekeeper, S3 Tables), your Iceberg tables on object storage, and your query engines — and supplies the operational loop the open components do not provide as one system: sense every table's health through continuous metadata analysis and cross-engine telemetry, decide what maintenance and physical layout each one needs, execute it in the correct dependency order on a dedicated engine, and feed the result back into the next decision.

This article covers why that layer is necessary, what it actually contains, what breaks without it, and how the current landscape of tools, platforms, and managed services fills or fails to fill the gap. For deep dives on individual concerns, see the companion articles on table maintenance, observability, and compaction strategies.

What Iceberg Ships and What It Does Not

It is worth being precise about what Iceberg's own maintenance surface covers, because the gap is both narrower and wider than most people assume.

The maintenance primitives (Iceberg 1.11.0)

Procedure	What it does
`rewrite_data_files`	Merges small files, applies sort or Z-order, resolves position deletes. 15+ tuning parameters including `target-file-size-bytes`, `min-input-files`, `partial-progress.enabled`, and `delete-file-threshold`. Branch support added in 1.11.0.
`rewrite_manifests`	Consolidates manifest files for faster query planning. 1.11.0 added `sort_by` to cluster manifest entries by partition transform rather than accepting default spec order.
`rewrite_position_delete_files`	Compacts position delete files and removes dangling deletes.
`expire_snapshots`	Removes old snapshots and their exclusively-referenced data files.
`remove_orphan_files`	Deletes files on storage not referenced by any live snapshot. Safety threshold: 3-day default minimum age.
`compute_table_stats`	Generates Puffin NDV sketches for cost-based optimization.
`compute_partition_stats`	Calculates partition-level statistics. Incremental from the last stats snapshot.

Trino's Iceberg connector adds optimize, optimize_manifests, expire_snapshots, and remove_orphan_files as ALTER TABLE ... EXECUTE commands, with hard minimum-retention floors that prevent accidental deletion. Flink embeds maintenance directly in streaming jobs through IcebergSink, with configurable triggers on commit count, file count, file size, and delete-file accumulation.

What none of them provide

Every one of these is a tool, not a system. The gap is not in execution capability. It is in the intelligence that decides when and how to use them:

No scheduling. No built-in interval, event, or condition-based trigger.
No health detection. No assessment of whether a table needs maintenance, how urgently, or what kind.
No cross-engine telemetry. No awareness of query patterns from Trino, Spark, Athena, DuckDB, Snowflake, or Flink. Sort-order selection is guesswork.
No sequencing. No enforced dependency order. Compacting before expiring snapshots wastes compute on files about to be garbage-collected. Rewriting manifests before compaction produces metadata that goes stale immediately.
No conflict coordination. OCC retries exist, but nothing coordinates maintenance with concurrent writers to avoid the conflict in the first place.
No fleet management. No multi-table prioritization, no resource allocation across a lake of hundreds or thousands of tables.
No policy model. No hierarchical defaults, no inheritance, no configuration that scales beyond one table at a time.

Apache Polaris — now an Apache Top-Level Project — illustrates the gap precisely. Since version 1.0, Polaris defines maintenance policy types: system.data-compaction, system.metadata-compaction, system.orphan-file-removal, system.snapshot-expiry, attachable at catalog, namespace, or table level with inheritance. But — and this is the critical detail most articles miss — Polaris does not execute them. The catalog stores the policy. Who runs it, when, and how is explicitly left to the user. This is a deliberate architectural choice and the right one for a catalog, but the gap between policy declaration and policy enforcement is exactly where a table management platform lives.

What Actually Breaks at Scale

These are not hypothetical concerns. Each is a pattern that emerges between 200 and 2,000 tables — not from any single tool failing, but from the gaps between tools.

Small files make queries exponentially worse

A table with the same data stored in 500,000 × 5 MB files performs 10–50× worse on scan planning, costs 10–100× more in S3 GET requests, and takes dramatically longer to maintain than 5,000 × 500 MB files. In lab benchmarks, compacting 2,500 small files into properly sized ones improved query time from 3.6 seconds to 0.4 seconds for aggregations (8.4× improvement) and from 1.2 seconds to 96 milliseconds for point lookups (12.7× improvement). A 1 TB table costs roughly $0.25 to scan when well laid out and $5.00 when fragmented — a 20× difference in S3 API spend alone, before you count the compute time.

Diagnose it:

SELECT partition,
       COUNT(*) AS file_count,
       AVG(file_size_in_bytes) / (1024*1024) AS avg_mb,
       SUM(CASE WHEN file_size_in_bytes < 8388608 THEN 1 ELSE 0 END) AS small_files,
       SUM(file_size_in_bytes) / (1024*1024*1024) AS total_gb
FROM prod.db.events.files
GROUP BY partition
HAVING small_files > 100
ORDER BY small_files DESC;

If a significant fraction of partitions show hundreds of files under 8 MB, every query against that table is paying a planning and API tax that compaction eliminates immediately. Streaming tables with per-minute commits are the most common source, but even daily batch tables fragment when they receive many small appends throughout the day.

Manifest bloat inflates planning time

Planning cost scales with the number of manifests, not the number of partitions. A table with 50,000 partitions and 20 manifests plans quickly. A table with 50 partitions and 20,000 manifests — which is what a per-minute streaming writer produces in two weeks — plans slowly, because the planner must open and parse every manifest in the snapshot before execution begins. Five hundred manifests tracking 200,000 data files requires hundreds of object-store GET requests before any data is read.

Iceberg 1.11.0 added server-side scan planning in the REST catalog spec, which moves manifest parsing from the engine to the catalog, and the sort_by parameter for rewrite_manifests, which clusters manifest entries by partition transform. Both help. Neither removes the need to keep the manifest tree healthy — they make a healthy tree faster and a bloated tree less catastrophic, but the underlying growth still requires periodic rewriting.

SELECT COUNT(*) AS manifest_count,
       SUM(added_data_files_count + existing_data_files_count) AS total_entries
FROM prod.db.events.manifests;

When manifest_count exceeds 500 and is growing, planning latency is measurably degraded.

Orphan files silently consume 25–40% of storage

Industry analyses consistently find that 25–40% of object-storage spend on typical production lakes covers bytes no query will ever read again: orphaned files from failed writes, superseded files from compaction, metadata from expired snapshots that nobody cleaned up. One lab measurement on a modest table found 127 MB of orphan metadata against 48 MB of actual data — a 2.8× metadata-to-data ratio. At Databricks, Predictive Optimization has vacuumed exabytes of unreferenced data, saving tens of millions of dollars — a good indication of how much accumulates when expiry does not keep pace with writes.

Delete files impose read amplification

Position delete files in Iceberg v2 require the engine to join each data file with its deletes at read time. Published benchmarks show that once even 3% of rows go through delete-and-insert cycles, query performance drops by roughly 1.5× purely from the file explosion, and the cost is dominated by the fixed overhead of having any delete files at all, not by the number of deleted rows.

Iceberg v3 introduced deletion vectors (DVs), which replace position delete files with a per-data-file bitmap. DVs reduce the read-time join cost dramatically but do not eliminate the storage and statistics impact of keeping deleted rows in data files. The rows stay on storage, consuming space and widening column min/max bounds (which degrades pruning), until compaction rewrites the affected files. DVs make the read penalty cheaper; compaction is still the mechanism that removes deleted data.

SELECT content,
       COUNT(*) AS file_count,
       SUM(file_size_in_bytes) / (1024*1024) AS total_mb
FROM prod.db.events.files
WHERE content IN (1, 2)
GROUP BY content;

Content 1 = position deletes, content 2 = equality deletes. Any non-trivial count against high-frequency update tables means compaction with delete resolution is overdue.

Snapshots compound everything

A streaming table committing every minute accumulates over 4,000 snapshots per month. Each snapshot extends the metadata chain that the planner walks. Unreferenced data files pile up behind expired snapshots that nobody removed. A table with 90 days of snapshots and no expiry can have 10× more metadata files than data files — and metadata reads are per-query overhead that no amount of caching eliminates for cold starts.

Maintenance storms degrade the lake they exist to fix

IOMETE's engineering team documented a subtler failure mode: at hundreds of tables, the maintenance workload itself degrades the lake. A cron job that compacts 500 tables at midnight generates a burst of metadata commits. Conflict rates spike, retries multiply, and the catalog becomes a bottleneck for production readers and writers that have nothing to do with maintenance.

The irony is structural: maintenance exists to improve query performance, but uncoordinated maintenance at scale degrades it. Production writers hit CommitFailedException because a compaction job holds a stale base snapshot. The compaction job retries, conflicts again, and eventually succeeds — but the writer's commit also retried, potentially doubling data. Or the compaction job succeeds, but the writer committed to a partition that compaction just rewrote, and now the compacted files are immediately fragmented again. Every CommitFailedException represents wasted compute, and at scale the waste is dominated by maintenance-vs-maintenance and maintenance-vs-writer conflicts, not by the actual compaction work.

Physical Layout: Why Binpack Is Not Enough

This distinction is where most Iceberg maintenance articles lose the plot, and it matters enough to state plainly: compaction fixes file count. It does not fix data skipping. If you bin-pack 5,000 small files into 50 correctly sized ones, you reduced planning overhead and API costs. But if every resulting file contains values from the full range of every column, no engine can skip any of them based on a WHERE clause. You have fewer files, but you still scan all of them.

The Iceberg scan planner discards files when column min/max bounds prove a file cannot contain matching rows. That only works when rows with similar values live in the same files. Getting them there requires clustering — and clustering is a separate operation from compaction, with different mechanics, costs, and decisions.

The pruning stack

Layer	Mechanism	What it decides
Partition pruning	Partition spec + hidden transforms	Which groups of files can be ignored entirely
File pruning (clustering / sort order)	Manifest-level and file-level column min/max bounds	Which files within a partition can be skipped
Z-order / multi-dimensional clustering	Space-filling curve bit interleaving	Balanced pruning across 2–4 equally important filter columns
Row-group pruning	Parquet footer column statistics	Which row groups within a file can be skipped
File size	Target 128–512 MB for analytical tables	Planning overhead vs pruning granularity

Partitioning decides which groups of files exist. Clustering decides whether the files inside those groups are skippable. These are different operations at different layers. Changing a partition spec is a metadata-only operation. Changing a sort order requires rewriting every data file in scope.

Sort order vs Z-order

A linear sort on customer_id produces files where each file covers a narrow range of customer IDs. Queries filtering on customer_id skip most files. Queries filtering on event_type skip nothing, because event_type values are scattered uniformly across the sorted files.

Z-order interleaves bits from multiple columns so that rows close in multi-dimensional space end up in the same files. This gives balanced pruning across 2–4 filter columns, at the cost of slightly worse pruning on any single column compared to a linear sort on that column. Past 4 columns, Z-order dilutes rapidly, and past 6 it provides little benefit over random distribution.

The decision between them depends on query patterns:

Workload	Layout
Queries overwhelmingly filter on one column	Linear sort on that column
Queries filter on 2–4 columns with roughly equal frequency	Z-order on those columns
Queries filter on many different columns unpredictably	Z-order on the 3–4 most common; accept limited pruning on the rest
High-cardinality point lookups on one key	Linear sort + Parquet bloom filter on that key

Why this requires telemetry

The right layout depends on how queries actually access the table — and that changes. A table sorted by order_date serves dashboard queries well. When a new fraud-detection pipeline starts filtering on merchant_id, the sort order becomes irrelevant for that workload. Without cross-engine telemetry aggregating filter, join, and group-by frequency across Trino, Spark, Athena, Snowflake, and Flink, sort-order decisions are intuition that nobody revisits.

This is the specific argument for a platform-level solution rather than per-table tooling. The optimal layout is not a property of the table. It is a property of the relationship between the table and every engine that reads it.

What "Table Management" Actually Means

The term gets used loosely. For Iceberg tables in production, it decomposes into five operational concerns that must work as a unified system rather than as independent tools. Each concern feeds the others; assembling them from disconnected parts is technically possible and operationally fragile.

1. Observability

You cannot maintain what you cannot measure. Iceberg has no native health dashboards, no cross-engine telemetry aggregation, no alerting. The metadata is there — snapshot history, manifest lists, file-level statistics — but extracting operational intelligence from it means running Spark or Trino SQL against table.snapshots, table.files, and table.manifests, one table at a time.

A management platform turns raw Iceberg metadata into continuous health classification. Every table is scored — Healthy, Warning, or Critical — based on signals that directly affect query performance: file count and size distribution, manifest depth, snapshot accumulation, delete-file ratio, partition skew, and sort-order alignment with actual query patterns.

The word continuous is the key. A quarterly audit catches problems months after they started costing money. Continuous scoring surfaces a streaming table's small-file explosion within hours, before downstream dashboards slow to a crawl.

Cross-engine telemetry is the other half. When Trino, Spark, and Snowflake all read the same table, understanding which columns each engine filters, joins, and groups on — aggregated into a single access profile per table — is the foundation for layout optimization. Without it, sort-key selection is intuition that nobody revisits.

2. Autonomous maintenance

The maintenance lifecycle for an Iceberg table is a dependency chain, not a checklist:

Expire snapshots — removes metadata references to old snapshots. This must go first because it shrinks the file set every subsequent step has to process.
Remove orphan files — deletes data files on storage that no snapshot references, including those just freed by expiry.
Compact data files — merges small files into optimally sized ones, applying sort or Z-order clustering.
Rewrite manifests — consolidates manifest files so the metadata tree indexes the layout that now exists.
Refresh statistics — recomputes Puffin NDV sketches so optimizer estimates match the current file set.

The ordering matters. Compacting before expiring wastes compute — you are rewriting files that expiry would have garbage-collected. Rewriting manifests before compaction produces metadata that goes stale as soon as compaction changes the underlying files. Refreshing statistics before compaction produces estimates that describe a layout that no longer exists.

Most teams start with Airflow DAGs or cron jobs calling Spark procedures. The architectural limitations emerge as the fleet grows:

Fixed schedules cannot adapt to table state. A table committing every 30 seconds needs compaction multiple times an hour. A weekly batch table needs it once. A uniform daily schedule over-compacts quiet tables and under-compacts hot ones.
Cross-DAG coordination is absent. Ensuring expiry completes before compaction starts, across 500 tables, requires manual orchestration Airflow was not designed for.
Commit conflicts accumulate. Compaction jobs running alongside production writers compete for the metadata pointer via OCC. At scale, conflict rates become the dominant source of failed maintenance, wasted compute, and on-call pages.
Linear configuration burden. Each new table needs its own sort columns, target file size, retention window, and schedule. At 2,000 tables, maintaining the configurations is a full-time job.

3. Intelligent compaction and layout

Standard bin-pack compaction merges small files into bigger ones. As established above, it does not improve data skipping. Query-aware compaction goes further: it builds a per-table access profile from all connected engines' query logs, identifies the filter, join, and grouping columns that drive data access, and physically re-sorts data files to match during compaction. When data is sorted by the columns queries actually hit, Parquet row-group min/max statistics become tight, and engines skip entire file groups without reading them. The difference is often an order of magnitude: a table that scanned 4 TB scans 400 GB after sort compaction, because min/max pruning eliminates 90% of files.

Layout simulation is the related capability: testing a proposed sort strategy against real query patterns on an Iceberg branch before committing it to production. Bad sort decisions are expensive to reverse since you are rewriting all the data files in scope, so the ability to evaluate projected scan reduction before a full rewrite is not a nice-to-have. It is the mechanism that makes automated sort-order selection safe rather than reckless.

4. Multi-engine coordination

Production Iceberg lakehouses rarely run a single engine. Trino handles interactive analytics. Spark runs ETL. DuckDB serves quick lookups. Snowflake powers BI. Athena covers ad-hoc. Flink processes streaming ingestion. Each has different cost characteristics, latency profiles, and strengths. Without coordination, every query goes to one engine regardless of whether it is the right one, and maintenance decisions cannot account for how all engines actually access the data.

A management platform provides a routing layer that dispatches each query to the optimal engine based on query shape, cost model, and latency target. This creates a reinforcing loop with compaction: a well-sorted table enables cheaper engines to serve queries that a fragmented table would require an expensive engine for. Routing telemetry feeds back into sort-order selection, so both improve simultaneously.

5. Policy and governance

At 50 tables, you configure each one individually. At 500, you need declarative policies that cascade hierarchically — organization, catalog, namespace, table — with inheritance and overrides. Define compaction targets, snapshot retention, orphan cleanup thresholds, and sort strategies once at the namespace level. New tables inherit the correct configuration automatically. Override individual tables when needed.

Governance also means audit trails. Every maintenance operation — what ran, when, duration, files before and after, bytes reclaimed, the signal that triggered it — needs to be logged in a structured, queryable format. For compliance environments the trail is evidence; without it, you are proving GDPR erasure completion from Airflow logs and Spark driver output.

Why these five must be unified

Each concern has standalone tools. The problem is that they are deeply coupled in practice:

Observability feeds maintenance. Health scores determine which tables need attention and how urgently.
Maintenance feeds compaction. Expiry and cleanup must complete before compaction to avoid rewriting files about to be garbage-collected.
Compaction depends on telemetry. The optimal sort order comes from knowing how all engines query the table.
Routing improves when compaction improves. A well-sorted table enables cheaper engines, and routing telemetry feeds the next sort decision.
Policies must span all four. A retention policy affects snapshots (maintenance), which affects orphan accumulation (observability), which affects how much compaction processes, which affects file layout for every engine (routing).

The argument for a platform is not that individual tools are bad. It is that the interactions between concerns are where the operational value compounds, and independent tools by definition cannot exploit feedback loops across them.

A Diagnostic Runbook: Is Your Lake Degraded?

Before choosing a tool, measure the problem. Run these queries against your most important tables and score the results.

Step 1: File fragmentation

SELECT COUNT(*) AS total_files,
       COUNT(CASE WHEN file_size_in_bytes < 8388608 THEN 1 END) AS small_files,
       AVG(file_size_in_bytes) / (1024*1024) AS avg_mb,
       STDDEV(file_size_in_bytes) / (1024*1024) AS stddev_mb
FROM prod.db.events.files;

Healthy: small_files < 5% of total_files, avg_mb in the 128–512 MB range. Warning: 5–30% small files. Critical: > 30% small files, or avg_mb under 32.

Step 2: Manifest depth

SELECT COUNT(*) AS manifest_count,
       AVG(length) / 1048576 AS avg_manifest_mb,
       SUM(added_data_files_count + existing_data_files_count) AS total_entries
FROM prod.db.events.manifests;

Healthy: under 200 manifests. Warning: 200–500. Critical: over 500 and growing.

Step 3: Snapshot accumulation

SELECT COUNT(*) AS snapshot_count,
       MIN(committed_at) AS oldest_snapshot,
       MAX(committed_at) AS newest_snapshot
FROM prod.db.events.snapshots;

Healthy: retention window is enforced and recent. Warning: snapshots older than 30 days with no policy. Critical: thousands of snapshots spanning months.

Step 4: Delete file burden

SELECT content, COUNT(*) AS files, SUM(record_count) AS rows
FROM prod.db.events.files
WHERE content > 0
GROUP BY content;

Any equality deletes outside a Flink pipeline are a red flag. Position delete counts exceeding 10% of data file count indicate compaction lag.

Step 5: Sort-order alignment

SELECT readable_metrics.customer_id.lower_bound AS lb,
       readable_metrics.customer_id.upper_bound AS ub,
       file_size_in_bytes / (1024*1024) AS mb
FROM prod.db.events.files
ORDER BY lb
LIMIT 20;

Replace customer_id with your primary filter column. If adjacent files have broadly overlapping bounds (lower_bound = 1, upper_bound = 10M on most files), the table is not clustered on that column and no amount of partitioning will fix file-level pruning.

Scoring

Score	Meaning	Action
All green	Table is well-maintained	Monitor; adjust cadence if write patterns change
1–2 warnings	Targeted fix	Run the relevant procedure(s); consider policy-based scheduling
Any critical	Active degradation	Immediate maintenance; evaluate platform-level management
Multiple criticals across tables	Systemic gap	The maintenance approach itself needs replacing

The Current Landscape

The Iceberg management space has converged on several architectural approaches, each making different trade-offs about scope, lock-in, and operational burden.

Catalog-bundled maintenance

Platform	What it provides	What it does not
Apache Polaris (TLP since Feb 2026)	Maintenance policies (data-compaction, metadata-compaction, orphan-removal, snapshot-expiry) with catalog-namespace-table inheritance	Does not execute them. Policy declaration without enforcement.
Dremio Open Catalog (on Polaris)	Automatic compaction, DV resolution, clustering, manifest rewrite, vacuum — five operations, ~3h optimize / 24h vacuum, dedicated engine	Only tables inside Dremio's catalog. Target file size not per-table configurable.
AWS Glue	Three optimizers: compaction (binpack/sort/z-order), snapshot retention, orphan deletion. $0.44/DPU-hour.	No manifest rewrite, no statistics, no cross-engine sort awareness, no sequencing, Parquet only, auto-suspends after 4 consecutive failures.
S3 Tables	Fully managed compaction (auto/binpack/sort/z-order), snapshot management, orphan cleanup — all enabled by default	No manifest rewrite, no statistics, no cross-engine telemetry. Object monitoring fee penalizes the fragmentation compaction exists to fix.
Snowflake	Data compaction (bin-pack) + manifest compaction (always-on, cannot disable). Free for Snowflake-only writes since May 2026.	No sort/z-order beyond Automatic Clustering. External-engine DML activates compaction credit charges. No exposed `expire_snapshots` or `remove_orphan_files` procedures.
Databricks	Predictive Optimization: OPTIMIZE + Liquid Clustering + VACUUM + ANALYZE, enabled by default since Nov 2024. Selects and evolves clustering keys from query telemetry.	Requires Unity Catalog. Telemetry is scoped to Databricks workloads.
Starburst Galaxy	"Icehouse LakeOps": automated compaction (hourly), snapshot expiry (daily, >30 days), manifest rewrite (daily), orphan removal. Per-table health dashboards.	Schedule-driven, not health-driven. No cross-engine sort awareness.
Lakekeeper	Automatic snapshot expiry (per-warehouse, configurable) and adaptive orphan cleanup	No compaction, no manifest rewrite, no layout optimization.
Gravitino	Table Maintenance Service: alpha, policy-driven compaction for identity-partitioned tables only	CLI-driven, not self-scheduling. No snapshot expiry, orphan cleanup, or sort maintenance.
Nessie	None	Pure catalog: Git-style branching and tagging for metadata.

The pattern: each platform solves maintenance within its boundary and is blind to everything outside it. Databricks is the furthest along with query-driven clustering, but it requires Unity Catalog and sees only Databricks workloads. In a multi-engine, multi-catalog environment — which is what the open lakehouse was designed to enable — each boundary creates a gap.

Open-source management tools

Tool	Language	Stars	What it does	Limitation
Apache Amoro (incubating)	Java	~1,170	Self-optimizing management service (AMS) with optimizer groups. Monitors tables, dispatches compaction, supports Iceberg + Paimon + Hudi.	AMS calls `loadTable()` on every planning cycle (default: every minute). At 2,000+ tables, this creates significant catalog and object-storage pressure. Still incubating (v0.8.1).
OpenHouse	Java	~400	LinkedIn's RESTful control plane: declarative table provisioning, policy-based retention, Spark maintenance on Kubernetes. Proven at LinkedIn scale.	Built on Iceberg 1.2.0 and Spark 3.1.2 — multiple major versions behind current Iceberg. Tied to LinkedIn's infrastructure patterns.
Nimtable	TypeScript	~479	Web platform for Iceberg observability and management by RisingWave Labs. Browse catalogs, visualize file and snapshot distribution, run SQL, acts as REST Catalog endpoint.	Observability-focused; delegates maintenance execution to Spark or RisingWave. Not a compaction engine.
Floe	Java	~29	Policy-based maintenance orchestrator. Declarative policies with glob-pattern table matching, health thresholds, scheduled execution of all four Iceberg procedures via Spark (Livy) or Trino. Multi-catalog. Web UI. Featured on Apache Polaris blog as the reference policy consumer.	Orchestrator, not engine — dispatches to Spark/Trino for execution. Solo developer.
nimtable/iceberg-compaction	Rust	~140	Purpose-built Rust compaction on DataFusion. Handles positional and equality deletes. Most active open-source compaction project.	Full-table bin-pack only; sort/Z-order, incremental compaction, and orphan deletion are roadmap items.
Bergman	Rust	1	Rust maintenance engine on `iceberg-rust` + DataFusion. All four operations plus dangling delete-file removal. Library-first.	Pre-release, solo project. Implements its own commit layer because upstream `iceberg-rust` lacks rewrite-commit support.
Firn	Go	~15	JVM-free daemon; shells out to DuckDB for compaction. Binpack/sort/z-order, snapshot expiry, orphan cleanup. Multi-catalog (Lakekeeper, Glue, Polaris, Nessie).	Pre-v1.0 and stale for several months.
Zamboni	Python	—	Single-process maintenance via PyIceberg + DuckDB. Declarative layout config. Covers compaction, Z-order, partition evolution, manifest rewriting, snapshot expiry, orphan removal.	Single-process; no fleet orchestration, no health-triggered scheduling.
IceGuard	TypeScript	3	Multi-catalog web console for browsing and scheduling maintenance pipelines (Airflow-style chained actions).	Very early stage.

A few deserve deeper examination because they represent distinct architectural philosophies:

Apache Amoro is the most ambitious open-source effort. Its architecture (AMS → Optimizer Groups → Optimizer containers) provides fleet-level visibility: a central dashboard shows every table's health, a scheduling layer assigns compaction tasks to optimizer groups, and the system detects "self-optimizing" conditions (small files, orphan snapshots, excessive delete files). The architectural concern is scale: at 2,000+ tables against S3-backed catalogs, the per-minute metadata load from loadTable() calls becomes the bottleneck the system was designed to alleviate.

OpenHouse proves the concept at LinkedIn scale. It is a full RESTful control plane: declarative table provisioning, policy-driven retention, column-lineage-based governance, Spark maintenance jobs on Kubernetes, and an API-first design where tables are requested through a service and automatically provisioned with the right policies. The limitation is that it was built for LinkedIn's specific infrastructure and remains on Iceberg 1.2.0 — five major versions behind.

nimtable/iceberg-compaction has emerged as the most active open-source compaction engine. Written in Rust on DataFusion, it handles positional and equality deletes, runs without a JVM, and benchmarks competitively. Its scope is intentionally narrow: full-table bin-pack compaction. This is a compaction library, not a management platform, and it is useful precisely because of that focus.

What none of them provides is the full loop: health detection driving prioritization, dependency-aware sequencing, cross-engine telemetry feeding layout decisions, conflict-safe execution, and a feedback cycle where outcomes improve future decisions. Each solves a slice.

Where Flink fits

Flink's TableMaintenance is the most sophisticated per-table automation in Iceberg: event-driven triggers on commit count, file count, file size, and delete-file accumulation, all embedded in the streaming topology. But it is per-table, per-sink maintenance — not a management platform. No fleet-wide visibility, no cross-engine telemetry, no policy management, no multi-table prioritization. It is one component of what a platform provides.

How a Control Plane Runs This Loop

A table management platform occupies a specific architectural layer:

┌──────────────────────────────────────────────────┐
│  Query Engines (Trino, Spark, Flink, DuckDB...)  │
├──────────────────────────────────────────────────┤
│  TABLE MANAGEMENT PLATFORM / CONTROL PLANE       │ ← This layer
│  Health · Sequenced maintenance · Telemetry ·    │
│  Layout optimization · Policies · Routing        │
├──────────────────────────────────────────────────┤
│  Catalog (Polaris, Nessie, Glue, Gravitino...)   │
├──────────────────────────────────────────────────┤
│  Table Format (Apache Iceberg)                    │
├──────────────────────────────────────────────────┤
│  Object Storage (S3, GCS, ADLS)                  │
└──────────────────────────────────────────────────┘

It does not replace the catalog, the engine, or the storage. It sits between them and supplies the operational intelligence that none of them individually possesses. A catalog manages metadata and access control. A query engine executes queries and maintenance procedures. A control plane decides which maintenance to run, when, in what order, with what parameters, and adapts those decisions continuously.

LakeOps is built for this layer. It connects to catalogs through standard APIs — Glue, Polaris, Nessie, Gravitino, Lakekeeper, S3 Tables — and to query engines for telemetry, without requiring data movement, pipeline changes, or an SDK. The architecture follows a closed-loop model:

Sense — reads Iceberg metadata from every connected catalog continuously. Each table is scored as Healthy, Warning, or Critical based on file layout, manifest depth, snapshot age, delete-file ratio, partition skew, and sort-order alignment with observed query patterns. Cross-engine telemetry from Trino, Spark, Snowflake, Athena, DuckDB, and Flink reveals which columns are filtered, joined, and grouped — per table, across the full engine fleet.

Plan — prioritizes tables by degradation severity. A production table with 50,000 small files gets compacted before a staging table with 200. Operations are sequenced in dependency order. Partitions with active writers are identified and excluded, eliminating the commit conflict failures that dominate maintenance at scale.

Optimize — compaction, snapshot expiration, orphan cleanup, manifest rewriting, and delete-file resolution execute on a Rust engine built on Apache DataFusion. No Spark clusters, no JVM overhead. Maintenance operations are staggered across the catalog to avoid the maintenance-storm pattern described earlier.

Learn — outcomes feed back into the system. Tables where sort compaction produced large scan reduction get prioritized for sort maintenance. Sort orders adapt as query patterns evolve. Cadence tunes to each table's actual write velocity. This is the closed loop that independent tools cannot form: the one where every maintenance decision improves the next one.

What this looks like concretely

Sort-order selection from telemetry. Rather than a human analyzing logs across three engines to guess which columns to sort by, the platform aggregates WHERE, JOIN, and GROUP BY column frequency across every connected engine and builds a per-table field-access profile. If 82% of Trino queries filter on user_id and 71% of Athena reports filter on campaign_id, the sort order maximizes pruning for that mix. When the mix changes — a new dashboard, a new pipeline — the sort order adapts in the next compaction cycle.

Layout simulation on Iceberg branches. Before a sort-order change touches production data, the candidate layout is tested against real query patterns on an Iceberg branch: comparing projected scan reduction and file distribution to the baseline. Only a strategy that measurably improves pruning gets applied. Bad sort decisions are expensive to reverse (you are rewriting every data file in scope), and this mechanism is what makes automated layout selection safe rather than reckless. Compaction strategy details cover how binpack, sort, and Z-order selection differ per table.

Conflict-aware compaction. Rather than failing four times and suspending (as Glue's optimizer does), the platform monitors per-partition commit activity across engines and excludes partitions with active writers from the compaction scope. When a commit does lose an OCC race, it backs off, re-evaluates partition state, and reschedules rather than immediately retrying into the same conflict.

Hierarchical policies. Maintenance configuration cascades from catalog to namespace to table, with more specific policies overriding broader defaults. New tables inherit the right configuration from their namespace automatically. This is the same model Polaris's policy framework uses for declaring intent, with the crucial difference that a control plane actually executes it.

Staged adoption. You can start in manual mode, executing operations per table and reviewing results. Then define scheduled policies at the namespace level. Then enable Adaptive Maintenance, where the platform monitors health signals and runs the right operations at the right time without fixed schedules or manual tuning.

Performance. The Rust/DataFusion compaction engine avoids the JVM entirely. Published benchmarks show 95% faster compaction than Spark on identical datasets: a 200 GB S3 Tables compaction completed in 221 seconds vs. 1,612 seconds for Spark (7.3×), while Spark OOMed on a 1.2 TB table that the Rust engine processed in approximately 11 minutes. A 5.5 TB, 10-table benchmark achieved 2,522 MB/s peak throughput with 81% file-count reduction. Sub-minute compaction cycles keep up with streaming writers instead of running as batch catch-up jobs.

The platform overview covers how these capabilities compose: from lake-wide observability through query-aware compaction, multi-engine query routing, declarative policy management, and AI-agent access with layered guardrails (ReadOnly mode, cost-estimate caps, PII masking, human approval for high-stakes operations).

Three architectural guarantees matter for teams evaluating this category:

No data movement. All operations happen through standard Iceberg commit APIs. Data stays on your storage.
No pipeline changes. Connect catalogs and engines through standard APIs. No SDK, no code changes.
No lock-in. Everything the platform produces is standard Iceberg: snapshots, manifests, data files. Disconnect it and your tables remain exactly as they are.

Evaluating the Options

The right approach depends on where you are, not where you want to be:

Your situation	Approach
Under 50 tables, single engine, single catalog, batch writes	Scripts + standalone tools (Spark procedures, Trino `EXECUTE`)
Glue catalog, Athena-only, Parquet, batch, and compaction works fine	AWS Glue optimizers. Add catalog-level defaults and stop there.
Glue optimizer suspends on streaming writes or manifests are never rewritten	Supplement Glue with external tools over the REST catalog for the missing operations
Single platform (Databricks, Snowflake) genuinely handles 90%+ of your workload	Platform-bundled optimizer. Accept the boundary.
Multiple engines, multiple catalogs, or streaming writes with cross-engine access	Control plane. The integration cost of assembling standalone tools exceeds the cost of a platform.
500+ tables, growing, with maintenance consuming more engineering time than pipelines	Control plane. Manual approaches break at this scale.

Build it yourself (Airflow + Spark)

Works when you have under ~100 tables, a single engine, and a platform team with capacity to own the scheduler, conflict handling, sequencing, health detection, and configuration management. The honest failure mode is not that it does not work. It is that it works until the person who built it changes teams.

One detail worth knowing: Iceberg's min-input-files default is 5, while Glue overrides it to 100. If you are moving from Glue compaction to Spark procedures, that difference explains why your tables feel under-compacted on Glue.

Adopt a catalog-bundled optimizer

Works well when a single engine or platform is genuinely your primary query layer. Databricks Predictive Optimization is the most complete, with query-driven clustering key selection that evolves over time, but it requires Unity Catalog and sees only Databricks workloads. Snowflake's managed compaction is zero-config but bills you permanently the moment an external engine touches the table. Glue is the path of least resistance on AWS but cannot rewrite manifests, select sort orders, or survive streaming write conflicts. Each is useful within its scope and blind outside it.

Use open-source management tools

Floe gives you policy-driven scheduling over Spark or Trino with multi-catalog reach. nimtable/iceberg-compaction gives you a fast Rust compaction engine. Apache Amoro gives you a self-optimizing management service with streaming-aware triggers. These are real options for teams that want to own the infrastructure and avoid vendor dependency, with the honest caveat that none provides the full loop and most are pre-1.0.

Adopt a dedicated control plane

Works when the table fleet is large enough that per-table configuration is intractable, when multiple engines hit the same tables and need coordinated layout decisions, when streaming writes make conflict handling a constant concern, and when the engineering time spent maintaining the maintenance system exceeds the cost of a platform. This is where LakeOps fits: a managed control plane that layers onto your existing stack, runs the full operational loop, and scales sublinearly — going from 500 to 5,000 tables should not require 10× more configuration or human attention.

The hybrid path

These approaches compose more than most comparisons suggest. You can run Glue compaction for bulk binpack on the tables it handles well, connect an open-source tool for manifest rewriting, and add a control plane for the tables with streaming writes and multi-engine access patterns. The Glue REST catalog makes this possible; the question is whether the integration cost exceeds the cost of a single system.

How Format Evolution Changes the Picture

Iceberg v3 and the early v4 proposals affect what management platforms need to handle, but they do not reduce the need for management itself.

Deletion vectors (v3) replace position delete files with per-data-file bitmaps. The read-time penalty drops from a file join to a bitmap lookup, but deleted rows stay in data files, consuming storage and widening column statistics until compaction rewrites them. DVs make the read penalty cheaper; compaction remains the mechanism that removes deleted data.

Variant shredding (v3) stores semi-structured data (JSON) in typed Parquet columns while preserving the original document. This creates wider tables with more columns, more partition statistics, and more metadata — making sort-order decisions more complex because shredded field access patterns may differ from top-level columns.

Row lineage (v3) assigns a monotonically increasing sequence number to every row, enabling engines to track individual rows across snapshots. For management platforms, lineage simplifies delete-file resolution: identify exactly which data files are affected by a batch of deletes and compact only those, rather than scanning the entire table.

Geospatial bounds (v3) add bounding-box statistics for geometry columns — a new class of statistics that management platforms need to maintain, and spatial clustering (Z-order on coordinates) interacts with them in ways that temporal sort does not.

Metadata compaction (v4 proposal) would move manifest consolidation into the format itself, potentially eliminating external rewrite_manifests calls. If adopted, this removes one of the five maintenance procedures but does not affect the other four.

The net effect: the individual procedures change — some get simpler, some get more complex, and new ones emerge. The need for an external system to coordinate them does not diminish.

Where the Market Is Heading

The Iceberg management platform category is emerging because the open lakehouse reached a maturity inflection. Iceberg is the format — every major cloud and engine treats it as the default. The catalog layer has consolidated around the REST spec with Polaris, Nessie, and managed offerings. What remains unsolved is the operational layer above catalogs and engines: the layer that keeps tables healthy, queries fast, and costs controlled without requiring every team to become an Iceberg internals expert.

The direction is visible from every side. Databricks acquired Tabular (the company founded by Iceberg's creators to build exactly this managed layer) and absorbed it into Predictive Optimization. Cyera paid $100–130M for Ryft, an Iceberg operations startup with $8M in seed funding — a security vendor paying nine figures for table-level operational intelligence, because governed, traceable data access cannot exist without maintained, well-managed tables underneath. Starburst launched "Icehouse LakeOps" as a named product for automated table health. SAP acquired Dremio, committing to Polaris and Iceberg as the foundation of its Business Data Cloud. Apache Amoro remains in incubation. Open-source projects are multiplying around the gap. And Iceberg v4 proposals include an adaptive metadata tree that would fold manifest consolidation into the format itself, potentially eliminating external rewrite_manifests calls.

The question is not whether Iceberg tables will be managed. It is whether that management intelligence lives inside a proprietary platform, gets assembled from scripts and tools by every team independently, or runs as a dedicated control plane that preserves the openness Iceberg was built to provide.

Getting Started

If you are feeling the operational weight — slow queries traced to small files, storage costs climbing from orphan accumulation, maintenance DAGs consuming more engineering time than the pipelines they were supposed to enable — run the diagnostic runbook above against your most critical tables. Score them against the health criteria. If two or more tables show warnings or critical scores across multiple dimensions, the maintenance approach itself needs attention, not just the individual tables.

LakeOps connects to your existing catalogs and surfaces health classification immediately — often revealing degradation you knew existed but could not quantify.

The adoption path is designed for progressive trust:

Connect a catalog. LakeOps connects to Glue, Polaris, Nessie, Gravitino, Lakekeeper, or S3 Tables through their standard APIs. No code changes, no SDK, no agents to deploy. Immediate visibility into every table's health, file distribution, manifest state, and snapshot history.
Run manual maintenance. Execute individual operations — expire snapshots, compact a single partition, rewrite manifests — and review the results. You control what runs and when.
Define policies at the namespace level. Set compaction targets, retention windows, sort strategies, and maintenance cadence that new tables inherit automatically. Override specific tables when needed.
Enable adaptive maintenance. The platform monitors health signals continuously and runs the right operations at the right time, in the correct dependency order, with conflict-safe execution. Tables that need attention get it immediately; tables that are healthy are left alone.

The lake becomes self-maintaining. The on-call burden shifts from firefighting fragmentation and orphan accumulation to reviewing a structured event log. The platform team gets its capacity back for the work that actually produces value — building data products, not babysitting compaction jobs.

AWS Glue Iceberg Optimizer Alternatives for a Data Lakehouse

joni sar — Fri, 25 Sep 2026 09:34:47 +0000

The AWS Glue Data Catalog's managed table optimizers are the path of least resistance for Iceberg maintenance on AWS. Three toggles, no Spark cluster to operate, $0.44 per DPU-hour. For a Glue-cataloged, Athena-queried, Parquet-only estate with batch write patterns, they are genuinely the right answer and this article is not trying to talk you out of them.

The reason you are reading this is that something broke that assumption. Compaction cannot keep pace with streaming ingestion and suspends itself. Trino and Athena both hit the same tables and the sort order helps neither. You have three catalogs and the optimizer only sees one. Or you finally looked at query planning time and realized nothing in the Glue optimizer rewrites manifests, which means nothing ever has.

This guide covers what the Glue optimizers actually do, with the real API parameters and documented defaults rather than the marketing summary; the specific constraints that cause teams to look elsewhere, including several that are genuine data-loss hazards rather than inconveniences; how to measure your real cost per terabyte instead of estimating it; and an honest survey of the alternatives, from Spark procedures through Flink, Trino, S3 Tables, the commercial platforms, four open-source projects, and dedicated control planes such as LakeOps that run maintenance as a coordinated loop across catalogs rather than as isolated jobs.

One framing worth setting up front, because it changes the conclusion: since the Glue Iceberg REST Catalog endpoint shipped, this stopped being an either/or decision. You can keep Glue's managed compaction doing bulk binpack work and attach an external tool over the REST protocol for the operations Glue does not offer. That hybrid is usually the lowest-risk answer, and it is the one most comparisons never consider.

The Short Version

Your situation	What to do
Glue catalog only, Athena only, Parquet, batch writes, under ~50 tables	Stay on the Glue optimizers. Add catalog-level defaults and stop there.
Glue optimizer works, but planning is slow and manifests are never rewritten	Supplement. Keep Glue compaction, add an external tool over the Glue REST catalog for `rewrite_manifests` and statistics.
Streaming ingestion; compaction keeps suspending itself	Replace compaction specifically. The auto-suspend has no auto-resume, and AWS's documented mitigations require control Glue does not expose.
Avro or ORC data files	You cannot use Glue compaction at all. It is Parquet-only.
Multiple catalogs, or multiple query engines with different filter patterns	Replace with something that spans catalogs and sees cross-engine query telemetry.
Cross-account or cross-Region tables, or us-west-1 / GovCloud	Glue optimizers are unavailable. This is an architectural constraint, not a tuning problem.

What the Glue Optimizer Actually Does

There are three optimizer types, and the API enum has not grown: compaction, retention, and orphan_file_deletion. Two adjacent managed features are billed on the same meter and configured through the same catalog API, which is why they get mistaken for a fourth optimizer: column statistics generation, which writes Iceberg Puffin files, and materialized view auto-refresh. Neither is a TableOptimizer, and the statistics one matters enough that it gets its own treatment below.

Compaction

Merges small data files into larger ones using Iceberg's rewriteDataFiles. The documented trigger is specific, and it is worth quoting because most write-ups paraphrase it into something vaguer:

In the Data Catalog, the compaction process starts when a table or any of its partitions have more than 100 files. Each file must be smaller than 75% of the target file size.

Three things that matter in that sentence. It is evaluated per partition, not per table. The 100 is the default of a configurable parameter, not a constant. And the 75% is not an AWS invention, it is Iceberg's stock min-file-size-bytes default surfacing through the managed wrapper.

The real configuration surface is smaller than you might expect:

Parameter	Default	Notes
`strategy`	`binpack`	`binpack`, `sort`, or `z-order`
`minInputFiles`	`100`	Minimum data files in a partition before compaction acts
`deleteFileThreshold`	`1`	Minimum deletes in a data file to make it eligible

Note what is not there: there is no targetFileSizeMB API parameter. Target file size is controlled only through the Iceberg table property write.target-file-size-bytes, which defaults to 512 MB. This is a real difference from S3 Tables, which exposes target size directly in its maintenance API.

Note also that sort and z-order require you to have already defined a sort_order on the table. Glue will apply a sort order. It will never choose one, infer one, or update one.

Snapshot retention

Parameter	Default	Range
`snapshotRetentionPeriodInDays`	`5`	Table property takes precedence if set
`numberOfSnapshotsToRetain`	`1`	Table property takes precedence if set
`cleanExpiredFiles`	—	If `false`, expiry is metadata-only and files stay on S3
`runRateInHours`	`24`	3 to 168

Both criteria must be satisfied for a snapshot to survive: it must be within the retention period and within the minimum count.

Orphan file deletion

Parameter	Default	Range
`orphanFileRetentionPeriodInDays`	`3`
`location`	table location	Can be scoped to a sub-prefix
`runRateInHours`	`24`	3 to 168

There is a safety semantic here that almost nobody documents and that surprises people badly: the optimizer only deletes files created after the optimizer itself was created. Files that predate the optimizer are never touched, regardless of age. If you enable orphan deletion on a table that has been accumulating cruft for two years, it will clean nothing until new orphans appear. Pre-existing orphans need a manual remove_orphan_files run.

The actual CLI shape

Configuration examples float around the internet with invented parameter names. The real structure nests the Iceberg config two levels deep:

aws glue create-table-optimizer \
  --catalog-id 123456789012 \
  --database-name analytics \
  --table-name events \
  --type compaction \
  --table-optimizer-configuration '{
    "roleArn": "arn:aws:iam::123456789012:role/GlueOptimizerRole",
    "enabled": true,
    "vpcConfiguration": { "glueConnectionName": "analytics-vpc-connection" },
    "compactionConfiguration": {
      "icebergConfiguration": {
        "strategy": "sort",
        "minInputFiles": 50
      }
    }
  }'

vpcConfiguration is the under-documented piece: it is how you run optimizers against tables reachable only inside a customer VPC.

Catalog-level defaults, and two traps

UpdateCatalog lets you set optimizer defaults for a whole catalog, which is the only thing that makes this manageable past a few dozen tables. Precedence runs table-level config, then catalog-level config, then Iceberg table property.

Two documented problems:

Catalog-level inheritance silently does not work when Data Catalog metadata encryption is enabled. Every table must then be configured individually.
AWS documents a known issue where tables without their own configuration may fail to inherit the disabled state from the catalog. Their own guidance is to audit the console and execution logs for optimizers running that you did not intend. That is a surprise-bill vector, not a cosmetic bug.

Pricing

$0.44 per DPU-hour, billed per second with a one-minute minimum per run. One DPU is 4 vCPU and 16 GB. AWS's worked example: 30 minutes on 2 DPUs costs $0.44.

The Glue 6.0 price reduction does not apply to optimizers. Glue 6.0 went GA in August 2026 with a headline 30% cut, and that cut is scoped to Spark ETL job DPU-hours on Glue 6.0. Table optimizers are not versioned, you cannot opt them into 6.0 pricing, and the optimizer meter still reads $0.44. Expect to see this conflated a lot over the next year.

The one-minute minimum deserves more attention than it usually gets. A high-churn table that triggers many small compaction runs pays a floor of 60 seconds of DPU each time, and the reports of Glue compacting fewer than ten files per run several times an hour are exactly this failure mode: you are billed a minimum-duration run to do almost nothing.

Which Iceberg version you get

	Glue 6.0	Glue 5.1	Glue 5.0	Glue 4.0
Spark	4.1.1	3.5.6	3.5.4	3.3.0
Iceberg	1.11.0	1.10.0	1.7.1	1.0.0
Java	17	17	17	8

Iceberg v3 deletion vectors and row lineage arrived in Glue 5.1, not 6.0. Glue 6.0 added the remainder of v3: VARIANT with shredding, geometry and geography types, and nanosecond timestamps. AWS Prescriptive Guidance explicitly lists Glue table maintenance as supporting v3, and states that after a v2 to v3 upgrade the next compaction removes the legacy v2 delete files.

The blocker nobody mentions: Athena cannot read Iceberg v3 at all. Glue 6.0 will happily write v3 tables that your Athena users cannot query. If Athena is in your read path, v3 is off the table regardless of what Glue supports, and since the v2 to v3 upgrade is atomic metadata-only but effectively irreversible once deletion vectors exist, this is a decision to make deliberately rather than discover.

Where it runs

Glue optimizers are available in 15 regions. Not available in us-west-1 (N. California) or any GovCloud region. If your lakehouse lives there, the evaluation is over before it starts.

Measure Your Real Cost Before You Decide

Almost every comparison of Glue optimizer costs against alternatives is built on estimates. It does not have to be, and this is the single most useful thing in this guide if you are trying to justify a change.

TableOptimizerRun returns an IcebergCompactionMetrics struct containing:

dpu_hours — actual DPU-hours consumed by that run
number_of_dpus — DPUs allocated, rounded up
job_duration_in_hour
number_of_bytes_compacted
number_of_files_compacted

So your real cost per terabyte is not a guess:

$/TB = (Σ dpu_hours × $0.44) / (Σ number_of_bytes_compacted / 2^40)

Pull the history with ListTableOptimizerRuns and BatchGetTableOptimizer, aggregate across your top tables for the last 30 days, and you have a defensible number instead of a vendor's. Do this before you evaluate anything, because it also tells you which tables are burning DPU-hours to accomplish very little, which is usually a more actionable finding than the total.

Do not forget the S3 request bill

Compaction is read-many-small-files, write-few-large-files, plus listing. In us-east-1:

PUT, COPY, POST and LIST: $0.005 per 1,000
GET: $0.0004 per 1,000
DELETE: free

Writes cost 12.5× more than reads per request, but reads dominate compaction because there are so many more of them, and they scale with input file count rather than bytes:

Scenario	GETs	PUTs	Request cost
1 TB as 1M × 1 MB files → 2,048 × 512 MB	1,000,000	2,048	~$0.41
1 TB as 10M × 100 KB files → 2,048	10,000,000	2,048	~$4.01

At ten million input files the S3 request bill rivals the compute bill. The modeling point that follows: compute cost scales with bytes, request cost scales with file count. Any comparison that models only DPU-hours understates the cost of pathologically fragmented tables, which are precisely the tables you are trying to fix.

LIST being billed at the PUT rate is also why orphan file deletion, a recursive prefix listing, is quietly the most expensive of the three optimizers on wide tables.

Where the Ceiling Is

I have grouped the documented considerations and limitations by how much they should worry you, rather than by feature area. The first group can destroy data.

Correctness hazards

Shared S3 locations cause cross-table deletion. If two Data Catalog tables point at overlapping S3 locations and both have retention or orphan optimizers enabled, one table's optimizer will delete files the other table still references. AWS documents this as "unrecoverable data loss from unintended deletion." Every table needs a unique, non-overlapping prefix, including across databases. This is the single most dangerous thing on the limitations page and it is almost never mentioned in comparison articles.

S3 lifecycle rules on Iceberg paths delete live files. A lifecycle expiration rule that touches an Iceberg prefix will remove manifests and data files still referenced by current snapshots. Exclude Iceberg paths from lifecycle rules entirely. AWS's suggested backstop is S3 versioning, which helps you recover but does not prevent the incident.

The one-million-file cap creates silent orphans. Retention and orphan deletion each delete at most 1,000,000 files per run. When snapshot expiry exceeds that cap, the overflow does not error, it becomes orphan files that a later orphan-deletion pass has to find. That is a two-stage cleanup dependency across two independently scheduled optimizers, and nothing sequences it.

Orphan deletion ignores everything older than the optimizer. Covered above, repeated here because it is the one that makes people think the optimizer is broken when it is behaving exactly as designed.

What the optimizer structurally does not do

Capability	Status	Consequence
Manifest rewrite	Genuinely absent	No API, no strategy, no mention anywhere. Manifest bloat degrades scan planning and is invisible in file-count metrics. This is the largest single gap.
Sort order selection	Absent	`sort` and `z-order` exist, but you must define `sort_order` yourself and Glue never infers or evolves it.
Partition statistics	Absent	Iceberg 1.11 has `compute_partition_stats`; there is no managed equivalent.
Standalone delete-file rewrite	Partial	`deleteFileThreshold` makes delete-heavy files eligible for data compaction, which resolves deletes as a side effect. There is no `rewrite_position_delete_files` equivalent; you cannot compact deletes without rewriting data.
Manual or on-demand trigger	Absent	You cannot force a run, and you cannot prevent one from overlapping with your ETL window.
Auto-resume after suspension	Absent	Manual re-enable only.
Non-Parquet compaction	Absent	Parquet only.
Target file size via API	Absent	Table property only.

Two of these deserve expansion.

Parquet-only compaction is buried on the compaction documentation page and absent from the limitations page, which is why nearly every comparison misses it. If any of your tables write Avro or ORC data files, Glue compaction simply does not apply to them. S3 Tables handles all three formats. This is a hard eligibility gate, not a performance consideration.

Puffin statistics are a subtlety, not a gap. It is tempting to write "Glue can't compute Iceberg statistics," and it would be wrong. Glue does generate Puffin NDV sketches, through a separate column-statistics feature configured via UpdateCatalog with ColumnStatistics.Enabled and ColumnStatistics.RoleArn. It runs weekly and samples 20% of records, and Athena needs use_iceberg_statistics=true on the table to consume them. So the accurate statement is narrower: the Glue table optimizer does not compute statistics, and the feature that does is on a separate schedule with no coupling to compaction. Since compaction is exactly what invalidates statistics, that decoupling is the actual problem. Your statistics can describe a file layout that a compaction run replaced four days ago.

No sequencing between the three

The three optimizers run independently. There is no dependency graph and no way to configure one.

Iceberg maintenance has a correct order, and it is not arbitrary. Expire snapshots first, so the later steps operate on a smaller file set and you are not rewriting files that are about to be dereferenced. Remove orphans next, including the ones expiry just created. Compact after that, against a clean current file set. Rewrite manifests last, so the metadata tree indexes the layout that now exists. Refresh statistics after that, so the optimizer's estimates match the files that will actually be read.

Glue gives you three of those five operations, on three independent schedules, in whatever order they happen to fire. The concrete waste is compaction rewriting files that snapshot expiry dereferences an hour later, and orphan cleanup running before expiry finishes and missing everything it was supposed to catch.

Architectural boundaries

Single catalog. Only tables in the Glue Data Catalog. REST catalogs (Polaris, Nessie, Gravitino, Lakekeeper), Hive Metastore, and anything else get nothing. Production lakehouses increasingly span several, and each additional catalog means another maintenance story with its own blind spots.
No cross-account tables and no cross-Region tables. Cross-account buckets work if the role has access, but cross-account tables and resource links do not.
No S3 Express One Zone.
Z-order does not support Decimal or TimestampWithoutZone.
Observability is CloudWatch job telemetry. Success, failure, duration, DPU usage. It tells you a job ran. It does not tell you whether the table is healthy: file size distribution, manifest depth, delete-file accumulation, snapshot growth relative to write velocity, or whether the sort order still matches how the table is queried. Diagnosing that means querying Iceberg metadata tables in Athena per table and correlating manually, which is tractable at 20 tables and is not at 500.

The Concurrency Problem

This is the failure mode that pushes the most teams off the managed optimizer, so it is worth understanding precisely rather than as "compaction sometimes fails."

What actually happens

Reports from teams running hourly MERGE jobs alongside Glue auto-compaction describe roughly one in ten compaction runs failing with:

Compaction optimizer failed. Error: partial-progress.enabled is true but no rewrite
commit succeeded. Check the logs to determine why the individual commits failed. If
this is persistent it may help to increase partial-progress.max-commits which will
break the rewrite operation into smaller commits.

That message leaks two things AWS does not document anywhere else.

First, Glue's managed compaction runs rewrite_data_files with partial-progress.enabled = true. That is otherwise invisible internal behavior, and it explains a lot about the observed run patterns.

Second, and this is the part that makes it maddening, the remediation the error recommends is not available to you. partial-progress.max-commits is not exposed through the optimizer API, and the managed Spark job is not accessible. The error tells you to turn a dial you cannot reach.

Then it gives up

From the AWS documentation, verbatim:

When compaction operations fail four consecutive times, AWS Glue catalog table optimization automatically suspends the optimizer to prevent unnecessary compute resource consumption.

Re-enabling is manual, through the console or API. There is no auto-resume, no documented backoff curve, and no alarm. A suspended optimizer is silent, and the table degrades in the background until somebody notices the query latency. If you run Glue compaction on tables with concurrent writers, build a CloudWatch alarm on optimizer state right now, independently of anything else in this article.

Why the usual advice does not work

The standard internet recommendation for Iceberg commit conflicts is to drop from serializable to snapshot isolation. AWS states plainly, in its own guidance on concurrent write conflicts, that this does not help here:

For conflicts between streaming ingestion and compaction operations, which is one of the most common scenarios, snapshot isolation does not provide any additional benefits to the default serializable isolation.

The two conflict classes behave differently and only one is retryable:

	Catalog commit conflict	Data update conflict
Exception	`CommitFailedException`	`ValidationException`
Cause	Concurrent metadata pointer update	Validation detected an overlapping data change
Auto-retryable	Yes, Iceberg retries the metadata commit	No, table state changed and retry safety is undecidable
Fix	Table properties	Application-level retry with backoff and jitter

The Iceberg-side properties you can actually set, with their defaults:

Property	Default
`commit.retry.num-retries`	4
`commit.retry.min-wait-ms`	100
`commit.retry.max-wait-ms`	60000
`commit.retry.total-timeout-ms`	1800000

AWS's recommended baseline for tables under frequent concurrent writes raises retries to 10 with a 10-second max wait; for maintenance operations they suggest keeping 4 retries but raising the minimum wait to 1,000 ms. Set these as table properties and Glue's compaction will honor them, since it is running stock Iceberg underneath.

The structural fix

Retry tuning treats the symptom. The structural fix is to not attempt compaction on a partition that has an active writer, and to make the compaction window short enough that overlap is unlikely in the first place. Glue exposes neither: you cannot scope compaction with a WHERE predicate through the managed optimizer, and you cannot control when it runs. AWS's own documented mitigation is to serialize your writes against compaction, which negates most of the value of "automatic."

What a Replacement Actually Needs

Before looking at options, it helps to name the criteria the gaps above imply:

Sequenced operations, where each step's output feeds the next, rather than three independent schedulers
Manifest rewrite and statistics refresh as first-class operations, coupled to compaction rather than on separate cadences
Query-aware layout, where sort keys come from the columns queries actually filter and join on, across every engine hitting the table
Health-driven triggers based on table state, not fixed file counts, so a streaming table with 40,000 small files gets attention before a batch table slightly under target
Writer-aware conflict handling, excluding hot partitions and backing off rather than failing four times and suspending
Multi-catalog reach, because the catalog boundary is an accident of history, not a meaningful maintenance boundary
Table health observability, not job telemetry
An execution engine sized for the work, since compaction is I/O-bound read-merge-write and a general-purpose JVM cluster is structurally over-provisioned for it

The category this describes

What that list adds up to is not a better compaction job. It is a control loop, and nothing in the open lakehouse stack ships one.

A lakehouse control plane is the layer that runs it. It sits above the components you already have rather than replacing any of them: your catalogs, your Iceberg tables in S3, and your query engines all stay where they are. It attaches through standard catalog and Iceberg metadata APIs, so there is no data movement, no pipeline rewrite, and no change to the table format, which also means leaving is a matter of disconnecting it.

The loop has four steps. Sense table structure and query telemetry across every connected catalog and engine. Classify each table's health from those signals. Plan the operations and physical layout each table needs, in dependency order. Execute, then measure the result and feed it back into the next decision.

Glue's optimizers implement a fragment of the execute step. Everything else, the sensing, the classification, the sequencing, and the learning, is the part you are currently doing by hand or not at all. Hold that model while reading the options below, because it is the axis they actually differ on.

The Alternatives Landscape

Spark procedures: the DIY baseline

Calling rewrite_data_files, rewrite_manifests, expire_snapshots, remove_orphan_files, and compute_table_stats yourself gives you complete control and complete operational ownership. Every capability on the criteria list is achievable. You are building the scheduler, the health signals, the conflict handling, the sequencing, and the observability.

Worth knowing: Iceberg's own min-input-files default is 5, while Glue overrides it to 100. If your tables feel under-compacted on Glue relative to a Spark job you used to run, that difference is probably why.

This is where most teams land after outgrowing the managed optimizer, and it trades a maintenance problem for a platform-engineering problem. Whether that is a good trade depends entirely on whether you have the headcount to own it properly. The honest failure mode is not that it does not work, it is that it works until the person who built it changes teams.

Trino: the underrated option if you already run it

Trino's Iceberg connector exposes maintenance through ALTER TABLE ... EXECUTE, and it covers something Glue cannot do at all:

ALTER TABLE analytics.events EXECUTE optimize(file_size_threshold => '100MB')
    WHERE event_date < DATE '2026-09-01';

ALTER TABLE analytics.events EXECUTE optimize_manifests;

ALTER TABLE analytics.events EXECUTE expire_snapshots(retention_threshold => '7d');

ALTER TABLE analytics.events EXECUTE remove_orphan_files(retention_threshold => '7d');

Three points in Trino's favor. optimize accepts a WHERE clause, which is exactly the partition-scoped compaction AWS recommends and Glue does not expose. optimize_manifests clusters manifests by partitioning columns, closing the biggest Glue gap. And the retention floors (iceberg.expire-snapshots.min-retention and iceberg.remove-orphan-files.min-retention, both 7d) hard-fail rather than letting you delete live files, which is a real safety property Glue lacks.

The catch for AWS shops: Athena is Trino-derived but does not expose optimize_manifests. You need actual Trino, and you are then running maintenance on your query cluster, competing with user workloads.

Flink TableMaintenance: only if Flink is already yours

Flink can run maintenance inside the streaming job that writes the data, which eliminates a separate cluster and makes maintenance naturally writer-aware:

IcebergSink.forRowData(dataStream)
    .table(table)
    .tableLoader(tableLoader)
    .rewriteDataFiles(Map.of(RewriteDataFilesConfig.MAX_BYTES, "1073741824"))
    .expireSnapshots(Map.of(
        ExpireSnapshotsConfig.RETAIN_LAST, "5",
        ExpireSnapshotsConfig.MAX_SNAPSHOT_AGE_SECONDS, "604800"))
    .deleteOrphanFiles(Map.of(DeleteOrphanFilesConfig.MIN_AGE_SECONDS, "259200"))
    .append();

Available tasks are ExpireSnapshots, RewriteDataFiles, and DeleteOrphanFiles. Scheduling triggers include scheduleOnCommitCount, scheduleOnDataFileCount, scheduleOnDataFileSize, and several delete-file variants. None of them have defaults, so nothing is scheduled until you set one.

Two accuracy notes, because this area moves fast. ConvertEqualityDeletes, which resolves Flink's equality delete backlog into v3 deletion vectors through a staging branch, is real and genuinely important for CDC pipelines, but as of now it appears only in Iceberg's nightly documentation and not the stable release. And RewriteManifests is not merged for Flink; the pull request is open and committers have leaned toward waiting for native metadata compaction in Iceberg v4. So Flink maintenance does not close the manifest gap either.

Amazon S3 Tables

The most hands-off option, and the defaults are the opposite of Glue's: all maintenance is enabled by default, so you are being billed for compaction from the moment you create a table.

Operation	Property	Default	Minimum
Compaction	`targetFileSizeMB`	512 MB	64 MB
Snapshot management	`minSnapshotsToKeep`	1	1
Snapshot management	`maxSnapshotAgeHours`	120	1
Unreferenced file removal	`unreferencedDays`	3	1
Unreferenced file removal	`nonCurrentDays`	10	1

Strategy accepts auto, binpack, sort, or z-order, and auto picks sort when the table has a sort order defined and binpack otherwise. That is a genuinely better default than Glue's unconditional binpack. S3 Tables also compacts Avro and ORC, which Glue does not, and it exposes target file size directly.

The pricing model is where it gets interesting, and it changed substantially after AWS cut rates:

Charge	Rate
Storage	$0.0265/GB/month, roughly 15% above S3 Standard
Compaction, data	$0.005/GB processed (binpack)
Compaction, objects	$0.002 per 1,000 objects processed
Object monitoring	$0.025 per 1,000 objects per month

Object monitoring is the one to watch. It is a recurring per-object charge on everything in the bucket, which means a fragmented table is penalized twice: once at compaction time and then forever on monitoring. It penalizes exactly the condition compaction exists to fix. AWS's own example puts a 1 TB, 10,486-object table at $0.26/month monitoring against a $27.14 storage bill, which is small, but it scales with object count rather than bytes.

AWS's pricing text qualifies the $0.005/GB rate as applying to "default binpack compaction," which implies sort and z-order cost more. I could not find a published rate for those. Verify before modeling a sorted-table budget.

Other constraints: 10 table buckets per account per Region by default, 10,000 tables per bucket, namespaces cannot nest, Parquet row group size is fixed at 128 MB, and data type Fixed plus brotli and lz4 compression are unsupported. The only way to stop compaction charges is to explicitly disable compaction.

On throughput, published comparisons put S3 Tables managed compaction well behind alternatives on the same data, which matters less if you are optimizing for zero operational involvement and matters a lot if compaction needs to keep up with ingestion.

The Glue Iceberg REST catalog changes the question

This is the strategic point most comparisons miss entirely.

Glue exposes an Iceberg REST Catalog endpoint at https://glue.{region}.amazonaws.com/iceberg, configured with type=rest, warehouse=<account-id>, rest.sigv4-enabled=true, rest.signing-name=glue:

catalog.type=rest
catalog.uri=https://glue.us-east-1.amazonaws.com/iceberg
catalog.warehouse=123456789012
catalog.rest.sigv4-enabled=true
catalog.rest.signing-name=glue
catalog.rest.signing-region=us-east-1

That decouples engine from catalog. PyIceberg, Trino, Flink, DuckDB, Snowflake, or a purpose-built maintenance daemon can all reach Glue-cataloged tables over a standard protocol while Glue's managed optimizers keep running.

Which means "Glue optimizer versus an alternative" is often the wrong framing. A perfectly reasonable production setup keeps Glue compaction doing bulk binpack on the tables it handles well, and attaches an external tool over REST for manifest rewriting, statistics refresh, query-aware sort, and the tables Glue cannot touch. You are not migrating off Glue, you are filling in what it does not do.

Engine-integrated maintenance

Vendor	What it does	Boundary
Dremio Automatic Optimization	Five operations in one pass: compaction, delete-file and DV resolution, clustering, partition-evolution alignment, and manifest rewriting. Partition-aware, workload-timed, roughly 3h optimize and 24h vacuum cadence	Only tables in Dremio's catalog; target file size not per-table configurable
Starburst Galaxy / Enterprise	Trino-native, scoped at table, schema, or catalog. Compaction, snapshot expiry, orphan cleanup, statistics	Schedule-driven, not health-driven. No table-health signal decides when to run
Databricks Predictive Optimization	Automatic `OPTIMIZE`, `VACUUM`, `ANALYZE` on Unity Catalog managed tables, Delta and Iceberg. Liquid Clustering selects clustering columns from query telemetry	Requires Unity Catalog; telemetry is scoped to Databricks workloads
Snowflake managed Iceberg	Data compaction plus manifest compaction, which cannot be disabled. `TARGET_FILE_SIZE` of `AUTO` or 16/32/64/128 MB, settable at account through table scope	See the billing cliff below
Cloudera Lakehouse Optimizer	Policy-based with schedule and event triggers; Spark binpack, sort, z-order	Cloudera ecosystem

Snowflake's arrangement has a trap worth flagging. For SNOWFLAKE_MANAGED external volume tables written only by Snowflake, compaction is bundled at no additional cost. Once an external engine performs DML or DDL through the Iceberg REST Catalog on or after 21 May 2026, that table begins incurring compaction credits permanently. One external write flips a free table into a billed one, and you can see it in ICEBERG_STORAGE_OPTIMIZATION_HISTORY.

The shared limitation across this whole category: each platform sees only its own workloads. In a multi-engine stack you get fragmented maintenance with no cross-engine coordination, which is the same problem as Glue with a nicer interface.

For completeness, since it still comes up: there is no standalone Tabular product. Tabular was acquired by Databricks in 2024 and its managed-maintenance thesis now lives inside Unity Catalog and Predictive Optimization.

Open source

This category is young and moving fast, so here is where each project actually stands rather than how its README reads:

floe — Java, Apache 2.0. A policy-based orchestrator rather than an engine: it discovers tables, scores health into a "debt score" from snapshot count, small-file percentage, and delete ratio, then dispatches REWRITE_DATA_FILES, EXPIRE_SNAPSHOTS, REMOVE_ORPHAN_FILES, and REWRITE_MANIFESTS to Spark via Livy or to Trino. Multi-catalog across REST (Polaris, Lakekeeper, Gravitino), Hive, and Nessie, with a web UI. The most complete conceptual model of the four. Pre-1.0, and development has been quiet for several months.
firn — Go, not Rust, despite how it is frequently described. A JVM-free daemon that shells out to DuckDB for compaction. Binpack, sort, z-order, snapshot expiry, orphan cleanup, across Lakekeeper, AWS Glue, Polaris, and Nessie, on S3, GCS, and Azure. Pre-1.0 and currently stale. Its README's claim of a "20-30x cost premium" for S3 Tables traces to a competitor's benchmark and predates the S3 Tables price cut; do not repeat it.
bergman — Rust, single binary on iceberg-rust and DataFusion, no JVM. Notable for implementing manifest rewrite and dangling delete-file removal in its own commit layer, on the correct observation that nothing else cleans those up. Triggered rather than scheduled, with Prometheus metrics and a daemon mode. Actively developed but effectively a solo pre-release project; interesting design, not something to put in front of production data yet.
nimtable/iceberg-compaction — Rust on DataFusion and iceberg-rust, handling positional and equality deletes. The strongest community signal of the four and the most actively maintained. Scope is narrower than it looks: full-table compaction only today, with partial and incremental compaction, binpack/sort/zorder strategies, and orphan deletion all still unchecked roadmap items.

None of these provide cross-engine query telemetry or lake-wide table health scoring. They are good options if you want to own the infrastructure and avoid another vendor, provided you are honest about maturity.

A structural note on where this is heading: Apache Polaris has become the de facto neutral catalog and explicitly does not execute maintenance, which is the gap these projects are racing to fill. Meanwhile Iceberg v4 discussions include native metadata compaction, which is part of why the Flink RewriteManifests work stalled. Manifest maintenance is a real gap today that the format itself may eventually close.

How a Control Plane Closes Each Gap

Taking the control-plane model from earlier and making it concrete against the specific Glue limitations above. LakeOps connects to the Glue Data Catalog through standard IAM credentials and runs alongside it, so Glue stays your catalog and your ETL engine.

Sequencing, as an actual dependency chain

The five-step order matters and Glue has no mechanism to express it. LakeOps runs snapshot expiration, then orphan cleanup, then compaction, then manifest optimization, with each step's output feeding the next. Cadence adapts per table to write velocity, so a streaming table may run the full cycle several times an hour while a weekly batch table runs once and an idle table is skipped entirely. The sequencing model is worth reading if you are building this yourself, because the ordering constraints are the part that is easy to get subtly wrong.

That also resolves the statistics decoupling described earlier. Puffin statistics are refreshed as part of the same chain, after the file layout changes, rather than on an independent weekly schedule that has no idea compaction happened.

Manifest rewrite as a first-class operation

The largest Glue gap is simply a supported operation here, with its own policy type rather than a Spark script you maintain separately. Because it is sequenced after compaction, the rewritten manifests index the file set that actually exists, which is the condition that makes manifest rewriting worth doing at all.

Query-aware sort instead of a sort order you guessed

Glue will apply a sort_order. It will never tell you what it should be. That question is unanswerable from inside any single engine, because the right layout depends on how Athena, Trino, Spark, and everything else query the table simultaneously.

LakeOps aggregates query telemetry across the connected engines, the AWS Glue integration covering Athena, Trino, Spark, Redshift, DuckDB, Snowflake, and Flink, and builds a per-table field-access profile from filter, join, and group-by frequency. Compaction then applies the sort order that maximizes min/max data skipping for that actual mix, and the strategy updates as access patterns move. When a new dashboard starts filtering on a different column, the next compaction pass reflects it rather than waiting for someone to notice.

Before a sort-order change touches production files, the candidate layout is validated on an Iceberg branch and compared against the baseline on real metadata, so a rewrite that would not improve scan reduction does not happen. That matters because sort-order changes rewrite every file in scope and are expensive to reverse.

Writer-aware conflict handling instead of suspension

This is the most direct answer to the auto-suspend problem. Rather than attempting compaction and failing four times, LakeOps monitors per-partition commit activity across connected engines and excludes partitions with active writers from the compaction scope, retrying them on a later cycle once commit frequency drops. When a commit does lose an optimistic-concurrency race, it backs off, re-evaluates partition state, and reschedules rather than retrying immediately into the same conflict.

There is also a mechanical contribution: a compaction pass that finishes in 221 seconds instead of 1,612 on the same 200 GB overlaps a far smaller window with your writers, which lowers conflict probability before any scheduling logic is involved. The commit conflict mechanics go deeper into why treating conflicts as expected behavior rather than as failures is the right operating model for multi-writer tables.

learn more:

Compaction Articles - LakeOps Blog

Iceberg compaction strategies, engines, and automation — binpack, sort, Z-order, streaming compaction, and production benchmarks.

lakeops.dev

Health-based triggers instead of a fixed file count

Glue's trigger is 100 files under 75% of target, per partition. That is a reasonable heuristic and it is the same heuristic for every table you own.

LakeOps evaluates each table across file count and size distribution, manifest depth, snapshot accumulation, delete-file ratio, partition skew, and sort-order alignment with observed query patterns, then prioritizes the most degraded tables first. The last signal has no equivalent in any threshold-based system, because it is not a property of the table at all. It is a property of the relationship between the table and its workload, and it degrades without a single byte being written.

An execution engine sized for the job

Compaction is I/O-bound read-merge-write. Running it on a general-purpose JVM cluster means JVM startup, GC pauses, executor provisioning, and idle capacity, compounded across hundreds of runs per day. LakeOps runs compaction on a Rust and DataFusion engine with bounded memory and disk spill instead. Published benchmarks on a 200 GB table:

Metric	Spark	S3 Tables	LakeOps
200 GB binpack	1,612s	6,300s	221s
Peak throughput	~350 MB/s	~32 MB/s	2,522 MB/s
Cost per TB	~$50	Managed	~$5
Memory model	JVM heap, OOM risk	Managed	Bounded, no OOM

Two production results from the same benchmark set: a 1,192 GB table that OOM'd Spark completed in 11 minutes, and a high-ingestion table went from 42,633 files to 69 in 138 seconds. Across 5.5 TB the file count dropped from 101,223 to 19,170. The full Spark replacement analysis has the methodology.

Set that against the $/TB number you calculated from dpu_hours earlier. That comparison is the whole business case, and you now have both sides of it measured rather than estimated. The broader cost picture also includes the storage side, since snapshot expiry and orphan cleanup reclaim bloat that never shows up in a compaction bill.

Multiple catalogs, one loop

Glue, REST catalogs (Polaris, Nessie, Gravitino, Lakekeeper), Hive Metastore, and S3 Tables are connected simultaneously, with every table discovered, health-scored, and maintained under consistent policies. If you run Glue for analytics, a REST catalog for streaming, and S3 Tables for a newer project, the catalog boundary stops being a maintenance boundary.

Policies instead of per-table CLI calls

Maintenance configuration cascades catalog-wide, then namespace, then table, with more specific policies overriding broader defaults. New tables inherit the right configuration from their namespace automatically. Compared with managing create-table-optimizer calls per table, and with the catalog-level inheritance caveats documented on the Glue side, this is the difference between configuration you can reason about and configuration scattered across an account.

Create a policy:

And let it run:

Table health, not job telemetry

Every maintenance operation is logged with before-and-after metrics: file counts, bytes reclaimed, duration, and the signal that triggered it. Severity-ranked insights flag excessive manifests, partition skew, and small-file buildup before query latency moves. The platform overview covers how observability, maintenance, compaction, governance, and routing compose into the single loop.

Side-by-Side

Capability	AWS Glue optimizer	S3 Tables	LakeOps
Compaction strategies	Binpack, sort, z-order	Auto, binpack, sort, z-order	Binpack, sort, z-order, query-aware
File formats	Parquet only	Parquet, Avro, ORC	Parquet, Avro, ORC
Trigger model	Fixed: >100 files per partition	Managed, continuous	Health-based, prioritized
Sort order selection	You define it	You define it	Cross-engine telemetry
Maintenance sequencing	Three independent schedules	Managed, opaque	Dependency-ordered pipeline
Manifest rewrite	Not supported	Not exposed	First-class operation
Statistics refresh	Separate weekly feature, 20% sample	Not exposed	Sequenced after compaction
Catalog scope	Glue only	S3 Tables only	Glue, REST, S3 Tables, HMS
Concurrent writes	Suspends after 4 failures	Managed	Hot-partition exclusion, backoff
On-demand trigger	Not supported	Not supported	Supported
Cross-account / Region	Not supported	Per-bucket	Supported
Target file size control	Table property only	API parameter	Policy
Observability	CloudWatch job metrics	CloudWatch	Per-table health, insights, audit trail
Execution engine	Managed Spark (JVM)	Managed	Rust + DataFusion
Layout validation	Not supported	Not supported	Branch-based simulation
Availability	15 regions, no us-west-1 or GovCloud	Region-dependent	Catalog-reachable

Making the Call

Stay with the Glue optimizers when

Every table is in the Glue Data Catalog and every data file is Parquet
Athena or Redshift is effectively your only query engine, so there is no cross-engine sort conflict to resolve
Write patterns are batch, and compaction is keeping up without suspending
You are under roughly 50 tables, where per-table configuration and manual metadata queries are still tractable
Catalog-level defaults cover your policy needs, and metadata encryption is off so inheritance actually works

That is a real and common environment. Glue's optimizers are zero-infrastructure and cost nothing to leave running. Enable them, set catalog-level defaults, put a CloudWatch alarm on optimizer suspension, exclude your Iceberg prefixes from S3 lifecycle rules, and move on to a more interesting problem.

Supplement rather than replace when

Glue compaction is working fine but planning time has grown and nothing rewrites manifests
Your statistics are stale relative to compaction
Most of your tables fit Glue's model and a handful do not

Connect over the Glue Iceberg REST catalog, leave the managed optimizers running, and add the operations Glue does not offer. This is the lowest-risk path and it is underused because most people frame the decision as a migration.

Replace when

Streaming ingestion. Compaction that suspends itself is worse than no compaction, because it fails silently. If you are seeing repeated partial-progress.enabled is true but no rewrite commit succeeded, the remediation the error suggests is not reachable through the managed optimizer.
Multi-engine reads. When Trino, Athena, Spark, and DuckDB hit the same tables, a static sort order set once at table creation is a guess that helps one workload and costs the others. The Athena cost analysis shows how directly bytes scanned tracks layout on scan-priced engines.
Multiple catalogs. The catalog boundary is not a meaningful maintenance boundary, and maintaining a separate pipeline per catalog is how drift starts.
Hundreds of tables. Per-table configuration and per-table metadata queries do not scale, and CloudWatch cannot tell you which table to look at first.
Cost at scale. Run the dpu_hours calculation. If your measured $/TB is materially above what a purpose-built engine delivers, and you are compacting petabytes, that gap is a line item rather than a rounding error.
Avro or ORC. Glue compaction does not apply. This one is binary.

The Bottom Line

The Glue optimizers are not a bad product. They are a narrowly scoped one: three independent operations, on one catalog, for one file format, triggered by one heuristic, with job-level telemetry. Inside that scope they work well and cost little.

What they are not is an operating model. Iceberg maintenance is a loop, not a set of jobs, and the parts Glue leaves out are the parts that make the loop work: sensing table health rather than job success, choosing a physical layout from how tables are actually queried, sequencing operations so each one's output feeds the next, and handling concurrent writers as normal rather than as failure. Those gaps are structural, not a roadmap item you can wait out.

The most useful next step is not choosing a tool. It is measuring what you have. Pull dpu_hours and number_of_bytes_compacted from ListTableOptimizerRuns for your top twenty tables, query the manifests and files metadata tables to see whether planning overhead and file distribution are actually where you assumed, and check whether any optimizer has quietly suspended itself. Most teams find at least one surprise, and the surprise usually decides the question.

If the answer is that you need the loop and not just the jobs, a control plane like LakeOps runs it on top of the Glue catalog you already have, without moving data or changing the table format. If the answer is that Glue covers you, that is a good outcome too, and now you know which limitations to watch for.

Apache Iceberg Query Optimization: Production Guide

joni sar — Thu, 24 Sep 2026 12:18:02 +0000

Every query against an Iceberg table is a negotiation about how much data the engine is allowed to ignore. Read the metadata, throw away everything that cannot match, scan what survives. A well-maintained table lets the planner discard 95–99% of files before a single Parquet footer is opened. A neglected table forces a near-full scan no matter how selective your WHERE clause looks.

That gap is almost never the engine's fault. Trino, Spark, Snowflake, and DuckDB all run the same fundamental pruning logic against the same Iceberg metadata. What differs is whether your physical layout and statistics give that logic anything to work with. A customer_id = 481 filter is worthless if every file in the table contains customer IDs 1 through 10,000,000, and no amount of cluster tuning will fix it.

This guide covers the entire surface: how Iceberg's scan planner actually works and what to measure inside it, how statistics get collected (and silently dropped), the difference between partitioning and clustering, why declaring a sort order often changes nothing, how bloom filters and Puffin statistics fit, what format v3 deletion vectors change for reads, and the engine-specific knobs that matter on Trino and Spark. It ends with a triage runbook and a checklist.

The Short Version

If you only read one section, read this table. It maps the symptom you are staring at to the mechanism that causes it.

Symptom	Most likely cause	Fix
Selective `WHERE` still scans most of the table	File min/max bounds overlap; data is not clustered on the filter column	Global sort or Z-order during compaction (not just binpack)
Seconds of latency before the first byte is read	Manifest fragmentation, or manifests clustered on the wrong partition field	`rewrite_manifests` with `sort_by` on the hot transform
Pruning does nothing and bounds are simply absent	Metrics mode is `none`, or the column fell past the 100-column inference cap	`write.metadata.metrics.column.<col>` = `truncate(16)`
Point lookup on a high-cardinality column is slow	Min/max cannot prune a scattered key	Parquet bloom filter on that column
Table got slower after a CDC backfill	Position or equality delete accumulation	Compact with a low `delete-file-threshold`; move to v3 deletion vectors
Joins pick catastrophic orders	Missing or stale Puffin NDV, or CBO is switched off	`compute_table_stats` / `ANALYZE`, refreshed after every compaction
Sort order is declared but nothing improved	Writers applied a local sort, so file ranges still overlap	Set `write.distribution-mode` to `range`, or sort globally during rewrite

That last row is the one that costs teams the most time, and it gets its own section below.

How Iceberg Actually Plans a Query

Iceberg planning is a funnel, and the table spec defines each stage precisely. Each stage is cheaper than the one beneath it, and each stage's usefulness depends entirely on the quality of the metadata written at commit time. Knowing the stages by name matters, because the diagnostics and the fixes are different at each level.

Stage 1: Manifest list filtering

Planning starts at the manifest list, a single Avro file that indexes every manifest in the snapshot. Each entry carries a partitions array of field_summary structs: lower_bound, upper_bound, contains_null, and contains_nan for each partition field across all data files that manifest tracks.

Iceberg converts your query predicate into partition space using inclusive projection, a deliberate widening that guarantees no file which might match is ever excluded. For a table partitioned by days(event_time), the filter event_time >= TIMESTAMP '2026-03-01' projects to a bound on event_time_day. ManifestEvaluator then tests that projected predicate against each manifest's summary. Manifests covering only January and February are dropped whole, along with every file they describe.

This is the cheapest prune available, and it is the one hidden partitioning buys you. Users filter on event_time; the planner does the transform arithmetic. Change the partition spec later and the queries keep working, because the projection is computed per spec at plan time.

Stage 2: Data file filtering

For each surviving manifest, the planner reads individual file entries. Every entry carries per-column metrics: lower_bounds, upper_bounds, null_value_counts, nan_value_counts, value_counts, and column_sizes, all keyed by field ID.

InclusiveMetricsEvaluator tests the predicate against those metrics and answers one question: could this file contain a matching row? It returns false only when the metrics prove no match. One detail dominates everything else in this guide:

If a column has no bounds recorded, InclusiveMetricsEvaluator returns true. Missing statistics do not fail loudly. They silently disable file pruning for that column.

Its counterpart, StrictMetricsEvaluator, answers the opposite question: do all rows in this file match? That powers metadata-only deletes and residual computation rather than read pruning.

A note on value_counts, because it is widely misdescribed: it is the total row count for that column including nulls and NaNs, not the count of non-null values. Null-aware pruning comes from comparing null_value_counts against record_count.

Stage 3: Residual evaluation

This stage is invisible in most articles and is the reason partition-aligned filters are effectively free.

Once a file survives, Iceberg computes a residual: the part of the predicate that still needs row-level evaluation after the partition value is taken into account. Take a table partitioned by days(ts) and a query with ts > TIMESTAMP '2026-03-01 08:00:00'.

For a file in partition ts_day = 2026-03-05, strict projection proves every row already satisfies the predicate. The residual collapses to alwaysTrue() and the engine skips per-row filter evaluation entirely.
For the boundary partition ts_day = 2026-03-01, the residual stays ts > '2026-03-01 08:00:00' and must be checked per row.

In Trino's EXPLAIN output you can see this split directly: predicates absorbed into the scan's constraint are enforced by Iceberg and never re-evaluated, while a residual Filter node above the scan is per-row work. If you see your main predicate in the Filter node rather than the constraint, the filter is not partition-aligned.

Stage 4: Inside the Parquet file

Once a file is opened, Iceberg's reader applies three row-group filters in a deliberate order, cheapest first:

ParquetMetricsRowGroupFilter — per-row-group min/max and null counts from the footer.
ParquetDictionaryRowGroupFilter — if the column chunk is dictionary-encoded, the dictionary is an exact membership test. No value in the dictionary means no matching row, with no false positives.
ParquetBloomRowGroupFilter — a probabilistic test, only consulted if the first two could not rule the group out, because it costs an extra read.

Worth knowing precisely: Iceberg's own reader prunes at row-group granularity only. It does not use Parquet's column index / offset index to skip individual pages. Trino uses its own Parquet reader and does apply page-level filtering, which is one real, structural reason the same table can prune differently on Trino than on Spark.

Column projection

Iceberg pushes the projected column set into the reader, fetching only the columns in your SELECT plus those referenced by predicates. Projection matches on stable field IDs, not names, so renames and schema evolution never break it. On a 300-column table, projection alone routinely removes 90%+ of I/O, which is why SELECT * on wide tables is an expensive habit rather than a stylistic one.

Where planning runs, and why it can be the bottleneck

Planning cost scales with the number of manifests opened, not the amount of data returned. Three execution models exist today:

Trino plans on the coordinator, multithreaded. iceberg.planning-threads (default 2× coordinator cores) controls manifest reads and iceberg.split-manager-threads controls split generation. There is no distributed planning, so a badly fragmented table turns the coordinator into a bottleneck for the whole cluster.
Spark supports distributed planning via spark.sql.iceberg.data-planning-mode and spark.sql.iceberg.delete-planning-mode, each accepting AUTO, LOCAL, or DISTRIBUTED and defaulting to AUTO. In DISTRIBUTED mode, manifest scanning becomes a Spark job over the entries metadata table. AUTO decides based on estimated manifest volume.
Server-side scan planning shipped in Iceberg 1.11.0 and is the development most guides still describe as "proposed." The REST catalog spec defines planTableScan, fetchPlanningResult, cancelPlanning, and fetchScanTasks endpoints, with a client-side scan-planning-mode property (client or server) and per-table override via LoadTableResult. Spark can now delegate planning to the catalog. Before you plan around it, check what your catalog actually advertises in GET /v1/config under endpoints; vendor support is uneven and claims run ahead of implementations.

Measure planning before you tune it

Do not guess at which stage is failing. Iceberg emits a ScanReport through its MetricsReporter interface with exactly the fields you need:

Metric	What it tells you
`totalPlanningDuration`	Planning latency, isolated from execution
`scannedDataManifests` / `skippedDataManifests`	Whether Stage 1 is working
`resultDataFiles` / `skippedDataFiles`	Whether Stage 2 is working
`resultDeleteFiles` / `skippedDeleteFiles`	Your merge-on-read burden
`totalFileSizeInBytes`	Bytes the scan committed to reading

The ratio skippedDataManifests / scannedDataManifests is the single best signal for manifest clustering quality, and skippedDataFiles / (skippedDataFiles + resultDataFiles) is the best signal for sort-order quality. In Trino, system.runtime.queries separates analysis_time, planning_time, and execution_time, and EXPLAIN ANALYZE VERBOSE gives you physicalInputBytes per operator. In Spark, the SQL tab's BatchScan node reports files read and files skipped directly.

The Layer Nobody Owns

Here is the structural problem the rest of this guide keeps running into.

Iceberg specifies how pruning works. Engines implement it. Neither decides what your table's physical layout should be. Nothing in the format observes that 80% of production queries filter on region and reorders your files accordingly. Nothing notices that last night's compaction invalidated your Puffin statistics. Nothing tracks that a table's manifest count crossed the threshold where planning latency became user-visible. Those are continuous operational decisions, and the open lakehouse ships them as your problem.

At ten tables you solve it with a Spark job and a cron entry. At a few hundred, across several engines and catalogs, you are running a control loop by hand with no telemetry, and the decisions go stale faster than you can revisit them.

A lakehouse control plane such as LakeOps is the layer that runs that loop. It sits above the stack you already have rather than replacing any of it: your catalogs (AWS Glue, DynamoDB-backed, REST catalogs like Polaris, Nessie, Gravitino, and Lakekeeper, or S3 Tables), your Iceberg tables in object storage, and your query engines. It attaches through standard catalog and Iceberg metadata APIs, so there is no data copy, no pipeline rewrite, and no change to the table format.

The loop it runs is the one the open stack leaves unowned: sense table structure and query telemetry, classify each table's health, plan the maintenance and layout each table needs, execute in dependency order, then measure the result and feed it back into the next decision.

Learn more:

Managed Iceberg Lakehouse | LakeOps

Connect your catalog — every Iceberg table maintained, compacted, and governed across your lakehouse. Trino, Spark, Snowflake, Athena.

lakeops.dev

Iceberg Lakehouse Table Maintenance | LakeOps

Autonomous Iceberg table maintenance for your data lakehouse. Snapshot expiry, orphan cleanup, manifest rewrite, and delete-file resolution — no cron.

lakeops.dev

Later in this guide, once the underlying mechanics are on the table, I will get specific about how that applies to sort-order selection, manifest health, and statistics lifecycle.

Keep reading first. Every fix below is worth understanding whether you automate it or not.

Statistics: The Fuel for Everything

Stage 2 pruning runs entirely on column bounds. If you optimize one thing before touching layout, make it this, because the failure mode is invisible and the fix is a table property.

What gets collected, and what does not

Metrics collection is governed by a small set of table properties:

Property	Default	Effect
`write.metadata.metrics.default`	`truncate(16)`	Mode for all columns without an explicit override
`write.metadata.metrics.column.<col>`	unset	Per-column override; wins over everything
`write.metadata.metrics.max-inferred-column-defaults`	`100`	Cap on how many columns get inferred metrics

Modes are none, counts, truncate(L), and full. And then there is the behavior that catches almost everyone:

The 100-column cap applies only to the inferred default. If you explicitly set write.metadata.metrics.default, the cap is bypassed completely. Setting it to 'truncate(16)' on a 2,000-column table looks like a no-op because it is "the default value," but it switches metrics on for all 2,000 columns and can inflate your manifests by an order of magnitude. This is a common self-inflicted wound: someone "tunes" statistics and makes planning dramatically slower.

Two more details from the implementation that are worth internalizing:

The column count uses projected field IDs, which includes struct fields themselves, not just leaves. A table with 60 leaf columns nested inside 45 structs is already past 100. Selection is pre-order, so it is the columns declared late in a wide schema that silently lose their bounds.
Sort-order columns are automatically promoted to truncate(16) when the effective default is none or counts. So write.metadata.metrics.default = 'none' still keeps bounds on your declared sort keys. Note the asymmetry: Z-order and Hilbert clustering do not get this promotion, because they do not change the table's declared SortOrder.

The configuration that is actually correct

For wide tables, be explicit in both directions. Turn the default off, then turn metrics on for the columns that matter:

ALTER TABLE db.events SET TBLPROPERTIES (
    'write.metadata.metrics.default' = 'none',
    'write.metadata.metrics.column.event_time' = 'truncate(16)',
    'write.metadata.metrics.column.customer_id' = 'full',
    'write.metadata.metrics.column.region' = 'full',
    'write.metadata.metrics.column.order_id' = 'truncate(16)'
);

This is the one case where setting default explicitly is right, precisely because you want to bypass the inference machinery and take control. Use full for short, high-value keys and truncate(16) for strings. Never use full on a column holding URLs, JSON blobs, or stack traces: an untruncated multi-kilobyte bound gets copied into both the lower and upper bound map for every data file, and the manifests become larger than some of the data.

Verifying which columns actually have bounds

Do not assume. Ask the table. Spark's files metadata table exposes a readable_metrics struct that decodes the binary bounds into their real types:

SELECT
    file_path,
    readable_metrics.customer_id.lower_bound AS cid_min,
    readable_metrics.customer_id.upper_bound AS cid_max,
    readable_metrics.customer_id.null_value_count AS cid_nulls,
    record_count,
    file_size_in_bytes
FROM db.events.files
ORDER BY 2
LIMIT 20;

If cid_min and cid_max come back NULL, that column has no metrics and no query filtering on it will ever prune. If they are populated but span nearly the full domain in every row, metrics exist but the data is not clustered. Those are two different problems with two different fixes, and this query distinguishes them in one shot.

To see the raw picture of which field IDs have bounds at all:

SELECT map_keys(lower_bounds) AS fields_with_bounds
FROM db.events.files
LIMIT 1;

Measuring clustering quality

Overlap is what kills Stage 2. A useful single number is how many distinct lower bounds exist relative to file count:

SELECT
    count(*) AS files,
    count(DISTINCT readable_metrics.customer_id.lower_bound) AS distinct_lower_bounds,
    avg(readable_metrics.customer_id.upper_bound
        - readable_metrics.customer_id.lower_bound) AS avg_range_width,
    max(readable_metrics.customer_id.upper_bound)
        - min(readable_metrics.customer_id.lower_bound) AS global_range
FROM db.events.files;

When avg_range_width approaches global_range, every file covers the whole domain and pruning is mathematically impossible. When distinct_lower_bounds is a small fraction of files, many files start at the same value and their ranges overlap heavily. Either result points at the same fix: cluster the data.

Partitioning: The Coarse Prune

Partitioning decides which groups of files can be ignored wholesale. It is the cheapest prune and the one with the most permanent consequences, because the partition spec shapes how every subsequent write lands.

Choosing transforms

Partition on the columns that appear in most query filters. Time nearly always wins, because nearly every analytical query bounds a time range:

CREATE TABLE events (
    event_id    bigint,
    event_time  timestamp,
    region      string,
    customer_id bigint,
    amount      decimal(12,2),
    payload     string
) USING iceberg
PARTITIONED BY (days(event_time));

The available transforms are year, month, day, hour, bucket(N, col), truncate(W, col), and identity. Format v3 adds multi-argument transforms for both partitioning and sorting.

Practical guidance that holds up in production:

Pick granularity from write volume, not from query habit. The target is partitions large enough to hold files near your target file size. A table ingesting 2 GB/day should be partitioned by day, not hour; hourly partitioning gives you 24 partitions of ~85 MB each and you have manufactured a small-file problem.
bucket(N, col) is for high-cardinality equality predicates, typically a join or lookup key: PARTITIONED BY (bucket(64, customer_id)). It also enables storage-partitioned joins on Spark and bucket-aware execution on Trino.
truncate(W, col) is underused for string keys with meaningful prefixes, such as truncate(4, country_code) or a hashed ID prefix.
identity on anything high-cardinality is a trap. PARTITIONED BY (customer_id) on a million customers produces a million partition directories, a million-plus manifest entries, and planning that is slower than the scan you were trying to avoid.

The over-partitioning failure mode

Over-partitioning is the most common self-inflicted Iceberg performance problem, and it is genuinely counterintuitive because each individual partition looks fine. The costs compound in three places at once: more files means more S3 GET requests and more open-file overhead per query; more partition values means more manifest entries to evaluate during planning; and small files means compression ratios and dictionary encoding both degrade, so the bytes on disk grow too.

Check for it directly:

SELECT
    partition,
    count(*)                    AS files,
    avg(file_size_in_bytes)/1048576 AS avg_mb,
    sum(record_count)           AS rows
FROM db.events.files
GROUP BY partition
ORDER BY files DESC
LIMIT 20;

If avg_mb sits well under 100 across most partitions, either the partition granularity is too fine or compaction is not keeping up. Both are fixable, but they are different fixes and the query above tells you which by showing whether the problem is uniform (granularity) or concentrated in recent partitions (compaction lag).

Partition evolution

Iceberg changes the partition spec without rewriting data. Old files keep their old spec; new writes use the new one. The planner evaluates each spec independently at plan time, so both layouts prune correctly and queries need no changes.

ALTER TABLE db.events REPLACE PARTITION FIELD days(event_time) WITH hours(event_time);
ALTER TABLE db.events ADD PARTITION FIELD region;

Two operational caveats. First, a table carrying several specs makes planning modestly more complex and makes partitions metadata harder to reason about, so evolution is not free forever. Second, if you want old data physically moved into the new spec, that is a rewrite_data_files pass with output-spec-id, and it costs a full rewrite of the affected range. Plan it deliberately rather than discovering it on next month's compute bill.

Sort Order and Clustering: Where Queries Are Actually Won

Partitioning determines which file groups exist. Clustering determines whether the files inside those groups are skippable. They are separate layers and collapsing them is the most common conceptual error in Iceberg performance work.

When data is clustered on a predicate column, each file holds a narrow contiguous slice of the value domain, the manifest bounds are tight, and InclusiveMetricsEvaluator eliminates most files. When it is not, every file holds a random sample of the full domain, bounds overlap completely, and Stage 2 becomes a no-op even though the statistics are present and correct.

This is also why binpack compaction is not clustering. Binpack fixes file count. Merging 900 unsorted 5 MB files into 40 unsorted 512 MB files makes your LIST and GET costs better and your planning faster, and changes file-level skipping by approximately nothing. Large unsorted files still fail file pruning. If you have been running scheduled binpack compaction and wondering why query latency never improved, this is why.

Local sort vs global sort: the distinction that decides everything

You can declare a sort order and get zero pruning improvement. This happens constantly and the reason is that where the sort happens matters more than that it happened.

A local sort orders rows within each writer task. You get N files, each internally sorted, whose [lower_bound, upper_bound] ranges overlap heavily, because each task saw a random sample of the data. InclusiveMetricsEvaluator cannot skip any of them for a point lookup. Local sort buys better compression and better in-file row-group pruning. It does not buy file-level skipping.
A global sort shuffles first so that each file receives a disjoint value range. Bounds do not overlap, and file-level skipping works.

The control for this is write.distribution-mode:

Mode	Shuffle	Result
`none`	None	Whatever the upstream partitioning gave you. Risks a small-file explosion unless you pre-sort or enable fanout writes.
`hash`	Hash exchange on the partition key	Each partition value lands in one task, so file counts stay sane. Does not order rows within the file.
`range`	Sample, then range-partition	Global ordering across files. Disjoint bounds. Real file skipping. More expensive to write.

Spark's effective default has been hash since Iceberg 1.2.0, and range when the table has a declared sort order. Per-operation overrides exist as write.delete.distribution-mode, write.update.distribution-mode, and write.merge.distribution-mode.

ALTER TABLE db.events WRITE ORDERED BY customer_id, event_time;

ALTER TABLE db.events SET TBLPROPERTIES (
    'write.distribution-mode' = 'range'
);

One more piece of honesty from the spec: writers should apply the declared sort order but are explicitly not required to, with streaming writes called out as the case where it is legitimately too expensive. A declared sort order is a hint, not a guarantee. Verify it against reality rather than trusting the table property:

SELECT sort_order_id, count(*) AS files, sum(file_size_in_bytes)/1073741824 AS gb
FROM db.events.files
GROUP BY sort_order_id;

sort_order_id = 0 means unsorted. A table that declares a sort order and shows mostly zeros is telling you its writers are ignoring it, and the fix is compaction with an explicit sort strategy rather than a table property change.

Choosing sort keys

Order matters, and it matters more than most teams expect. A linear sort gives the leading column tight, nearly disjoint bounds. The second column is only clustered within runs of equal values in the first, so its bounds are meaningfully wider. By the third or fourth column, the clustering benefit is close to noise.

Query pattern	Recommended layout	Reasoning
One dominant filter column	Linear sort, that column leading	Tightest possible bounds on the column that matters
Entity plus time range (`customer_id` + `event_time`)	`ORDERED BY customer_id, event_time`	Entity clustering first, time ordering inside each entity
Joins on a stable key	Sort by the join key, or `bucket()` partition it	Enables merge joins and storage-partitioned joins
2–4 filter columns used in varying combinations	Z-order	Balanced pruning across dimensions instead of one good dimension and three bad ones
More than 4 filter columns	Pick the top 2–3	Curves dilute; the rest is wishful thinking
High-cardinality point lookups on a non-sort column	Bloom filter, not sort	You only get one leading column; spend it wisely

Z-order, and the gotcha that silently breaks it

Z-order interleaves the bits of several columns into a single Morton code, producing a space-filling curve that clusters data across all of them at once. No single dimension gets bounds as tight as a dedicated linear sort would give it, but every dimension gets usable bounds instead of one winner and several losers.

CALL catalog.system.rewrite_data_files(
    table       => 'db.events',
    strategy    => 'sort',
    sort_order  => 'zorder(customer_id, region, product_id)',
    options     => map(
        'target-file-size-bytes', '536870912',
        'max-concurrent-file-group-rewrites', '10',
        'partial-progress.enabled', 'true'
    )
);

Z-order has two options of its own, and one of them is a landmine:

var-length-contribution (default 8) is the number of bytes taken from each String or Binary column when building the curve. Only the first 8 bytes participate. If your string keys share a long common prefix, and prefixed UUIDs, hierarchical paths, and evt_2026_03_... style identifiers all do, then every value contributes identical bits and Z-order degenerates into expensive noise. Raise the value, or truncate the column to its distinguishing portion first.
max-output-size (default 2147483647) caps total interleaved bytes.

Two more properties of curve clustering worth knowing. It is applied per rewrite and is not persisted as the table's SortOrder, so subsequent ordinary writes will not maintain it. And because the declared sort order is unchanged, Z-ordered columns do not receive the automatic truncate(16) metrics promotion described earlier. If your metrics default is restrictive, set explicit per-column metrics on every column you Z-order, or you will do the clustering work and record no bounds to exploit it.

Hilbert curves: the accurate status

Hilbert curves are a genuine improvement over Z-order for multi-dimensional locality, because consecutive points on a Hilbert curve are always true spatial neighbors, eliminating the long jumps a Morton curve makes across the value space.

Be precise about availability, because the internet is not. As of Iceberg 1.11.0, SparkHilbertUDF exists on the project's main branch under spark/v4.1 only. It is not present in any released version, it is absent from spark/v4.0 and spark/v3.5, and it is documented only in the nightly docs, not the release docs. Backports to Spark 4.0 and 3.5 are tracked as an open issue. An earlier 2022 attempt was closed without merging, so blog posts citing it as evidence of support are wrong.

The nightly documentation records one design difference worth noting for when it does ship: unlike Z-order, every column contributes its full 8-byte primitive width to the Hilbert index, so var-length-contribution and max-output-size do not apply. Hilbert and Z-order also cannot be combined with each other or with plain column sorts in a single sort_order.

For production work today, Z-order is the multi-dimensional option. Plan for Hilbert; do not build on it yet.

One thing no Iceberg property can tell you

Declaring a sort order is a one-line DDL statement. Knowing which columns to declare requires knowing what your queries actually filter on, across every engine, every dbt model, every BI tool, and every notebook that touches the table. Most teams choose a sort order at table creation from intuition and never revisit it, while the query mix drifts underneath them for two years.

That is a telemetry problem, not an Iceberg problem, and it is the first place a control plane earns its keep. Because LakeOps sits above all connected engines rather than inside one, it collects WHERE, JOIN, and GROUP BY column frequency from Trino, Spark, Snowflake, Athena, DuckDB, and Flink together, and scores candidate layouts against the combined mix rather than against whichever engine an individual team happens to use. The mechanics of how that turns into an applied sort order come later in this guide.

File Size and Row Group Size

File sizing is usually discussed as a storage-cost topic. It is also a pruning-resolution topic, and the interaction between file size and row group size sets a hard ceiling on how much data any query can skip inside a file.

The relevant defaults:

Property	Default	Notes
`write.target-file-size-bytes`	536870912 (512 MB)	Compaction target
`write.parquet.row-group-size-bytes`	134217728 (128 MB)	Granularity of in-file skipping
`write.parquet.page-size-bytes`	1048576 (1 MB)
`write.parquet.page-row-limit`	20000	Matters for engines that use the page index
`write.delete.target-file-size-bytes`	67108864 (64 MB)
`read.split.target-size`	134217728 (128 MB)	Scan parallelism unit
`read.split.open-file-cost`	4194304 (4 MB)	Planner's cost model for opening a file

Do the arithmetic on the first two together. A 512 MB file at the default 128 MB row group size contains exactly four row groups, so the best possible in-file skip is 75%, and a point lookup that matches one row still reads a quarter of the file. Drop to 32 MB row groups and you get sixteen groups, finer skipping, a larger footer, and slightly worse compression.

That trade resolves differently by workload, which is the actual guidance:

Scan-heavy analytical tables: keep 128 MB row groups. You are reading most of the file anyway and you want the compression and the low footer overhead.
Point-lookup and selective-filter tables: 32–64 MB row groups genuinely help, and they compound with bloom filters, since both operate at row-group granularity.
Streaming and CDC tables: smaller target files (128–256 MB) reduce write amplification during compaction, at the cost of more files.

Two more layers of parallelism control sit on top. read.split.target-size determines how a large file is divided across tasks, and it uses the split_offsets recorded in each manifest entry, which are the Parquet row group boundaries, so splitting does not require reading footers at plan time. Iceberg 1.11.0 added adaptive split sizing for Spark 4.1 through spark.sql.iceberg.read.adaptive-split-size.enabled, plus a session-level spark.sql.iceberg.read.split-size override. Adaptive sizing fixes the chronic case where a medium scan produces ten splits on a two-hundred-core cluster.

If you use AQE to control output file sizes during writes, note that spark.sql.adaptive.advisoryPartitionSizeInBytes is measured as in-memory row size, not on-disk compressed size. It must be set larger than write.target-file-size-bytes by a data-dependent ratio, and Spark cannot write a file larger than a single task or spanning an Iceberg partition boundary.

Bloom Filters: The Point-Lookup Escape Hatch

Bloom filters solve the one case that min/max statistics structurally cannot: equality predicates on a high-cardinality column that is not your sort key. A trace_id or user_id column with millions of scattered distinct values has bounds spanning the whole domain in every file. Sorting on it would fix pruning but you only get one leading sort column, and it is usually already spent.

A Parquet bloom filter is a compact probabilistic structure per row group. It answers "definitely not present" with certainty and "possibly present" with a tunable false-positive rate. False negatives cannot occur, so it is always safe.

Configuration

ALTER TABLE db.events SET TBLPROPERTIES (
    'write.parquet.bloom-filter-enabled.column.trace_id' = 'true',
    'write.parquet.bloom-filter-fpp.column.trace_id'     = '0.01',
    'write.parquet.bloom-filter-ndv.column.trace_id'     = '5000000',
    'write.parquet.bloom-filter-max-bytes'               = '1048576'
);

bloom-filter-ndv is the property most guides omit and the one that decides whether the filter works. Without an expected distinct-value count, Parquet has to infer the bitset size, and you routinely end up either with a filter so small its false-positive rate approaches 100% (making it useless) or one clipped by bloom-filter-max-bytes (same outcome, wasted bytes). Set it to a realistic per-row-group cardinality estimate. There is also an undocumented write.parquet.bloom-filter-adaptive-enabled (default false) present in TableProperties but absent from the published configuration table.

These properties affect newly written files only. Existing files need a rewrite pass to gain filters:

CALL catalog.system.rewrite_data_files(
    table    => 'db.events',
    strategy => 'binpack',
    where    => 'event_time >= TIMESTAMP ''2026-08-01''',
    options  => map('rewrite-all', 'true')
);

When they help, and when they are pure overhead

Scenario	Useful?	Why
`WHERE trace_id = 'a7f...'`, high cardinality, unsorted	Yes	Min/max is useless; the bloom filter is the only mechanism left
`WHERE status = 'active'`, low cardinality	No	Dictionary filtering already handles this exactly, with no false positives
`WHERE amount > 500`	No	Bloom filters only answer equality and `IN`
`WHERE customer_id = 481` on the leading sort column	No	Min/max already prunes; the filter is redundant footer bloat
`WHERE user_id IN (...)` from a runtime join filter	Yes	Spark's runtime filtering pushes these down as equality sets

The sweet spot is two or three columns. Enabling bloom filters across a wide schema bloats footers, slows every write, and adds a read cost on every row group that the first two filter stages could not already resolve.

Which engines read them

Iceberg's own reader (Spark via the Iceberg data source, Flink, generic readers) uses ParquetBloomRowGroupFilter.
Trino reads them too, which is often assumed otherwise. The catalog property is parquet.use-bloom-filter, default true, with session property parquet_use_bloom_filter. Trino can also write them for Iceberg tables via the parquet_bloom_filter_columns table property.

Verify empirically rather than trusting either statement: run EXPLAIN ANALYZE on Trino and compare physicalInputBytes before and after writing filters. If the number does not move, the filter is not being consulted for your predicate shape.

Puffin, NDV, and Cost-Based Optimization

Everything above optimizes scanning. Puffin statistics optimize planning decisions: join order, join strategy, and aggregation sizing. On a three-way join, the difference between the best and worst join order can be two orders of magnitude in intermediate data volume, and no amount of file skipping recovers from a plan that builds a billion-row hash table first.

What is actually in a Puffin file

Correcting a persistent misconception: Puffin does not store column min/max bounds. Those live in manifest entries, as covered above. The Puffin spec standardizes exactly two blob types:

apache-datasketches-theta-v1 — an Apache DataSketches Theta sketch for NDV (number of distinct values). The estimate is carried in the blob's ndv property.
deletion-vector-v1 — the format v3 Roaring bitmap deletion vector, discussed in the merge-on-read section below.

Collecting them

-- Spark
CALL catalog.system.compute_table_stats(
    table   => 'db.events',
    columns => array('customer_id', 'region', 'product_id')
);

-- Trino
ANALYZE db.events WITH (columns = ARRAY['customer_id', 'region']);

The three ways this silently fails

1. Spark computes the stats and nothing uses them. spark.sql.iceberg.report-column-stats defaults to true, so Iceberg does report them. But Spark's cost-based optimizer is gated on spark.sql.cbo.enabled, which defaults to false. You compute sketches, Iceberg hands them over, and the optimizer ignores them. This is the single most common "statistics don't work" report, and the fix is one config:

spark.sql.cbo.enabled=true
spark.sql.cbo.joinReorder.enabled=true

2. Trino ignores multi-column sketches. Trino's reader filters for the standard theta blob type and requires fields().size() == 1, so any sketch covering multiple columns is skipped entirely. It is gated by iceberg.table-statistics-enabled (default true) and maintained on write by iceberg.extended-statistics.collect-on-write (default true). Re-analyzing a subset of columns after a full ANALYZE requires calling drop_extended_stats first, or the new run is rejected.

3. There is no invalidation mechanism at all. This is the design weakness and it deserves emphasis. A Puffin blob pins the snapshot-id it was computed from, and engines walk the snapshot ancestry to find the most recent statistics file. Nothing marks it stale. A sketch computed at snapshot N is served unchanged at snapshot N+10,000, so if the table doubled in size the optimizer is planning against fiction. Worse, a statistics file whose snapshot is no longer an ancestor, after a rollback or a branch rewrite, is simply not found, and the table silently drops to no statistics with no error.

Compaction makes this acute: rewriting files changes the cardinality landscape the sketches describe. Treat statistics refresh as a mandatory step in your maintenance sequence, not an occasional chore. In Trino, SHOW STATS FOR db.events tells you what the optimizer currently believes; compare it against reality before you debug a bad plan. The Puffin statistics lifecycle has more on incremental sketch computation, which matters once full recomputation costs more than the compaction that triggered it.

Manifest Health and Planning Latency

Manifest fragmentation is the most overlooked cause of slow Iceberg queries, because the symptom does not look like a data problem. Queries that should plan in milliseconds take seconds, with perfect partitioning and a good sort order, and the time disappears before any data file is touched.

How fragmentation happens

Every commit, whether append, overwrite, delete, or compaction, produces at least one new manifest. Streaming, micro-batch, and CDC tables accumulate thousands of them, each tracking a handful of files. The planner must open and evaluate every manifest in the snapshot. More manifests means more metadata round trips to object storage, more Avro parsing, and a longer serial-ish critical path before execution begins.

SELECT
    partition_spec_id,
    count(*)                                          AS manifests,
    sum(added_files_count + existing_files_count)     AS files,
    avg(length)/1048576                               AS avg_manifest_mb
FROM db.events.manifests
GROUP BY partition_spec_id;

Manifests well under commit.manifest.target-size-bytes (8 MB) mean fragmentation. Hundreds or thousands of manifests on a table with moderate file counts means your planning time is dominated by metadata I/O.

Rewriting manifests, and the `sort_by` argument almost nobody uses

Manifest rewriting is metadata-only. It does not read or write data files, which makes it the cheapest maintenance operation available with the largest effect on planning latency.

CALL catalog.system.rewrite_manifests(
    table       => 'db.events',
    sort_by     => array('event_time_day'),
    use_caching => false
);

sort_by arrived in Iceberg 1.11.0 and takes an array of partition transform names, as documented in the Spark procedures reference. It is arguably the highest-leverage planning knob in Iceberg and it is barely known, mainly because the default behavior sounds harmless and is not.

The default sorts manifests by all partition transforms in spec order. Consider a table partitioned by (region, day(ts)) where every production query filters on time and no query filters on region. The default clusters manifests by region first, so each manifest ends up containing files spanning the full time range. The manifest-list field_summary bounds on ts_day overlap across every manifest, and ManifestEvaluator prunes nothing at Stage 1. Passing sort_by => array('ts_day') reorders the clustering to match how the table is actually queried, and Stage 1 starts working.

You can see the effect directly in the ScanReport: skippedDataManifests goes from near zero to most of the manifest count.

Trino's equivalent is ALTER TABLE db.events EXECUTE optimize_manifests, which clusters manifests by partitioning columns and reports rewritten_manifests_count, added_manifests_count, kept_manifests_count, and processed_manifest_entries_count.

Sequencing, and why it is not arbitrary

Maintenance operations feed each other, and running them in the wrong order wastes compute or produces metadata describing files that no longer exist. The correct sequence:

Expire snapshots beyond your retention window. This shrinks the file set every later step has to consider, and it is why expiry goes first: compacting files that are about to be garbage-collected is pure waste.
Remove orphan files left by failed writes and by the snapshots you just expired. Use a grace period, never a zero threshold, or you will race in-flight writers.
Compact data files, applying the sort or curve strategy, ideally scoped to the partitions that need it rather than the whole table.
Rewrite manifests, so the metadata tree indexes the layout that now exists rather than the one that did.
Refresh Puffin statistics, so the optimizer's cardinality estimates match the files it will actually read.

Run step 4 before step 3 and your manifests describe a dead file set. Skip step 5 and your CBO plans against a pre-compaction world, which is arguably worse than having no statistics at all, because a confidently wrong estimate produces a confidently wrong plan.

Merge-on-Read: The Delete Tax

On merge-on-read tables, deletes are recorded rather than applied, and every reader pays to reconcile them. This is a read-performance topic that gets filed under "write strategy" and then surprises people when a table that was fast last quarter is not.

Position deletes and the shape of the cost

The v2 position delete file records (file_path, position) pairs, sorted by both so readers can push down by file and stream the merge without holding deletes in memory. The critical property is that cost scales with the number of delete files, not the number of deleted rows. A table with 50 uncompacted small delete files means a full scan opens 51 files where it should open one.

Published benchmarks on position-delete overhead show the cost curve jumping roughly 4× between 0% and 1% deleted rows, then rising only gradually from 1% to 100%. The fixed cost of having any delete files dominates the marginal cost of having more deleted rows. That shape is the argument for compacting on file count aggressively rather than waiting for a deletion-ratio threshold to trip.

Equality deletes, and why they are much worse

An equality delete file records the values of identifier fields rather than positions. That makes writes cheap, which is exactly why Flink's streaming upsert mode defaults to them: locating (file_path, pos) would require a read-before-write at CDC rates.

The cost lands entirely on readers, and two scoping rules make it severe. First, an equality delete applies when the data file's sequence number is strictly less than the delete's, regardless of locality. Second, and this is the one that bites, if the delete file's partition spec is unpartitioned, it is a global delete applied against every data file in the table.

Equality delete files do carry their own metrics, so InclusiveMetricsEvaluator can prune them by identifier bounds. Most streaming writers do not optimize for tight bounds, which means most real deployments do not get that benefit.

Check where you stand:

SELECT
    content,
    file_format,
    count(*)                          AS files,
    sum(record_count)                 AS deleted_rows,
    sum(file_size_in_bytes)/1048576   AS mb
FROM db.events.delete_files
GROUP BY content, file_format;
-- content = 1 -> position deletes or deletion vectors
-- content = 2 -> equality deletes
-- file_format = 'puffin' -> v3 deletion vectors

Any content = 2 rows on a table that is not fed by Flink is a red flag worth chasing down.

Controls

ALTER TABLE db.events SET TBLPROPERTIES (
    'write.delete.mode'         = 'merge-on-read',
    'write.update.mode'         = 'merge-on-read',
    'write.merge.mode'          = 'copy-on-write',
    'write.delete.granularity'  = 'file'
);

All three mode properties default to copy-on-write. write.delete.granularity defaults to partition at the table level; setting it to file produces one delete file per data file, which is more files but each is precisely scoped and converts to a deletion vector without a merge step.

For compaction, the two relevant rewrite_data_files options are delete-file-threshold (default 2147483647, effectively off) and delete-ratio-threshold (default 0.3). Given the cost curve above, delete-file-threshold is the more useful lever. Set it low, around 2 or 3, so any data file accumulating delete files gets rewritten rather than waiting for 30% of its rows to be deleted:

CALL catalog.system.rewrite_data_files(
    table   => 'db.events',
    options => map(
        'delete-file-threshold', '2',
        'remove-dangling-deletes', 'true',
        'partial-progress.enabled', 'true',
        'partial-progress.max-commits', '20'
    )
);

Note that remove-dangling-deletes costs an extra commit and, per the documentation, works solely from data sequence numbers. It does not cover global equality deletes, invalid equality deletes, or multi-file position delete files.

What Format v3 Changes for Reads

Format v3 is complete and adopted. Version 4 is under active development and has not been formally adopted. For query performance specifically, v3 matters in four places.

Deletion vectors replace position delete files. The spec allows at most one DV per data file per snapshot, enforced by writers merging on commit. That converts delete application from "open N delete files and sort-merge by (file_path, pos)" into "load one Roaring bitmap, test a bit." Manifests carry referenced_data_file, content_offset, and content_size_in_bytes, so the reader issues a byte-range request for exactly the DV blob with no Puffin footer scan. Position delete files are deprecated in v3 and must be folded into a DV when one is created.

Variant shredding turns semi-structured data into something prunable. With write.parquet.shred-variants (default false), variant sub-fields become real Parquet columns with real statistics, and manifest bounds for variant columns are keyed by normalized JSON path such as $['location']['latitude']. That is file skipping on JSON payloads, which was previously impossible. Bounds are not written for mixed-type fields.

Geospatial bounding boxes give geometry and geography columns real lower and upper bounds as X/Y/Z/M points, with intersects checking added in 1.11.0. geography permits xmin > xmax to express antimeridian crossing.

Row lineage (_row_id, _last_updated_sequence_number) is always on for v3, not optional. It is inheritance-based, so readers materialize the values from first_row_id + _pos and the manifest sequence number. Cheap, but it is work engines must implement, and coverage arrived incrementally across 1.9 through 1.11.

The engine support matrix, and why it is a one-way door

Upgrading to v3 is a single atomic metadata change with no data rewrite. Rolling back, once deletion vectors have been written, is not practical. Check your entire read path first.

Engine	v3	Deletion vectors	Notes
Spark (Iceberg 1.9+)	Yes	Yes	Can rewrite v2 deletes into v3 DVs
Flink	Yes	Yes	`IcebergSink` writes DVs as of 1.11.0
Trino 483	Experimental	Read only	Row-level updates, deletes, and `OPTIMIZE` are not supported on v3
Snowflake	Yes (GA May 2026)	Read and write	Managed and externally managed tables
Amazon Athena	No	No	Creates and operates on v2 only; v3 errors out
DuckDB 1.5.3+	Yes	Read and write	Selects DV vs positional automatically from `format-version`
StarRocks 4.1	Partial	No	Supports defaults and row lineage; fails fast on DVs, encryption, and geo types
ClickHouse	No	No	v1 and v2 only

The practical reading: if Athena, StarRocks, or ClickHouse are anywhere in your read path, v3 breaks them. StarRocks' fail-fast behavior is deliberate and was introduced after cached DVs produced silently wrong results, which is a good reminder that "unsupported" is safer than "partially supported."

One forward-looking note worth tracking. Today's manifest metrics are stored as map<int, binary>, so every bound is length-prefixed binary that must be deserialized per value and the untyped map prevents Avro from projecting individual columns out of it. Version 4 moves these into a typed content_stats struct with deterministic field IDs, which would make column projection over manifest statistics possible: a planner filtering on one column could read only that column's stats instead of deserializing every bound in the file. It also adds tight_bounds (a boolean asserting the bounds are exact, enabling metadata-only MIN/MAX) and avg_value_size_in_bytes for memory estimation. Foundational types for this landed in 1.11.0.

Related and immediately relevant: Iceberg 1.11.0 disabled min/max aggregation pushdown for string and binary columns. It was a correctness fix, because truncated bounds produced wrong answers. If you documented metadata-only MIN(varchar_col) as a trick, it now reads data.

Engine-Specific Tuning

Table-level optimization benefits every engine that reads the table. Engine tuning is the layer on top, and the defaults are frequently wrong for lakehouse workloads.

Trino

These are the properties from the Iceberg connector docs that actually move query time.

Property	Default	Why it matters
`iceberg.dynamic-filtering.wait-timeout`	`1s`	How long split generation waits for join-side filters to arrive. Too short and dynamic filtering silently does nothing on large builds.
`iceberg.max-split-size`	unset → table's `read.split.target-size`	Note the name; it is not `iceberg.split-size`. Session override is `experimental_split_size`.
`iceberg.parquet-footer-cache.type`	`none`	Off by default. Setting it to `memory` is a free latency win on high-file-count tables.
`iceberg.metadata-cache.enabled`	`true`	Coordinator-side. Deactivated when `fs.cache.enabled = true`.
`iceberg.query-partition-filter-required`	`false`	Only enforced for schemas listed in `iceberg.query-partition-filter-required-schemas`. Setting the boolean alone does nothing.
`iceberg.planning-threads`	2× coordinator cores	Manifest read parallelism
`iceberg.bucket-execution`	`true`	Trino's answer to storage-partitioned joins on bucketed tables
`iceberg.minimum-assigned-split-weight`	`0.05`	Raise for skewed aggregations; lower for many-small-file tables
`parquet.use-bloom-filter`	`true`	Bloom filters are consulted by default
`parquet.ignore-statistics`	`false`	Escape hatch when a writer produced corrupt stats

Dynamic filtering is Trino's highest-value Iceberg feature and the one most affected by that first timeout. Trino collects values from the build side of a join at runtime and pushes them into the Iceberg scan, pruning files by join key rather than only by WHERE predicate. On a star schema, a small dimension filter can eliminate most of the fact table. If the build side takes longer than the wait timeout to materialize, splits are generated without the filter and the benefit vanishes.

File caching is separate from metadata caching and requires more setup than a single flag: fs.cache.enabled (default false), fs.cache.directories (must exist on the coordinator and every worker), and one of fs.cache.max-sizes or fs.cache.max-disk-usage-percentages. Node affinity comes from node-scheduler.cache-preferred-hosts-count (default 2) in the coordinator config; raising it spreads load and adds resilience at the cost of effective cache size. Each catalog needs its own cache directory.

Spark

Runtime filtering is Iceberg's implementation of dynamic partition pruning, via SupportsRuntimeV2Filtering. Spark re-plans the scan after the build side materializes, and Iceberg re-runs InclusiveMetricsEvaluator with the runtime value set. The underappreciated part: this works on any column with bounds, not just partition columns, which makes it strictly more capable than Hive-style DPP.

Storage-partitioned joins eliminate the shuffle entirely when both sides are bucketed on the join key. Getting them to actually engage requires more than the one flag most guides cite:

spark.sql.sources.v2.bucketing.enabled=true
spark.sql.iceberg.planning.preserve-data-grouping=true
spark.sql.sources.v2.bucketing.pushPartValues.enabled=true
spark.sql.requireAllClusterKeysForCoPartition=false
spark.sql.sources.v2.bucketing.partiallyClusteredDistribution.enabled=true
spark.sql.autoBroadcastJoinThreshold=-1

Three failure modes from real deployments. preserve-data-grouping defaults to false and is the one people forget, so the join silently falls back to a shuffle. Partition columns must appear in the join condition or you get an UnsupportedOperationException about retrieving values from an empty struct. And MERGE INTO with a WHEN NOT MATCHED clause still requires a full shuffle and can OOM under skew; the documented workaround is splitting matched and not-matched into two statements.

Other Spark settings with real read impact:

Config	Default	Effect
`spark.sql.iceberg.vectorization.enabled`	table default (Parquet: `true`)	Vectorized Parquet reads
`spark.sql.iceberg.data-planning-mode`	`AUTO`	`LOCAL` / `DISTRIBUTED` manifest scanning
`spark.sql.iceberg.executor-cache.enabled`	`true`	Caches delete files on executors so the same DV is not re-read by every task
`spark.sql.iceberg.executor-cache.max-entry-size`	64 MB	Silently excludes large equality delete files
`spark.sql.iceberg.aggregate-push-down.enabled`	`true`	Metadata-only aggregates where bounds permit
`spark.sql.iceberg.report-column-stats`	`true`	Reports Puffin stats; needs `spark.sql.cbo.enabled` to matter

The executor cache is the quietly valuable one on merge-on-read tables with concentrated deletes, and the 64 MB per-entry cap is the reason it sometimes appears not to work.

Everything else

Snowflake reached v3 GA in May 2026 with DV read and write on both managed and externally managed tables. Its internal pruning and caching behavior is proprietary and unpublished, so treat claims about how it layers on Iceberg statistics with suspicion, including this one.
Amazon Athena is v2-only. Its documentation cites an Iceberg library version, which is frequently misread as a format version; do not conflate them.
DuckDB 1.5.3+ has full v3 read and write support including DVs, variant, and row lineage, and it is genuinely excellent for single-node work on well-compacted tables. iceberg_metadata() is a fast way to inspect layout without a cluster.
StarRocks 4.1 adds native Iceberg DELETE, global late materialization for v3, and a metadata cache covering tables, partition names, data files, and delete files.
ClickHouse does not enable use_iceberg_partition_pruning by default. Setting it to 1 is an easy and commonly missed win.

The point that survives all of this: table-level layout, statistics, and manifest health benefit every engine simultaneously. Engine tuning benefits one. Fix the table first.

A Runbook for One Slow Query

When a specific query is slower than it should be, work down the funnel in order. Each step either finds the problem or rules out a stage.

1. Split planning from execution

In Trino, check planning_time against execution_time in system.runtime.queries, or read the query detail page. In Spark, look at the gap between job submission and first task launch. If planning dominates, skip to step 4. If execution dominates, the problem is how much data you are reading, so continue.

2. Confirm the predicate is reaching the scan

Run EXPLAIN and look at where your predicate landed. In Trino, a predicate absorbed into the scan's constraint is enforced by Iceberg; one sitting in a Filter node above the scan is evaluated per row after the data is read. A filter wrapped in a function the engine cannot push down, an implicit cast between timestamp precisions, or a comparison against a non-deterministic expression will all quietly demote your predicate to a post-scan filter.

3. Check whether the filter column has usable bounds

SELECT
    count(*) AS files,
    count(readable_metrics.customer_id.lower_bound) AS files_with_bounds,
    count(DISTINCT readable_metrics.customer_id.lower_bound) AS distinct_lowers
FROM db.events.files;

files_with_bounds well below files means a metrics configuration problem: fix it with write.metadata.metrics.column.<col> and a rewrite. distinct_lowers well below files means a clustering problem: fix it with a sorted compaction.

4. Check manifest count and clustering

SELECT count(*) AS manifests, avg(length)/1048576 AS avg_mb
FROM db.events.manifests;

Hundreds of manifests averaging well under 8 MB means fragmentation. Run rewrite_manifests, and pass sort_by naming the transform your queries actually filter on rather than accepting the spec-order default.

5. Check the delete burden

SELECT content, count(*) AS files, sum(record_count) AS rows
FROM db.events.delete_files
GROUP BY content;

High delete file counts, or any equality deletes outside a Flink pipeline, mean every scan is paying reconciliation cost. Compact with a low delete-file-threshold.

6. Check partition and file size distribution

SELECT
    partition,
    count(*) AS files,
    avg(file_size_in_bytes)/1048576 AS avg_mb
FROM db.events.files
GROUP BY partition
ORDER BY files DESC
LIMIT 20;

Thousands of tiny files in a few partitions means compaction lag. Tiny files uniformly across all partitions means the partition spec is too fine.

7. Only now, tune the engine

Dynamic filtering timeouts, footer caches, split sizes, and join configs are real wins, but they are multipliers on a healthy table. Applying them first means tuning the constant factor on an exponential problem.

How a Control Plane Runs This Loop

Everything above is tractable by hand on ten tables. The structure of the problem changes as the lake grows, and it changes in a specific way worth naming: each individual decision stays easy, but the number of decisions grows with tables × engines × time, and every one of them expires.

Consider what "keep sort orders correct" actually requires at a few hundred tables. You need query telemetry from every engine, because the optimal layout for a table depends on how Trino queries it, how Spark ETL writes to it, and how a Snowflake dashboard reads it, simultaneously, and no single engine has that view. You need to score candidate layouts against the combined mix. You need to validate a proposed layout before rewriting terabytes, because a sort order change is expensive to reverse. You need to sequence expiry, cleanup, compaction, manifest rewrite, and statistics refresh correctly per table. You need to avoid compacting a partition with an active writer, and handle optimistic-concurrency conflicts when you lose the race anyway. And you need to redo all of it when the query mix shifts.

That is a control loop, and LakeOps exists to run it as one. Here is what that means concretely for the specific problems in this guide.

Apache Iceberg Lakehouse Control Plane | LakeOps

Manage your Iceberg lakehouse from one control plane — health-driven compaction, table maintenance, multi-engine routing, and observability.

lakeops.dev

Sort order selected from telemetry, not from intuition

The hardest question in this guide is which columns to cluster on. LakeOps answers it from observation: it collects SELECT, WHERE, JOIN, and GROUP BY column frequency across the connected engines, builds a field-access profile per table, and evaluates single-column sort, multi-column sort, and Z-order strategies independently against that profile rather than applying one policy lake-wide. When a new dashboard ships that filters on a different column, the profile shifts and the next compaction cycle applies the updated layout. The query performance results published for this are 51% less data scanned per query, 12× faster queries, and 76% less CPU.

Iceberg Lakehouse Governance | LakeOps

Iceberg lakehouse governance — compliance automation, declarative policies, rollback safety, and audit trails across every catalog and engine.

lakeops.dev

Layout changes validated on a branch before production files move

This is the part that makes automated layout changes defensible rather than reckless. Rewriting every data file in scope is expensive and awkward to undo, so the decision needs evidence before it is applied, not after.

LakeOps runs layout simulations on Iceberg branches: it branches from the latest snapshot, applies a candidate layout, replays observed production query patterns against it, compares scan reduction and file distribution to the baseline, and discards the branch. Only the strategy that measurably wins gets applied to production. Iceberg's branching model is what makes this possible at all, and it is a good example of a format capability that most teams never operationalize because wiring it up is more work than the layout question seemed to be worth.

Iceberg Lakehouse Observability | LakeOps

Iceberg table health scores, cross-engine monitoring, and proactive alerts for your data lakehouse. Track file layout, snapshot depth, and partition skew.

lakeops.dev

Compaction on a dedicated engine rather than a Spark cluster

Sorted compaction is the expensive half of everything above, and running it on Spark means cluster startup, idle capacity, GC pressure, and OOM risk on large rewrites. LakeOps runs compaction on a Rust and DataFusion engine instead, with conflict-aware Iceberg commits, bounded memory with disk spill, and partitions that have active writers excluded from the rewrite set. Its published benchmark compacts 200 GB in 221 seconds against 1,612 seconds for Spark on the same hardware, at a peak throughput of 2,522 MB/s. The compaction strategy deep dive covers how binpack, sort, and Z-order selection differ per table.

Apache Iceberg Lakehouse Compaction | LakeOps

Query-aware Iceberg compaction on a Rust engine. Sorts Parquet files by production query patterns. 95% faster than Spark. Lakehouse ready.

lakeops.dev

The maintenance sequence, executed in dependency order

The five-step sequence earlier in this guide is not hard to understand and is hard to hold correct across hundreds of tables with different retention windows, write patterns, and SLAs. LakeOps sequences expiry, orphan cleanup, sorted compaction, manifest rewrite, and statistics refresh automatically, per table, with each step's output feeding the next. Puffin NDV sketches are recomputed after the file layout changes rather than on a fixed schedule, which is the only way the staleness problem described earlier stays solved rather than solved once.

Health classification, so you know which tables to care about

Rather than a schedule, maintenance triggers on structural signals: file count and size distribution, manifest depth, snapshot accumulation, delete-file ratio, partition skew, and sort-order alignment with observed query patterns. That last signal is the one that has no equivalent in schedule-driven maintenance, because it is not a property of the table at all. It is a property of the relationship between the table and its workload, and it can degrade without a single byte being written. Lake-wide observability surfaces it alongside the structural signals so a table whose layout has drifted out of alignment shows up before users file a ticket.

Routing, which closes the loop

Physical layout determines which engines are viable for a query. A well-clustered table with tight bounds makes a DuckDB point lookup faster and cheaper than the same query on a distributed engine; a table needing a full scan does not. LakeOps routes queries from a single SQL endpoint to the engine best suited to each query shape, and those routing decisions feed back into the same telemetry that drives layout, so agent and application query patterns inform the next sort-order decision. The platform overview covers how the observe, maintain, compact, govern, and route capabilities compose.

Routing Multiple Query Engines with Iceberg - LakeOps Blog

Route queries across Trino, Spark, DuckDB, Athena, and more on shared Iceberg tables. Routing architecture, strategies, dialect translation, and patterns.

lakeops.dev

Iceberg Lakehouse Multi-Engine Query Routing | LakeOps

One SQL endpoint for your lakehouse. Route Iceberg queries to Trino, Spark, DuckDB, Snowflake, or Athena by cost, latency, or workload.

lakeops.dev

None of this is a substitute for understanding the mechanics. It is a substitute for running the loop by hand, on a schedule you set once and never revisited, across a lake that changed underneath you.

Optimization Checklist

Lever	Typical impact	When to apply
Verify metrics exist on filter columns	Enables Stage 2 at all	First. Before any layout work.
Partition on time, plus one high-value dimension	90–99% coarse elimination	Table creation; evolve as query patterns shift
Global sort on the dominant filter column	Large improvement in file-level skipping	Once you know the dominant predicate
`write.distribution-mode = range`	Turns a declared sort into real disjoint bounds	Whenever a sort order is declared
Z-order on 2–4 columns	Balanced pruning across dimensions	Varied multi-column filters; check `var-length-contribution` on string keys
Bloom filters on 2–3 high-cardinality keys	Decisive for point lookups	Equality predicates on non-sort-key columns; always set `bloom-filter-ndv`
Row group size 32–64 MB	Finer in-file skipping	Selective and point-lookup workloads only
`rewrite_manifests` with `sort_by`	Seconds to milliseconds of planning	Manifest count in the hundreds, or `skippedDataManifests` near zero
Compact with low `delete-file-threshold`	Removes merge-on-read reconciliation cost	Any MoR table; urgently if equality deletes are present
Refresh Puffin stats after compaction	Correct join orders	Every time the file layout changes
Enable `spark.sql.cbo.enabled`	Makes collected stats actually count	If you compute stats and see no plan change
Project only needed columns	Linear I/O reduction	Always on wide tables

The Takeaway

Iceberg query performance is not an engine property. It is a function of how much data the planner is able to discard, and every layer of that decision is something you control at write and maintenance time: whether the statistics exist, whether the bounds are tight enough to be useful, whether the manifests are clustered the way your queries filter, whether the deletes have been applied, and whether the optimizer's cardinality estimates describe the table that exists today.

Those layers compound rather than add. A table with sensible partitioning, globally sorted data, bloom filters on its lookup keys, consolidated manifests sorted on the hot transform, resolved deletes, and current statistics hands the engine everything it needs to eliminate 99% of the work before execution starts. The same table missing any one of them can lose most of that benefit, which is why the diagnostic order in the runbook matters more than any individual knob.

The engineering is the same whether you run it with carefully sequenced Spark procedures or hand the loop to a control plane like LakeOps that senses query patterns, plans layout per table, validates on a branch, and measures the result. Make data skipping work at every level, keep the metadata honest, and make sure the statistics still describe reality.

Apache Iceberg Governance: Access Control, Policies, and Audit for Open Lakehouses

joni sar — Wed, 23 Sep 2026 09:45:20 +0000

Governance in an Apache Iceberg lakehouse is not a feature you enable. It is an architecture you assemble.

In a data warehouse — Snowflake, BigQuery, Redshift — governance is built in. The vendor controls access, enforces policies, manages retention, and produces audit trails. You configure roles and trust the system. The trade-off is lock-in: your data lives in a proprietary format, governed by a single vendor's rules.

Iceberg inverts this. The table format is open, engine-agnostic, and deliberately governance-unaware. Any engine can read any table. No single system owns access control. That openness creates a governance vacuum that you have to fill with a stack of purpose-built tools — and filling it correctly is one of the hardest operational problems in production lakehouses.

This guide covers the governance tools, patterns, and architectural layers that production Iceberg lakehouses need — from catalog-level RBAC and credential vending to the new Read Restrictions spec, policy engines, compliance patterns, and the operational governance layer that most teams overlook.

Why Iceberg Governance Works Differently

The Iceberg community made a deliberate choice: the table format is responsible for data portability, not governance. Governance is someone else's job.

Three reasons this is the right boundary:

Portability would be destroyed. If Iceberg metadata included ACLs and masking rules, every engine would need to implement the same governance model — the same RBAC structure, the same masking functions, the same row-filter syntax. Any engine that doesn't implement the full model either breaks compatibility or silently bypasses security.

Evolution would be frozen. Governance requirements change faster than storage formats. GDPR, CCPA, DORA, the AI Act — regulatory regimes evolve, classification taxonomies expand, organizational structures shift. Encoding governance in metadata means every policy change requires metadata migration across potentially thousands of tables.

Enforcement would be unverifiable. When rules live in the format, enforcement depends on the reader. A well-behaved Spark job respects the ACL. A misconfigured Trino deployment ignores it. A custom Parquet reader bypasses it entirely. There is no enforcement authority — only conventions that cooperative engines follow and adversarial engines ignore.

The result: governance in an Iceberg lakehouse must be decomposed into separate, independently evolvable layers. The ecosystem has converged on three.

The Three-Layer Governance Model

Every production Iceberg governance architecture decomposes into three layers:

Layer	Responsibility	Tools
Table format	Data portability and structural integrity	Apache Iceberg (metadata, snapshots, schema)
Catalog	Enforcement and coordination	Apache Polaris, Lakekeeper, Nessie, Gravitino, AWS Glue, Unity Catalog
Policy engine	Rules, classification, and context	OPA, Cedar, OpenFGA, Apache Ranger, cloud IAM

The format defines what the data looks like — how data files, manifests, snapshots, and metadata are organized. It carries structural metadata that governance systems can leverage (column names for classification, snapshot history for audit, table properties for labels), but it never interprets these as rules.

The catalog enforces governance. Every operation — table creation, schema alteration, data read, snapshot commit — routes through the catalog. This makes the catalog the mandatory intermediary between engines and data. If the catalog denies access, the engine never sees the metadata, let alone the data files.

The policy engine defines the rules the catalog enforces. This separation is critical: the catalog is the enforcement mechanism, but it should not be the rule-authoring system. Policy engines are purpose-built for defining, versioning, auditing, and distributing access policies.

The three-layer model means you can swap Ranger for OPA or Cedar without changing the catalog. You can migrate from Glue to Polaris without rewriting your policies. You can upgrade from Iceberg v2 to v3 without touching your governance model. Each layer evolves independently — which is exactly the modularity that makes open lakehouses viable in the first place.

But governance alone does not guarantee that data is actually usable. A table that passes every access control check but takes 45 minutes to query because of 200,000 small files is technically governed but operationally useless. That is where operational governance — table health, maintenance policies, and lifecycle management — becomes essential.

A control plane like LakeOps fills this gap: it handles autonomous table maintenance, structural health monitoring, and compaction across all engines — ensuring that governed data is also operationally sound.

For teams designing governance architectures, understanding how access governance and operational governance interact is a recurring theme — and one we will return to.

Layer 1: The Catalog as Enforcement Point

The catalog is where governance gets teeth. Without a governance-capable catalog, no amount of policy sophistication upstream will matter.

Apache Polaris: The Reference Implementation

Apache Polaris (graduated from incubation in early 2026) is the most fully-realized governance catalog for Iceberg. It implements a four-level RBAC model:

Privileges are granted to catalog roles (e.g., finance_reader grants TABLE_READ_DATA on the finance namespace)
Catalog roles are granted to principal roles (e.g., data_engineer_role)
Principal roles are assigned to service principals (e.g., the Spark ingestion service, the Trino BI cluster)
Privileges never attach directly to a service principal — the two-level role indirection means identity management and permission management evolve independently

The privilege model is granular: TABLE_READ_DATA, TABLE_WRITE_DATA, TABLE_CREATE, TABLE_DROP, TABLE_LIST, NAMESPACE_CREATE, NAMESPACE_LIST, CATALOG_MANAGE, plus administrative privileges for managing grants. Privileges scope to the namespace hierarchy — granting TABLE_READ_DATA on analytics implicitly covers analytics.clickstream and analytics.sessions. This enables the principle of least privilege: a data scientist's service principal can read tables in analytics without being able to create tables, alter schemas, or access the raw namespace.

Polaris 1.7.0 also introduced a Policy framework for defining lifecycle and operational rules directly in the catalog:

POST /api/catalog/v1/{catalog}/namespaces/{namespace}/policies
{
  "name": "production-snapshot-retention",
  "type": "system.snapshot-expiry",
  "content": {
    "max-snapshot-age-ms": 604800000,
    "min-snapshots-to-keep": 5
  }
}

Policies attach at catalog, namespace, or table level, with inheritance flowing downward. A table-level policy overrides namespace defaults, which override catalog defaults. This gives teams declarative, auditable lifecycle governance without per-table manual configuration.

Credential Vending: Zero-Trust Storage Access

Credential vending is the foundation of zero-trust lakehouse security, and it solves one of the hardest governance problems.

Without credential vending, engines need direct access to the storage bucket — which means they can read any file in the bucket regardless of table-level access controls. With credential vending, the catalog vends short-lived, scoped storage credentials (AWS STS temporary credentials, GCS signed URLs, Azure SAS tokens) that only allow access to the specific data files in the specific table the engine is authorized to read.

The flow:

Engine authenticates to Polaris with OAuth2 credentials
Polaris evaluates the request against its RBAC model
If authorized, Polaris calls AWS STS AssumeRole with an inline session policy scoped to the exact table locations
Engine receives temporary credentials valid for minutes — read-only, specific path prefix, short-lived
Engine accesses storage directly with scoped credentials

No long-lived storage credentials are distributed. A compromised engine token is scoped to one table and expires quickly. The catalog is the single chokepoint through which all storage access flows.

Why Catalog-Level Enforcement Matters for Multi-Engine

This is where catalog-level governance pays for itself. When Spark, Trino, Flink, DuckDB, and Snowflake all access the same Iceberg tables, the alternative is configuring access control separately in each engine. Consider what that looks like in practice:

Spark has its own ACL model via TableCatalog security
Trino has access-control.type in its catalog configuration, with its own rule syntax
Flink relies on whatever identity the job was submitted with
DuckDB has no native access control model at all

Any drift — a permission granted in Spark but not in Trino — creates a bypass path. A data engineer discovers they can read restricted.financial_transactions through DuckDB even though Trino denies the same query. The policy was configured in Trino's access control file but not in the DuckDB deployment, because DuckDB has no mechanism to enforce it. This is not hypothetical — it is the default failure mode in multi-engine lakehouses without catalog-level enforcement.

With catalog-level enforcement, every engine that connects through the REST Catalog protocol inherits the same governance model. A data scientist denied access to financial.transactions is denied regardless of whether they query through Spark, Trino, or DuckDB — because the catalog never returns the metadata, and credential vending never issues scoped storage credentials. A new engine inherits the full governance model the moment it connects. A policy change propagates immediately because the enforcement point is singular.

For a detailed comparison of catalog governance capabilities, see the catalog comparison on LakeOps.

Layer 2: Policy Engines — OPA, Ranger, and Cloud IAM

The catalog enforces decisions. Policy engines make them. This separation matters because production enterprises already have policy systems, and a governance architecture that requires replacing all of them is dead on arrival.

Open Policy Agent (OPA)

OPA is the standard for policy-as-code in cloud-native environments. Policies are written in Rego, versioned in Git, tested with unit tests, and deployed through CI/CD — the same workflows that manage application code. This is governance-as-code: auditable, reviewable, reproducible. In 2026, Polaris's OPA integration is maturing, and Lakekeeper includes a native OPA bridge that translates its OpenFGA permissions into OPA format for engines like Trino.

An OPA policy for Iceberg table access can evaluate rich context beyond identity:

package iceberg.authz

default allow = false

allow {
    input.action == "TABLE_READ_DATA"
    input.resource.classification != "restricted"
    input.principal.team == input.resource.domain
}

allow {
    input.action == "TABLE_READ_DATA"
    input.resource.classification == "restricted"
    input.principal.clearance == "pii_cleared"
    is_business_hours(input.request.timestamp)
}

This is attribute-based access control (ABAC) — policies that evaluate the requesting principal's team membership and clearance, the table's data classification, the columns being accessed, the time window, and the query pattern. ABAC goes where RBAC cannot: purpose limitation under GDPR (restricting access based on declared query purpose), time-restricted production access, classification-conditional column masking.

Lakekeeper: Cedar and OpenFGA

Lakekeeper — a Rust-based Iceberg REST catalog — takes a different approach to authorization by supporting two policy backends:

OpenFGA (open-source default) uses relationship tuples for ReBAC — relationship-based access control. Permissions are expressed as relationships between principals and objects, with bi-directional inheritance flowing through the namespace hierarchy. An object owner can grant access to their own tables through the API or UI, combining RBAC with discretionary access control. Lakekeeper requires OpenFGA v1.11+ for idempotent-write semantics and includes an OPA bridge that translates OpenFGA permissions into OPA format for query engines like Trino.

Cedar (available with Lakekeeper+) embeds authorization directly — no external service required. Policies are versioned, reviewable code with per-decision audit traces. Cedar supports ABAC natively: policies can evaluate resource attributes, tags, time conditions, and request context. A powerful pattern is storing access-control lists as table properties (access-readers, access-owners) that Cedar evaluates at query time — access control metadata lives with the data it governs.

	OpenFGA	Cedar
Extra service	Yes (OpenFGA + database)	No, built into Lakekeeper
Permission model	Relationships stored as data	Policies deployed as code
ABAC support	No	Yes (time, tags, attributes)
Runtime changes	API/UI grants	Redeploy policy source

Apache Ranger

Ranger has been the governance standard in Hadoop ecosystems for over a decade. Organizations with existing Ranger deployments — policies, audit logs, classification taxonomies — need a migration path that preserves their investment. Polaris delegates authorization decisions to Ranger's REST API, passing table identity, namespace, operation type, and requesting principal. Ranger evaluates against its policy store and returns the decision. The enforcement point changes; the rules remain.

Cloud IAM Integration

AWS IAM, Google Cloud IAM, and Azure RBAC are the default authorization systems for cloud infrastructure. The key principle: the catalog does not replace cloud IAM — it layers table-level governance on top. Cloud IAM controls who can reach the catalog. The catalog controls who can access specific tables. The policy engine defines the rules for both.

Iceberg Read Restrictions: Spec-Level Fine-Grained Access Control

The most significant governance development in the Iceberg ecosystem in 2026 is Read Restrictions — an extension to the Iceberg REST Catalog specification that moves column masking and row-level security into the catalog protocol itself.

How It Works

When a client loads a table, the catalog now returns the Iceberg metadata along with two additional governance instructions:

required-row-filter — an Iceberg predicate (e.g., country = 'USA') that the reader must apply. Any row evaluating to false must not appear in results.
required-column-projections — a list of columns the reader must mask per a specified action.

Both are scoped to the principal who made the request. The same table returns different restrictions for different callers as policies and roles change.

{
  "read-restrictions": {
    "required-row-filter": {
      "type": "eq",
      "left": { "type": "reference", "id": 3 },
      "right": { "type": "literal", "value": "USA" }
    },
    "required-column-projections": [
      { "field-id": 4, "action": "show-last-4" },
      { "field-id": 7, "action": "sha-256-global" },
      { "field-id": 12, "action": "replace-with-null" }
    ]
  }
}

The spec defines nine standardized masking actions — replace-with-null, show-last-4, sha-256-global, mask-alphanum, truncation, and others. Two engines implementing the spec independently produce identical masked output, making governance truly interoperable.

The Trust Contract

Read Restrictions shift governance from a deny-or-allow model to a nuanced instructions model. The catalog evaluates its policies and returns caller-specific restrictions; the trusted engine is responsible for applying them. Trust is established through mTLS or OAuth delegation — the catalog must know that the engine will actually enforce the restrictions before returning data access.

This is a pragmatic design. It avoids the impossible requirement that every engine implement the same governance model natively. Instead, the catalog speaks a standard governance protocol, and engines that support it participate in fine-grained access control automatically.

The Identity Gap

One practical limitation: Read Restrictions work cleanly for engines with federated identity where the requesting user's identity flows through to the catalog. Spark with connect-style setup and Trino with OAuth pass-through can propagate per-user identity. But the identity models are far from uniform:

Spark authenticates per-application (one identity per Spark session, shared across all users of a notebook or ETL job)
Trino supports OAuth pass-through for interactive queries but uses service accounts for scheduled jobs
Flink connects per-job — a streaming pipeline runs as a single service principal for its entire lifetime
DuckDB, ClickHouse, Postgres typically connect with a single set of catalog credentials — no per-query identity pass-through at all

For engines without per-user identity, Read Restrictions apply uniformly to the service principal, not to the human who submitted the query. A show-last-4 mask on SSN protects data from the service account — but if that account has full access, every user behind it sees unmasked data. Identity-mapping infrastructure that translates per-engine session identity to catalog-level principals mostly doesn't exist yet.

This is not a flaw in Read Restrictions — it is a gap in the identity fabric that the ecosystem is still building. For production deployments, it means Read Restrictions are most effective when paired with engines that support identity federation, and least effective for the embedded and lightweight engines where governance gaps tend to be largest.

Compliance Patterns: GDPR, Retention, and Audit

Governance in regulated environments goes beyond access control. Compliance requires proving that data lifecycle policies are enforced continuously.

GDPR Right to Be Forgotten

Iceberg's immutability model creates a direct tension with GDPR Article 17. When you run DELETE FROM users WHERE user_id = 48213, Iceberg marks the rows as deleted (via delete files or data file rewrites) but creates a new snapshot while keeping all old snapshots intact. The old data is still accessible through time travel.

Physical erasure requires a three-step pipeline:

Step	Operation	What Happens
1	`DELETE`	Logically removes the row (creates delete file)
2	Compaction	Rewrites data files without the deleted rows
3	Snapshot expiration + orphan cleanup	Removes old data files that contained the row

Only after all three steps is the personal data physically, irreversibly removed from storage. But the Iceberg table is not the only place the data lives. S3 versioning preserves old object versions even after Iceberg deletes the file. Disaster-recovery replicas, downstream data marts, ML feature stores, BI caches, and backups may all contain copies. A complete erasure pipeline must account for every replication path — not just the source table.

A table with 14-day snapshot retention cannot complete erasure in less than 14 days without expiring snapshots early — which removes the recovery window for every other row. The practical solution: set retention windows shorter than the regulatory deadline on any table holding personal data. GDPR allows one month; a table with seven-day snapshot retention, a seven-day S3 versioning lifecycle, and weekly maintenance completes the sequence in under three weeks.

Cryptographic Shredding: The Alternative

Cryptographic shredding inverts the problem. Instead of finding and destroying every copy of the data across snapshots, replicas, and backups, you encrypt personal data under a key unique to each data subject, store the keys in a small, mutable key store, and delete the key when an erasure request arrives.

Every copy of the ciphertext — in every snapshot, version, replica, backup, and downstream extract — becomes unreadable at the same instant. The ciphertext is still there. The information is gone.

The pattern:

Each data subject gets a data encryption key (DEK), wrapped by a KMS master key
PII columns are encrypted with the subject's DEK before being written to the lakehouse
The key store is a small table (bucketed on user_id) with aggressive retention — one-day snapshot retention, copy-on-write deletes
Erasure = delete the subject's row from the key store

This is distinct from Iceberg's native table encryption (which protects entire tables at rest with per-file keys). Crypto-shredding is an application-level pattern for per-subject erasure. The two compose: a table can use Iceberg encryption for at-rest protection and per-subject keys for erasure.

The trade-offs are real: encryption adds read/write overhead to every query touching PII columns, the key store becomes a critical dependency, and AWS KMS enforces a 7–30 day waiting period for key deletion. But for large tables where full compaction is expensive, or environments with many downstream copies, crypto-shredding can be the only way to meet erasure deadlines reliably.

Why Governance and Maintenance Are Inseparable

Without automated, continuous maintenance — compaction running after deletes, snapshot expiration enforced on schedule, orphan cleanup reclaiming storage — GDPR erasure deadlines cannot be met reliably across hundreds of tables. An erasure queue with deadlines and completion tracking is the audit story. Without it, you cannot prove compliance. Manual maintenance means manual compliance, which means eventual non-compliance.

Retention Policies

Different data classifications require different retention periods. Financial data under SOX may need seven years of queryable history. Operational data may need 90 days. PII-containing tables may need the shortest retention possible to limit exposure surface.

Snapshot retention governance should enforce:

Minimum retention periods per classification level
Maximum retention periods for PII-containing tables
Approval workflows for expiring snapshots on compliance-sensitive tables
Audit logging recording which snapshots were expired, by whom, when

Cross-Engine Audit Trails

With engine-level governance, audit logs are scattered across Spark logs, Trino logs, Flink logs, and engine-specific monitoring systems. Compliance auditors do not want to correlate six different logging systems to reconstruct who accessed what. They want a single, authoritative record.

Catalog-level governance provides this: every access request — regardless of which engine initiated it — is logged in one place. But access audit is only half the story. Maintenance operations — compaction, snapshot expiration, orphan cleanup, manifest rewriting — are also governance-relevant actions. A compaction job rewrites data files. A snapshot expiration permanently removes time-travel capability. An orphan cleanup deletes files from storage.

Most governance systems handle read and write access well. Few handle maintenance operations at all. The result is a gap: the governance system controls who can query and who can ingest, but maintenance operations run outside the governance model, often as overprivileged service accounts with blanket access.

For a deep dive into how the three-layer model addresses this, see the governance separation of concerns guide.

Schema Evolution Governance: Valid Is Not Safe

Iceberg's schema evolution is mechanically safe — field IDs ensure that column renames, reorders, and drops never corrupt data. A renamed column still maps to the same field ID. A dropped column simply disappears from the current schema while old snapshots retain it. No data files are rewritten.

But mechanically safe is not operationally safe. A column rename from ship_date to shipped_at is a metadata-only operation that preserves the field ID — and simultaneously breaks every Spark job, dbt model, Trino query, and BI dashboard that references the old name. The catalog does not know what consumers have open query plans. The Iceberg spec has no built-in mechanism to enumerate downstream impact before a commit lands.

Schema governance requires treating schema changes as releases, not commits:

Diff — generate a field-ID-aware diff (not a string diff) that distinguishes renames from drop-then-add
Classify — label each change as non-breaking (column add with default), potentially breaking (rename), or definitely breaking (type narrowing, drop of a non-nullable column)
Enumerate impact — check against query history, lineage metadata, dbt manifests, and BI connections to identify affected consumers
Gate — require approval for potentially breaking and definitely breaking changes
Stage rollback — pin a snapshot tag before the change lands so rollback has a named target, not a guessed snapshot ID
Verify — run smoke tests against the new schema before promoting to production

Partition evolution carries the same risk profile. A partition-spec change that requires rewrite_data_files should be flagged before publish, not discovered from the next compute bill.

Neither Polaris nor any current catalog provides this release-gate workflow natively. Teams either build custom CI checks against the catalog API, use emerging tools like Kastor for change management, or accept the risk of deploying schema changes without impact analysis.

Operational Governance: The Layer Most Teams Miss

Here is the gap that trips up most teams designing Iceberg governance: access governance tells you who can read and write data. It says nothing about who can maintain it — and maintenance operations have direct governance implications.

Who Can Compact?

Compaction reads existing data files, merges them, and writes new ones. This requires both read and write access. A malicious or buggy compaction job could introduce data corruption, alter sort order, or fail mid-operation leaving the table degraded. Restricting compaction to a trusted maintenance service — and auditing every compaction operation — is a governance requirement.

Who Can Expire Snapshots?

Snapshot expiration permanently removes time-travel capability. In regulated environments, prematurely expiring snapshots on a financial table can violate retention requirements. Governance policies must enforce minimum retention periods per classification, and the expiration service must be auditable.

Who Can Delete Orphan Files?

Aggressive orphan cleanup can delete files still needed by in-progress operations. Orphan cleanup requires a grace period, validation against active snapshots, and restriction to service accounts that understand the table's operational state.

Closing the Gap with a Dedicated Control Plane

Instead of granting maintenance privileges to multiple engines and hoping they coordinate, a dedicated operational control plane runs all maintenance through a single authority that respects table-level policies, avoids conflicts with active writers, sequences operations correctly, and logs every action.

LakeOps fills this role as a control plane for Apache Iceberg lakehouses. Its governance capabilities are specifically designed for operational governance:

Declarative maintenance policies define compaction thresholds, snapshot retention windows, orphan cleanup schedules, and manifest rewrite targets at organization, catalog, namespace, or individual table scope. Policies cascade with clear precedence: a table-level override wins over namespace defaults, which override catalog defaults, which override organization baselines. New tables inherit the correct configuration from their namespace automatically.

Configuration policies enforce consistent Iceberg table settings — format version, file format, write distribution mode, commit retry behavior — across all tables in scope. Without these, table configurations drift as different teams create tables with different defaults, resulting in inconsistent behavior across the lake.

Lake-wide event audit trail logs every maintenance operation across all catalogs and namespaces — what ran, when, duration, files before/after, bytes reclaimed, and the signal that triggered it. The same feed powers compliance evidence for SOC 2 and GDPR: proof that retention policies are enforced continuously, not just configured.

Learn more:

Agentic AI for Iceberg Lakehouse | LakeOps

AI agents query your Iceberg data lakehouse via MCP with read-only, cost-cap, and PII guardrails. Engine routing keeps the lake optimized for AI workloads.

lakeops.dev

Policy versioning tracks every policy change — who modified what, when, and what the previous settings were. The combination of operation logs and policy versioning creates an auditable governance record.

AI agent guardrails address an emerging governance surface: AI agents querying the lakehouse via MCP, Postgres wire protocol, or REST API. LakeOps routes agent queries through layered guardrails — ReadOnly blocks DDL/DML from agent sessions, CostEstimate rejects queries exceeding scan thresholds, PIIMask hashes sensitive columns before results reach the model, and HumanApproval pauses high-stakes operations for review. Agent query telemetry feeds back into compaction and sort-order decisions, closing the loop between agent access patterns and table optimization.

Learn more:

Safe AI Agent Access to Apache Iceberg in Production - LakeOps Blog

How to give AI agents safe access to Apache Iceberg data — guardrails for ReadOnly, CostEstimate, PIIMask, and HumanApproval with governance and routing.

lakeops.dev

For the full operational model, see the LakeOps platform page. For a deep dive into how operational governance intersects with access governance in the three-layer model, see the data lake governance guide.

Building the Governance Stack: A Practical Path

For teams designing governance for a production Iceberg lakehouse, the three-layer model suggests a clear implementation path:

Step 1: Choose a Governance-Capable Catalog

The catalog determines your governance ceiling. If it does not support fine-grained access control, credential vending, and policy engine integration, no amount of policy sophistication upstream will help.

Catalog	RBAC	Credential Vending	Policy Engine Integration	Read Restrictions
Apache Polaris	Full (namespace, table, column)	AWS, GCS, Azure	OPA, Ranger, custom	Yes (spec support)
Project Nessie	No built-in RBAC	No (engines bring own creds)	Pair with Polaris or OPA	No
Lakekeeper	RBAC, ReBAC, ABAC (OpenFGA + Cedar)	AWS, GCS, Azure	OpenFGA, Cedar, OPA bridge	Planned
Apache Gravitino	Namespace/table grants	Via federated catalogs	Custom integration	In progress
AWS Glue	IAM-based	IAM role assumption	IAM policies	Limited
Unity Catalog	Full (row, column, table)	Native	Built-in	Proprietary

Step 2: Establish Namespace and Classification Strategy

Before writing policies, define the organizational structure. A practical starting point:

Three sensitivity tiers: public, internal, restricted
One namespace per data domain per tier: e.g., internal.marketing, restricted.finance
Column-level classification for PII fields

This structure is simple enough to implement in a week and comprehensive enough to cover most governance requirements. It also maps naturally to RBAC grants: data engineers get write access to raw.*, data scientists read curated.*, analysts read published.*.

Step 3: Implement RBAC as the Baseline

Start with role-based access control using the catalog's native privilege model. Define roles mapping to organizational functions, grant namespace-level and table-level privileges, enforce through the catalog. This covers 80% of access control requirements and is auditable, understandable, and manageable without external tooling.

Step 4: Layer ABAC Through a Policy Engine

Once RBAC is in place, add attribute-based policies for requirements roles cannot express — classification-based access, column masking, time restrictions, purpose limitation. Deploy OPA, Cedar, or Ranger depending on your catalog choice (OPA/Ranger for Polaris, Cedar or OpenFGA for Lakekeeper), integrate with the catalog, and version policies in Git alongside application code.

Step 5: Automate Maintenance Governance

Governance without operational health is incomplete. A governed table with millions of small files, thousands of accumulated snapshots, and orphaned data consuming storage passes every access control check but fails every performance expectation. And a table without automated maintenance cannot meet GDPR erasure deadlines — compaction, snapshot expiration, and orphan cleanup must run continuously, not manually.

Close the loop by connecting the governance catalog (who can access) with an operational control plane (ensuring the data is healthy, fast, and compliant). The governance catalog enforces access policies. LakeOps ensures the underlying tables are structurally healthy, compacted, and query-ready — with full audit trails that satisfy the same compliance requirements as your access governance. For tables holding personal data, the same maintenance automation that keeps query performance healthy also drives the three-step erasure pipeline to completion within regulatory deadlines. For a complete production readiness checklist that includes governance alongside maintenance, see the Iceberg production readiness checklist.

The Bottom Line

The open lakehouse disaggregated storage, compute, and metadata into independent layers. Governance must follow the same decomposition — rules separate from enforcement, enforcement separate from the format, operational governance separate from access governance.

The tooling exists. Apache Polaris provides catalog-level RBAC, credential vending, and policy engine integration. Lakekeeper adds Cedar and OpenFGA for policy-as-code and relationship-based access control. Read Restrictions bring interoperable column masking and row filtering into the REST Catalog spec. OPA and Ranger provide external policy engines with full audit trails.

LakeOps provides the operational governance layer — declarative maintenance policies, AI agent guardrails, and lake-wide audit trails that ensure governed data is also structurally healthy.

Start with the catalog choice — it determines your governance ceiling. Layer policies incrementally: RBAC first, ABAC when needed. Treat schema changes as releases, not commits. Build an erasure pipeline that accounts for every replication path — or invest in cryptographic shredding for tables where full compaction cannot meet deadlines. And close the operational loop from day one. The teams that treat access governance, schema governance, and operational governance as three facets of the same architecture — each with its own tooling and audit trail — are the ones running multi-engine lakehouses at scale without governance gaps or compliance drift.

The Real Cost of Apache Iceberg Table Maintenance

joni sar — Tue, 22 Sep 2026 09:15:42 +0000

If you operate Apache Iceberg tables in production, you already know maintenance is not optional. What most teams underestimate is how much it actually costs — and how much of that spend is avoidable.

Iceberg gives you the primitives: rewrite_data_files, expire_snapshots, remove_orphan_files, rewrite_manifests. It does not give you an operational layer that runs them efficiently, in the right order, at the right frequency, on the tables that actually need work. That gap between primitives and practice is where maintenance costs compound silently — across storage, compute, and engineering time.

This article breaks down exactly where Iceberg maintenance dollars go, how each cost category interacts with the others, and what you can do to cut total maintenance spend without sacrificing table health.

Four cost categories that compound

Iceberg maintenance costs are not one line item. They come from four distinct sources that interact with and amplify each other. Fixing one in isolation delivers partial results.

1. Storage waste

Every Iceberg snapshot persists until you explicitly expire it. Each snapshot pins references to data files from that point in time, preventing garbage collection of superseded bytes. Failed writes and aborted Spark jobs leave orphan files — storage objects not referenced in any metadata — that accumulate indefinitely.

The cost math is straightforward. S3 Standard storage costs $0.023/GB/month ($23.55/TB/month) in us-east-1. On mature production lakes with streaming ingestion, unreferenced data from retained snapshots and orphan files routinely represents 25–40% of billable object-storage spend. A table with 30-day snapshot retention and 10-minute commit intervals accumulates over 4,000 snapshots per month. Each holds references to data files the current table state no longer needs.

The numbers from real deployments are concrete. One production scan found approximately 200 TB of orphan data across 324 tables — roughly 1.8 million unreferenced files from aborted transactions, failed Spark jobs, and stale tables. At S3 Standard rates, that was ~$4,700/month in storage for data no query would ever read. Another team found 120 TB removable from expired snapshots alone — approximately $34,000/year in pure waste. Amazon Ads reported a 74% reduction in orphan data and 32% lower storage costs after optimizing their Iceberg file layout — confirming that orphan accumulation is a systemic problem, not an edge case.

2. Query compute waste

This is the hidden maintenance cost most teams miss entirely. Unmaintained tables are expensive to query, not just to store.

Every Parquet file requires at least one S3 GET request to read its footer. S3 GET requests cost $0.0004 per 1,000 — seemingly cheap until you multiply by file count and query frequency. A table with 47,000 small files forces 47,000 metadata round trips before reading a single row. After compacting that table to 280 files, query time dropped from 52 seconds to 5.8 seconds — a 9× improvement that translates directly to 9× less CPU time per execution.

Sort order has an equally large impact. When data is physically sorted by the columns queries filter on, engines use min/max statistics in Parquet row group metadata to skip irrelevant data without reading it. On scan-priced engines like Athena ($5/TB scanned), the difference is stark: a well-sorted 1 TB table queried with a selective filter might cost $0.25 per query. The same query on an unsorted, fragmented copy costs $5.00 — a 20× difference. For a daily reporting pipeline scanning 5 TB across 20 tables, that is $25/run on Athena ($750/month), versus $3–5/run on a right-sized Trino cluster ($90–150/month).

Delete files from merge-on-read operations add another layer. Every query must reconcile data files against position delete files at read time. One production table had 23,433 delete files covering 551 million rows of deleted data — every query paid the full cost of reading and filtering that entire history.

The takeaway: the cost of not maintaining your tables shows up as inflated query compute bills, not as a maintenance line item. It is the largest single cost most teams fail to attribute to table health.

3. Compaction compute

Maintenance itself consumes compute. The dominant pattern — Spark-based compaction on scheduled Airflow DAGs — works, but costs significantly more than it needs to.

Compaction is a narrow, I/O-bound read-merge-write operation. Running it on Spark means paying for JVM startup, garbage collection overhead, executor provisioning, and idle cluster time between runs. EMR Serverless charges $0.052624/vCPU-hour — which adds up fast when compaction runs are over-provisioned (as they usually are, because under-provisioning causes OOM failures, and failures mean 2 AM pages).

The benchmark comparison on a 200 GB table (~1 TB uncompressed): Spark-based binpack takes 1,612 seconds at ~$1.54 in compute. AWS S3 Tables' built-in compaction takes 6,300 seconds. A purpose-built Rust engine on Apache DataFusion completes the same job in 221 seconds at ~$0.21 — roughly one-tenth the compute cost. Independent benchmarks from RisingWave confirm the magnitude: their Rust/DataFusion engine completed the same workload 5.5× faster at 82% lower cost, and succeeded on delete-heavy workloads where Spark failed with OOM errors.

Directional monthly cost comparison for a mid-size estate (~80 TB, 200+ tables, daily compaction):

Approach	Est. monthly compaction cost	Operational overhead
Spark on EMR	~$15,000	High (clusters, DAGs, on-call)
Spark on Glue	~$12,000	Medium (job configs, scheduling)
Athena OPTIMIZE	~$8,000	Low (SQL only, no sort support)
Purpose-built Rust engine	~$1,500	Minimal (policies, autonomous)

Estimates based on $50/TB compaction cost for Spark and $5/TB for purpose-built engines, with daily runs across 200+ tables. Actual costs vary by table size, compaction frequency, and instance pricing.

The gap is structural, not incremental. JVM-based engines carry startup, GC, and idle-cluster overhead that compounds across hundreds of daily compaction runs. Native engines built for the specific workload of reading, sorting, and writing Parquet files operate at a fundamentally different cost point.

4. Engineering time

Someone built those Airflow DAGs. Someone maintains them, monitors them, debugs them when compaction conflicts with a streaming writer at 2 AM, and investigates when snapshot expiration fails silently.

At 50 tables, this is manageable side work — as Alex Merced's table maintenance economics analysis notes, a small estate (40 tables, 3 TB) needs one DAG and a week of initial setup. At 500 tables, maintenance scripts become a dedicated role. At medium scale (600 tables, 80 TB), the work concentrates on the ~30 streaming tables that generate most maintenance needs, but table discovery, per-table policy, and failure handling stop being a side project. Merced estimates the ongoing cost at roughly a quarter of an engineer indefinitely, rising with table count.

Fully loaded — salary, on-call burden, opportunity cost — this vector frequently exceeds infrastructure spend on lakes with 300+ tables. It never appears on a cloud bill, which is exactly why it is routinely underestimated.

How the costs interact

These four categories do not operate independently. Each amplifies the others:

Small files inflate both query and compaction costs. Every query pays per-file overhead. Compaction has more file handles to manage and more footers to read.
Skipped snapshot expiration inflates storage and compaction costs. Compacting before expiring snapshots rewrites data files that are about to become unreferenced anyway — wasted work.
Orphan cleanup before expiration is a no-op. The files are still referenced by unexpired snapshots. Running operations out of order wastes compute and leaves storage unreclaimable.
Manifest fragmentation and small files compound. Tables with many small files tend to have many manifests. Every query planner reads every manifest before deciding which files to scan. At 200+ manifests, planning overhead often exceeds the actual scan.

The implication: sequencing matters as much as execution. Running each operation independently on a fixed schedule yields partial results. Running them as a coordinated loop — in the order Iceberg's architecture requires — is where the full cost reduction materializes.

Learn more:

Iceberg Lakehouse Optimization with LakeOps - LakeOps Blog

Optimize your Apache Iceberg lakehouse end to end — compaction, snapshots, manifests, routing, and governance. Production results: 80% cost reduction.

lakeops.dev

Apache Iceberg Cost Optimization in 2026 - LakeOps Blog

Apache Iceberg cost breakdown — small files, snapshot bloat, orphans, and compaction overhead. How autonomous management cuts lakehouse costs up to 80%.

lakeops.dev

State of Iceberg FinOps and Cost Reduction in 2026 - LakeOps Blog

State of Iceberg FinOps and Cost Reduction in 2026 — lakehouse spend leaks, chargeback metrics, maintenance economics, and tools from LakeOps to AWS Glue.

lakeops.dev

Reducing AWS S3 Cost with Iceberg: A Guide - LakeOps Blog

Reduce AWS S3 costs for Iceberg — compaction, snapshot expiration, orphan cleanup, layout optimization, and storage tiering with production numbers.

lakeops.dev

The correct maintenance sequence

The cost-optimal order is not arbitrary. It follows from Iceberg's metadata dependencies:

1. Expire snapshots    → releases references to superseded data files
2. Remove orphan files → deletes unreferenced storage (now including newly dereferenced files)
3. Compact data files  → merges remaining small files into optimal sizes
4. Rewrite manifests   → consolidates metadata against the new layout
5. Refresh statistics  → generates Puffin files for aggressive engine-level pruning

Running this out of order inflates every cost category. You pay compute to rewrite files that expiration would have freed. You miss reclaimable storage because orphan cleanup runs before expiration dereferences anything. You burn metadata I/O rebuilding manifests that compaction is about to invalidate.

Most Spark-based maintenance setups run each operation as an independent cron job with no awareness of the others. The operations execute on their own schedules, in whatever order the DAG happens to trigger them. This is where a control plane like LakeOps becomes relevant — it sequences maintenance operations in dependency order per table, triggered by actual table health signals rather than fixed schedules, so cleanup always precedes compaction and metadata optimization always follows it.

Here's a real data scnapshot of an enterprise lake with 7k tables runing LakeOps:

Measuring your actual maintenance cost

Before optimizing, measure. Most teams cannot answer "what does Iceberg maintenance cost us?" because the spend is fragmented across cloud line items.

Storage audit

Compare logical table size (total-data-files-size-in-bytes from Iceberg metadata) to actual billable storage for each table prefix. A gap exceeding 20% typically indicates orphans, unexpired snapshot references, or incomplete compaction rewrites. Multiply the gap in TB by $23.55 (S3 Standard monthly rate per TB) for a rough dollar estimate.

-- Check snapshot count and age per table (via Spark)
SELECT count(*) as snapshot_count,
       min(committed_at) as oldest_snapshot,
       max(committed_at) as newest_snapshot,
       datediff(current_date(), min(committed_at)) as oldest_snapshot_age_days
FROM catalog.db.my_table.snapshots;

-- Estimate orphan file waste: dry-run before deleting (via Spark)
CALL catalog.system.remove_orphan_files(
  table => 'db.my_table',
  dry_run => true
);

Tables with unbounded snapshot growth — streaming tables can accumulate 4,000+ snapshots per month — are your highest-priority storage cost targets.

Compute audit

Track two metrics:

Maintenance compute as percentage of query compute. If your compaction clusters cost more than 30% of your query compute, the maintenance strategy needs a faster engine or signal-driven triggers. Alex Merced's table maintenance economics analysis puts the healthy range at 10–25% of query compute for write-heavy tables. Teams that have never measured this are usually surprised: either it is zero (no maintenance running, query costs inflated) or above 40% (naive schedule, wasted runs).
Cost per TB compacted. This is the unit-economics metric. Spark-based compaction typically runs ~$50/TB on EMR. Purpose-built Rust engines run ~$5/TB. Compare your actual cost across engines on identical tables — synthetic benchmarks understate the JVM overhead at production scale.

Query waste audit

The most underrated measurement: how much query compute is wasted on poorly maintained tables?

-- Check file count and average size per partition (via Spark)
SELECT partition, count(*) as file_count,
       avg(file_size_in_bytes) / (1024*1024) as avg_file_size_mb,
       sum(file_size_in_bytes) / (1024*1024*1024) as total_size_gb
FROM catalog.db.my_table.files
GROUP BY partition
HAVING count(*) > 100 OR avg(file_size_in_bytes) < 33554432  -- < 32 MB
ORDER BY file_count DESC;

-- Check delete file overhead (via Spark)
SELECT count(*) as delete_file_count,
       sum(record_count) as total_delete_records
FROM catalog.db.my_table.delete_files;

Tables where >30% of files fall below 32 MB are costing you measurably more on every query. On Athena, you can calculate the exact dollar waste: (bytes scanned on fragmented table - bytes scanned on compacted equivalent) × $5/TB × daily query count. Tables with delete-file-to-data-file ratios above 1:10 are candidates for immediate compaction.

Reducing compaction compute cost

Compaction is typically the largest controllable maintenance cost. Three strategies reduce it:

Replace the engine

The structural mismatch between Spark and compaction is the single biggest cost lever. Compaction is a narrow I/O operation. Spark is a general-purpose distributed computation framework with JVM overhead, GC pauses, and executor provisioning designed for arbitrary DAG workloads.

Purpose-built compaction engines built on Rust and Apache DataFusion eliminate that overhead structurally. LakeOps's Rust compaction engine processes the same 200 GB benchmark in 221 seconds versus 1,612 seconds for Spark — 7.3× faster, at roughly one-tenth the compute cost ($0.21 vs. $1.54). A 1.2 TB table that caused Spark to OOM completed in 11 minutes on the same hardware.

The cost gap compounds: Spark costs scale linearly with data volume at ~$50/TB. Self-improving engines that skip tables not needing work grow sublinearly — one team cut their per-table compaction cost from $8,400/year to $750.

Compact only what needs work

Fixed-schedule compaction (every 4 hours, every night) wastes compute on tables that have not degraded since the last run. Signal-driven triggers — firing when file count, average file size, or delete-file ratio crosses a threshold — skip healthy tables entirely.

A lake with 600 tables where 30 receive continuous streaming writes generates almost all maintenance work from those 30 tables. The other 570 need snapshot expiration and little else. Cron-based compaction treats all 600 equally.

Sort during compaction

Sort compaction costs more than binpack per run, but the downstream savings dwarf the upfront investment. Sorted data enables min/max pruning on every subsequent query — unsorted tables scan roughly 50% more data than sorted equivalents in production benchmarks. The ROI calculation: sort compaction cost is paid once per sort-order cycle; the scan reduction applies to every query until the next compaction.

The challenge is choosing the right sort key. The wrong choice accelerates one query pattern while degrading another. LakeOps addresses this by validating proposed sort orders against actual query patterns before committing to a full rewrite — ensuring the compute investment produces a net-positive return on query performance.

For a detailed comparison of compaction strategies and tooling options, the LakeOps blog has a practical guide on Iceberg compaction strategies.

Reducing storage waste cost

Storage cost reduction is the highest-ROI starting point because it requires no sort-order decisions or parameter tuning. You scan, identify waste, and remove it.

Snapshot retention policies

Most tables do not need 30 days of snapshots. Define retention based on actual requirements:

Time-travel window: How far back do queries actually look? Usually 7–14 days.
Rollback safety window: How long before you would detect a bad write? Usually 24–72 hours.
ML reproducibility: Use named tags for specific training snapshots instead of retaining all history.

A streaming table producing 288 snapshots/day with 30-day retention holds 8,640 snapshots. Each pins references to data files. Reducing retention to 7 days and a minimum of 5 retained snapshots cuts snapshot count to ~2,000 — and releases the storage those 6,600 expired snapshots were preventing from being garbage-collected. On a table where expired snapshots pin 10 TB of superseded data, that is ~$235/month in recoverable S3 Standard storage.

-- Expire snapshots older than 7 days, keeping at least 5
CALL catalog.system.expire_snapshots(
  table => 'db.my_table',
  older_than => TIMESTAMP '2026-09-09 00:00:00',
  retain_last => 5
);

Orphan file cleanup

Orphan files accumulate from failed writes, aborted compaction, and dropped tables. They are invisible to query engines but billable on your storage invoice. OOMed compaction jobs are the most prolific source — a run writing 500 new files that fails before the atomic commit leaves 500 orphans per failure.

-- First: dry-run to see what would be removed
CALL catalog.system.remove_orphan_files(
  table => 'db.my_table',
  older_than => TIMESTAMP '2026-09-09 00:00:00',
  dry_run => true
);

-- Then: remove orphan files older than 7 days
CALL catalog.system.remove_orphan_files(
  table => 'db.my_table',
  older_than => TIMESTAMP '2026-09-09 00:00:00'
);

Always run orphan cleanup after snapshot expiration. Expiration dereferences files that then become orphans — if cleanup runs first, it misses them. The older_than threshold of 3–7 days is critical: it protects files from in-flight writes that have not yet committed.

At lake scale, the impact is significant. One cleanup pass across 324 tables removed approximately 200 TB of orphan data — ~$56,000/year in S3 Standard storage savings from a single operation.

Metadata bloat

Iceberg's metadata.json stores all historical schemas and partition specs. Tables with frequent schema evolution accumulate hundreds of schema versions — a table with thousands of columns that has undergone 200+ schema changes stores every complete schema version in full. Production reports show metadata.json files reaching 10 MB compressed (250 MB uncompressed), consuming approximately 4 GB of heap on load in engines like Trino. Each snapshot entry also adds 200–500 bytes; at 288 commits/day, that is 56–140 KB/day of snapshot metadata alone.

Mitigations: set write.metadata.delete-after-commit.enabled to true and write.metadata.previous-versions-max to a reasonable limit (e.g., 10–20). Keep snapshot expiration aggressive to reduce the snapshot list in metadata.json. For a deeper analysis, see the LakeOps blog on Iceberg metadata lifecycle maintenance.

Reducing query compute through better table health

The most impactful long-term cost reduction comes not from cheaper maintenance, but from maintaining tables well enough that every query costs less.

Three levers:

File sizing. Compact small files to 128–512 MB targets. This alone reduces per-file API call overhead by orders of magnitude. A table going from 47,000 files to 280 files saves ~46,700 S3 GET requests per query — and at $0.0004 per 1,000 GETs, that is ~$0.019 saved per query just in API calls, before accounting for the far larger CPU and scan savings.

Sort order. Physical sort by high-selectivity filter columns enables row-group skipping. Engines use min/max statistics in Parquet row group metadata to skip irrelevant file groups entirely. Sorted data also compresses better (163 GB vs. 178 GB for 1 TB TPC-H — a 9% reduction), lowering storage costs as a side effect.

Delete file cleanup. Physically apply position deletes through compaction so queries stop paying reconciliation cost at read time. Tables with delete-file-to-data-file ratios above 1:10 are candidates for immediate compaction.

The compound effect: a table that is compacted to the right file size, sorted by the right columns, with delete files cleaned up and manifests consolidated, delivers dramatically faster queries than the same table in a degraded state. On Athena at $5/TB scanned, the cost difference is direct and measurable.

From scripts to policies: cutting engineering time

The engineering time cost of Iceberg maintenance scales linearly with table count when you manage it through scripts. Every table needs a compaction DAG, an expiration schedule, an orphan cleanup job, monitoring, alerting, and on-call coverage.

The alternative is policy-based maintenance: define rules at the catalog, namespace, or table level and let the system enforce them.

A practical policy set for a mid-size estate:

All tables: Expire snapshots daily (7-day window, minimum 5 retained). Remove orphan files weekly (7-day older_than threshold).
Streaming tables (>10 commits/hour): Compact when average file size drops below 64 MB or small-file ratio exceeds 30%. Rewrite manifests after every compaction cycle.
Batch tables: Compact when file count exceeds 500 per partition. Rewrite manifests monthly.
CDC tables: Compact when delete-file count exceeds threshold. Priority: position-delete rewrite before data-file compaction.

The LakeOps platform implements this as declarative policies scoped by catalog, namespace, or individual table — with a specificity hierarchy where table-level overrides take precedence. New tables inherit policies automatically. Every execution is logged with duration, impact, and status for governance and chargeback.

The engineering time savings at scale are substantial. Instead of maintaining hundreds of Airflow DAGs and Spark scripts, you define policies once and the system handles sequencing, scheduling, conflict avoidance, and adaptation. The on-call burden drops from "compaction failed on table X at 2 AM" to reviewing an event log.

What a control plane changes

The underlying problem is that Iceberg gives you maintenance primitives but not an operational system. Each procedure runs in isolation. Nothing sequences them, nothing triggers them from table health signals, nothing coordinates across tables, and nothing tells you which tables are actually costing you money.

A lakehouse control plane like LakeOps connects to your catalogs (Glue, REST/Polaris, Nessie, S3 Tables) and query engines (Trino, Spark, Snowflake, Athena, DuckDB) and targets each cost vector through specific mechanisms:

Storage waste → lake-wide observability and automated cleanup. Every table assessed for structural health — file fragmentation, snapshot age, manifest bloat, write velocity — with the most degraded tables surfaced first. Snapshot expiration and orphan cleanup run in dependency order automatically — so the orphan cleanup that reclaimed 200 TB at one production lake does not require a manual audit to trigger.

Compaction compute → Rust engine replacing Spark. Compaction on a purpose-built Rust/DataFusion engine at ~$5/TB instead of Spark at ~$50/TB. No JVM startup, bounded memory, no OOM failures. The engine skips tables that have not degraded since the last run, so costs grow sublinearly with table count.

Query compute → query-aware sort optimization. Sort decisions driven by cross-engine query telemetry — which columns your Trino, Spark, and Athena queries actually filter on — not guesswork at table-creation time. Layout changes are tested on Iceberg branches before touching production.

Engineering time → policy-driven automation replacing DAGs. Maintenance rules defined at the namespace level replace per-table Airflow configurations. New tables are covered from the moment they appear in the catalog. Every execution is logged with duration, impact, and status.

Cross-engine savings → multi-engine routing. Each query routed to the cheapest engine meeting its latency target. A nightly batch pipeline scanning 5 TB drops from $25/run on Athena to $3–5/run on Trino — without changing application code.

The LakeOps platform integrates with your existing infrastructure through standard catalog and engine APIs — setup is measured in minutes, not sprints.

For a comprehensive breakdown of all four cost vectors with production numbers, the Apache Iceberg cost optimization guide covers each in detail. For a broader view of the tooling landscape — cloud-native optimizers, engine-integrated maintenance, and dedicated control planes — the State of Iceberg FinOps survey compares approaches across the ecosystem.

The highest-ROI sequence

For most production lakes, the highest-ROI sequence is:

Audit storage waste (week 1). Compare logical table sizes to billable storage. Run remove_orphan_files with dry_run => true on your top 20 tables by object count. Multiply the gap in TB by $23.55 for your monthly waste estimate.
Run snapshot expiration and orphan cleanup (week 1–2). These release storage at near-zero compute cost. No sort-order decisions required. One pass can reclaim tens of thousands of dollars in annual storage spend.
Benchmark compaction cost per TB (week 2–3). Compare your current engine against alternatives on identical tables. The engine choice is a first-order cost decision — $50/TB vs. $5/TB is a 10× difference that compounds across every table and every run.
Compact and sort your highest-traffic tables (week 3–4). Start with the tables that generate the most latency complaints — they are typically your highest-cost tables from a compute perspective.
Replace cron with health-driven policies (month 2). Move from fixed schedules to triggers based on file count, average file size, and delete-file ratio. This is where engineering time savings materialize.

The total reduction from addressing all four cost categories as a coordinated system — not as isolated optimizations — is where production teams consistently report the largest improvements. Storage waste disappears. Query compute drops because every engine scans less. Maintenance compute drops because the engine is cheaper and skips healthy tables. Engineering time drops because policies replace scripts.

Iceberg's table format is production-ready. The maintenance layer should be too — and it should cost a fraction of what cron-based Spark jobs spend to keep your lake healthy. LakeOps is one way to get there.

Thanks for reading :)

Apache Iceberg Maintenance Tools: From Spark Procedures to Autonomous Control Planes

joni sar — Thu, 17 Sep 2026 11:34:50 +0000

Every write to an Apache Iceberg table creates a new snapshot. Every snapshot pins references to data files. Over weeks of production workloads, this produces thousands of small files, bloated metadata, orphaned storage artifacts, and dangling delete records — all of which silently degrade query performance, inflate storage costs, and slow down every engine connected to the table.

The fix requires five distinct maintenance operations, executed in the right order, at the right frequency, across every table in the lake. The tooling landscape for this has matured substantially — from raw engine procedures to standalone Rust and Go binaries to fully autonomous control planes. This guide covers the complete spectrum, with enough technical depth to help you choose the right approach for your stack.

The five maintenance operations

Before evaluating tools, you need a clear picture of what maintenance actually involves. Iceberg maintenance is not a single operation — it's a coordinated pipeline of five distinct actions, each solving a different structural problem.

Here's an example of such operations using an autonomous and intelligent Lakehouse control plane:

Managed Iceberg in 2026: Autonomous Data Lake - LakeOps Blog

A deep dive into nine components of a modern Iceberg optimization platform — full-stack observability, query-aware compaction, snapshot lifecycle management, manifest optimization, orphan file cleanup, organization-wide policies, multi-engine query routing, agentic AI enablement, and layout simulations — with product walkthroughs showing how LakeOps delivers autonomous table maintenance, up to 28x faster compaction, and up to 80% cost reduction.

lakeops.dev

1. Snapshot expiration

Every commit — write, update, delete, compaction — creates a new snapshot. Snapshots enable time travel and rollback, but they also prevent data files from being garbage-collected. A streaming table with 5-minute commits accumulates over 8,600 snapshots per month.

expire_snapshots removes snapshots older than a retention threshold and deletes the manifest files and manifest lists that were exclusively referenced by those expired snapshots. Data files that are no longer referenced by any retained snapshot become eligible for orphan cleanup (they are not automatically deleted by expiration alone on all engines — always pair with remove_orphan_files).

Key parameters:

older_than — the retention boundary. Set this to at least 2× the duration of your longest-running query. A snapshot expired mid-scan produces a FileNotFoundException that surfaces minutes later, making root cause analysis painful.
retain_last — the minimum snapshot count regardless of age. Never set this to 1 — that leaves zero rollback targets. For streaming tables, 25–100 is typical.
stream_results — set to true on large tables. Without it, the full list of deleted files is collected in the Spark driver's memory before being returned, which causes driver OOM on tables with millions of expired files.

2. Orphan file cleanup

Distributed engines produce orphan files — data and metadata files that exist in object storage but are not referenced by any snapshot. They accumulate from failed Spark tasks, aborted compaction jobs, and commit conflicts. On active tables, orphan storage can grow to hundreds of gigabytes without anyone noticing.

remove_orphan_files lists every file in the table's storage location and compares what exists against what the metadata tree references. Anything unreferenced and older than the retention interval is deleted.

The default retention is 3 days, but production teams use 7+ days. The reason: a Spark job that writes files at the start of a 4-hour run but commits at the end has unreferenced files for 4 hours. A retention shorter than that window deletes live data mid-write, silently corrupting the table.

Three less-obvious failure modes to watch for:

URI scheme mismatch — if your metadata records paths as s3:// but the listing returns s3a://, the diff considers every file an orphan and deletes your entire table's data. Always verify schemes match before first run, and use prefix_mismatch_mode => 'ERROR' (the default since recent Iceberg versions) to catch this.
S3 rate limiting — orphan cleanup issuing thousands of concurrent deletes can trigger S3's 3,500 DELETE/s per-prefix throttle, producing SlowDown errors that affect production reads. Use max_concurrent_deletes to cap throughput.
Driver OOM on large tables — set stream_results => true to stream file lists in partitions rather than collecting them all in the driver.

3. Data file compaction

Small files are the most visible maintenance problem. Each file adds metadata overhead, and each file open during a scan adds latency. A partition with 500 files averaging 4 MB each is orders of magnitude slower to scan than the same data in 8 files of 256 MB.

rewrite_data_files reads small files and rewrites them into larger, optimally-sized targets. Three strategies are available:

Binpack — merges files without changing row order. Fast, low-risk, and sufficient when queries scan full partitions.
Sort — rewrites files with rows physically ordered by specified columns. Maximizes data skipping via Parquet min/max statistics. More expensive (requires a full shuffle) but delivers the largest performance gains for filtered queries.
Z-order — interleaves sort columns for multi-dimensional range queries. Useful when queries filter on multiple independent columns.

Target file size depends on workload: 64–128 MB for point-lookup tables, 256–512 MB for analytical scans. Most production tables land at 256 MB as a default.

Two parameters are critical for avoiding OOM during compaction: max-file-group-size-bytes controls how much data is processed in a single rewrite group — set it too high on a partition with thousands of small files and your Spark executors run out of memory. rewrite-job-order (e.g., bytes-asc) lets you process smallest groups first, catching configuration problems early before committing resources to large rewrites. Note that Spark disables Adaptive Query Execution (AQE) during rewrite_data_files, so size your shuffle resources explicitly rather than relying on auto-tuning.

4. Manifest rewriting

Manifests are the metadata index — they map which data files exist in which partitions with which column statistics. When writes arrive in a pattern that doesn't match query patterns (e.g., data written hourly but queried by customer ID), manifests become fragmented. Query planners must scan more manifests than necessary, adding seconds of planning overhead on large tables.

rewrite_manifests reorganizes manifest files to align with the actual file layout, reducing query planning time. The operation is metadata-only — it rewrites Avro manifest files but never touches data files, making it fast (seconds on most tables) and low-risk. The conflict window with concurrent writers is correspondingly narrow, so run it immediately after compaction before the next batch of writes lands.

5. Delete file maintenance

Iceberg's merge-on-read mode writes position delete files instead of immediately rewriting data files on row-level deletes. This is efficient for small deletes but accumulates overhead: every scan must reconcile delete files against base data at read time.

rewrite_position_delete_files compacts small delete files and filters out dangling delete records — entries pointing to data files that no longer exist after compaction. This is the most frequently overlooked maintenance operation, and on tables with frequent row-level deletes, it can account for a significant portion of read overhead.

Why sequencing matters

These operations have dependencies. Running them in the wrong order wastes compute or misses reclaimable storage:

Expire snapshots first — dereferences files held exclusively by old snapshots, converting them from "referenced" to "orphan-eligible." Without this step, orphan cleanup misses the largest category of reclaimable storage entirely.
Remove orphans second — catches everything expiration just released, plus any debris from failed writes. Reduces the file set before compaction.
Compact data files third — operates on a clean, minimal dataset. Compacting files that expiration would have removed is the single largest source of wasted compute in maintenance pipelines.
Rewrite manifests last — reflects the final compacted layout. Rewriting manifests before compaction produces a metadata alignment that becomes stale the moment compaction runs.

Running these as independent cron jobs on separate schedules is the most common source of maintenance collisions and wasted compute. See Automating Apache Iceberg Table Maintenance for a detailed treatment of the sequencing rationale.

Commit conflicts during maintenance

Iceberg uses optimistic concurrency control (OCC) for all metadata commits. When a maintenance operation and a writer commit simultaneously, one loses the race and must retry. On streaming tables producing multiple commits per minute, compaction jobs that run for 10+ minutes will almost certainly hit at least one conflict.

The critical distinction: catalog commit conflicts (two writers race to swap the metadata pointer) are transient and resolve through automatic retries. Data conflicts (compaction rewrites files in a partition that received new writes during the rewrite) cause ValidationException under serializable isolation and require either custom retry logic or switching to snapshot isolation.

Production guidance:

Exclude hot partitions from compaction. On time-partitioned tables, compact WHERE event_date < current_date() — never today's partition while a streaming writer is appending to it. This single filter eliminates the most common source of compaction failures.
Use snapshot isolation for tables with concurrent streaming writes and compaction. It allows appends to the same partition that compaction is rewriting, as long as the appended files don't overlap with the rewritten set.
Increase commit.retry.num-retries on streaming tables from the default 4 to 10–20. The retry only repeats the metadata commit, not the entire rewrite.
Watch for retry storms. Contention causes retries, retries increase catalog load, higher catalog load slows commits, which widens the contention window. If your catalog request rate climbs while successful commit rate falls, reduce concurrency rather than increasing retries.

Engine-native maintenance: Spark

For most teams, Spark is the first maintenance tool they use with Iceberg. The Iceberg connector exposes stored procedures covering every core operation:

-- 1. Expire snapshots older than 7 days, keep at least 10
CALL catalog.system.expire_snapshots(
  table => 'db.events',
  older_than => TIMESTAMP '2026-09-09 00:00:00',
  retain_last => 10,
  stream_results => true  -- prevents driver OOM on large tables
);

-- 2. Remove orphan files older than 7 days (always dry-run first)
CALL catalog.system.remove_orphan_files(
  table => 'db.events',
  older_than => TIMESTAMP '2026-09-09 00:00:00',
  dry_run => true,
  stream_results => true,
  max_concurrent_deletes => 1000  -- throttle to avoid S3 rate limits
);

-- 3. Compact data files (exclude today's hot partition)
CALL catalog.system.rewrite_data_files(
  table => 'db.events',
  strategy => 'binpack',
  where => 'event_date < current_date()',
  options => map(
    'target-file-size-bytes', '268435456',
    'min-input-files', '5',
    'max-file-group-size-bytes', '3221225472',
    'partial-progress.enabled', 'true',
    'partial-progress.max-commits', '10'
  )
);

-- 4. Rewrite manifests
CALL catalog.system.rewrite_manifests(
  table => 'db.events'
);

-- 5. Clean up dangling position delete files
CALL catalog.system.rewrite_position_delete_files(
  table => 'db.events',
  options => map('rewrite-all', 'true')
);

What Spark does well: Full control over every parameter. Support for all five operations. The partial-progress.enabled option is critical for production — it commits compaction incrementally, so a failure at 80% doesn't lose all prior work. The stream_results option prevents driver OOM on tables with millions of files.

What Spark doesn't do: Spark provides procedures, not a maintenance system. There is no scheduling, no health detection, no multi-table orchestration, no sequencing logic, and no awareness of what other engines or operations are running against the same table. You build all of that yourself — typically as Airflow DAGs, cron jobs, or custom scripts. That works at 10 tables. At 500, it becomes its own infrastructure project.

The other major limitation is cost: Spark requires a running cluster. Running maintenance on a dedicated Spark cluster that sits idle between jobs, or sharing a cluster with production queries and risking commit conflicts, is a tradeoff every team manages differently.

Engine-native maintenance: Trino and Flink

Trino

Trino exposes maintenance through ALTER TABLE ... EXECUTE, keeping operations inside the same catalog, permissions model, and deployment pipeline as queries:

ALTER TABLE db.events EXECUTE optimize
  WHERE event_date < current_date - INTERVAL '1' DAY;

ALTER TABLE db.events EXECUTE expire_snapshots(retention_threshold => '7d');

ALTER TABLE db.events EXECUTE remove_orphan_files(retention_threshold => '7d');

The advantage is simplicity — no separate Spark cluster for maintenance. The disadvantage is that maintenance competes with query resources on the same Trino cluster, and Trino's maintenance surface is narrower than Spark's (no sort compaction, no position delete file rewriting, fewer tuning knobs).

Flink

Flink provides native streaming maintenance through the TableMaintenance API. Maintenance runs as a Flink streaming job — triggered by commit counts or file thresholds rather than external schedules. Since Iceberg 1.12, external lock factories are deprecated; Flink uses its own coordination locks by default:

// Standalone maintenance job (Flink-managed coordination lock)
TableMaintenance.forTable(env, tableLoader)
    .uidSuffix("my-maintenance-job")
    .rateLimit(Duration.ofMinutes(10))
    .lockCheckDelay(Duration.ofSeconds(10))
    .add(ExpireSnapshots.builder()
        .scheduleOnCommitCount(10)
        .maxSnapshotAge(Duration.ofDays(7))
        .retainLast(10)
        .parallelism(8))
    .add(RewriteDataFiles.builder()
        .scheduleOnDataFileCount(10)
        .targetFileSizeBytes(128 * 1024 * 1024)
        .partialProgressEnabled(true)
        .partialProgressMaxCommits(10))
    .add(DeleteOrphanFiles.builder()
        .minAge(Duration.ofDays(5)))
    .append();

Flink also supports post-commit maintenance directly in IcebergSink — compaction, expiration, and orphan cleanup run automatically after each data commit, eliminating external scheduling entirely for Flink-written tables:

IcebergSink.forRowData(dataStream)
    .table(table)
    .tableLoader(tableLoader)
    .rewriteDataFiles(Map.of(
        RewriteDataFilesConfig.MAX_BYTES, "1073741824"))
    .expireSnapshots(Map.of(
        ExpireSnapshotsConfig.RETAIN_LAST, "5",
        ExpireSnapshotsConfig.MAX_SNAPSHOT_AGE_SECONDS, "604800"))
    .deleteOrphanFiles(Map.of(
        DeleteOrphanFilesConfig.MIN_AGE_SECONDS, "259200"))
    .append();

For tables with row-level deletes in upsert/CDC workloads, Flink also provides ConvertEqualityDeletes — a maintenance task that rewrites expensive equality deletes into position deletion vectors (requires Iceberg format v3). This can significantly reduce read amplification on tables with frequent upserts.

When engine-native works

Engine-native tools are a good fit when:

You have a small number of tables (under ~50)
A single engine handles both writes and maintenance
Your team has the capacity to build and maintain orchestration (DAGs, cron, monitoring)
You don't need cross-engine coordination

They start to break down when multiple engines touch the same tables, when table counts grow beyond what manual orchestration can cover, or when the operational cost of maintaining the maintenance system itself becomes significant.

Standalone maintenance tools

A newer category of tooling has emerged: standalone maintenance binaries that run outside any query engine. They connect directly to Iceberg catalogs and object storage, execute maintenance operations, and exit. No Spark cluster, no Trino session, no JVM in some cases.

Bergman

Bergman is a Rust-native maintenance engine built on iceberg-rust and Apache DataFusion. One static binary. No JVM.

It supports all core operations — compaction, snapshot expiration, orphan cleanup, manifest rewriting, and dangling delete removal — driven by declarative TOML policies:

bergman inspect   # table health: file counts, sizes, delete ratios
bergman plan      # what maintenance would do, and why
bergman run       # execute the plan (same code path as plan)

Bergman's design is library-first: the binary is a thin wrapper around a public Rust API, so catalogs and control planes can embed the engine in-process. Destructive operations (orphan deletion, compaction, file cleanup) are off by default and must be explicitly opted into via policy. It currently supports REST catalogs and is early-stage — all operations are functional but it has not yet seen large-scale production use.

Firn

Firn is a single Go binary that provides writer-agnostic maintenance — compaction (binpack, sort, and z-order via a DuckDB subprocess), snapshot expiration, and orphan cleanup. It supports AWS Glue, Lakekeeper, Apache Polaris, and Nessie catalogs, with S3-compatible, GCS, and Azure storage backends. Firn positions itself as an open-source alternative to AWS S3 Tables' automatic maintenance — the same operations, without AWS lock-in or the cost premium. Pre-v1.0.

iceberg-zamboni

iceberg-zamboni uses PyIceberg for metadata and DuckDB for sorting, providing all six maintenance operations as a single Python process. It supports Z-order clustering, partition evolution, dangling-delete removal, and manifest rewriting — locally by default, with optional Trino or Spark Connect backends for scale. Install with pipx install iceberg-zamboni. Declarative table configs describe the target layout; every mutating command requires --yes to execute.

Ice-keeper

Ice-keeper wraps Iceberg's Spark procedures and orchestrates them across many tables using rules stored in Iceberg tblproperties. It handles table discovery, parallelizes work across large inventories, and records every action in a journal for auditability. It runs wherever Spark runs — the benefit is multi-table orchestration, not engine independence.

Floe

Floe is a policy-based maintenance system that adds declarative policies over Spark or Trino. Define rules matching tables by glob or regex pattern, set cron schedules or manual triggers, and Floe orchestrates compaction, snapshot expiration, orphan cleanup, and manifest optimization across multiple catalogs (REST, Polaris, Lakekeeper, Gravitino, DataHub, Nessie, Hive). Policies support priority-based resolution when multiple patterns match the same table and optional health-based trigger conditions. Apache 2.0 licensed.

What standalone tools add

The common thread: these tools treat maintenance as a first-class concern separate from query execution. They add table discovery, policy-driven configuration, multi-table orchestration, and operational logging — the layer that raw engine procedures leave entirely to you.

What they generally don't provide: health-driven triggering (knowing when a table needs maintenance based on its actual state), cross-engine awareness, intelligent sort-order selection based on query patterns, or the ability to adapt cadence to write velocity.

Platform maintenance layers

If you run within a managed platform, maintenance tooling is increasingly built in:

Platform	Maintenance approach	Engine	Key characteristics
Cloudera CLO	Policy-based, schedule or event triggers	Spark	Compaction, expiration, orphan cleanup. REST API for policy management.
Starburst/Galaxy	Trino-native scheduled maintenance	Trino	No separate Spark cluster. Scope-based scheduling. Limited to Trino's Iceberg connector surface.
Dremio Open Catalog	Autonomous compaction and cleanup	Polaris-based	Built on Apache Polaris. Auto-compaction, auto-vacuum, Iceberg Clustering (Z-order).
AWS Glue Table Optimizers	Managed compaction and retention	Spark (managed)	Automatic compaction and snapshot management for Glue Catalog tables. AWS-only.
Athena	`VACUUM` and `OPTIMIZE`	Athena	Serverless maintenance within Athena sessions. Simple but narrow.
Snowflake	Automatic for managed Iceberg tables	Snowflake	Invisible for Snowflake-managed tables. Open Catalog tables require external maintenance.
Databricks	`OPTIMIZE` and `VACUUM`	Spark (managed)	Deeply integrated with Delta, Iceberg support through UniForm. Tied to the Databricks runtime.
IOMETE	Health-driven detect → evaluate → execute	Spark	Threshold-based triggers for internal REST catalog tables. Table-level overrides.

Platform tools excel when you're fully committed to one ecosystem. They break down when your data lake spans multiple engines, catalogs, or clouds — which is increasingly common in production architectures that use Iceberg specifically to avoid vendor lock-in.

Lakehouse control planes

The gap in all the approaches above is the same: none of them treat maintenance as a lake-wide coordination problem. Engine procedures don't know about other engines. Standalone tools orchestrate operations but don't understand query patterns. Platform layers are confined to their ecosystem.

A lakehouse control plane
bridges your catalogs, engines, and storage into a single operational layer — the coordination fabric that individual components were never designed to provide. LakeOps is built specifically for this role.

Learn more:

What Is a Data Lakehouse Control Plane? - LakeOps Blog

What is a data lakehouse control plane? An intelligent layer that autonomously maintains, optimizes, and governs Apache Iceberg tables across all engines.

lakeops.dev

Health-driven vs. schedule-driven

The fundamental architectural difference between a control plane and a cron-based approach is what triggers maintenance.

Scheduled maintenance runs on fixed intervals regardless of table state. A table that receives no writes still gets compacted on Tuesday. A streaming table that accumulated 10,000 small files since the last run waits until the next scheduled window.

Health-driven maintenance evaluates each table's structural signals continuously — file count per partition, average file size relative to target, snapshot count, manifest depth, delete-file ratio — and triggers operations only when indicators cross thresholds. Streaming tables may be maintained hourly. Batch tables daily. Idle tables are skipped entirely. The cadence adapts to write velocity, not wall-clock time.

How LakeOps approaches the problem

LakeOps plugs into your existing Iceberg catalogs (REST, Glue, Polaris, Nessie, S3 Tables) and engines (Spark, Trino, Flink, Snowflake, Athena, DuckDB) through their standard APIs. No infrastructure changes, no code changes — only metadata is processed, and your data stays in your storage account.

For maintenance specifically, the system:

Continuously assesses every table's structural condition — file layout, snapshot accumulation, manifest depth, write velocity — and flags degradation before it reaches queries. The table health scoring model evaluates signals that individual engine procedures have no visibility into, surfacing the tables that need maintenance most urgently.

Triggers Operations based on tresholds or events and sequences operations in dependency order per table: snapshot expiration → orphan cleanup → compaction → manifest rewriting. Each step runs to completion before the next begins, eliminating the conflict scenarios and wasted-compute problems described above. There is no possibility of expiration conflicting with in-flight compaction, because they never run concurrently on the same table.

Applies query-aware sort orders during compaction. Rather than blindly merging small files, LakeOps watches which columns your queries actually filter, join, and group on across all connected engines — then physically re-sorts data to match. Engines skip entire file groups via min/max pruning instead of scanning everything. The sort strategy adapts as query patterns evolve, and proposed layout changes are validated on Iceberg branches before being applied.

Runs compaction on a purpose-built engine (Rust + Apache DataFusion) rather than requiring a Spark cluster. This eliminates JVM startup latency, executor provisioning, and resource contention with production queries — a significant operational and cost reduction.

Governs maintenance through policies rather than per-table configuration. Thresholds, retention windows, and sort strategies propagate through a namespace hierarchy, so new tables comply automatically. Execution modes range from fully autonomous to manual approval.

The platform page shows the full maintenance pipeline in detail — from health scoring through coordinated execution to the event trail that logs every operation's duration, file impact, and bytes reclaimed.

Real example from a 7k tables Iceberg Lakehouse:

For teams dealing specifically with small file accumulation or the operational burden of table cleanup at scale, the control plane approach replaces per-table cron jobs with a single system that handles the entire maintenance lifecycle across the lake.

Choosing the right approach

The right tool depends on three variables: how many tables you manage, how many engines touch them, and how much operational overhead your team can absorb.

Scenario	Recommended approach	Why
< 20 tables, single engine	Engine-native procedures + cron/Airflow	Simple, direct, full control. The orchestration overhead is manageable.
20–100 tables, single engine	Standalone tool (Floe, Ice-keeper) or platform layer	Multi-table orchestration is necessary. Manual per-table configuration stops scaling.
100+ tables, single engine	Platform layer or control plane	Health-driven triggering matters. You can't manually monitor 100+ tables for maintenance signals.
Any count, multiple engines	Control plane (LakeOps)	Cross-engine coordination is the critical requirement. No single engine has visibility into what the others are doing.
Fully managed platform	Built-in platform tools	If you're all-in on Databricks, Snowflake, or Cloudera, use what's already there.
No-JVM requirement	Bergman, Firn, iceberg-zamboni, or LakeOps	Bergman (Rust), Firn (Go), and iceberg-zamboni (Python+DuckDB) are open-source standalone tools. LakeOps is a managed control plane with a Rust/DataFusion engine.

There's no single correct answer. The common mistake is underestimating how quickly the operational surface area grows. Ten tables with four operations each, sequenced correctly, monitored for failures, adapted to write patterns, across two engines — that's already a non-trivial system to build and operate.

Practical implementation checklist

Regardless of which tool you choose, these principles apply:

Always dry-run orphan cleanup before executing. A misconfigured older_than or a URI scheme mismatch can delete live data. Verify the count is reasonable — if it's close to your total file count, something is wrong.
Expire before compact. This is the single most impactful sequencing rule. Compacting files that expiration would have removed wastes the most compute.
Enable partial-progress for compaction on any table larger than a few gigabytes. Without it, a single OCC conflict at 90% completion loses all prior work.
Exclude hot partitions from compaction. Use where => 'event_date < current_date()' or equivalent. Compacting partitions with active streaming writers is the primary cause of commit conflicts and wasted compute.
Set retain_last to at least 2. Setting it to 1 leaves zero rollback targets after the current snapshot. For streaming tables, 25–100 is appropriate.
Use stream_results => true for expire_snapshots and remove_orphan_files on any table with more than a few thousand files. Without it, the full file list is collected in the Spark driver's memory.
Use a 7+ day orphan retention. The 3-day default is too aggressive for environments with long-running Spark jobs or Flink recovery scenarios. Verify URI schemes match before first run.
Monitor compaction output, not just completion. A compaction job that succeeds but produces files 10× larger than your target is masking a configuration problem.
Run rewrite_position_delete_files regularly on tables with row-level deletes. Dangling deletes are invisible in most monitoring but accumulate meaningful read overhead.

Where the ecosystem is heading

The Iceberg maintenance ecosystem is converging on a clear pattern: from manual procedures toward automated, health-driven systems. Flink's native TableMaintenance API builds maintenance into the streaming pipeline. Standalone tools like Bergman, Firn, and iceberg-zamboni remove the JVM dependency entirely. Control planes like LakeOps close the loop between table health signals and maintenance execution across the full lake.

The underlying insight across all of these is the same: maintenance is a continuous structural concern, not an ad-hoc operational task. The teams that treat it as infrastructure — with proper sequencing, conflict awareness, adaptive cadence, and per-table tuning — avoid the silent degradation that catches everyone else by surprise six months into production.

Amazon S3 Tables Alternatives for Production Iceberg

joni sar — Wed, 16 Sep 2026 12:30:56 +0000

Amazon S3 Tables solves a real problem. If you have ever spent a weekend debugging a failed compaction job, or watched a table accumulate 40,000 small files because nobody remembered to schedule rewrite_data_files, the pitch is compelling: create a table bucket, write data, and AWS handles compaction, snapshot expiration, and orphan cleanup automatically. No Spark clusters. No Airflow DAGs. No on-call for maintenance failures.

For greenfield deployments with straightforward requirements — especially AWS-native stacks using Athena, EMR, and Redshift — that trade-off works well and can save weeks of infrastructure setup. But production Iceberg estates tend to outgrow what S3 Tables can deliver, and the constraints become apparent exactly when you need flexibility most.

This article breaks down what S3 Tables does well, where it falls short, and what alternatives exist for teams that need more from their Iceberg operations.

What S3 Tables actually gives you

S3 Tables, generally available across most AWS regions since 2025, merges Iceberg storage and catalog into a single managed service. You create a "table bucket" (a dedicated S3 bucket type that only stores Iceberg tables), and AWS provides:

Built-in Iceberg REST Catalog — each table bucket exposes a REST endpoint that any Iceberg-compatible engine can connect to. S3 Tables also integrates with the Glue Data Catalog through a federated s3tablescatalog, making tables discoverable alongside standard Glue-managed tables.
Automatic compaction — background binpack, sort, or z-order compaction with configurable target file sizes (64 MB to 512 MB). As of July 2025, compaction costs were reduced by up to 90%.
Snapshot expiration and orphan cleanup — managed automatically without any user-side scheduling.
Higher write throughput — AWS reports up to 10x higher transactions per second compared to general-purpose S3 buckets. For high-concurrency streaming ingestion pipelines, this can be the deciding factor — Glue Catalog limits concurrent UPDATE TABLE calls to roughly 10 per second per table by default.
Intelligent-Tiering — automatic storage cost optimization based on access patterns. Compaction runs only on files in the Frequent Access tier, reducing maintenance costs on cold data.
SageMaker Lakehouse integration — S3 Tables integrates with SageMaker Lakehouse for unified analytics across S3 Tables, Redshift, and third-party sources through a single management experience.
Cross-region replication — read-only replicas of tables across regions and accounts, preserving full snapshot history.

For a team of two or three engineers standing up their first Iceberg lakehouse on AWS, this eliminates weeks of infrastructure work. Credit where it is due: zero-configuration maintenance is a genuine step forward for the ecosystem.

Where S3 Tables falls short

The constraints surface as soon as requirements move beyond the simple case.

Black-box maintenance

S3 Tables handles compaction, but you cannot see what it did. There are no detailed compaction logs, no before/after file counts, and no way to correlate a compaction run with a change in query latency. The trigger conditions for auto-compaction are undocumented — understanding when and why compaction fired (or didn't) requires an AWS support case.

For streaming workloads with freshness SLAs, this opacity is a production risk. Compaction that runs during a peak query window can temporarily increase latency. Without visibility into the schedule, you cannot coordinate maintenance with critical workloads.

There is also a subtler issue: commit contention. S3 Tables runs maintenance operations continuously in the background, which can advance table metadata and conflict with your writes. Pipelines see CommitFailedException: branch main has changed — a retry-and-backoff situation that requires explicit handling in your ingestion code. AWS documents that maintenance jobs themselves can fail due to "out of date metadata" and get retried. For high-frequency writers, this contention is an operational reality that the "zero-ops" pitch does not surface.

Snapshot management has its own gotchas. If any user-defined tag or branch exists on a table, snapshot management fails for the entire table — no snapshots are expired or removed until you delete all tags and branches. Setting Iceberg snapshot retention properties (history.expire.max-snapshot-age-ms or history.expire.min-snapshots-to-keep) as table properties also causes snapshot management to fail silently. These are not edge cases for teams using Iceberg's branching features for testing or auditing.

The maintenance API lets you configure target file size and compaction strategy per table. That is the extent of your control.

Engine compatibility gaps

S3 Tables works natively with Athena, EMR, and Redshift. Through the REST Catalog API and the Glue Data Catalog federation, AWS also lists Spark, Trino, Flink, Snowflake, and DuckDB as compatible. Engine breadth has improved significantly since launch. But practical gaps remain:

dbt: dbt-athena 1.11.0 (July 2026) shipped end-to-end S3 Tables support via catalogs.yml v2. This is real progress. However, S3 Tables does not support ALTER TABLE RENAME, which breaks dbt's standard near-zero-downtime table swap. The materialization uses drop-and-recreate on every run, S3 cleanup is skipped (since S3 Tables manages storage), and tables must be lowercase-only. For teams where dbt is the primary transformation framework — which is most modern data teams — this is workable but not equivalent to standard Glue-backed Iceberg.

Views: CREATE VIEW is not supported against the S3 Tables catalog endpoint. Views are fundamental to most transformation and BI layers.

DDL restrictions: The REST endpoint does not support stage-create (blocking CTAS in some engine configurations), multi-level namespaces, or OAuth-based authentication. Metadata files over 50 MB return a 400 Bad Request.

Catalog separation: S3 table bucket catalogs surface as a federated s3tablescatalog in the Glue Data Catalog, which is an improvement over full isolation. But the hierarchical mapping (table bucket → child catalog → database → table) means cross-catalog joins between standard Glue tables and S3 Tables still require fully qualified references. It is more integrated than at launch, but not seamless.

Permission failure modes: A notable gotcha from production PoCs: when Lake Formation permissions are missing, SHOW TABLES returns empty results rather than an access-denied error. There is no way to distinguish between "table doesn't exist" and "you lack permissions." This was reported as the single biggest time sink in at least one published S3 Tables evaluation.

The structural concern remains: new tools in the data ecosystem integrate against the open Iceberg spec first. When a query engine ships Iceberg support, it works against a standard Glue-registered table on day one. S3 Tables has narrowed the gap significantly through its REST endpoint and Glue federation, but some tools still need specific adapter work for the s3tablescatalog path.

Security: Lake Formation support with caveats

S3 Tables now integrates with Lake Formation through the Glue Data Catalog federation, supporting column-level, row-level, and cell-level security. This is a significant improvement over the table-level-only IAM policies available at launch.

The caveat: when you enable "full table access" for external engines (third-party engines outside the AWS analytics stack), Lake Formation returns credentials without enforcing column or row filters. Fine-grained access control only applies when engines use Lake Formation credential vending without the full-table-access flag. For teams running mixed engine stacks — which is common in production — this means security enforcement depends on how each engine connects, not just what permissions are defined. Standard AWS engines (Athena, EMR, Redshift) enforce Lake Formation grants; external engines may bypass them depending on configuration.

AWS-only, exit requires rewrite

To be clear about what is and is not locked in: the data itself is standard Apache Iceberg on Parquet. The format is open. The lock-in is at the storage and catalog layer. S3 table buckets are not accessible via standard S3 object APIs — you cannot aws s3 ls the underlying files or point another catalog at the same path. Moving data out of S3 Tables requires a full data export (typically via Spark CTAS), not a catalog pointer swap.

For a small table, this is a manageable migration. For a lake at scale, it is a significant project. As one detailed comparison put it: "With self-managed Iceberg, your files are on standard S3, accessible via standard S3 APIs. The catalog is a pointer. Switching to any other catalog means re-registering the table path. Nothing moves. S3 Tables is a different commitment."

There is also a subtler ecosystem portability dimension. When your catalog is proprietary, ecosystem support depends on AWS's roadmap and release schedule. On the self-managed Iceberg path, integrations happen in the open, on the community's timeline. That structural difference matters when the ecosystem is moving as fast as it is now.

No cross-estate observability

S3 Tables provides basic CloudWatch metrics per table — object count, storage size, maintenance job status. There is no cross-table health dashboard, no severity-ranked alerting, and no way to see which tables are degrading before they affect query performance or cost.

At scale, the question is not "is compaction running?" — it is "which of my 500 tables has a small-file ratio above 40%, and what is the impact on query latency across the three engines that read it?"

Here's a preview of table health observability with a control plane like LakeOps:

Incomplete maintenance coverage

S3 Tables runs compaction, snapshot expiration, and orphan cleanup. It does not handle manifest rewriting or Puffin column-level statistics computation. Both directly affect query planning performance:

Manifest bloat: Frequent commits accumulate hundreds of small manifest files. Each one must be opened during query planning, adding seconds of latency before any data is read.
Missing statistics: Without fresh Puffin NDV, min/max, and null-count statistics, engines cannot prune files effectively. They scan more data than necessary.

These are not niche operations. They are part of the standard Iceberg maintenance lifecycle, and their absence means S3 Tables' "fully managed" maintenance is, in practice, partial.

The alternatives

When engineers search for S3 Tables alternatives, they are typically looking for one of three things: more engine flexibility, more operational control, or vendor neutrality. The answer depends on which constraint is binding.

Self-managed Iceberg on standard S3

The traditional approach: write Iceberg tables to standard S3 buckets, register them in AWS Glue Data Catalog (or a REST catalog like Apache Polaris or Nessie), and run maintenance yourself.

What you gain:

Every Iceberg-compatible engine works on day one — Trino, Spark, Flink, DuckDB, Snowflake, Databricks, ClickHouse, dbt.
Full Lake Formation support for column-level security and row-level filtering without the external-engine caveats that affect S3 Tables.
Standard S3 object access — no proprietary storage layer, no exit rewrite. Switching catalogs means re-registering the table path; nothing moves.
Complete maintenance control — you decide when compaction runs, what sort order to use, and how aggressively to expire snapshots.

What it costs:

You own the entire maintenance lifecycle. Compaction is a Spark job or Athena OPTIMIZE call. Snapshot expiration and orphan cleanup are separate procedures. Getting the sequence wrong — for example, running orphan cleanup before snapshot expiration finishes, or skipping manifest rewrites after compaction — wastes compute or risks deleting files still referenced by active snapshots.
At 10–50 tables, this is manageable with a few DAGs. At 500+ tables with different ingestion rates, it becomes the primary source of operational incidents.
No built-in observability across the estate.

Self-managed Iceberg is the default for most production teams today. It works — but the operational burden scales linearly with the number of tables, and most teams underestimate the ongoing cost until they are deep into it.

Open-source REST catalogs (Apache Polaris, Lakekeeper, Nessie)

If the concern is catalog lock-in rather than maintenance automation, an open REST catalog can replace S3 Tables' built-in catalog without the proprietary coupling.

Apache Polaris (graduated to ASF Top-Level Project in 2026) implements the Iceberg REST Catalog spec with RBAC, credential vending, multi-engine support, and external catalog federation. It can be self-hosted on Kubernetes with PostgreSQL, or used through managed offerings like Snowflake Open Catalog or Dremio Catalog.

Lakekeeper is a Rust-based REST catalog focused on authorization — OPA-based policies, audit events, and fine-grained governance.

Nessie adds Git-like branching and tagging to the catalog layer, enabling table versioning and isolated experimentation.

What you gain:

Vendor-neutral catalog that works on any cloud.
Full Iceberg REST spec compliance — every engine integrates without custom adapters.
Credential vending, RBAC, and (in Polaris) external catalog federation.

What they don't solve:

None of these catalogs handle compaction, snapshot expiration, or orphan cleanup on their own. They are metadata management services, not maintenance automation. You still need a separate system for the operational lifecycle.
Self-hosting requires infrastructure: PostgreSQL backend (Polaris) or Rust binary (Lakekeeper), Kubernetes deployment, monitoring, upgrades.

Emerging: open-source maintenance layers. Floe connects to Polaris and provides policy-driven maintenance (compaction, snapshot expiration, orphan cleanup, manifest optimization) executed via Spark or Trino. Firn is a Go-based maintenance daemon that works with Lakekeeper, Polaris, Nessie, or Glue — using DuckDB for compaction instead of Spark. Both are pre-1.0 but represent the open-source ecosystem's response to the maintenance gap.

An open catalog solves the portability problem. With Floe or Firn, you can start to solve the maintenance problem too — though both require more assembly than managed alternatives.

Lakehouse control plane (LakeOps)

A control plane complements the storage and catalog layers by handling the operational lifecycle that neither S3 Tables nor open catalogs address on their own.

LakeOps connects to any Iceberg catalog — Glue, S3 Tables, Polaris, Nessie, Gravitino — and provides autonomous maintenance, observability, and optimization across the entire estate. The distinction from S3 Tables is that LakeOps works with your existing infrastructure rather than replacing it, and provides the visibility and intelligence that managed services abstract away.

What this approach covers:

Cross-estate observability — a single dashboard covers every table across every catalog (Glue, S3 Tables, REST) with severity-ranked insights. You see which tables are degrading before the impact reaches query dashboards or invoices.

Full maintenance sequence in correct dependency order — snapshot expiration → orphan cleanup → compaction → manifest rewrite → statistics computation. Each step's output feeds the next, ensuring cleanup always precedes compaction and metadata optimization always follows it.

Query-aware compaction — instead of blind binpack, the system analyzes which columns your queries actually filter, join, and group on, then physically sorts data to match those patterns. Engines skip entire file groups through min/max pruning, reducing I/O significantly. The sort strategy adapts as query patterns change.

Purpose-built execution engine — compaction runs on a native Rust/DataFusion runtime that bypasses the JVM stack entirely. Production benchmarks show 7× higher throughput than Spark and over 20× faster than S3 Tables' built-in compaction on equivalent datasets.
Health-driven triggers, not cron schedules — every table is scored along multiple structural dimensions (file fragmentation, delete-file depth, snapshot age, manifest bloat) and maintenance fires only when indicators cross actionable thresholds. Streaming tables might be compacted hourly, batch tables daily, and well-maintained tables are skipped entirely.

Policy-driven governance — maintenance rules defined once at the namespace or catalog level, with per-table overrides where needed. Every execution is logged with before/after metrics for auditability.

For a deeper look at how each capability works in practice, the LakeOps platform page walks through the full system from observability through compaction, routing, and governance.

Comparison: S3 Tables vs. alternatives

Dimension	S3 Tables	Self-Managed (Glue + S3)	Open Catalog (Polaris)	Control Plane (LakeOps)
Compaction	Automatic, opaque	Manual (Spark/Athena)	Not included	Autonomous, query-aware, Rust engine
Snapshot / orphan mgmt	Automatic (fails if tags/branches exist)	Manual procedures	Not included	Autonomous, sequenced
Manifest rewrite	Not available	Manual	Not included	Automatic
Statistics (Puffin)	Not available	Manual	Not included	Automatic post-compaction
Engine support	Athena, EMR, Redshift, Spark, Trino, Flink, Snowflake, DuckDB via REST	All Iceberg engines	All Iceberg engines	All engines + routing
dbt	Supported (dbt-athena 1.11+, drop-recreate)	Full production support	Full	Full
Security model	Lake Formation via Glue (caveats for external engines)	Lake Formation (column, row, tag)	RBAC (Polaris-native)	Works with Lake Formation + operational policies
Observability	CloudWatch basics	Whatever you build	None	Health classification, insights, audit trails
Multi-cloud	AWS only	Any cloud (with catalog choice)	Any cloud	Any cloud
Vendor lock-in	Open format, proprietary storage layer, exit = data rewrite	Standard S3, portable	Fully open	No lock-in, works with existing stack
Maintenance control	Config only (target size, strategy)	Full control, full burden	N/A	Policy-driven with full visibility

What matters for your evaluation

The right alternative depends on which S3 Tables constraint is actually blocking you.

If engine compatibility is the issue

S3 Tables' engine compatibility has improved significantly — Spark, Trino, Flink, Snowflake, and DuckDB all connect through the REST endpoint. But if you need standard dbt table-swap semantics, full CREATE VIEW support, or CTAS via stage-create, self-managed Iceberg on standard S3 with Glue Data Catalog remains the straightforward path. Every engine works without adapter caveats. The trade-off is owning the maintenance lifecycle, which is well-understood and automatable — but requires sustained engineering investment.

If vendor lock-in is the concern

An open REST catalog like Apache Polaris gives you a vendor-neutral metadata layer that works on any cloud and any engine. You still need to solve maintenance separately. For teams that already have mature Spark-based maintenance pipelines and just want catalog portability, this can be the right layer to change.

If operational complexity is the bottleneck

This is where most teams land after running Iceberg in production for more than a few months. The problem is not any single capability — it is the combinatorial complexity of managing compaction schedules, monitoring health across hundreds of tables, keeping sort orders aligned with evolving query patterns, and ensuring the maintenance sequence runs correctly every time across different ingestion cadences.

S3 Tables addresses this by abstracting the problem away behind a black box. The control plane approach addresses it by automating the same operations with full visibility and policy governance — plus the query-aware optimization and cross-engine telemetry that no black-box service can provide, because they require understanding how your tables are actually queried.

If you are already using S3 Tables

LakeOps connects to S3 Tables as a catalog source alongside Glue and other catalogs. You do not have to migrate off S3 Tables to get cross-estate observability, health classification, or policy governance. For tables that work well in S3 Tables — high-throughput streaming writes that benefit from managed compaction and higher TPS, AWS-native stacks where the Glue federation provides sufficient engine access — you can keep them there. For tables that need standard dbt semantics, view support, or unrestricted Lake Formation enforcement across external engines, those live in standard S3 with Glue. LakeOps provides the unified operational layer above both.

The maintenance gap is the real problem

The deeper pattern behind the "S3 Tables alternative" search is not really about S3 Tables at all. It is about the gap between Iceberg's capabilities as a format and what it takes to operate Iceberg well in production.

Iceberg gives you ACID transactions, time travel, schema evolution, and engine-agnostic access. It does not give you a system that keeps tables healthy as data flows in. That requires compaction, cleanup, manifest optimization, statistics computation, health monitoring, and policy enforcement — coordinated correctly across every table in the estate.

S3 Tables fills part of this gap — and it fills it well for the use cases it targets. If you are building an AWS-native lakehouse with Athena and Redshift, want zero-ops maintenance, and can work within the DDL and view restrictions, S3 Tables is a strong starting point. The ecosystem has improved steadily since launch: Lake Formation integration, Glue federation, dbt support, and broader engine compatibility have addressed several early limitations.

But many production estates outgrow the managed-service model. They run multiple engines. They need views and standard dbt semantics. They need to understand what maintenance did and why. They need policies that work across catalogs, and observability that covers the entire lake — not just one storage layer.

Open catalogs like Apache Polaris fill the metadata portability piece. Emerging tools like Floe and Firn add open-source maintenance automation. Self-managed deployments give maximum control at the cost of operational burden.

A lakehouse control plane fills the gap end to end: the full maintenance lifecycle with correct sequencing, the observability to know what is happening, the policy layer to govern it at scale, and the execution engine to do it efficiently — across every catalog type, every engine, and every cloud. That is the architectural layer the Iceberg ecosystem has been missing, and it is the one most teams end up needing regardless of where their data physically sits.

If you are evaluating alternatives to S3 Tables, start by clarifying which constraint matters most — portability, engine support, security, or operational maturity. Then pick the combination of storage, catalog, and operational layers that matches. LakeOps handles the operational layer above any combination of the others.

Managed Data Lake: A Guide for 2027

joni sar — Sun, 23 Aug 2026 09:05:42 +0000

Managed Data Lake: A Guide for 2027

Apache Iceberg is the standard table format for production data lakes in 2027. Every major engine reads and writes it natively. The catalog ecosystem standardized on REST. You own your data on commodity storage with no lock-in.

But Iceberg deliberately separates the table format from the system that keeps tables healthy. It gives you the primitives for maintenance — rewrite_data_files, expire_snapshots, remove_orphan_files, rewrite_manifests — but not the intelligence to decide when, how, and in what order to run them. Without that operational layer, every Iceberg table degrades over time: small files accumulate, snapshots bloat metadata, sort orders drift from query patterns, orphan files inflate storage costs, and query performance decays silently until something breaks visibly.

This operational gap is the central challenge of running a data lake at production scale. Netflix built four internal services to address it — Autotune for compaction strategy selection, Polaris for catalog management, janitors for garbage collection, Metacat for cross-service observability — each staffed by dedicated teams over multiple years. Google engineered automatic compaction and garbage collection directly into BigLake, so their managed Iceberg tables stay healthy regardless of write volume or query pattern changes.

In 2027, you do not need to replicate that investment. This guide covers what "managed" actually means for a data lake, the degradation mechanics that make it necessary, the control plane architecture that solves it, and the practical paths to getting there — whether you are running 50 tables or 5,000.

Why Lakes Degrade — The Mechanics

The degradation pattern is predictable and present in nearly every Iceberg lake running for more than three months without dedicated maintenance. Understanding these mechanics is necessary regardless of which management approach you choose.

The Small-File Problem

Every streaming writer — Flink, Spark Structured Streaming, Kafka Connect, RisingWave, CDC connectors — produces files sized by checkpoint intervals, not by optimal read patterns. A Flink job committing every 60 seconds across 100 active partitions creates 144,000 new files per day. Each of those files might be 1–5 MB. The target for efficient reads is 128–512 MB.

Every query pays per-file overhead: an S3 GET request ($0.0004 per 1,000), a Parquet footer parse, a task-scheduling slot in the engine. This means query compute scales with file count rather than with actual data volume. You are paying for how the data was written, not for how much data exists.

The insidious part is the rate. A batch table that loads once daily accumulates files slowly — degradation takes months to become visible. A streaming table crosses into problematic territory within days. A high-throughput streaming table (hundreds of partitions, sub-minute commits) can reach a degraded state within hours of going live if no background compaction runs alongside the writer. The longer you wait to compact, the heavier and more disruptive the operation becomes — it conflicts with active writers, consumes significant cluster resources, and takes hours instead of seconds.

Snapshot Accumulation

Every Iceberg commit creates a new snapshot — that is the mechanism that enables time travel and atomic rollback. A streaming table at 5-minute commits accumulates ~288 snapshots per day. Without expiration, counts reach tens of thousands within weeks.

Query planning reads the snapshot chain to resolve the current table state. Once you exceed roughly 1,000–2,000 retained snapshots, planning time degrades measurably. The metadata.json file grows — production environments have seen it reach 400+ MB. The symptom is slow EXPLAIN, slow first-row-time, and users reporting that "the query hangs for a few seconds before anything happens." The data scan itself is fine. It is the planning that has become expensive.

The correct retention window depends on your time-travel requirements. For most analytics workloads, 3–7 days of snapshots is sufficient. Regulated industries (financial, healthcare) may require 30–90 days — but should understand the planning-time trade-off and invest in more aggressive manifest rewriting to compensate. Whatever your retention policy, it must be actively enforced. Iceberg does not expire snapshots on its own.

Manifest Fragmentation

Manifests are Iceberg's lookup structure — they tell engines which data files exist in which partitions and what their column-level statistics are. Every commit appends at least one manifest entry. Over months of streaming writes, you accumulate hundreds or thousands of small manifest files.

The query planner must read all of them to build an execution plan. This adds seconds of latency before any data file is opened. When Iceberg 1.11 introduced server-side scan planning (shifting metadata traversal off query engines entirely), lean manifests became even more valuable — the planning server benefits proportionally from fewer, larger manifest files.

Most teams misdiagnose slow planning as an engine problem — they try tuning memory, scaling coordinators, or upgrading engine versions. It is almost always the manifests. Rewriting them into a smaller number of well-merged files is one of the highest-leverage operations available — and one of the least commonly automated.

Orphan Files

Failed Spark stages, interrupted compaction, aborted writes, schema-evolution retries, OOM kills during large rewrites, and partial commits all leave files on object storage that no Iceberg snapshot references. They are invisible to SELECT queries, invisible to Iceberg metadata, invisible to your data quality checks — but fully visible on your cloud bill.

You cannot find them by querying your tables. You must compare what exists in object storage against what Iceberg metadata references — a set difference operation that requires listing entire prefixes and cross-referencing with the catalog. This is why orphan accumulation goes undetected for months. Without a dedicated process that runs this comparison regularly, you have no way to know how much you are spending on dead data.

Production orphan sweeps routinely reclaim 20–40% of a lake's storage. For a 100 TB lake, that is 20–40 TB of billable storage serving no purpose. At $0.023/GB/month (S3 Standard), that is $460–$920 per month for one lake — just in orphan storage. Multiply by the number of accounts, regions, and environments your organization operates.

Sort-Order Drift

Bin-pack compaction — merging small files into larger ones — solves the file-count problem but does nothing for data organization. If your data is not physically sorted by the columns queries actually filter on, Parquet min/max statistics span wide ranges and engines cannot skip irrelevant row groups.

Consider a transactions table sorted by transaction_id (because it was the natural append key) but queried primarily by customer_id and event_date. Every file contains a wide range of customer IDs, so a predicate like WHERE customer_id = 'abc123' cannot skip any files based on statistics alone — the engine scans everything.

Sort the same data by customer_id, event_date and the statistics become tight. A query for one customer touches one or two files instead of hundreds. The engine eliminates 95%+ of scan volume before reading a single row. This is not an incremental improvement. It is a categorical difference in query performance.

The challenge is that sort orders set at table creation drift. The column that was the primary filter six months ago might not be today. New consumers join. Dashboard patterns shift. ML feature pipelines add access patterns the original designer never anticipated. Across hundreds of tables with different consumers, manual sort-order tuning is impractical — and sort orders that are wrong silently cost you compute on every query, every day, without generating any error or warning.

Delete File Accumulation

Merge-on-read tables accumulate position and equality delete files with every UPDATE or DELETE operation. This is the mechanism that makes updates fast at write time — rather than rewriting the entire file, Iceberg records "this row is deleted" in a separate file.

The cost lands on reads. Every query must reconcile live rows against all accumulated delete files for the partitions it touches. This reconciliation cost grows linearly with each write cycle. At high delete volumes — CDC workloads doing frequent upserts, GDPR compliance deleting records, slowly changing dimensions — the read-time overhead becomes significant. Queries that should complete in seconds spend most of their time reconciling deletes rather than reading actual data.

Iceberg 1.11's deletion vectors (backed by Roaring bitmaps) improve the mechanism by storing deletes more compactly and enabling faster reconciliation. But the fundamental accumulation problem remains: unless compaction regularly applies those deletes (physically rewriting files to exclude deleted rows and then removing the delete files), read performance degrades continuously.

The Organizational Cost

Beyond the technical debt, the team cost scales worst.

Maintenance responsibilities accumulate incrementally: compaction scripts, expiration logic, monitoring, alerting, per-table configuration for heterogeneous workloads, debugging writer conflicts, handling OOM failures, on-call rotation for overnight maintenance failures, documentation, and onboarding. Each table added to the lake adds marginal load to a team that is not growing proportionally.

Platform engineers who should be building data products, self-service tooling, or pipeline reliability improvements are instead maintaining an ever-growing library of operational scripts. The knowledge concentrates in a few heads — when those people leave, the lake degrades within weeks because nobody else knows what the DAGs do or why they are configured that way.

This cost does not show up on cloud bills but manifests in team velocity, retention risk, and the growing gap between what the data platform promises and what it delivers.

What "Managed" Actually Means

The term is overloaded. Vendors use it to mean proprietary platforms you migrate into. Cloud providers use it for single-service table maintenance. Some teams use it to mean a weekly compaction cron job. Others equate it with having a catalog.

A more useful definition: a managed data lake is one where every table stays healthy, every query hits well-organized data, and operational cost does not grow linearly with table count — without requiring a platform team that scales proportionally with the lake.

This definition has three important implications. First, it is about outcomes, not tools. Running compaction nightly does not make your lake managed if half your tables are still degraded. Second, it covers the full lifecycle — not just file compaction, but snapshot management, orphan cleanup, manifest optimization, sort alignment, policy enforcement, and observability. Third, it must scale sublinearly: managing 500 tables should not require 10x the effort of managing 50.

The Architecture

Your data lake already has three layers: commodity object storage (S3, GCS, ADLS), a table format and catalog (Iceberg plus Polaris/Glue/Nessie/Gravitino/Lakekeeper), and compute engines (Spark, Trino, Snowflake, Athena, DuckDB, Flink). These give you storage economics, structural reliability, and query flexibility.

What you are missing is the fourth layer: an operations control plane.

This is not the same as the catalog. The catalog (Polaris, Glue, BigLake Metastore) is a metadata control plane — it resolves table locations, handles access control, and provides the REST interface for multi-engine access. Every major catalog now implements the REST spec: Polaris (Apache top-level project since February 2026), Glue's REST adapter, BigLake Metastore, Nessie, Gravitino, Lakekeeper.

The operations control plane sits above the catalog. It uses metadata as input, but its job is fundamentally different: observe structural health across every table, classify what needs attention, decide what maintenance to run, execute it safely and in the correct sequence, adapt compaction to real query patterns across all engines, enforce governance at lake scale, route queries to optimal engines, and feed outcomes back into future decisions.

LakeOps is this operations control plane — a purpose-built product for Apache Iceberg that does nothing else.

It is not an engine, not storage, not a catalog. It connects to your existing catalogs (AWS Glue, Polaris, Gravitino, Nessie, Lakekeeper, S3 Tables) and engines (Spark, Trino, Flink, Snowflake, Athena, DuckDB), adds autonomous management across the entire lake, and requires no data movement or pipeline changes. Most teams have their first lake-wide health assessment within ten minutes of connecting a catalog.

The key architectural property: it does not replace anything. If you remove the control plane, everything continues working exactly as before — just without the automation. Your storage, your engines, your catalog, your data — all unchanged. The control plane adds intelligence to an existing stack. It does not create dependency.

The Maturity Spectrum

Teams operating Iceberg follow a predictable progression. Identifying where you sit determines what to invest in next.

Stage 1: No maintenance. Iceberg adopted for format-level benefits. Pipelines land data. Nobody explicitly owns table health. Performance degrades and storage costs rise gradually, but the symptoms are not connected to a root cause. Common in organizations that migrated to Iceberg within the last year.

Stage 2: Reactive fixes. After an incident — slow dashboard, cost anomaly, failed SLA — an engineer writes a one-off compaction script or snapshot expiration job. The immediate pain resolves, but only for the tables that broke visibly. The rest of the lake continues degrading silently. The cycle repeats every few months on different tables.

Stage 3: Scheduled automation. The team builds Airflow DAGs or cron jobs running maintenance on a fixed schedule. Compaction nightly. Snapshot expiration weekly. Orphan cleanup monthly. This works for stable batch workloads but has structural limitations at scale:

Wastes compute on healthy tables that do not need maintenance
Misses tables that degrade between cycles (a streaming table needing hourly compaction gets nightly)
Cannot adapt to changing workloads — new query patterns do not trigger sort-order changes
Runs operations as independent jobs without sequencing coordination, creating race conditions
The maintenance infrastructure itself requires ongoing maintenance: updating DAGs, handling failures, adjusting schedules, onboarding new tables

Stage 4: Autonomous management. A dedicated system continuously observes every table, classifies health based on combined structural, storage, and query signals, sequences operations in the correct order, adapts compaction strategy to real query patterns from cross-engine telemetry, enforces policies lake-wide, and feeds outcomes back into decisions. Maintenance is signal-driven rather than schedule-driven. Tables that need attention get it immediately. Healthy tables cost zero compute.

Most production lakes sit between Stage 2 and Stage 3. The gap from Stage 3 to Stage 4 is where teams either build a dedicated platform (6–12 months, 2–3 senior engineers full-time, plus ongoing maintenance of the platform itself) or adopt an existing control plane. Stage 3 is where DIY efforts stall — building cross-engine telemetry collection, health classification logic, sequencing and conflict resolution, adaptive compaction strategy, and policy inheritance is a non-trivial distributed systems project that most teams underestimate.

How the Control Plane Works

A control plane is not a collection of scripts with a dashboard. It is a closed-loop system where each stage feeds the next. The difference between Stage 3 automation (scheduled scripts) and Stage 4 automation (autonomous management) is entirely in this loop: observe → decide → act → optimize → govern → route → enable → learn.

Observation: Unified Telemetry

Everything starts with signals. Three telemetry streams feed every decision the system makes:

First, structural metadata from Iceberg — file counts per partition, average file sizes, manifest depth, snapshot count and velocity, delete file ratios, current sort order configuration, partition layout, and schema history. This tells you the table's physical state — the raw structural facts that indicate health or degradation.

Second, object storage metrics — total bytes per prefix, orphan file counts, access patterns, request rates, storage class distribution. This reveals the cost reality that metadata alone does not capture. A table might look healthy in Iceberg metadata while harboring terabytes of orphaned files in its storage prefix.

Third, and most critically, cross-engine query telemetry — which columns appear in WHERE, JOIN, and GROUP BY clauses from every engine reading your tables. Trino knows its own queries. Spark knows its own. Snowflake knows its own. Athena knows its own. None of them knows what the others are doing on the same table. None of them can tell you what the combined access pattern looks like across your full consumer base.

A control plane collects this telemetry from every connected engine, correlates it per table, and uses the combined signal to drive optimization decisions that no single engine could make alone. This is the fundamental capability that separates an operations control plane from single-engine maintenance tools — it sees the whole picture.

This unified telemetry is what enables the system to classify every table as Healthy, Warning, or Critical — and to surface specific, actionable insights ranked by severity. You know exactly which of your 500 tables needs attention right now, what the problem is, what caused it, and what the fix should be.

Decision: Health-Driven Prioritization

Raw telemetry becomes actionable through health classification. Instead of blindly running compaction on every table overnight (Stage 3), the system targets precisely the tables that need attention — and leaves healthy tables alone.

A table's health score reflects the combined severity of its degradation signals: file count relative to target, manifest fragmentation ratio, snapshot depth relative to retention policy, sort-order alignment with actual query patterns, orphan volume, and delete file accumulation. Each dimension has independent thresholds, and the composite determines the classification.

When a table crosses from Healthy to Warning (file count exceeds target, sort order drifts from query patterns, manifest fragmentation passes threshold), maintenance triggers. When it returns to Healthy, operations pause. Critical tables get immediate attention regardless of scheduling. Warning tables are queued by priority. Healthy tables cost zero compute — no maintenance runs against them until conditions change.

This feedback loop eliminates the two failure modes of Stage 3 automation: wasting compute on healthy tables that do not need attention, and missing tables that degrade between fixed-schedule cycles. A streaming table that crosses into Warning two hours after the last scheduled maintenance run does not wait until tomorrow night. It gets addressed when the signal fires.

Execution: Sequenced Maintenance

The individual operations are well-understood — Iceberg provides the procedures. What matters is the order, the coordination, and the safety.

The correct sequence is: expire snapshots → remove orphan files → compact data files → rewrite manifests → refresh statistics. Each step depends on the previous:

Expiration dereferences files and snapshots that are past their retention window
Orphan cleanup then safely removes those dereferenced files from storage — only possible after expiration has run
Compaction operates on live data only — not on files that are about to be expired — reducing wasted work
Manifest rewriting consolidates the new manifest structure that compaction created
Statistics refresh ensures the newly organized data has accurate min/max values for predicate pushdown

Running these in the wrong order wastes compute (compacting data that will be expired in the next step), creates safety risks (removing files that unexpired snapshots still reference), or produces suboptimal results (rewriting manifests before compaction means they immediately fragment again on the next compact).

At Stage 3, these operations typically run as independent cron jobs — compaction in one DAG, expiration in another, orphan cleanup in a third, each with its own schedule and no awareness of the others. This creates exactly the sequencing failures described above. A control plane runs them as a coordinated pipeline per table, every time, conflict-aware (never compacting partitions with active writers), with OCC-aware retry logic for optimistic concurrency conflicts.

Adaptive maintenance bundles this further: rather than configuring each operation's frequency separately, a single data-driven policy monitors commit frequency, file accumulation rate, query load, and seasonal patterns — then runs the right operations at the right frequency, automatically. A high-velocity streaming table gets frequent bin-pack compaction to keep pace with the write rate. A stable batch table gets occasional deep sort compaction when access patterns shift. A table that received a sudden burst of writes gets immediate attention. One policy toggle replaces dozens of per-table DAG configurations.

Optimization: Query-Aware Compaction

This is where the control plane delivers its highest-impact differentiation from scripts and scheduled jobs.

Basic bin-pack compaction normalizes file sizes — merging many small files into fewer large ones. That is necessary (it solves the per-file overhead problem) but insufficient (it does nothing about data organization). A table with perfectly sized files but random data ordering still forces engines to scan everything because min/max statistics span the full value range.

The real performance leverage is in sort compaction: physically rewriting data so records are ordered by the columns queries actually filter on. When data is correctly sorted, Parquet row-group min/max statistics become tight — each row group contains a narrow range of values for the sort columns. Engines read these statistics during planning and skip entire file groups whose ranges do not match the predicate. The difference is not incremental. It is the difference between scanning 100% of a table and scanning 5–15%.

The challenge at scale is knowing what to sort by — and keeping it current. Static sort orders set at table creation drift within weeks as query patterns evolve. A column that was the primary filter six months ago might be secondary today. New consumers arrive. Dashboard logic changes. ML pipelines add new access patterns. Across hundreds of tables with different consumers across different engines, manual sort-order tuning is impractical.

Cross-engine telemetry solves this. The control plane observes which columns are actually filtered, joined, and grouped across all engines hitting each table — Trino, Spark, Snowflake, Athena, DuckDB — then physically re-sorts data to match the real combined access pattern. As patterns change, the sort order adapts. The optimization is continuous and automatic, not a one-time configuration that decays.

LakeOps executes compaction on a purpose-built Rust/DataFusion engine — no JVM, no garbage collection pauses, no Spark clusters to provision or decommission. Compaction is event-driven (triggered by health signals, not cron schedules) and conflict-aware (never touching partitions with active writers, with built-in OCC retry). The engine completes bin-pack and sort operations in a fraction of the time Spark requires for the same workload, without the cluster management overhead.

Before committing to a sort-order change on a large table — where a wrong decision means rewriting terabytes of data for no benefit — layout simulations let you test strategies safely. Simulations create a real Iceberg branch from the latest snapshot, apply the proposed layout to that branch, replay actual query patterns from cross-engine telemetry against it, and compare results against the current baseline. You see projected file counts, data-skipping improvement percentage, and scan reduction for each candidate strategy — before writing a single byte to production data. The branch is discarded after analysis. This is the verification step that makes autonomous sort optimization trustworthy at scale.

Governance: Policies That Scale

At one table, configuration is a one-time decision. You pick a compaction strategy, set a retention window, choose a target file size, and move on. At 50 tables it becomes repetitive. At hundreds of tables across multiple catalogs and namespaces, it becomes governance entropy.

The symptoms: some tables have compaction configured, others use defaults nobody verified. Retention windows are inconsistent — production tables with 90-day retention next to staging tables with no expiration at all. New tables inherit nothing because nobody templated the configuration. An engineer leaving means their table-specific tuning knowledge leaves with them. Auditors ask "what is your data retention policy?" and nobody can answer precisely because the truth is scattered across table properties, DAG configs, and Terraform state files.

A control plane enforces declarative policies with hierarchical inheritance: compaction strategy, target file size, snapshot retention, cleanup safety windows, and maintenance frequency defined at organization, catalog, namespace, or table level. Everything cascades downward — set a policy at the namespace level and every table in that namespace inherits it. New tables automatically pick up the correct configuration. Exceptions override at the leaf level when specific tables need different treatment. Policies are versioned, auditable, and diff-able — changes are tracked, reviewable, and reversible.

Routing: Multi-Engine Cost Optimization

Running multiple engines on the same Iceberg tables is standard practice in 2027. The REST catalog spec means any engine can read any table. Most production lakehouses use at least three: Trino for interactive analytics, Spark for heavy ETL, and one of Snowflake/Athena/DuckDB for BI or ad-hoc queries. Each has radically different cost and latency profiles.

A point lookup costs fractions of a cent on DuckDB running locally and significantly more on Snowflake's warehouse compute. A multi-table join with complex aggregations completes in seconds on Trino but might take minutes on DuckDB. A 10 TB batch transformation is cheapest on Spark with spot instances. Without routing, applications hardcode engine connections — every query goes to one engine regardless of whether a cheaper or faster option exists for that specific query shape.

A control plane's routing layer classifies query shapes (point lookups, analytical aggregations, complex joins, full-table scans) and sends each to the optimal engine based on cost targets, latency requirements, and current table health. Applications connect to a stable SQL endpoint; backend engine selection adapts as conditions change, as table health improves, and as new engines are added to the pool.

Because the control plane already has cross-engine telemetry and table health data, routing decisions use the same signals that drive compaction — creating a virtuous cycle. Better-compacted tables unlock more routing options (a well-sorted table can serve point lookups from DuckDB that would be too slow on an unsorted table). Better routing data informs compaction priorities (tables with high cross-engine query volume get priority optimization).

AI Readiness: Agent Access at Scale

AI agents are becoming primary consumers of lakehouse data — not a future prediction, but a present reality in many production environments. They issue SQL iteratively, at high frequency, without human review of each query. They explore schemas, join across tables, aggregate at scale, and do it hundreds of times per analysis cycle. This creates requirements that traditional table maintenance does not address: structured schema discovery (agents need to understand table relationships programmatically), cost and safety guardrails (an agent with no limits will issue expensive full-table scans), and storage that self-optimizes for agent access patterns that may differ from human patterns.

LakeOps provides an agent-native MCP interface — any MCP-compatible agent (Claude, GPT, custom models) connects with zero integration code and auto-discovers available tools for schema discovery, health analysis, and query execution. Guardrails stack composably: ReadOnlyGuard prevents DDL and DML, CostEstimateGuard rejects queries that would scan more than a cost threshold, PIIMaskGuard scrubs sensitive columns before results reach the agent, HumanApprovalGuard gates high-stakes operations for human review. These are not all-or-nothing — you compose them per use case.

Agent query telemetry feeds back into the compaction engine alongside human telemetry — the lake self-optimizes for the combined workload. If agents start heavily filtering on a column that humans rarely use, the sort order adapts to serve both access patterns. The lake becomes agent-ready not by adding a separate AI layer, but by extending the same optimization loop to a new class of consumers.

Learning: The Closed Loop

The final stage is what transforms a management tool into a system that improves with time and scale. Every maintenance operation produces measurable outcomes: file count reduction, scan volume improvement, query latency change, storage reclaimed, compute saved. These outcomes are not just metrics to display — they feed back into future decisions.

Sort compaction that improved dashboard latency by 8x on one table informs strategy selection for structurally similar tables. Routing decisions that reduced costs without impacting latency reinforce the model for similar query shapes. Compaction schedules that were too aggressive for a table's write rate — causing frequent OCC conflicts — get automatically dialed back. Target file sizes that produced optimal performance on one storage class get recommended for similar tables.

This is why a control plane gets more effective as the lake grows. More tables under management means more signal about which strategies work for which table shapes. More engines means richer telemetry about cross-engine access patterns. More queries means more precise sort-order recommendations. The operational burden stays flat (or decreases) while the optimization quality increases. The system scales better than a team can — because it learns from the entire lake, not just the tables one engineer happened to look at.

Choosing Your Path

Three approaches to managed data lakes exist in 2027. Each suits different architectural choices and organizational realities. The right one depends on your engine diversity, cloud strategy, and team structure.

Vendor-Managed Platforms

Databricks (Predictive Optimization, Liquid Clustering) and Snowflake (automatic clustering, table maintenance) bundle operational management inside their platforms. If you run exclusively on one vendor's stack — all compute through their engine, all tables in their catalog — this is the lowest-friction path. Maintenance is handled within the ecosystem, tuned for their engine's access patterns, and requires no additional tooling.

The trade-off is architectural scope. Optimization runs only within that vendor's engine. If you later want to query those tables from Trino, DuckDB, or a different vendor — the core value proposition of choosing Iceberg in the first place — maintenance optimization will not account for those external access patterns. The vendor optimizes for its own queries. Cross-engine workloads are invisible to it.

Cloud-Native Services

AWS S3 Tables and Google BigLake provide managed Iceberg maintenance within their respective clouds. Compaction, cleanup, and optimization run automatically for tables registered with that service. No additional tooling needed for those tables.

The trade-off is scope. Maintenance applies to tables within that cloud service only. If your lake spans multiple catalogs, clouds, or includes self-managed Iceberg tables outside S3 Tables or BigLake, you need a separate solution for the rest. Cross-engine query-aware optimization is not typically available because the service lacks visibility into query workloads from external engines — it knows what Athena does but not what Trino or Spark are doing on the same tables.

Dedicated Control Plane

A platform like LakeOps operates across your entire stack — any catalog, any engine, any cloud — because it layers onto existing infrastructure rather than replacing it. It provides the full closed loop (observe → decide → act → optimize → govern → route → enable → learn) without constraining architectural choices.

The trade-off is that you add a service to your stack rather than relying on built-in capabilities. The benefit is that optimization accounts for your actual cross-engine workload, management applies lake-wide regardless of where tables live, policies are consistent across catalogs and clouds, and you retain full engine flexibility — add or remove engines without changing your management layer.

When to Choose Each

The decision framework is practical:

If you run 5–20 tables on a single engine from a single vendor, use that vendor's built-in management. The tooling is sufficient, it is already there, and the additional complexity of a separate service is not justified.

If you run 50+ tables across multiple engines, multiple catalogs, or multiple clouds, you need a layer that spans your stack. No single vendor or cloud service has visibility into your complete workload. Cross-engine telemetry, unified policies, and lake-wide governance require a purpose-built operations layer.

Many organizations run hybrid approaches — vendor-managed tables for workloads that live entirely within one ecosystem, and a dedicated control plane for the open multi-engine portion. The approaches coexist because they solve different scoping problems.

The Economics

Four cost vectors compound in an unmanaged lake. Left unchecked, each grows independently and they reinforce each other. They are also the four that a properly managed lake eliminates:

Storage drops 40–60% as orphan files, stale snapshots, and unreferenced data are continuously cleaned. This is not theoretical — it is the immediate, measurable result of running a proper orphan sweep and snapshot expiration. Production deployments routinely discover that a significant portion of their billable storage is serving no analytical purpose. The savings are recurring: every month you do not clean, the waste accumulates further.

Query compute drops as sort-aware compaction aligns data to real access patterns. When engines skip 80% of row groups via tight min/max statistics instead of scanning everything, compute consumption falls proportionally. Queries that scanned 100 GB to return 100 rows now scan 5 GB. The per-query savings compound across every query, every user, every dashboard refresh, every scheduled report — thousands of times per day across the organization.

Maintenance compute drops when a purpose-built Rust engine replaces Spark clusters for compaction workloads. No JVM startup overhead, no garbage collection pauses, no idle executor cores between compaction runs, no over-provisioning for peak load. Event-driven execution means compute runs only when needed (health signal fires) rather than on a fixed schedule (cluster running overnight regardless of whether any table needs attention).

Engineering time redirects from maintenance firefighting to building data products. Policies replace scripts. Observability replaces guesswork. Health classification replaces manual triage. On-call incidents for table degradation approach zero because degradation is prevented rather than repaired. The platform team stops being reactive (fixing things that broke) and becomes proactive (building capabilities that generate value).

The combined effect for teams running at production scale: the lake becomes cheaper to operate as it grows larger. More data, more tables, more engines — but costs grow sublinearly because the automation handles the marginal complexity without marginal engineering investment. This is the opposite of what happens without management, where cost and complexity grow linearly (or worse) with lake size.

Getting Started

The practical path, regardless of where you sit on the maturity spectrum:

Start with visibility. Connect your catalogs and see the health of every table before any maintenance runs. You will likely be surprised by how many tables are degraded — most teams discover Critical and Warning states on tables they thought were fine. The first insight alone (which tables are Critical, what orphan volume exists, where manifest bloat lives, which sort orders have drifted) changes how you prioritize the next sprint. This step costs nothing and requires no maintenance operations — it is purely diagnostic.

Fix the highest-impact tables first. You do not need to manage everything on day one. Identify the tables with the most file fragmentation, the highest query volume, or the largest orphan accumulation — ideally, tables that score high on all three. Run sequenced maintenance on those: expire, clean, compact, rewrite manifests. The ROI is immediate: dashboards that were slow become fast, storage bills drop measurably, and the team sees what managed looks like on real tables.

Establish policies early. Define your retention windows, target file sizes, compaction strategies, and cleanup schedules — even if you start with manual or semi-automatic execution. These decisions become the governance layer when you move to full automation. A clear, documented policy also prevents the entropy that accumulates when ten engineers configure tables differently based on individual judgment. Consistency across the lake matters more than perfection on any single table.

Expand to full automation. Once you trust the signals and see the impact on your initial tables, enable adaptive maintenance across the lake. Every table gets the right maintenance at the right frequency — coordinated, sequenced, conflict-aware, and fully observable. At this point, your platform team stops maintaining DAGs and scripts and starts building the data products and self-service capabilities that generate value for the organization.

Add routing and AI access last. Multi-engine routing and agent access depend on consistently healthy tables and predictable data layouts. They are the payoff of the operational foundation — not a starting point. A routing layer cannot optimize what it cannot predict. An AI agent cannot reliably query tables whose performance is unpredictable. Get the foundation right first; the advanced capabilities become straightforward once table health is reliable.

The constraint throughout: your data never leaves your storage. Your engines remain yours to choose and change. Your catalogs remain yours. The control plane adds operational intelligence without replacing anything in your stack — and if you ever remove it, everything continues working exactly as before, just without the automation.

Conclusion

Every production data lake faces the same operational reality: without an active system keeping tables healthy, Iceberg's open architecture degrades into slow queries, rising costs, and engineering time spent on maintenance instead of product work.

The problem is not the format. Iceberg provides exactly what it promises — reliable, open, engine-agnostic tables. The operational layer that continuously compacts, cleans, sorts, monitors, governs, and optimizes must exist separately. It must be built or adopted.

Netflix and Google proved the architecture works by building it internally over years with dedicated teams. In 2027, the same architecture is available without that investment. The approaches range from vendor-managed (Databricks, Snowflake) to cloud-native (S3 Tables, BigLake) to dedicated control planes — each fitting a different point on the architectural complexity spectrum.

The question is not whether your lake needs management — the degradation mechanics are inherent to how Iceberg works at scale. The question is which path fits your architecture, engine diversity, and team structure.

For teams running open, multi-engine Iceberg lakehouses at production scale, LakeOps is the control plane that closes the operational loop — across any engine, catalog, and cloud. Connect your existing infrastructure, define your policies, and let the lake manage itself.

Further learning:

Data Lakehouse: How and Why

joni sar — Wed, 19 Aug 2026 14:09:23 +0000

How the data lakehouse works in production — the internal mechanics, the data flow patterns, why tables degrade under real workloads, and the closed-loop control plane that keeps the architecture performing at scale.

What a data lakehouse solves

Every enterprise data platform eventually hits the same wall. Analytical data exists in two places: a data lake (cheap, scalable, ungoverned) and a data warehouse (expensive, fast, governed). Between them sits an ETL pipeline that copies, transforms, and reconciles — creating a permanent tax on engineering time, data freshness, and infrastructure spend.

The symptoms are familiar. Dashboards show yesterday's numbers because the warehouse loads overnight. ML models train on stale copies because the warehouse does not support Python-native access. Storage bills grow because the same curated datasets live in both systems simultaneously. Governance fragments because access controls, lineage, and audit trails cannot span two architecturally different platforms.

The data lakehouse resolves this by collapsing both systems into one: transactional guarantees applied directly to data on object storage. One copy. One governance model. Any engine — SQL, ML, streaming, AI — reads and writes the same tables with full ACID safety. The warehouse is not replaced by a different product; the lake becomes the warehouse through a metadata innovation called a table format.

The practical result: BI dashboards, Spark ML pipelines, Flink streaming jobs, and autonomous AI agents all query the same governed tables — without ETL between them, without stale copies, without paying warehouse markup on storage that costs $0.023/GB/month in its native form.

The internal mechanics: how a lakehouse actually works

A data lakehouse is not a monolithic system. It is four cooperating layers — each handling a specific responsibility, each independently evolvable. Understanding how they interact is the foundation for building one that performs in production.

The storage layer

All data resides on cloud object storage — S3, GCS, or ADLS. This is where the economics originate. Object storage provides eleven-nines durability at $20–25 per TB per year. A petabyte of analytical data costs roughly $20,000/year to persist — compared to $500,000+ when that same data sits inside a traditional warehouse with bundled compute.

Storage is decoupled from everything above it. Shut down every query engine and the data remains. Add a new engine tomorrow — it reads from the same bucket. Move clouds — the files copy byte-for-byte with no format conversion.

Data lands as Apache Parquet files — a columnar format that stores values by column rather than by row. A query touching 5 columns from a 300-column table physically reads only those 5 columns. Parquet also embeds per-column min/max statistics in each file header, enabling query engines to skip entire files whose value ranges cannot satisfy the WHERE clause. This skip-scanning is what makes lakehouse queries competitive with warehouse queries — but only when files are properly sized and organized.

The table format layer

A directory of Parquet files on S3 is a data lake — readable, but not transactional. The table format is the metadata innovation that adds the missing guarantees:

Apache Iceberg — the dominant format as of 2026, supported natively by Spark, Trino, Flink, DuckDB, Snowflake, Databricks, Athena, and StarRocks — provides ACID commits, schema evolution, time travel, partition evolution, and concurrent multi-writer safety through a metadata tree architecture.

How the metadata tree works:

Each Iceberg table maintains a hierarchy of metadata files on the same object storage as the data:

A metadata pointer (tracked by the catalog) — points to the current table version
Snapshot files — each representing the table's complete state at a point in time. Every write creates a new snapshot.
Manifest lists — each snapshot references a manifest list that indexes the active manifests
Manifests — each tracking a batch of data files with per-file statistics (path, partition values, row count, column min/max)
Data files — the actual Parquet files containing rows

This tree enables two properties that make the lakehouse viable:

Atomic commits. Every write produces new data files and a new snapshot. The catalog atomically swaps the pointer from old snapshot to new. Readers always see a consistent state. Failed writes leave no trace — the pointer was never advanced.

Statistical pruning. Query engines read manifests (kilobytes) to determine which data files (gigabytes) are relevant. A query filtering on country = 'DE' checks manifest-level statistics and skips every file whose country column's min/max range excludes 'DE'. No data read. No I/O wasted. This is why lakehouse queries can match warehouse speed — but only when the metadata is current and the data layout aligns with query patterns.

The critical implication: every commit grows the tree. Every streaming write adds files. Every mutation adds delete markers. The tree that enables fast queries also accumulates entropy that degrades query speed over time — unless something actively maintains it.

The catalog layer

The catalog is the coordination service that answers: where is the current metadata pointer for each table, and how do multiple engines safely coordinate writes?

Every engine — Spark, Trino, Flink, DuckDB — consults the catalog before each read (to locate current metadata) and during each write (to commit atomically). Without the catalog, concurrent writers could corrupt table state. With it, they race safely: one commits, the other retries against updated state.

Modern lakehouses use the Iceberg REST Catalog specification — a standard HTTP API that every engine implements. This means adding a new engine is a configuration change (point it at the catalog URL), not an integration project. Catalog options include managed services (AWS Glue), open-source servers (Apache Polaris, Lakekeeper), federated solutions (Apache Gravitino), and Git-branching models (Project Nessie) — all speaking the same REST protocol.

The catalog is also where access control, retention policies, and audit logging materialize. It is the governance anchor for multi-engine environments where every engine must see consistent permissions.

The compute layer

The lakehouse separates compute from storage, enabling multiple specialized engines to operate on the same data concurrently:

Spark — batch ETL, ML training, complex multi-stage transformations. The workhorse for heavy writes and joins.
Trino — interactive SQL with sub-second response times. Excels at dashboard queries and ad-hoc exploration.
Flink — streaming ingestion with exactly-once semantics. Continuous CDC from operational databases into lakehouse tables.
DuckDB — embedded analytics for notebooks, CI/CD pipelines, and single-node processing. Zero infrastructure overhead.
Snowflake / Athena / StarRocks — managed engines for specific workload profiles (high-concurrency BI, serverless ad-hoc, real-time OLAP).

Each engine discovers tables through the same REST catalog and reads/writes through Iceberg's transactional protocol. They coexist safely on the same tables through snapshot isolation. This is the key architectural advantage over the warehouse model: instead of one vendor's engine for everything, you use the best tool for each workload shape.

But multi-engine access multiplies operational pressure. More engines means more write patterns, more file fragmentation profiles, and more divergent query patterns competing for the same table's physical layout — creating the operational challenge that defines production lakehouse management.

How data flows through a production lakehouse

Theory aside — how does data actually move from source systems through the lakehouse to consumers? The medallion architecture (bronze → silver → gold) is the standard pattern for organizing this flow, and each layer creates different operational characteristics that matter for long-term health.

Bronze: the raw capture layer

Source data lands in its original form. CDC events from PostgreSQL (captured by Debezium, written by Flink). Clickstream events from Kafka. Daily batch exports from SaaS APIs. Partner file drops. Everything arrives as append-only, schema-on-read, immutable records.

Bronze is the replay source. When silver-layer transformation logic has a bug discovered months later, you rebuild from bronze. When a new use case requires fields that were previously ignored, they are already preserved in bronze. This immutability is non-negotiable — teams that apply transformations at ingestion lose the ability to retrospectively correct logic errors.

Common bronze ingestion patterns:

Streaming CDC — Debezium captures row-level changes (inserts, updates, deletes) from PostgreSQL, MySQL, or MongoDB. Flink consumes these events and writes them to Iceberg tables with exactly-once semantics at sub-minute commit intervals. Each commit is small (a few thousand rows) but creates a new data file and a new snapshot.
Event streams — Application events (clickstreams, transactions, IoT sensor data, ad impressions) flow from Kafka topics into Flink or Spark Structured Streaming jobs that write to Iceberg. Event volumes are high and bursty — Black Friday traffic spikes create 10x the normal file accumulation rate.
Batch loads — Daily or hourly exports from ERP systems, CRM platforms, and third-party APIs. These produce larger files but less frequently. Batch tables fragment slower but still accumulate snapshot overhead.
File drops — Partners or internal systems deposit CSV/JSON/Parquet files in S3. A detection pipeline validates schema, converts to Iceberg-managed Parquet, and registers the files in the catalog.

What this means operationally: Streaming CDC with 5-minute commits creates approximately 8,600 new files per table per month. A lakehouse ingesting from 20 operational databases at sub-minute latency accumulates hundreds of thousands of files across its bronze layer within weeks. Individual file sizes average 5–20 MB — far below the 256–512 MB target where engines perform optimally. File count — not data volume — is what degrades query planning speed.

Silver: the governed truth layer

Silver transforms bronze into a data model that the organization commits to: one row per entity, resolved foreign keys, deduplicated events, enforced types, quarantined bad records. Silver is not a cleanup step — it is an architectural commitment to a domain model that all downstream consumers depend on. If a table exists because one dashboard needs a specific aggregation, it belongs in gold, not silver.

Key transformations:

Deduplication — Late-arriving records, duplicate events from at-least-once delivery, and replay scenarios resolved to a single authoritative version. Deduplication runs as MERGE INTO with match conditions, producing delete markers for superseded records.
Type enforcement and validation — Strings cast to proper types. Nulls validated against business rules. Referential integrity checked against dimension tables. Records failing validation quarantined to error tables with reason codes — never silently dropped, never propagated downstream.
Slowly changing dimensions (SCD Type 2) — History tracked for entities that evolve over time. When a customer's tier changes, the old row is closed (end_date set) and a new row opened. This produces update operations on every change event.
Reference data enrichment — Raw IDs joined to meaningful attributes. Country codes resolved to names. Product SKUs expanded to categories, brands, and pricing tiers. Enrichment creates wider rows with higher value per query.
Temporal alignment — Events from different sources (orders, payments, shipments) aligned to consistent time zones, granularity, and business calendars.

Silver tables are the enterprise source of truth. Every downstream consumer — dashboards, ML models, AI agents, compliance reports — starts from the same clean, governed layer.

What this means operationally: Silver tables receive MERGE INTO operations — upserts that create delete markers (position-delete files or V3 deletion vectors) that every subsequent read must reconcile. A CDC table receiving 50,000 updates per hour generates thousands of delete markers per day. Query engines must read both the data files and the delete files, reconcile which rows are current, and filter accordingly — adding I/O overhead that increases linearly with delete-file count. Without periodic resolution of these markers, query latency creeps 2–5x higher while the table reports the same row count and appears unchanged to basic monitoring.

Gold: the consumer-optimized layer

Gold tables exist for performance: pre-aggregated metrics for dashboards, denormalized feature tables for ML training, materialized snapshots for compliance reporting. Gold trades storage redundancy for read speed — accepting wider tables and duplicated columns to eliminate joins at query time.

Gold layer patterns:

BI aggregations — Daily/weekly/monthly rollups pre-joined by all relevant dimensions. A dashboard that previously ran a 30-second join across 5 silver tables now reads a single pre-materialized gold table in under 1 second.
ML feature tables — Wide, denormalized tables with one row per entity and all features materialized. Training jobs read a single scan rather than orchestrating multi-table joins. Feature freshness controlled by the gold refresh cadence.
Compliance snapshots — Point-in-time materializations capturing exactly what was known at a specific date. Regulatory filings, board reports, and audit evidence stored as immutable gold snapshots with indefinite retention.
Serving tables — Pre-computed results for user-facing analytics (product recommendations, customer health scores, real-time dashboards) with sub-second SLA requirements.

What this means operationally: Gold tables are typically overwritten on a schedule — daily, hourly, or triggered by upstream freshness signals. Each OVERWRITE operation atomically replaces the table's content: it writes new data files, creates a new snapshot pointing to them, but leaves the previous snapshot's files in storage until explicitly expired. A daily gold refresh creates 365 snapshots per year. At 50 GB per refresh, that is 18 TB of stale data files persisting in storage — even though only the latest 50 GB is logically current. Without snapshot lifecycle management, storage grows linearly forever from data that was replaced weeks or months ago.

The maintenance gradient

The three layers create a gradient of operational need:

Bronze → small-file pressure (compaction priority)
Silver → delete-file pressure (merge resolution priority)
Gold → stale-snapshot pressure (expiration priority)

No uniform maintenance schedule handles this correctly. Each table needs the right operation at the right cadence — determined by its ingestion velocity, mutation pattern, and query load. This is the insight that leads directly to the control plane model.

What the lakehouse unlocks: multiple workloads, one platform

The economic case (10–50x cheaper storage) is straightforward. The architectural case is more powerful: eliminating the boundaries between workload types that the two-system model enforced.

BI and reporting

Dashboards query silver and gold tables directly through Trino or Snowflake. No warehouse sync delay. No overnight ETL window creating stale numbers. Schema changes in silver propagate to consumers without separate migration scripts — because the table format handles schema evolution as a metadata-only operation.

Time travel lets analysts reproduce the exact state of any report at any prior date. In regulated industries (financial services, healthcare, insurance), "what did we report last quarter" must be answerable precisely and auditably. The lakehouse provides this natively through snapshot pinning — no separate archival infrastructure required.

Performance depends on table health: well-compacted tables with accurate column statistics and sort-order alignment with dashboard filter patterns enable engines to prune 90%+ of files on selective queries. When a BI dashboard filters on region = 'EMEA' AND quarter = 'Q2', a properly maintained table skips every file that cannot contain matching rows. A degraded table forces a full scan regardless of how selective the filter is. This is why maintenance directly impacts end-user experience — and why it cannot be deferred.

ML and feature engineering

ML pipelines read training data from the same governed tables that power dashboards — eliminating training-serving skew, the most common source of ML model degradation in production. When the training pipeline and the serving pipeline read from different copies (one from the warehouse, one from the lake), subtle differences in transformation logic cause the model to encounter data distributions in production that it never saw during training. The lakehouse eliminates this by providing one table for both.

Experiment reproducibility comes free: pin to a snapshot ID, retrain on byte-for-byte identical data months later. Feature freshness improves because features and serving data live in the same transactional system — no nightly export, no stale copies, no drift between what the model trained on and what it serves against.

Feature stores built on lakehouse tables inherit versioning, access control, lineage, and time-travel capabilities from the table format. Feature computation runs as standard Spark or Flink jobs writing to Iceberg tables — no separate feature-store infrastructure, no proprietary feature format.

Streaming analytics

Flink and Spark Structured Streaming write directly to lakehouse tables with exactly-once transactional guarantees. Near-real-time dashboards and alerting read from the same tables — no separate streaming infrastructure, no Kafka-to-materialized-view pipeline, no divergent dual-path architecture.

The same transformation logic runs as streaming (for freshness) or batch (for cost) by changing only the trigger configuration. This eliminates the lambda architecture's core problem: maintaining two parallel pipelines (one batch, one stream) with identical logic that inevitably drift apart. In the lakehouse model, there is one pipeline with a tunable latency/cost knob.

AI agents as first-class consumers

In 2026, AI agents autonomously discover schemas, formulate analytical queries, and feed results into multi-step reasoning chains. The lakehouse is their knowledge infrastructure — providing structured, governed, queryable data that agents access through standard protocols.

The requirements for agent access differ from human access: agents generate unpredictable query patterns, cannot diagnose slow responses caused by table degradation, and burn tokens (and money) on retries when they hit poorly-maintained tables. They also need guardrails — limits on what they can read, how much they can scan, and what they can modify.

Why lakehouses degrade: the operational reality

Here is what most guides skip. The four layers give you the architecture. They do not give you a system that stays healthy. Every production lakehouse — without exception — degrades over time unless something actively maintains it. This is not a flaw; it is a consequence of how append-only transactional systems work.

The mechanics of degradation

Iceberg never modifies files in place. Every write creates new files. Every snapshot preserves a complete view. This is what provides transactional safety — but it means the system accumulates structural debt continuously:

File fragmentation. Each commit adds files. Streaming ingestion at 5-minute intervals into a table receiving 2 GB/hour creates ~8,600 files per month. Each file averages 12 MB — well below the 256–512 MB target where query engines perform optimally. The engine must open, plan, and coordinate reads across 100x more files than necessary.

Planning overhead. Engine query planners traverse the manifest tree to identify relevant files. At 1,000 files, planning takes milliseconds. At 50,000 files, planning takes 10–20 seconds — sometimes exceeding the query execution itself. The metadata hierarchy that enables fast pruning becomes a bottleneck when it indexes too many undersized files.

Delete-file accumulation. Every UPDATE and DELETE writes markers that subsequent readers must reconcile against data files at read time. A heavily-mutated table (CDC, SCD, late-arriving corrections) accumulates thousands of delete markers. Query latency increases 2–5x while the table's row count and size appear unchanged — making degradation invisible to basic monitoring.

Storage waste. Expired logical content leaves physical files behind. Orphaned files from failed writes, aborted compaction, and expired snapshots accumulate in storage. Without active cleanup, production lakehouses accumulate terabytes of unreferenced files — pure cost with no analytical value.

Compounding pressure. These four forces are not independent — they compound. More files means larger manifests. Larger manifests means slower planning. Slower planning means longer maintenance operations. Longer maintenance operations means less frequent runs. Less frequent runs means more file accumulation. The system has a natural tendency toward runaway degradation that accelerates once it begins.

Why manual maintenance does not scale

The instinct is to write scripts: a Spark job that compacts every table nightly, a cron that expires old snapshots weekly. This works for 5–10 tables. It fails at production scale for structural reasons:

Tables need different cadences. A streaming table needs compaction hourly. A daily-refresh table needs it weekly. A slowly-growing dimension needs it monthly. One schedule cannot serve all three without either wasting compute on healthy tables or neglecting degraded ones.

Operations have dependencies. Compacting files that will be expired next hour is wasted work. Rewriting manifests before compaction invalidates the rewrite. Computing statistics on pre-compaction layouts produces stale metadata. The five maintenance operations must run in a specific sequence — and re-sequence when conditions change.

Maintenance competes with production. Running compaction on the same Spark cluster that serves production queries creates resource contention. Running it concurrently with active writers on the same table causes commit conflicts. Scheduling around production windows requires coordination that scripts cannot perform.

Static thresholds drift. "Compact when file count exceeds 1,000" is reasonable for one table's current state. But partition cardinality changes, ingestion velocity shifts, query patterns evolve. The threshold that was correct last month is wrong today — and nobody adjusts it until the table is already degraded.

The control plane: a closed-loop system

The answer is not better scripts or more sophisticated cron. It is an architecturally different approach: a control plane that operates the lakehouse as a closed-loop system — continuously sensing table state, reasoning about what each table needs, executing the right operations in the right sequence, and learning from outcomes to improve future decisions.

    ┌──────────────────────────────────────────────────────┐
    │                                                      │
    │   SENSE  →  ASSESS  →  PLAN  →  EXECUTE  →  LEARN   │
    │     ↑                                         │      │
    │     └─────────────────────────────────────────┘      │
    │                (continuous per table)                 │
    │                                                      │
    └──────────────────────────────────────────────────────┘

This is the mental model for LakeOps — an autonomous control plane for data lakehouses that implements this closed loop across every table in your lake, continuously, without cron jobs or manual intervention.

Sense: collecting structural telemetry

The control plane continuously reads metadata from every connected catalog — file counts per partition, file size distributions, manifest depth, snapshot velocity, delete-file ratios, partition-level skew, and cross-engine query patterns. This is metadata inspection, not data access — the control plane never reads or moves your actual data.

The sensing is passive and lightweight. It adds no load to production engines. It works across catalogs (Polaris, Glue, Gravitino, Nessie, Lakekeeper) and across engines (Spark, Trino, Flink, DuckDB, Snowflake, Athena) — aggregating a unified view that no single component can provide alone.

Assess: health classification

Raw telemetry becomes actionable through health scoring. Every table is continuously classified:

Healthy — file sizes within target, manifests compact, delete ratios low, sort order aligned with query patterns. No action needed.
Warning — metrics drifting toward thresholds. File count growing. Delete ratio increasing. Not yet impacting queries — but will if left unaddressed.
Critical — active degradation impacting performance. Planning latency elevated. Read amplification high. Immediate maintenance required.

The thresholds are not static defaults. They account for each table's partition cardinality, ingestion velocity, engine mix, and historical maintenance response — adapting automatically as workloads evolve.

LakeOps observability surfaces this classification as a lake-wide dashboard: how many tables are healthy, warning, critical — with drill-down into specific metrics, proactive insights at severity levels, and cross-engine telemetry that reveals optimization opportunities no single engine can see.

Plan: sequenced maintenance decisions

For each table classified as Warning or Critical, the control plane determines which operations to run and in what order.

The sequencing is not arbitrary — it follows a dependency chain where each step's output becomes the next step's input:

Step 1 → Expire snapshots. Remove snapshots beyond the retention policy. This dereferences files that are no longer needed — reducing the scope for subsequent operations and preventing compaction from processing files about to be garbage-collected.

Step 2 → Remove orphan files. Delete unreferenced files older than the safety window (default: 7 days, configurable). This captures files newly dereferenced by expiration, plus accumulated waste from failed writes and aborted operations.

Step 3 → Compact data files. Merge small files into optimally-sized ones. Resolve delete markers so subsequent reads skip reconciliation overhead. Optionally sort data by query-relevant columns to enable statistical pruning.

Step 4 → Rewrite manifests. Consolidate the manifest tree to reflect the new compacted layout. After compaction reduces file count from 50,000 to 2,000, the manifest tree should reflect 2,000 files — not carry forward metadata from the old layout.

Step 5 → Compute statistics. Refresh Puffin column statistics (distinct values, null counts, extended min/max) on the compacted files. This ensures every engine benefits from accurate metadata for pruning decisions.

LakeOps managed maintenance runs this five-step pipeline as a coordinated sequence per table — respecting dependencies, avoiding conflicts with active writers, and targeting only the partitions that need work.

Execute: fast, conflict-aware action

The control plane executes maintenance on a purpose-built engine rather than borrowing production compute:

LakeOps compaction runs on a dedicated Rust binary powered by Apache DataFusion — not Spark. The practical differences:

Speed: A 1 TB TPC-DS compaction benchmark completes in under 4 minutes — roughly 29x faster than equivalent Spark operations on the same data. This speed means maintenance runs in minutes rather than hours, so tables never accumulate enough debt to degrade between cycles.
Cost: Sub-$5 per TB compacted — roughly 90% cheaper than running equivalent Spark clusters. This economic efficiency is what makes continuous compaction viable rather than forcing teams into overnight-only windows.
Reliability: Bounded memory with graceful spill-to-disk. No JVM garbage collection pauses. No OOM crashes on large tables. No cluster provisioning delay.
Conflict safety: Aware of active writers and in-flight transactions. Never compacts partitions with concurrent writes. Never expires snapshots that active readers depend on.

Intelligent sort: learned from production queries

Beyond file consolidation, LakeOps compaction physically sorts data by the columns that production queries actually filter on — learned automatically from cross-engine telemetry.

This is the highest-leverage optimization in the entire stack. When data within each file spans a narrow value range for the sort columns, statistical pruning becomes surgical: the engine checks file-level min/max metadata and eliminates 90%+ of files before reading any data. The difference between an unsorted table (full scan on every query) and a properly sorted table (targeted reads on 5–10% of files) is typically 8–12x in query speed.

The key insight is that the right sort order is not static. Trino queries might filter on customer_id and order_date. Spark ML pipelines might filter on event_type and region. AI agents might filter on combinations nobody predicted. No single engine has the full picture of how the table is accessed. The control plane aggregates telemetry from all engines and computes the optimal sort strategy for the combined workload — updating it automatically as patterns shift over time.

Learn: outcome-driven improvement

After each maintenance cycle, the control plane measures outcomes against expectations. Did file count reach the target? Did planning latency improve? Did the health score advance? Results feed back into the assessment model:

Tables that responded well to sort compaction on specific columns get those columns reinforced in future cycles
Tables where expiration freed significant storage get more aggressive retention recommendations
Tables where manifest rewriting dramatically improved planning get lower rewrite thresholds

This learning loop means the control plane improves over time — each cycle's outcomes inform the next cycle's decisions. Tables with unusual characteristics (very high partition cardinality, extremely bursty ingestion, mixed engine access) get customized strategies without manual configuration.

Governance: policies that run themselves

Visibility without enforcement is just monitoring. The control plane turns observability into automated governance through declarative policies:

LakeOps governance supports policies at multiple scopes:

Organization-wide defaults — baseline compaction targets, snapshot retention, orphan cleanup schedules applied to every table unless overridden
Namespace-level rules — production namespaces get aggressive maintenance; staging gets relaxed thresholds
Per-table exceptions — compliance tables with 365-day retention; hot tables with 15-minute compaction cadence

Policies inherit downward: new tables automatically receive the governance rules of their namespace. No manual configuration per table. No tables falling through the cracks because someone forgot to add them to the maintenance script.

Every policy execution is logged with full audit trail — what ran, when, what changed, duration, bytes before/after. This is critical for compliance environments where "prove your data lifecycle management" is a regulatory requirement.

Query routing: making multi-engine economically rational

Having multiple engines on the same data is architecturally elegant. But without routing intelligence, it becomes accidentally expensive: queries dispatched to the wrong engine pay the wrong pricing model and get the wrong performance profile.

A point lookup for 100 rows dispatched to Spark burns 30 seconds of cluster startup. That same query on DuckDB resolves in 0.3 seconds. A dashboard query running on Snowflake at $2/credit is 10x more expensive than the same query on a self-hosted Trino cluster. A full-table scan on DuckDB crashes; on Spark, it completes in minutes.

LakeOps query routing provides a single SQL endpoint that dispatches queries to the optimal engine based on:

Workload type — define routing groups (BI/Reporting → cost-optimized, Data Science → performance-optimized, ETL → throughput-optimized) with different backend engine pools
Table health state — well-compacted tables are eligible for lightweight engines (DuckDB, Athena). Fragmented tables route to engines that tolerate planning overhead (Spark).
Cost/latency targets — constraints per group that the router respects (under $0.05/query for BI, under 2s latency for dashboards)

The routing layer and the maintenance layer form a reinforcing loop: as the control plane compacts and sorts tables, more engines become eligible per query shape. More routing options mean lower per-query cost. Lower cost justifies more frequent querying. More frequent querying generates better telemetry for compaction decisions.

AI agent enablement: the lakehouse as knowledge infrastructure

In 2026, AI agents are not just consumers of pre-built reports — they autonomously explore data, formulate queries, and feed results into reasoning chains. The lakehouse becomes the knowledge layer that agents operate on.

This creates specific requirements that a control plane addresses:

Access interface. LakeOps provides an MCP-native interface (Model Context Protocol) with PostgreSQL, MySQL, and Arrow Flight wire compatibility. Any MCP-compatible agent discovers catalogs, browses schemas, executes queries, and receives results — without custom integration per agent framework.

Layered guardrails. Agents need boundaries. The control plane enforces per-session constraints:

ReadOnly — blocks DDL and DML; agents explore but cannot modify
CostEstimate — rejects queries whose estimated scan exceeds configurable thresholds
PIIMask — hashes sensitive columns before results reach the model
HumanApproval — pauses high-stakes operations for human review before execution

Closed-loop optimization. Agent query telemetry feeds back into the control plane's sort and compaction decisions. Tables that agents query heavily get optimized for agent access patterns. The lake adapts to AI workloads without manual intervention — queries get faster as agents use the system more.

Implementing: the practical sequence

You do not need the full architecture on day one. Each step delivers independent value and sets up the next:

1. Start with Iceberg. Choose Apache Iceberg for all new analytical tables. The ecosystem support is universal — every major engine reads and writes it natively. Existing Delta Lake tables interoperate through UniForm.

2. Deploy a REST catalog. This is the coordination point — get it right early. AWS Glue for managed simplicity. Self-hosted Polaris for full control. Gravitino if you need to federate across existing catalogs.

3. Build the ingestion layer (bronze). CDC from operational databases via Flink. Event streams from application telemetry. Batch loads from external sources. Keep it append-only and immutable.

4. Build the semantic layer (silver). Domain modeling, deduplication, quality enforcement. This becomes the organization's governed truth — every downstream consumer starts here.

5. Connect the control plane. Once data flows and tables accumulate, connect LakeOps to your catalog. Ten minutes, no data movement, no infrastructure changes. Instant visibility into every table's health state. Start in manual-approval mode — review what the control plane recommends before enabling autonomous execution. Then enable autopilot and let the closed loop run.

6. Add specialized engines. As workloads diversify, add Trino for interactive SQL, DuckDB for notebooks, Snowflake for governed BI access. The REST catalog makes each addition a configuration step.

7. Build consumer-optimized tables (gold). Pre-aggregations, feature tables, reporting snapshots. Optimize for specific access patterns once you understand what consumers need.

8. Enable routing. With 3+ engines, define routing groups. Let the control plane dispatch each query to the optimal backend automatically.

9. Open AI access. Expose tables to agents via MCP. Configure guardrails per agent type. Agent telemetry feeds back into optimization priorities.

Comparing the options: lake, warehouse, lakehouse

Characteristic	Data lake	Data warehouse	Data lakehouse
Storage model	Open files on object storage	Proprietary format, vendor storage	Open files on object storage
Transactions	None	Full ACID	Full ACID (table format)
Governance	Manual, fragmented	Vendor-managed, centralized	Catalog-unified, multi-engine
Query speed	Depends entirely on maintenance	Vendor-optimized, consistent	Matches warehouse when maintained
Engine flexibility	Any tool reads files	Vendor's engine only	Multiple engines, REST catalog
ML/AI workloads	Native but ungoverned	Export required	Native and governed
Streaming	Separate infrastructure	Separate infrastructure	Same tables, same transactions
Cost at petabyte	~$20K/yr storage	~$500K+/yr storage+compute	~$20K/yr storage + operations
Maintenance burden	None (no transactions)	Zero (vendor manages)	Requires control plane
Vendor lock-in	Low	High	Low (open formats + catalog)

The lakehouse occupies a specific position: warehouse guarantees at lake economics, with the trade-off that operational maintenance is your responsibility rather than a vendor's. The control plane is what makes this trade-off viable at scale — providing the vendor-grade operational ease without the vendor lock-in.

The economic model

The lakehouse's cost advantage operates at four levels that compound:

Level 1: Storage decoupling. Object storage at $20–25/TB/year vs. warehouse storage at $500–2,000/TB/year. For 200 TB of curated data, that is $4,000–5,000/year vs. $100,000–400,000/year. At petabyte scale, the savings fund entire platform teams. But storage cost is only the starting point — the larger savings come from operational efficiency.

Level 2: Elimination of duplication. The lake-to-warehouse ETL that copies, transforms, and reconciles data disappears. This eliminates: the compute cost of the ETL pipeline itself, the engineering hours maintaining it, the storage cost of the warehouse copy, and the reconciliation overhead when copies drift. For a 200 TB estate, duplication elimination saves $100K–400K/year in direct warehouse storage costs plus 1–3 FTEs worth of engineering time maintaining the pipeline.

Level 3: Multi-engine routing efficiency. When every query runs on the cheapest engine that meets its latency requirement, aggregate compute cost drops dramatically. A DuckDB point lookup costs fractions of a cent. A self-hosted Trino dashboard query costs pennies. A Spark batch join costs proportional to actual data processed. Without routing, organizations default to Snowflake pricing for everything — $2+ per credit regardless of query complexity. With routing, simple queries route to lightweight engines and only expensive workloads pay for expensive compute.

Level 4: Maintenance economics. The control plane's Rust-based engine compacts at sub-$5/TB — roughly 90% cheaper than equivalent Spark maintenance clusters. This cost reduction is not incremental — it is structural. It makes continuous maintenance economically viable rather than forcing overnight-only windows where tables degrade for 23 hours between passes. Continuous maintenance means tables never accumulate enough degradation to require expensive recovery operations, queries never slow enough to warrant over-provisioned clusters as compensation, and engineering teams never spend emergency weekends rebuilding collapsed tables.

Level 5: Recovered engineering time. Without autonomous operations, platform teams spend 20–40% of their capacity on table health — writing maintenance scripts, debugging failures, tuning thresholds, responding to "the dashboard is slow" tickets. With a control plane handling the closed loop, that time returns to building data products, onboarding new sources, and serving consumers. For a five-person platform team, that is 1–2 engineers' worth of recovered capacity.

Combined impact: Organizations operating at 500+ TB report 3–5x total cost reduction compared to the warehouse-plus-lake architecture — combining storage savings, duplication elimination, routing-driven compute efficiency, cheaper maintenance, and recovered engineering time.

When to build a data lakehouse

Not every team needs this architecture. A single-engine, single-workload team at moderate scale gets reasonable value from a managed warehouse without the operational complexity. But the lakehouse becomes the correct choice when:

Multiple workload types coexist. BI, ML, streaming, and AI workloads need the same data with different access patterns and different engines.
Scale makes economics matter. Above 50–100 TB of analytical data, the storage cost differential (10–50x) between warehouse and lakehouse becomes significant enough to fund an entire platform team.
Multi-engine is a requirement. Different teams or workloads genuinely need different engines — Spark for ETL, Trino for interactive, Flink for streaming, DuckDB for development. The lakehouse makes this safe; the warehouse makes it impossible.
Vendor independence is a priority. Open formats on your own storage mean no single vendor controls your exit path. Every component decision is reversible.
Data freshness matters. Streaming ingestion directly into governed tables eliminates the ETL delay that warehouses impose between source and dashboard.
AI/ML is a first-class workload. Models and agents need native, governed access to analytical data — not exports, not API wrappers, not stale copies in separate feature stores.

If three or more of these apply, the lakehouse is not a future consideration — it is the architecture to build toward now.

Conclusion

The data lakehouse is not a concept — it is the production standard for teams that need BI, ML, streaming, and AI on the same governed data without warehouse pricing or lake-quality compromises.

Building one is an assembly problem: object storage for economics, a table format (Iceberg) for transactional guarantees, a REST catalog for coordination, and specialized engines for execution.

Operating one is a systems problem: tables degrade through file fragmentation, metadata growth, delete-file accumulation, and storage waste. Bronze layers fragment through streaming commits. Silver layers accumulate delete-file debt through mutations. Gold layers waste storage through stale snapshots. The degradation is inherent to append-only transactional design — not a bug to fix but a force to continuously counteract.

The control plane is the system that counteracts it. LakeOps implements the closed loop — sense structural telemetry across every table and catalog, assess health with adaptive scoring, plan sequenced maintenance respecting operation dependencies, execute on a dedicated Rust engine that is fast enough to run continuously, and learn from outcomes to improve future decisions. Intelligent compaction that sorts by production query patterns across all engines. Sequenced maintenance that runs the right operations in the right order. Observability that classifies every table's health and surfaces degradation before users feel it. Routing that dispatches queries to the optimal engine for each workload. Governance that enforces retention, maintenance, and access policies lake-wide without per-table configuration. And AI enablement that makes the lake agent-ready with discoverable schemas and layered guardrails.

The architecture works because something continuously keeps it working. That something is the control plane.

Thanks for reading!

Further learning:

Autonomous Iceberg Table Maintenance — the closed-loop maintenance system in depth
Iceberg Lakehouse Observability Guide — seven pillars of lake monitoring
Intelligent Lakehouse: Build Like Netflix — how Netflix designed autonomous lakehouse operations
Fixing Small Files in Apache Iceberg — root causes, measurement, and automated resolution
How AI Agents Operate a Data Lake via MCP — the agent-native lakehouse interface
Apache Iceberg Production Readiness Checklist — ten operational dimensions for enterprise Iceberg
Routing Multiple Query Engines with Iceberg — multi-engine dispatch patterns and optimization

Apache Iceberg Query Performance: A Practical Guide

joni sar — Sun, 02 Aug 2026 09:31:14 +0000

Iceberg is the standard. Snowflake, Databricks, AWS, and every major query engine read and write it natively. The format question is settled.

The performance question is not.

Most production Iceberg tables are slower than they need to be — not because of anything wrong with the format, but because the physical state of the table has degraded over time. Thousands of small files from streaming writes. Manifests fragmented across hundreds of snapshots. Data scattered randomly across files with no correlation to how it's actually queried. Delete files piling up from CDC pipelines.

The result: engines scan 5–10x more data than necessary. Query planning takes seconds instead of milliseconds. S3 API costs spike. And every engine — Trino, Spark, Snowflake, Athena, DuckDB — is equally affected because they all read the same physical data layout.

This guide covers what determines Iceberg query performance, how to fix it, and the two paths available: intelligent continuous optimization through a control plane, and the manual approach with SQL and cron. Both paths target the same physical levers — the difference is whether they adapt over time or require ongoing human attention.

How Iceberg scan planning works

Understanding the performance machinery is essential for both approaches. When a query engine receives a SELECT against an Iceberg table, it runs a scan planning process on a single node that determines which files to read. This is one of Iceberg's core advantages over Hive-style directory listing — no distributed scan needed just to find your data.

Planning operates in three progressive levels of elimination:

Level 1: Manifest list pruning. The engine reads the snapshot's manifest list and checks partition value ranges in each manifest's summary. If a manifest's date range doesn't overlap with the query's filter, it's skipped entirely — along with all data files it references. One check eliminates hundreds of manifest files.

Level 2: File-level data skipping. For surviving manifests, the engine reads per-file column statistics (min/max bounds, null counts, row counts) and evaluates predicates against them. A query for amount > 500 skips every file where the amount column's maximum is below 500. This is where sort order determines everything — tight, non-overlapping min/max ranges mean more files eliminated.

Level 3: Row group pruning. Within surviving files, Parquet row group statistics and bloom filters skip sub-file data blocks. If only one of four row groups matches, 75% of the file's I/O is avoided.

The effectiveness of each level depends entirely on the physical state of the table. A detailed walkthrough covers these internals in depth.

Why tables degrade

Iceberg tables don't start slow. They become slow because the physical layout drifts away from what's optimal for the queries hitting them.

Streaming writes create files at every checkpoint interval (every 1–5 minutes). A table receiving continuous events generates hundreds or thousands of files per day — each one tiny, each one adding a manifest entry that must be evaluated during planning.

Static sort orders lose relevance. A sort key chosen at table creation doesn't reflect what queries actually filter on six months later. As access patterns evolve, the sort becomes meaningless and min/max statistics become wide and useless for pruning.

Maintenance runs in isolation. Compaction, snapshot expiration, manifest rewriting, and orphan cleanup each affect each other's effectiveness. Running them independently — or in the wrong order — means wasted work and suboptimal results.

CDC pipelines accumulate delete files. Every update writes a small delete file. After weeks, hundreds of delete files must be reconciled at read time — each query pays the cost.

No feedback loop. Without observing which columns queries actually filter on, which tables are degrading fastest, and which maintenance actions produce the most benefit — optimization is guesswork.

These problems don't happen once and get fixed. They're continuous. They happen every day, on every table, as long as data is being written and queried.

Two paths to query performance

There are fundamentally two approaches to keeping Iceberg tables performant:

Path 1: Intelligent continuous optimization. A system that observes query patterns, understands table state, and takes the right action at the right time — autonomously, adaptively, and in the correct sequence. This is the control plane approach.

Path 2: The manual path. SQL procedures, cron jobs, and configuration tuning. You run the same operations — compaction, manifest rewrites, snapshot expiration — but you decide when, how, and on which tables. You choose the sort order. You monitor for degradation. You sequence the operations correctly.

Both paths target the same physical levers described above. The difference is not what gets done — it's whether it adapts continuously or requires ongoing engineering attention.

Path 1: Intelligent continuous optimization

LakeOps is an autonomous control plane for Apache Iceberg. It connects to your existing catalogs and query engines, observes table state and query patterns, and continuously applies the optimizations that make queries faster — without human intervention, without Spark clusters, and without static configurations that go stale.

Here's what it does and why each piece improves query performance.

Query-aware data layout

This is the highest-impact capability. Sort order is the single most impactful lever for Iceberg query performance — sorted tables scan 51% less data per query than unsorted ones. But choosing the right sort order requires knowing which columns production queries actually filter on. That knowledge changes over time.

LakeOps captures query telemetry from every connected engine — Spark, Trino, Flink, Snowflake, Athena, DuckDB, StarRocks — and identifies which columns appear in WHERE, JOIN, and GROUP BY clauses for each table. During compaction, data is physically re-sorted by those columns.

The result: Parquet row-group min/max statistics become maximally tight. The scan planner eliminates the vast majority of files at Level 2 because each file covers a narrow, non-overlapping value range for the columns queries actually use.

The sort strategy adapts. If a new dashboard starts filtering on customer_segment alongside event_date, the next compaction pass incorporates it. No manual ALTER TABLE SET SORT ORDER needed. On three consecutive runs of a 1.2 TB table, the system improved runtime from 22 min → 18 min → 11 min as it converged on optimal column ordering — zero configuration changes.

Production numbers: a table with 47,000 scattered files, after query-aware sort compaction to 280 files, saw query time drop from 52 seconds to 5.8 seconds. A 9x improvement from layout optimization alone.

Compaction at streaming speed

Sort compaction is the most impactful maintenance operation — it solves both the small-file problem and the sort-order problem in one pass. But on Spark, it's slow and expensive. A 200 GB sort compaction takes ~25 minutes and costs ~$3.50 on EMR. At that speed, running it multiple times per day on streaming tables is impractical. Most teams default to binpack (size-only) because sort is too expensive — leaving the biggest performance lever unused.

LakeOps's compaction engine is built in Rust on Apache DataFusion. No JVM. No garbage collection. No OOM crashes. The speed difference makes sort compaction viable even for streaming tables:

Binpack: 221 seconds vs 1,612s (Spark) vs 6,300s (S3 Tables) on identical 200 GB dataset
Sort: 780 seconds vs estimated 3,000+ for Spark
Cost: $0.21 vs $1.54 for 200 GB binpack
Peak throughput: 2,522 MB/s — TB-scale tables in minutes
Bounded memory: spills to disk gracefully, no OOM regardless of table size

Because it's fast enough to run frequently, tables stay both consolidated and sorted continuously — never degrading between maintenance windows.

Coordinated maintenance pipeline

Individual maintenance operations interact. The order matters:

Expire snapshots — release references to old data files
Remove orphans — delete unreferenced files from storage
Compact — merge remaining files with optimal sort order
Rewrite manifests — consolidate metadata over the new, clean file set

Running these in the wrong order wastes work. Compacting before expiring snapshots means you might rewrite files that should be deleted. Rewriting manifests before compaction means the structure changes again during the compaction commit.

LakeOps sequences operations automatically. Each step produces a clean input for the next. The manifest rewrite runs on an already-consolidated, properly-sorted file set — producing the leanest possible metadata layer. This coordination eliminates the redundant work that independent cron jobs inevitably create.

Event-driven triggers

A streaming table that accumulates 500 files per hour needs compaction multiple times per day. A stable batch table with weekly loads needs compaction monthly. A CDC table with accumulating delete files needs cleanup when the read-time overhead crosses a threshold.

LakeOps triggers operations based on actual table state — file count thresholds, delete-file ratios, manifest fragmentation, partition skew — not arbitrary cron schedules. Tables get exactly the maintenance they need, when they need it. No wasted runs on healthy tables, no missed runs on degrading ones.

Multi-engine query routing and workload optimization

Different engines have different performance profiles: a point lookup takes 0.5s on DuckDB vs 2.3s on Athena. A full-table scan costs $5 on Athena vs $50 on Trino. Without routing, every query goes to the same engine regardless of its shape.

LakeOps includes QueryFlux — an open-source, Rust-based SQL proxy with 0.35ms overhead — for multi-engine routing. Queries route based on shape, latency targets, cost ceilings, engine availability, and table health status. The routing layer learns from execution history: if a query shape consistently runs faster on one engine, future executions route there.

In benchmarking, workload-aware routing reduced total query cost by up to 80%, with individual queries sometimes dropping by 90%.

Observability and self-improvement

You can't optimize what you can't see. LakeOps classifies every table into health tiers (Critical, Warning, Healthy) based on file fragmentation, manifest depth, snapshot velocity, delete ratios, and sort-order staleness. Insights surface at four severity levels — before degradation becomes noticeable in query times.

The system records per-table throughput, partition structure, and memory usage from each compaction run. Subsequent passes execute faster as the planner converges on optimal resource allocation. Every operation feeds back into the next decision.

What setup looks like

Connect your catalogs (AWS Glue, REST catalogs like Polaris/Nessie/Lakekeeper, S3 Tables) and storage. ~10 minutes. No agents to deploy, no data movement, no pipeline changes. Your data stays in your account. The system discovers tables, classifies health, begins autonomous maintenance according to policies you define.

Production results across customers: up to 12x average query acceleration, up to 80% total cost reduction, 786+ tables managed autonomously across 112+ PB.

Path 2: The manual approach

Every optimization LakeOps automates is also achievable with SQL procedures, engine configuration, and scheduling. Here's how to implement each one by hand.

Diagnosing the bottleneck

Before optimizing, use Iceberg's metadata tables to identify what's actually wrong:

-- File health: count, size distribution, small file ratio
SELECT 
  COUNT(*) as file_count,
  AVG(file_size_in_bytes) / 1048576 as avg_mb,
  COUNT(CASE WHEN file_size_in_bytes < 67108864 THEN 1 END) as small_files
FROM prod.db.events.files;

-- Manifest health: fragmentation
SELECT COUNT(*) as manifest_count,
       AVG(added_data_files_count) as avg_files_per_manifest
FROM prod.db.events.manifests;

-- Delete file accumulation (MoR tables)
SELECT COUNT(*) as delete_file_count
FROM prod.db.events.delete_files;

-- Snapshot accumulation
SELECT COUNT(*) as snapshot_count,
       MIN(committed_at) as oldest
FROM prod.db.events.snapshots;

Thresholds: avg file size < 128 MB → compaction needed. Manifest count > 200 → rewrite manifests. Delete files > 50 → compact to apply. Snapshots > 1,000 → expire.

Use EXPLAIN to verify predicates push down to the scan level rather than appearing as residual filters.

Sort order configuration

Choose columns based on your most common query patterns:

-- Set sort order (first column gets tightest clustering)
ALTER TABLE prod.db.events WRITE ORDERED BY event_date, user_id;

-- Sort compaction to apply the order to existing data
CALL system.rewrite_data_files(
  table => 'prod.db.events',
  strategy => 'sort',
  sort_order => 'event_date ASC NULLS LAST, user_id ASC NULLS LAST'
);

-- For tables with diverse multi-column filters, use Z-order
CALL system.rewrite_data_files(
  table => 'prod.db.events',
  strategy => 'sort',
  sort_order => 'zorder(event_date, user_id, region)'
);

Guidelines for choosing sort columns:

First key: Column most frequently in WHERE clauses. Gets tightest min/max ranges
Second key: Diminishing returns, but meaningful for two-column filters
Z-order: When no single column dominates filter patterns

The challenge: you must manually identify which columns queries filter on, and update the sort order when access patterns change. Most teams don't revisit this after initial setup.

File consolidation (compaction)

-- Binpack: merge small files by size (fast, no reordering)
CALL system.rewrite_data_files(
  table => 'prod.db.events',
  strategy => 'binpack',
  options => map('target-file-size-bytes', '268435456', 'min-file-size-bytes', '67108864')
);

-- Sort compaction: merge + apply sort order (slower, better for queries)
CALL system.rewrite_data_files(
  table => 'prod.db.events',
  strategy => 'sort',
  sort_order => 'event_date ASC NULLS LAST, user_id ASC NULLS LAST',
  options => map('target-file-size-bytes', '268435456')
);

Target: 256–512 MB files for analytical workloads. Schedule via Airflow/cron — daily for streaming tables, weekly for batch.

Preventing small files at write time

The cheapest compaction is the one you never need:

ALTER TABLE prod.db.events SET TBLPROPERTIES (
  'write.target-file-size-bytes' = '268435456',    -- 256 MB
  'write.distribution-mode' = 'hash',              -- Group by partition before write
  'write.parquet.compression-codec' = 'zstd'       -- Better compression
);

Critical: write.distribution-mode = 'none' on partitioned tables is the #1 source of small files. With 200 Spark tasks writing to 365 daily partitions, you get 73,000 tiny files per job. Always use hash for partitioned tables.

For Spark, align AQE with your target file size:

spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "true")
spark.conf.set("spark.sql.adaptive.advisoryPartitionSizeInBytes", "268435456")

For Flink streaming writes, increase checkpoint intervals to reduce file frequency:

SET 'execution.checkpointing.interval' = '5min';
SET 'sink.committer.operator.commit-interval' = '5min';

Bloom filters for point lookups

When queries use equality predicates on high-cardinality columns (WHERE user_id = 12345), bloom filters eliminate row groups that provably don't contain the value:

ALTER TABLE prod.db.events SET TBLPROPERTIES (
  'write.parquet.bloom-filter-enabled.column.user_id' = 'true',
  'write.parquet.bloom-filter-enabled.column.session_id' = 'true',
  'write.parquet.bloom-filter-fpp.column.user_id' = '0.05'
);

-- Generate filters on existing data via compaction
CALL system.rewrite_data_files(
  table => 'prod.db.events',
  options => map(
    'write.parquet.bloom-filter-enabled.column.user_id', 'true',
    'write.parquet.bloom-filter-fpp.column.user_id', '0.05'
  )
);

Limit to 2–3 columns per table (each adds ~1 MB to every file footer). Only useful for equality predicates — range queries don't benefit.

Manifest consolidation

-- Check if needed
SELECT COUNT(*) FROM prod.db.events.manifests;

-- Consolidate (run AFTER compaction)
CALL system.rewrite_manifests('prod.db.events');

Target: 10–50 manifests. Run after compaction stabilizes the file set — not before.

Snapshot expiration and orphan cleanup

-- Expire snapshots older than 7 days, keep at least 5
CALL system.expire_snapshots(
  table => 'prod.db.events',
  older_than => TIMESTAMP '2026-07-26 00:00:00',
  retain_last => 5
);

-- Remove orphan files from failed writes
CALL system.remove_orphan_files(
  table => 'prod.db.events',
  older_than => TIMESTAMP '2026-07-26 00:00:00'
);

Guidelines: 3–7 days retention for streaming tables, 30–90 days for batch. Always retain_last >= 2 to avoid breaking concurrent readers.

Delete file cleanup (MoR tables)

-- Compact to apply pending deletes
CALL system.rewrite_data_files(
  table => 'prod.db.events',
  options => map('delete-file-threshold', '3')
);

Monitor regularly. A table accumulating 200+ delete files/day will see read times increase from 3s → 22s over two weeks.

Partition strategy

-- Set partition (at table creation or via evolution)
CREATE TABLE prod.db.events (
  event_id BIGINT,
  event_time TIMESTAMP,
  user_id BIGINT,
  region STRING
) USING iceberg
PARTITIONED BY (day(event_time));

-- Evolve without data rewrite (metadata-only, instant)
ALTER TABLE prod.db.events SET PARTITION SPEC (day(event_time));

Rules: each partition should hold at least 128 MB. Daily partitioning on a 10 MB/day table → use monthly. bucket(col, 1024) on a 50 GB table → try 16–64 buckets instead.

Puffin statistics for join planning

-- Generate NDV statistics for cost-based optimization
CALL system.compute_table_stats(table => 'prod.db.events');

-- Or via ANALYZE TABLE (Spark)
ANALYZE TABLE prod.db.events COMPUTE STATISTICS FOR COLUMNS
  user_id, event_type, region;

Recompute after major data changes. For details: Puffin Statistics Guide.

Parquet-level tuning

ALTER TABLE prod.db.events SET TBLPROPERTIES (
  'write.parquet.row-group-size-bytes' = '134217728',   -- 128 MB row groups
  'write.parquet.page-size-bytes' = '1048576',          -- 1 MB pages
  'write.parquet.dict-size-bytes' = '2097152',          -- 2 MB dict threshold

  -- Disable stats for columns never filtered on (reduces manifest bloat)
  'write.metadata.metrics.column.raw_payload' = 'none',
  'write.metadata.metrics.column.debug_info' = 'none'
);

For read-side tuning in Spark:

spark.conf.set("spark.sql.files.maxPartitionBytes", "268435456")  # 256 MB per task
spark.conf.set("spark.sql.files.openCostInBytes", "4194304")      # 4 MB open cost

Copy-on-Write vs. Merge-on-Read

Choose your update strategy based on your read/write ratio:

Workload	Strategy	Why
Batch reporting (updated nightly, read all day)	Copy-on-Write	Zero read overhead, clean files always
Streaming CDC (Debezium/Kafka)	Merge-on-Read + compaction	Write speed critical, compact to restore reads
Infrequent GDPR/compliance deletes	Merge-on-Read + periodic cleanup	Fast compliance, clean up later
High-frequency MERGE INTO	Deletion Vectors (V3)	Best write + read performance

The correct maintenance sequence

Whether automated or manual, the order of operations matters:

Expire snapshots → releases references to dead files
Remove orphans → deletes unreferenced files from storage (cheaper subsequent compaction)
Compact (with sort) → merges files, applies sort order, resolves delete files
Rewrite manifests → consolidates metadata over the clean, final file set

Running these out of order wastes compute. Compacting before expiring rewrites files that should be deleted. Manifest rewrites before compaction get invalidated immediately.

Performance troubleshooting checklist

Symptom	Likely cause	Fix
Slow range queries	Unsorted data, many small files	Sort compaction on filter columns
Slow point lookups	No bloom filters	Enable per-column bloom filters + compact
Slow query planning (>5s)	Too many manifests	`rewrite_manifests`
Full table scans	Wrong partition scheme	Partition evolution
Gradually slowing reads	Accumulating delete files	Compact to apply deletes
High S3 API costs	Thousands of small files	Compaction + write-side tuning
Dashboard timeouts	Multiple degradation factors	Full maintenance pipeline + routing

Choosing your path

The manual path works. Every SQL procedure shown above is production-proven. The challenge is sustaining it:

Choosing the right sort order requires analyzing query logs across all engines — and updating when patterns change
Scheduling compaction requires different frequencies for different tables — streaming tables need it hourly, batch tables weekly
Sequencing operations correctly requires understanding their dependencies
Monitoring 50+ tables for degradation signals requires dashboard infrastructure
Scaling to hundreds of tables means the maintenance burden grows linearly with your data estate

For teams with fewer than 10–20 tables and stable access patterns, the manual path is manageable. Beyond that, the operational burden compounds — and the sort order inevitably goes stale because nobody has time to analyze query patterns per table per quarter.

The control plane approach eliminates this scaling problem. It handles the continuous observation, adaptation, and execution at any table count — and because its compaction engine is 95% faster than Spark, it can apply sort compaction at frequencies that would be cost-prohibitive manually.

Both paths lead to the same destination: tables where queries are 12x faster because the physical layout matches how data is actually accessed. The question is whether you want that as a continuous property of your lake, or as a point-in-time optimization that needs periodic human re-tuning.

Summary

Iceberg query performance is determined by the physical state of your tables — not by the format spec, not by the engine, and not by hardware. A well-maintained table is fast on every engine. A degraded table is slow on all of them.

The physical levers that matter:

Sort order aligned to query patterns — 51% less data scanned, up to 12x faster queries
File consolidation — 5–10x faster planning, reduced I/O
Bloom filters — near-instant elimination for equality lookups
Manifest optimization — 2–5x faster planning
Write-side configuration — prevent degradation at source
Delete file cleanup — restore read performance on MoR tables
Snapshot lifecycle — lean metadata, lower storage costs
Partition strategy — coarse-grained elimination at the foundation
Multi-engine routing — right engine per query, up to 56% cost reduction

The intelligent path keeps all of these optimal continuously. The manual path gives you the tools to do it yourself. Either way — the performance ceiling is the physical state of your tables, and now you know how to raise it.

Further reading:

AI Data Lake and Lakehouse

joni sar — Fri, 24 Jul 2026 14:07:16 +0000

The word "AI" appears in every data platform pitch now. Most of the time, it means nothing — a chatbot bolted onto a dashboard, an LLM that generates SQL, or a marketing slide that says "AI-powered" next to a cron job.

This article is about something different: what happens when you put actual intelligence — machine learning, adaptive optimization, closed-loop feedback systems — at the core of how a data lakehouse operates. Not AI as a feature. AI as the operating model.

The result is a lakehouse that understands its own workloads, optimizes its own storage layouts, routes its own queries, heals its own tables, and continuously improves — without human intervention. That is what an AI data lake actually is. Everything else is automation with better branding.

Automation Is Not Intelligence
The AI Control Plane
How AI Transforms Each Layer of the Lakehouse
AI-Driven Compaction: Learning How Data Is Read
AI-Driven Maintenance: Predicting What Breaks
AI-Driven Query Routing: Matching Queries to Engines
AI-Driven Observability: From Dashboards to Diagnosis
AI-Driven Governance: Policies That Adapt
The Closed Loop: How Every Query Makes the Lake Smarter
AI Agents as Lake Citizens
AI-Native vs. AI-Washed: How to Tell the Difference
The Compounding Effect
Where This Is Heading

Automation Is Not Intelligence

Most data lake "automation" today works like this: a cron job runs compaction every night at 2 AM. It merges small files into bigger ones. It does not know if the table needs compaction. It does not know which partitions are degraded. It does not know what columns queries filter on. It does not know whether last night's compaction actually helped.

That is automation. It follows a fixed rule, regardless of context.

Intelligence is different. Intelligence observes, learns, decides, acts, measures the outcome, and adjusts. A system that compacts a table because its file health crossed a threshold, sorts by the columns that real queries actually filter on, sequences the operation after snapshot expiration to avoid wasted work, and then verifies that query latency improved — that is intelligent.

The distinction matters because data lakes are not static systems. Workloads shift. New dashboards appear. AI agents start querying tables that were designed for nightly batch. Streaming pipelines change write patterns. A schema evolves.

Fixed rules cannot keep up. Intelligence can.

The AI Control Plane

An AI data lake needs a brain — a system that continuously ingests telemetry from every table, every engine, and every query, and uses that signal to drive optimization decisions across the entire lake.

LakeOps is that system. It is an autonomous control plane for Apache Iceberg that puts AI at the center of lakehouse operations — compaction, maintenance, query routing, observability, governance, and agent enablement, all driven by continuous learning rather than static rules.

What makes it a control plane and not another tool: LakeOps is not an engine, not a catalog, and not storage. It connects to whatever you already run — AWS Glue, Polaris, Nessie, Gravitino, Lakekeeper, or S3 Tables as catalogs; Trino, Spark, Snowflake, Athena, DuckDB, Flink as engines; S3, GCS, or ADLS as storage — and adds the intelligence layer on top. No data movement. No code changes. No vendor lock-in. Setup takes about ten minutes.

What makes it AI-native: every decision LakeOps makes is informed by learned behavior, not hardcoded rules. It observes query patterns across all engines, predicts table degradation from structural signals, adapts data layouts as workloads shift, learns which engine performs best for each query shape, and measures the outcome of every optimization to improve the next one. It also exposes a native MCP (Model Context Protocol) server so AI agents can query the lake directly — with schema discovery, intelligent routing, and layered guardrails built in. The more the lake is used — by humans and agents alike — the smarter the control plane gets.

Three operating modes let teams choose how much autonomy the AI gets:

Autopilot — the AI decides and executes. Define policies and thresholds; the system handles everything.
Recommendations — the AI analyzes and recommends with full context (what, why, expected impact). A human approves.
Policy-bounded — the AI operates within declared boundaries, escalating edge cases.

Most teams start with recommendations to build trust, then move tables to autopilot as confidence grows. The intelligence is the same in all modes — the only difference is who clicks "execute."

With that foundation, here is how AI transforms each layer of lakehouse operations.

How AI Transforms Each Layer of the Lakehouse

When we talk about an "AI data lake," we are talking about specific capabilities that require learning, prediction, and adaptation — not features that could be implemented with a shell script.

Layer	What AI Does	What Scripts Cannot
Compaction	Learns query patterns across engines, sorts data by the columns actually used in WHERE/JOIN/GROUP BY, adapts as patterns change	Static sort keys, fixed file-size targets, blind to query patterns
Maintenance	Predicts which tables will degrade next, sequences operations by dependency, prioritizes by business impact	Fixed schedules, independent jobs, no awareness of health trajectory
Query routing	Learns which engine performs best for each query shape, uses LLMs to reason about new templates, adapts as tables improve	Static rules, manual engine assignment, no cost optimization
Observability	Correlates structural signals with query performance, predicts degradation before user impact	Threshold alerts, no predictive capability, no cross-signal correlation
Governance	Adapts policy enforcement to workload patterns, recommends changes from observed behavior	Static policies, manual updates, no feedback from operations
Agent integration	Feeds agent telemetry back into optimization, shapes the lake around AI access patterns	No awareness of agents, no closed-loop optimization

Each of these deserves a deeper look.

AI-Driven Compaction: Learning How Data Is Read

Traditional compaction is a janitor: it sweeps up small files and merges them. It does not know why the files exist or how they are read. An intelligent compaction engine does.

Query Pattern Analysis

LakeOps continuously observes every query that hits every table — across Trino, Spark, Snowflake, Athena, DuckDB, Flink, and any other connected engine. It extracts which columns appear in WHERE, JOIN, and GROUP BY clauses. It tracks how these patterns change over time. It weights recent queries more heavily than old ones.

This is not a one-time analysis. It is a continuous learning loop.

From this analysis, the AI knows — per table — which columns matter most for data layout. A table where 80% of queries filter on event_date and region should have its data physically sorted by those columns. A table where queries mostly join on customer_id should sort differently.

From Analysis to Action

During compaction, data files are physically re-sorted by the learned columns. The result: Parquet row-group min/max statistics become effective, and engines skip irrelevant data without reading it. Predicate pushdown and column pruning actually work — not because someone manually configured sort keys, but because the AI figured out what the right sort order is.

On production tables, this AI-driven sorting reduces data scanned by 51% and delivers 12x faster queries compared to unsorted compaction. The numbers are large because the difference between "sort by whatever was configured at table creation" and "sort by what queries actually need" is enormous.

Adaptive Sort Evolution

Here is where intelligence really separates from automation. When query patterns shift — a new dashboard starts filtering on product_category instead of region, a data science team changes their join keys, AI agents start accessing different columns — the sort strategy evolves on the next compaction pass. No configuration change. No ticket. No human intervention.

A table whose workload stabilizes gets compacted less frequently — the AI recognizes that the current layout is already optimal and avoids redundant work. A table whose workload spikes gets prioritized. This is resource allocation driven by learned behavior, not fixed schedules.

Speed as an AI Enabler

The AI's compaction decisions need an engine fast enough to execute them continuously — not queue them for a nightly batch window. LakeOps runs compaction on a purpose-built Rust engine powered by Apache DataFusion: 221 seconds where Spark takes 1,612 seconds (95% faster), at roughly $5/TB versus $50/TB (90% cheaper). Peak throughput reaches 2,522 MB/s. Tables that OOM Spark — a 1.2 TB table with 12,000 files, for example — complete without special configuration because memory is bounded, not heap-dependent.

That speed is not just a performance metric. It is the enabler for AI-driven compaction: when compaction takes minutes instead of hours, the system can respond to workload changes on the same day they happen rather than falling behind. The AI can iterate. Fast execution makes continuous learning practical.

AI-Driven Maintenance: Predicting What Breaks

Iceberg tables require six maintenance operations: data compaction, snapshot expiration, manifest consolidation, orphan cleanup, position delete resolution, and statistics computation. Every platform team knows this. The question is when and in what order to run them — and that is where AI changes the game.

Health Prediction, Not Health Reaction

Traditional monitoring tells you a table is degraded after queries slow down. By then, users have already filed tickets and data pipelines are late.

LakeOps predicts degradation before it hits. The AI continuously scores every table on structural signals — file count trajectory, partition balance trends, manifest growth rate, snapshot accumulation velocity, delete-file ratios. It sees the pattern: "this table's small-file ratio is climbing at 4% per day; at current trajectory it will hit Critical in 72 hours."

This is a classification and prediction problem that humans cannot solve manually at scale. A platform team with 800 tables across multiple catalogs cannot eyeball metadata trends for each one. An AI system that ingests every commit, every query, and every structural change can.

Intelligent Sequencing

The order of maintenance operations matters. Running orphan cleanup before snapshot expiration risks deleting referenced files. Running manifest rewrites before compaction wastes the rewrite since compaction will invalidate file references.

The AI learns the correct dependency graph and sequences operations per table:

Compaction first (merge small files, apply learned sort order)
Snapshot expiration after (remove old references safely, respecting active readers)
Manifest consolidation next (consolidate after file references stabilize)
Orphan cleanup last (reclaim newly unreferenced storage)

This is not just a fixed ordering — the AI evaluates whether each step is needed based on current table state. A table with healthy manifests skips manifest consolidation. A table with no orphans skips cleanup. The maintenance plan is tailored per table, per cycle, based on real signals.

Operation Outcome Learning

After each operation completes, the AI measures the outcome: did compaction actually reduce query latency? Did snapshot expiration free the expected storage? Did manifest consolidation improve planning time?

These outcomes feed back into future decisions. If compacting a particular table's hot partition consistently delivers a 5x query speedup, the system learns to prioritize that partition. If expiring snapshots on a rarely-queried table has minimal impact, it deprioritizes that work. Over time, the maintenance engine gets better at deciding what to do — not just executing a predefined list.

AI-Driven Query Routing: Matching Queries to Engines

Most production lakehouses use multiple query engines. Trino for interactive analytics. Spark for heavy ETL. Snowflake for BI dashboards. DuckDB for ad-hoc exploration. Athena for pay-per-query workloads. The problem is deciding which query goes where.

Without intelligence, this is a manual decision — or worse, no decision at all, and every query hits the same engine regardless of whether it is a 50 MB selective filter or a 500 GB full table scan.

The Three-Router AI Stack

LakeOps implements intelligent routing through QueryFlux, an open-source Rust-based SQL proxy, extended with a three-layer AI stack:

1. Adaptive routing (learned behavior). The AI tracks historical execution data for every query template — which engine ran it, how long it took, how much it cost, whether it succeeded. For previously seen query shapes, routing is instant: the system already knows which engine delivers the best cost-performance ratio. Cached decisions take 0ms.

2. LLM-based routing (reasoning). For new query templates the system has not seen before, an LLM reasons over the query structure combined with live table statistics — file count, data size, sort order, partition layout, health score. It predicts which engine is the best fit. This handles the cold-start problem that pure historical routing cannot.

3. Semantic routing (intent matching). When neither historical data nor structural analysis alone is sufficient, semantic matching infers the query's intent — exploratory analysis, dashboard refresh, ETL pipeline, or AI agent retrieval — and routes accordingly.

The three routers compose. Over time, more query templates enter the adaptive cache and routing gets faster and more accurate.

Data-Quality-Aware Routing

This is where AI routing connects to AI compaction and AI observability. The routing layer knows — from LakeOps's health scoring — whether a table is well-compacted, fragmented, or degraded. It adjusts routing decisions accordingly:

A well-compacted table with good sort order? Route to a lightweight, cheaper engine that benefits from efficient data skipping.
A degraded table with 40,000 small files? Route to an engine that can handle the overhead while compaction runs in the background.
A table that was just compacted and is now healthy? Update the routing weights so future queries use cheaper engines.

As the AI compaction engine improves table health, the AI routing engine shifts queries to cheaper engines. The cost savings compound. Production benchmarks show up to 56% workload cost reduction from intelligent routing alone.

AI-Driven Observability: From Dashboards to Diagnosis

Traditional observability is a dashboard: it shows metrics, you interpret them. AI-driven observability is a diagnostic system: it interprets the metrics for you, tells you what is wrong, explains why, and either fixes it or recommends a fix.

Cross-Signal Correlation

LakeOps does not monitor file counts and query latency independently. The AI correlates signals across layers:

Storage signals — file count, size distribution, access patterns, orphan volume
Metadata signals — manifest count and depth, snapshot velocity, partition skew, statistics coverage
Engine signals — query latency, scan volume, CPU utilization, per-engine field access
Operation signals — maintenance duration, compaction throughput, health score changes

From these, it produces actionable insights with causal chains. Example: "Query latency on raw_clickstream increased 8x because 312 partitions exceed the file count threshold — scan amplification is 8x. Compaction is recommended and will reduce query time to baseline."

That is not a threshold alert. That is a diagnosis.

Severity-Ranked Insights

The AI surfaces per-table findings at four severity levels — CRITICAL, HIGH, WARNING, LOW — each with the specific metric, current value, threshold violated, and a remediation action. Critical and High Insights trigger automated remediation. Warning and Low Insights surface for human review.

The key AI element: the system does not just check "is file count above X?" It evaluates the combination of signals. A table with many small files but no queries gets a lower severity than a table with fewer small files but high query frequency. Business impact is inferred from workload patterns, not just structural metrics.

Predictive Degradation

Instead of alerting after a table degrades, the AI predicts when it will degrade based on trajectory. A table whose snapshot accumulation rate will hit its retention limit in 48 hours triggers an alert now — with enough lead time to act before users notice.

AI-Driven Governance: Policies That Adapt

Static governance policies are set-and-forget — which usually means set-and-break. Workloads change, new tables appear, query patterns shift, and the policies that made sense six months ago no longer match reality.

Intelligent Policy Recommendations

LakeOps's AI observes actual workload behavior and recommends policy adjustments:

"Table orders_archive has not been queried in 90 days. Current retention policy keeps 30 days of snapshots. Recommend reducing to 7 days to reclaim 18 GB of metadata."
"Namespace analytics.* consistently produces small files from streaming writes. Recommend adding an auto-compaction policy with a 1-hour trigger window."
"Table customer_events is queried by 3 AI agents with PII columns. Recommend adding PIIMaskGuard policy to agent-facing routing groups."

These are not generic suggestions. They are derived from observed behavior — query patterns, write patterns, access patterns, and health trajectories specific to each table.

Policy Inheritance with Workload Awareness

Policies cascade from organization to catalog to namespace to table — but the AI ensures inheritance makes sense. A compaction policy designed for batch tables is not blindly applied to streaming tables in the same namespace. The system flags conflicts between inherited policies and actual workload patterns, and either adjusts automatically (in autopilot mode) or recommends changes (in manual mode).

Policies apply uniformly across catalogs — AWS Glue, Polaris, Nessie, Gravitino, Lakekeeper — without per-catalog scripts or engine-specific logic. Every policy is auditable, versioned, and controllable with a single toggle.

The Closed Loop: How Every Query Makes the Lake Smarter

The most important concept in an AI data lake is the closed-loop feedback system. This is what separates an intelligent system from a collection of automated scripts — and it is the core of how LakeOps operates.

Here is the loop:

    Queries hit the lake
    (engines, agents, dashboards, pipelines)
            │
            ▼
    Telemetry captures patterns
    (columns, engines, latency, cost, frequency)
            │
            ▼
    AI analyzes and learns
    (sort order, routing weights, health scores,
     maintenance priorities, policy fitness)
            │
            ▼
    Optimizer acts
    (compact with new sort, reroute queries,
     adjust policies, trigger maintenance)
            │
            ▼
    Outcomes measured
    (query faster? cost lower? health improved?)
            │
            ▼
    Learnings feed back ──────► next cycle

Every query — whether from a BI dashboard, a Spark ETL job, or an AI agent — feeds signal into this loop. The more the lake is used, the better it gets at organizing itself. This is the defining characteristic of an AI data lake: usage improves performance, rather than degrading it.

In a traditional lake, usage degrades performance — more queries means more load, more writes means more small files, more engines means more fragmentation. In an AI data lake, usage is signal. More queries means better sort-order decisions. More engines means richer routing data. More writes means more opportunities for the optimizer to learn and act.

AI Agents as Lake Citizens

AI agents introduce a fundamentally new workload pattern. They issue SQL iteratively — dozens of queries per reasoning step — at high frequency, without human review, with expectations of sub-second latency. A single customer-support agent answering one ticket can issue 30–50 SQL statements across user history, orders, product metadata, and support logs. Multiply that across hundreds of concurrent agent sessions and the workload profile looks nothing like the BI traffic your lake was designed for.

Read: Routing Multiple Query Engines with Iceberg

The Infrastructure Problem

Agents querying uncompacted tables pay a 5–10x latency penalty. When an agent runs 12 sequential queries in a reasoning loop, each 3-second query delay adds up to 36 seconds of wall-clock time for a single interaction. Users perceive this as the agent being broken. And unlike a human analyst who learns to avoid expensive queries, an agent in a loop repeats the same mistake indefinitely — compounding costs every iteration.

The LakeOps MCP: Agent-Native Connectivity

The Model Context Protocol (MCP) is the emerging standard for how AI agents interact with data infrastructure — filling the same role for agent-to-lakehouse communication that REST catalogs fill for engine-to-catalog communication.

Read: Agentic AI MCP for Iceberg

LakeOps exposes a native MCP server that any compatible agent — Claude, LangChain, LlamaIndex, or custom — can connect to with zero integration code. The MCP server provides four purpose-built tools rather than a generic "run anything" endpoint (agents that get a single execute_sql tool tend to skip schema discovery and hallucinate table names):

list_schemas — returns all available schemas and databases across connected engines. Agents call this before constructing queries to avoid guessing at namespace structures.
describe_table — returns column names, types, partition specs, sort orders, and optionally sample rows in a single call. Eliminates the multiple round-trips agents typically make to understand a table's shape.
execute_query — runs SQL through the full routing and guardrail pipeline, with optional engine_hint and enforced max_rows to prevent unbounded result sets from flooding the LLM context window.
explain_query — returns estimated cost, row count, and execution plan without running the query. Agents use this for pre-flight cost checks.

Each routing endpoint gets a stable URL — for example, storefront-analytics.lakeops.dev. Agents connect to that URL and inherit the full policy stack (guardrails, routing rules, cost limits) automatically. No per-agent configuration. Wire compatibility spans PostgreSQL, MySQL, and Arrow Flight SQL, plus async query support with SSE streaming so agents in tool-use loops can handle long-running queries gracefully.

Intelligent Guardrails

Unsupervised agents need enforcement at the query level, not at the application level. LakeOps provides a composable guard chain — every query passes through it regardless of which agent or frontend issued it:

ReadOnlyGuard — blocks DDL and DML from agent sessions using SQL parsing (not string matching), catching CTEs that wrap mutations and function calls with side effects.
RowLimitGuard — injects LIMIT N when a query lacks one. Agents in reasoning loops often forget to limit results, leading to multi-gigabyte result sets that overflow LLM context.
CostEstimateGuard — issues EXPLAIN before execution and rejects queries where estimated scanned bytes exceed a threshold. This is the primary defense against the cartesian-join and missing-WHERE patterns that agents frequently generate.
PIIMaskGuard — rewrites queries to exclude, hash, or null-out columns tagged as sensitive. PII never enters the LLM context window.
HumanApprovalGuard — pauses high-stakes queries and sends a webhook (Slack, email, or custom) for manual review.

Guards stack per routing group. A typical agent-facing configuration: read_only → row_limit → cost_estimate → pii_mask. A data engineering agent gets: cost_estimate → human_approval for DDL. Every guard action is recorded with full auditability — what fired, what was rewritten, what was rejected.

AI Solving the AI Problem

This is where the AI data lake concept becomes recursive. The same intelligence that optimizes the lake for human workloads also optimizes it for AI workloads — and agents produce particularly rich signal:

Higher query frequency — more data points for the adaptive optimizer per unit of time
Predictable patterns — roughly 80% of agent queries are repeated templates (same SQL structure, different literal values), making sort-order learning fast
Agent context propagation — every query carries structured metadata (agent_id, conversation_id, step_index, tool_call_id) that enables per-agent cost attribution, session replay for debugging, and workload-specific routing optimization

The feedback loop tightens: agents query the lake → telemetry captures their patterns → the AI reshapes data to match → agents get faster results → the agents produce even better signal → the cycle accelerates. Production deployments show agent query p95 latency dropping from 5–10 seconds to under 500ms as tables are compacted and sorted for observed access patterns, with per-query compute cost dropping 65% through intelligent routing.

AI-Native vs. AI-Washed: How to Tell the Difference

Not every product that claims "AI-powered" actually uses intelligence. Here is how to evaluate:

Signal	AI-Native	AI-Washed
Compaction	Learns sort order from cross-engine query patterns; adapts as patterns change	Fixed sort keys configured manually; schedule-based execution
Maintenance	Predicts degradation; sequences by dependency; learns from outcomes	Runs on cron; independent jobs; no feedback loop
Routing	Learns cost-performance per query shape; uses LLM for cold-start; adapts weights	Static rules; manual engine assignment
Observability	Correlates signals across layers; predicts issues; diagnoses root cause	Threshold alerts; dashboard without interpretation
Governance	Recommends policy changes from observed behavior; adapts inheritance	Static policies; manual updates
Feedback loop	Closed-loop: every query improves the system	Open-loop: no learning from outcomes
Agent support	Native MCP with intelligent guardrails and self-optimization	REST API bolt-on; no workload-aware optimization

The litmus test: does the system get better over time without human configuration changes? If yes, there is actual intelligence. If no, it is automation with a marketing label.

The Compounding Effect

The individual components are valuable. The compound effect is transformational.

AI-driven compaction reduces data scanned by 51%. That means every query across every engine benefits. Routing then shifts those improved queries to cheaper engines, because well-compacted tables do not need heavyweight compute. Storage costs drop because compaction reduces total file count and orphan cleanup reclaims unreferenced data. Observability improves because the AI has better signal to work with. Agents respond faster, which means fewer retries, lower token costs, and better AI application quality.

Each AI component amplifies the others. Better compaction → better routing → lower cost → more headroom for AI agents → richer signal → better compaction. The loop runs continuously.

Production numbers from LakeOps deployments:

80% reduction in compute and storage costs
12x faster queries after AI-driven compaction and sorting
95% faster compaction than Spark (221s vs. 1,612s on 200 GB)
90% cheaper compaction ($5/TB vs. $50/TB for Spark)
56% workload cost reduction from intelligent routing
100% of tables continuously monitored and maintained
~10 minutes to connect and start optimizing

These numbers compound because the system is a closed loop, not a collection of independent tools.

Where This Is Heading

The trajectory is clear. Data lakehouses are moving from manually operated infrastructure to self-managing systems where intelligence handles the operational layer end-to-end.

Today, LakeOps's AI handles compaction, maintenance sequencing, query routing, health prediction, and policy adaptation. The direction extends to:

Automatic schema evolution — the AI detects that a new column is consistently added to queries and recommends schema changes or materialized views
Predictive capacity planning — forecasting storage and compute needs based on workload growth trends and seasonal patterns
Cross-lake optimization — intelligence that spans multiple lakes, regions, and clouds, optimizing data placement and replication based on global access patterns
Agent-native data contracts — AI agents negotiate data freshness, latency, and cost SLAs directly with the control plane, and the system optimizes to meet them

The foundation for all of this is the closed-loop feedback system. A lake that learns from every query is a lake that can evolve to meet requirements that do not yet exist.

Summary

An AI data lake is not a data lake with an AI chatbot. It is a data lake where intelligence drives operations:

AI decides how to compact — learning from cross-engine query patterns, not static configuration
AI decides when to maintain — predicting degradation, sequencing by dependency, learning from outcomes
AI decides where to route — matching query shapes to engines using learned cost-performance models
AI diagnoses problems — correlating signals across storage, metadata, engines, and operations
AI adapts governance — recommending policy changes based on observed behavior
AI optimizes for agents — feeding agent telemetry back into the optimization loop

The closed-loop feedback system is the defining characteristic. Usage improves performance. Every query makes the lake smarter.

LakeOps is the autonomous control plane that implements this intelligence stack for Apache Iceberg — connecting to your existing catalogs, engines, and storage in minutes, with no data movement and no vendor lock-in.

Explore the platform · Read the docs · Get a demo

Managed Data Lake: A Guide for 2026

joni sar — Tue, 21 Jul 2026 11:32:48 +0000

There are two ways to run a data lake. You can manage every table by hand — writing Airflow DAGs, scheduling Spark compaction jobs, building monitoring dashboards, and staffing a platform team that grows linearly with table count. Or you can deploy a control plane that continuously observes every table in the lake, classifies health, sequences maintenance operations, optimizes data layout for real query patterns, enforces policies at lake scale, and adapts as workloads change — while your data stays in your storage account and your engines stay yours. This guide covers both strategies and explains why teams that start with the first keep arriving at the second.

The data lake format question is settled. Apache Iceberg is the default table format for the modern lakehouse. Every major engine — Spark, Trino, Snowflake, Athena, DuckDB, Flink, Databricks — reads and writes Iceberg natively. Adoption is near-universal among data platform teams building on open storage.

The operations question is not settled. Iceberg gives you ACID semantics, schema evolution, time travel, hidden partitioning, and partition evolution on your own object storage. What it does not give you is a system that keeps tables healthy. Compaction, snapshot expiration, orphan cleanup, manifest optimization, sort-order alignment, governance, and lake-wide observability are left to you — the official maintenance guide is explicit about that. At a handful of tables, scripts work. At hundreds of tables across multiple engines, catalogs, and write patterns, the lake quietly degrades into a swamp — and the engineering hours spent holding it together scale faster than the team.

"Managed data lake" is an overloaded term. Some products use it to mean a closed platform. Others mean one automated job. This guide defines what managed actually requires in 2026, then walks through two strategies to get there. First, the control-plane approach — using LakeOps, a dedicated autonomous control plane for Apache Iceberg that handles observability, compaction, snapshot lifecycle, orphan cleanup, manifest optimization, governance policies, multi-engine query routing, and agentic AI readiness across your existing stack without moving data or replacing engines. Then, the manual approach using open-source tooling and custom scripts. Both paths are real. The difference is in what scales.

What "managed" actually means in 2026

Iceberg on S3 is not a managed data lake. A nightly Spark rewrite_data_files job is not either. "Managed" means the operational contract is closed continuously, safely, and at lake scale — without engineering time growing linearly with table count.

An open data lake has four architectural layers:

Storage — S3, GCS, ADLS, or on-premises object storage. Commodity infrastructure you own.
Table format + catalog — Iceberg plus a catalog (Glue, Polaris, Nessie, Gravitino, Lakekeeper, S3 Tables, Unity Catalog). The structural contract every engine speaks.
Compute engines — Spark, Trino, Snowflake, Athena, DuckDB, Flink, Databricks, ClickHouse. Chosen per workload, not locked to a single vendor.
Operations control plane — the layer that observes table health, decides what maintenance to run, executes it safely, enforces policy, and learns from query patterns across engines. This is the layer that most lakes are missing.

Closed platforms like Databricks and Snowflake bundle the last three layers. Open lakes decouple them on purpose — you pick the best engine for each workload, keep data in your own storage, and avoid vendor lock-in. The trade-off: you inherit the operations that the closed platform handled for you. That trade-off is the subject of this guide.

In practice, a managed data lake means: tables stay compact and sorted for real query patterns. Snapshots and orphans do not inflate storage. Manifests stay lean enough that planning never dominates query time. Policies apply lake-wide and new tables inherit them automatically. Health is visible in one place. Correct results are the floor. Predictable cost and latency are the product.

Why data lakes degrade — the mechanics of silent failure

The failure mode is predictable, well-documented, and present in nearly every Iceberg lake that has been running for more than a few months without dedicated maintenance. The financial impact compounds from four directions at once: storage costs balloon from orphan files and stale snapshots, compute costs spike as engines scan thousands of small files instead of a few optimally-sized ones, query latency degrades as tables that returned results in seconds now take minutes, and engineering time evaporates as someone has to write, maintain, debug, and be on-call for the scripts that hold it all together. Understanding the mechanics is necessary for both strategies.

Small file accumulation. Streaming writers (Flink, Spark Structured Streaming, Kafka Connect Iceberg sink, CDC connectors) produce files sized by checkpoint throughput, not by optimal read size. A streaming pipeline at 1 commit per minute across 10 partitions creates approximately 14,400 files per day. Each query pays an object-store GET request, a Parquet footer parse, and task setup overhead per file. Cost and latency scale with file count, not with data volume.

Snapshot accumulation. Every Iceberg commit creates a new snapshot. A streaming table at 5-minute commits creates ~288 snapshots per day. Without expiration, snapshot counts reach tens of thousands within weeks. Query planning reads the snapshot chain; above ~1,000–2,000 retained snapshots, planning time degrades measurably — from sub-second to 30+ seconds. The mechanism is detailed in how Iceberg query planning works.

Manifest fragmentation. Every commit adds at least one manifest file to the manifest list. Streaming tables accumulate thousands of manifests per month. Each manifest must be read during planning; at 500+ manifests, planner latency alone can exceed multiple seconds. This is the layer most teams diagnose last because the symptom (slow EXPLAIN, slow first row) does not point at data files.

Orphan file growth. Failed writes, aborted Spark jobs, interrupted compaction, and schema-evolution retries leave data files on object storage that no snapshot references. These orphans are invisible to Iceberg but billable by the cloud provider. Production orphan sweeps routinely reclaim a significant share of billable storage on affected prefixes. A detailed breakdown of these nine degradation vectors and how to address each is in the Managed Iceberg in 2026 guide.

Delete file amplification. Merge-on-read tables accumulate position and equality delete files with every UPDATE or DELETE. Without compaction that physically applies deletes, every read must reconcile live rows against all accumulated delete files. Above a certain delete-to-data file ratio, reads pay measurable reconciliation tax and users feel it on every query.

Layout drift. Binpack compaction normalizes file sizes but does not sort data. Engines cannot skip irrelevant file groups because Parquet min/max statistics span wide ranges. A table compacted by size but not sorted by its most-filtered columns still forces full scans on common predicates. And the sort order that accelerated one workload six months ago may not match today's query patterns — layout needs to adapt as usage evolves.

Write amplification in metadata. Every Iceberg write — even a tiny append — produces a new metadata.json, a new manifest list, and one or more new manifest files. For a batch job running hourly, this overhead is invisible. For a streaming job committing every few seconds, metadata writing dominates I/O. Object storage starts throttling. Query planning suffers badly because every new metadata version adds to the chain the planner must traverse.

Governance entropy. Some tables have compaction configured; others do not. Some have snapshot retention; others use defaults that never expire. Retention windows are too aggressive for long-running queries, or too conservative for GDPR. No single view shows the health of the entire lake across catalogs and engines. Every new table starts from scratch unless someone remembers to configure it.

These failure modes do not trigger alarms. They compound silently over weeks and months — and the first real symptom is usually an analyst asking why their dashboard takes 40 seconds instead of 3. By the time someone investigates, the debt has accumulated across dozens of tables and unwinding it is a project, not a task.

Strategy 1: Smart automation with a control plane

A control plane is an architectural layer that sits between your storage, catalogs, and engines — observing the state of every table, understanding cross-engine query patterns, and applying the right maintenance at the right time. It does not replace anything in your stack. It adds the operational intelligence that open-source components do not provide on their own. Netflix built this layer in-house over years — Autotune, janitors, Metacat, Polaris. That story is in Intelligent Lakehouse: Build Like Netflix. Most teams cannot staff that.

LakeOps is a dedicated control plane for Apache Iceberg — the only product that exists purely as this ops layer. Not an engine, not storage, not a catalog. It connects to your existing catalogs (AWS Glue, Polaris, Nessie, Gravitino, Lakekeeper, S3 Tables) and query engines (Trino, Spark, Flink, Snowflake, Athena, DuckDB) without moving data or changing pipelines. Setup takes roughly ten minutes. Your data stays in your storage account.

What you get is a closed-loop system covering:

Lake-wide observability — continuous health scoring (Critical / Warning / Healthy) across every table and catalog, proactive insights ranked by severity, and cross-engine telemetry in one dashboard

Query-aware compaction — a purpose-built Rust/DataFusion engine that sorts data by the columns your queries actually filter on, orders of magnitude faster and cheaper than Spark-based compaction
Snapshot lifecycle management — automated, concurrency-safe expiration with configurable retention policies that respect active readers

Orphan file cleanup — safe detection and removal with age-threshold protection, sequenced after expiration

Manifest and metadata optimization — consolidation, position delete rewrites, and Puffin statistics computation, triggered automatically after compaction

Declarative policies and governance — compaction, retention, cleanup, and configuration rules scoped at organization, catalog, namespace, or table level with cascade inheritance

Multi-engine query routing — route queries across engines optimized for cost, latency, or throughput via QueryFlux

Agentic AI readiness — native MCP interface with schema discovery, layered guardrails (ReadOnly, CostEstimate, PIIMask, HumanApproval), and agent query telemetry feeding back into optimization

Layout simulations — preview compaction impact on Iceberg branches before committing to production

The operating model repeats continuously in four phases: connect and collect telemetry, classify table health and raise insights, execute maintenance in the correct sequence (expire snapshots → remove orphans → compact data files → rewrite manifests), and observe outcomes while enforcing governance. Each cycle the system learns — sort orders adapt as query patterns evolve, the compaction planner improves throughput on repeated runs, and policies self-enforce as new tables are created.

Observability and health classification

After connecting a catalog, every table is continuously scored and classified. The dashboard shows aggregate health across catalogs, recent operations, storage trends, and a summary of proactive insights. Platform teams can triage the entire lake in one screen without writing a single query.

Health is computed from the same Iceberg signals that matter in manual debugging: file count and size distribution per partition, manifest count and depth, snapshot accumulation, delete-file ratio, partition skew, and sort-order alignment with real query patterns. The difference is that the control plane evaluates these signals continuously across every table — not one table at a time when someone asks "why is this slow?"

Proactive insights go beyond classification to tell you why a table is degraded and what to do about it. Each insight is tied to a specific table, a severity level (CRITICAL, HIGH, WARNING, LOW), and a recommended action. In a manual workflow, you discover these conditions after a user files a ticket. With Insights, you discover them while they are still at Warning — before they escalate to a production incident. Details on the observability surface.

Intelligent compaction

Compaction is the highest-impact maintenance operation. LakeOps approaches it differently from generic Spark-based compaction in four ways.

Query-aware sorting. LakeOps tracks which columns appear in WHERE, JOIN, and GROUP BY clauses across every connected engine for every table. During compaction, data is physically sorted by those columns — so Parquet row-group statistics enable engines to skip irrelevant data without reading it. When query patterns change, the sort order adapts on subsequent compaction passes without manual configuration.

Rust/DataFusion engine. Compaction runs on a purpose-built engine written in Rust with Apache DataFusion. Arrow columnar buffers, bounded memory, zero garbage collection, lock-free parallelism, native Parquet I/O with predicate pushdown. Orders of magnitude faster and cheaper than Spark-based compaction on identical datasets. Partitions that OOM Spark complete in minutes because memory is bounded, not heap-dependent. Full benchmark details are in the compaction product page.

This cost reduction is what makes frequent compaction economically viable. Instead of one nightly job that lets streaming tables degrade for 24 hours between passes, LakeOps runs compaction multiple times per day when file health signals warrant it. Tables stay continuously healthy instead of oscillating between degradation and cleanup.

Conflict-aware execution. The system inspects active writer state and targets only cold partitions — partitions without active streaming appends. If a boundary-case conflict occurs, only the affected file group retries on the next cycle. No data loss, no full-job restarts, no CommitFailedException storms.

Self-improving planner. The compaction planner learns from workload telemetry across consecutive runs. Same table, zero config changes — throughput and runtime improve with each pass as the planner adapts its internal parallelism, file-group sizing, and sort strategy.

Snapshot, orphan, and manifest lifecycle

LakeOps automates the full maintenance lifecycle with correct sequencing:

Snapshot expiration. Policy-based retention (time and count) enforced continuously. Concurrency-safe — expiration respects active readers and in-flight queries, preventing the most common production failure in teams that automate snapshot expiration without accounting for concurrent readers. Version history, snapshot comparison, and one-click rollback remain available within the retention window.

Orphan cleanup. Age-threshold safety (7+ days by default) protects in-flight writes. Sequenced after expiration so newly dereferenced files are captured in the same sweep. Full audit trail of every removed file.

Manifest and metadata optimization. Three operations triggered automatically after compaction: manifest consolidation (fewer manifests = faster planning), position delete file rewrite (clean reads for merge-on-read tables), and Puffin statistics computation (column-level NDV, min/max, null counts for aggressive file-level pruning). Manifests are rewritten after compaction, not before, so the index reflects the final file set.

Every maintenance operation is logged lake-wide and per table with duration, impact, and status — the audit trail that answers "what ran, when, and did it succeed?" during incident response, without parsing executor logs across multiple systems.

Policies, routing, and AI readiness

Declarative policies scope compaction targets, snapshot retention, orphan cleanup, and manifest optimization at four levels with cascade inheritance: organization → catalog → namespace → table. Adaptive Maintenance bundles the full lifecycle into one data-driven policy — the system decides which operations each table needs from structural signals and runs them in the correct sequence. Cross-catalog enforcement works uniformly across AWS Glue, Polaris, Nessie, and Gravitino.

Multi-engine routing via QueryFlux connects all engines to a unified routing layer that makes per-query decisions based on cost, latency, table health, and historical patterns. Without routing, teams either standardize on one engine (losing optimization) or let each user choose (creating cost chaos). Routing decisions are enriched by table health — if a table is currently degraded, queries can be directed to engines that handle fragmentation better until compaction resolves the issue.

Agentic AI readiness provides a native MCP (Model Context Protocol) interface for AI agents with schema discovery, wire compatibility (PostgreSQL, MySQL, Arrow Flight), and layered guardrails configurable per agent session. Agent query telemetry feeds back into compaction and sort-order decisions — when agents start filtering on columns that no human dashboard uses, the compaction planner detects the new pattern and adjusts accordingly. Details in the AI agents guide.

The compound effect

The capabilities above are not independent features. They form a closed loop where each component's output feeds the next: observability detects degradation, the system runs expiration to unpin files, orphan cleanup captures newly dereferenced objects, compaction runs sorted by the columns that real queries filter on, manifests are rewritten against the final file set, Puffin statistics are recomputed, the table moves from Warning to Healthy, and the event is logged with before/after metrics. Next cycle, the compaction planner runs faster because it learned from the previous pass.

The engineering hours that used to go into writing, scheduling, monitoring, and debugging maintenance scripts are returned to product work. Policies are defined once and the system executes them continuously — every action logged, auditable, and reversible. Data stays in your storage account. No pipeline rewrites. Minutes to install.

Strategy 2: Manual management with open-source tooling

Not every team needs a control plane on day one. If you have a handful of Iceberg tables, one or two engines, batch-only writes, and a platform engineer who owns maintenance, manual management works. This section is the complete playbook — the same methodology that works whether you have a control plane or not.

The manual stack

The typical setup: an orchestrator (Airflow, Dagster, or Prefect) scheduling maintenance as DAG tasks, Spark (on EMR, Dataproc, or Kubernetes) running Iceberg maintenance procedures, a catalog (Glue, Polaris, Nessie), custom SQL queries against metadata tables piped into Grafana or Datadog, and threshold-based alerting. This stack is well-understood and runs at many organizations today. The constraint is not capability but coverage: every new table, every new write pattern, and every new engine requires incremental engineering work to bring under management.

Compaction by hand

Use Spark's rewrite_data_files procedure:

CALL catalog.system.rewrite_data_files(
  table => 'db.events',
  strategy => 'binpack',
  where => 'event_date >= current_date() - INTERVAL 7 DAYS',
  options => map(
    'target-file-size-bytes', '268435456',
    'min-input-files', '3',
    'partial-progress.enabled', 'true',
    'partial-progress.max-commits', '20',
    'max-concurrent-file-group-rewrites', '15'
  )
);

Key decisions: binpack for file-size normalization, sort for query-pattern optimization, zorder for multi-column predicates. Target 256–512 MB files. Always enable partial-progress. Never compact the active write partition on a streaming table — compacting the hot partition while Flink is appending is the primary cause of CommitFailedException storms. Run compaction on a separate cluster. Schedule cadence must match write cadence — if you write 14,400 files per day and compact once at midnight, the table spends 23 hours degraded. Decision frameworks are in the compaction strategies guide.

For sort compaction, you must determine sort columns yourself by analyzing query patterns across engines. When patterns change, you must notice and update the sort order manually. This is the gap a control plane closes automatically with cross-engine telemetry.

CALL catalog.system.rewrite_data_files(
  table => 'db.events',
  strategy => 'sort',
  sort_order => 'event_date ASC, region ASC, user_id ASC',
  where => 'event_date >= current_date() - INTERVAL 7 DAYS',
  options => map(
    'target-file-size-bytes', '268435456',
    'partial-progress.enabled', 'true',
    'partial-progress.max-commits', '20',
    'max-file-group-size-bytes', '10737418240'
  )
);

Snapshot expiration, orphan cleanup, and manifests by hand

Expire snapshots regularly. Retention window must exceed your longest-running query — a Spark job that runs for 3 hours will fail if its snapshot is expired mid-flight. This is one of the most common production failures in teams that automate aggressively.

CALL catalog.system.expire_snapshots(
  table => 'db.events',
  older_than => TIMESTAMP '2026-07-14 00:00:00',
  retain_last => 50
);

ALTER TABLE db.events SET TBLPROPERTIES (
  'write.metadata.delete-after-commit.enabled' = 'true',
  'write.metadata.previous-versions-max' = '100'
);

Orphan cleanup runs after expiration, never before. Always dry-run first. The older_than threshold must be far enough in the past to protect in-flight writes — production teams use 7+ days. Verify URI schemes match (s3:// vs s3a:// vs s3n://) before your first run.

CALL catalog.system.remove_orphan_files(
  table => 'db.events',
  older_than => TIMESTAMP '2026-07-07 00:00:00',
  dry_run => true
);

Rewrite manifests after every compaction cycle. This is the step most teams skip. Without rewriting, planning time remains elevated even though the data files are healthy. On extreme fragmentation, rewriting alone can drop planning from 30+ seconds to under a second. Full methodology in the orphan cleanup guide and the table health guide.

CALL catalog.system.rewrite_manifests(
  table => 'db.events'
);

Health monitoring and the safe sequence

Manual monitoring means querying Iceberg metadata tables and setting alert thresholds. Start with four numbers per table: file count and average size per partition, snapshot count, manifest count, and delete-file ratio for MOR/CDC tables.

SELECT partition, COUNT(*) AS file_count,
  ROUND(AVG(file_size_in_bytes) / 1048576, 1) AS avg_size_mb
FROM catalog.db.events.files
GROUP BY partition ORDER BY file_count DESC LIMIT 20;

SELECT COUNT(*) AS snapshot_count
FROM catalog.db.events.snapshots;

SELECT COUNT(*) AS manifest_count
FROM catalog.db.events.manifests;

The correct maintenance order — consistent across the official Iceberg docs and the automating table maintenance guide — is:

Expire snapshots — release metadata references to files nobody needs for time travel
Remove orphan files — delete unreferenced objects from storage (with 7+ day safety window)
Compact data files — merge small files and physically apply deletes
Rewrite manifests — consolidate the metadata index against the final file set
Compute statistics (optional) — refresh Puffin stats so planners see current NDVs

Compacting before expiration rewrites files about to be discarded — wasted compute. Orphan cleanup before expiration misses the largest reclaimable set. Manifest rewrite before compaction produces an index stale minutes later. Never invert the order under pressure.

Where manual management breaks down

Manual management does not fail because the tools are inadequate. Spark procedures work. Airflow can schedule them. Grafana can chart health. The breakdown happens at the system level.

Coverage gap. Every new table must be added to the DAG, configured, monitored, and validated. New tables slip through. Policies do not inherit automatically.

Workload blindness. Manual compaction sorts by columns you choose at configuration time. When a new dashboard goes live, when a new team starts querying with different filters, when AI agents start accessing columns nobody filtered before — the sort order becomes stale.

Sequence violations under pressure. At 3 AM during an incident, the instinct is to compact first. Compacting before expiration wastes compute. Running orphan cleanup with a too-short safety window corrupts data.

Concurrency danger. Snapshot expiration that does not respect active readers kills long-running queries. Compaction that targets active write partitions causes commit conflicts. Manual scripts do not know what other processes are doing.

Cost of execution. Spark compaction means spinning up JVM clusters, allocating executors, and paying for compute that is orders of magnitude more expensive than a purpose-built engine. Most teams run on over-provisioned clusters because under-provisioning causes failures and 2 AM pages.

Linear engineering scaling. Maintenance time grows linearly with table count. The team size does not. The data swamp to modern lakehouse guide walks through the transition in detail.

Fix the writers, not just the table

This section applies to both strategies. Many data lake incidents are writer-configuration debt — the pipeline creates structural damage faster than maintenance can repair it.

Checkpoint interval. The single most impactful setting for streaming Iceberg tables. Flink at 30-second checkpoints with 16 write tasks produces ~46,000 files per day per table. Move to 3–5 minute checkpoints and file creation drops dramatically. For staging tables, feature stores, and compliance archives, 10–15 minute intervals are often appropriate. Details in the Flink Iceberg optimization guide.

SET 'execution.checkpointing.interval' = '5min';
SET 'execution.checkpointing.mode' = 'EXACTLY_ONCE';
SET 'execution.checkpointing.timeout' = '10min';

Write distribution mode. Set write.distribution-mode=hash for partitioned streaming tables. Hash distribution shuffles records by partition key before writing, cutting file count by the parallelism factor.

ALTER TABLE catalog.db.events SET TBLPROPERTIES (
  'write.distribution-mode' = 'hash',
  'write.target-file-size-bytes' = '268435456'
);

Sink parallelism. Decouple source parallelism from sink parallelism. A Kafka topic with 64 partitions does not need 64 writer subtasks. Setting source at 64 and sink at 8–16 with hash distribution yields far fewer files while maintaining consumption throughput.

Partition grain. Over-partitioning is the leading cause of small files at scale. Prefer days(ts) over hours(ts). Use bucket(N, col) instead of identity transforms on high-cardinality columns. Decision frameworks in the partitioning best practices guide.

Delete mode vs mutation rate. Merge-on-read (MOR) is right for high-churn CDC where write latency matters more than read performance. Copy-on-write (COW) is right when read latency is sacred. Mixing MOR with no compaction policy is how delete-ratio incidents start.

Fixing writers halves the maintenance burden for both strategies. Skipping this step turns your control plane into a very expensive mop — and makes your manual scripts a full-time job.

Choosing your strategy

Both strategies work. The question is scale, cost, and where engineering time delivers the most value.

Strategy 2 (manual) fits when: you have fewer than ~50 tables, one or two engines, batch-only or low-frequency writes, a platform engineer who owns maintenance, and predictable query patterns that do not change frequently. The metadata queries and safe sequence in this guide are your on-call playbook.

Strategy 1 (control plane) fits when: table count crosses ~50, you run streaming or CDC pipelines, multiple engines hit the same tables, engineering time spent on maintenance scripts exceeds the cost of a purpose-built system, GDPR or compliance requirements need policy-based enforcement, or query patterns change and layout optimization must adapt without manual intervention.

The strategies compose. Many teams start with manual management, adopt a control plane for the highest-traffic catalogs first, and expand as value is proven.

The decision framework reduces to a few questions: Does the system see all my catalogs in one place? Is compaction query-aware across engines? Are operations coordinated in the correct sequence? Do policies inherit to new tables automatically? Does maintenance compete with my query clusters for compute? Can I simulate layouts before committing? Are agent guardrails first-class? Do I have to move data?

If most answers are "no" on an open multi-engine Iceberg stack, you do not have a managed data lake yet — you have Iceberg plus hope.

The migration path: manual to control plane

The transition from manual management to a control plane is incremental, not disruptive:

Connect a catalog. Discovery and health classification begin immediately without touching existing maintenance infrastructure.
Inspect and learn. Use the observability dashboard to understand the current state of your lake. Teams often discover tables they did not know were degraded.
Run maintenance on the worst tables. Trigger compaction, expiration, or orphan cleanup on the most Critical tables. Watch the before/after metrics.
Enable policies per namespace. Define compaction targets, retention windows, and cleanup thresholds. Every table in scope inherits automatically.
Enable Adaptive Maintenance for streaming tables. Let the system decide when to compact based on file health signals rather than cron schedules.
Retire manual scripts. As each namespace is covered by policies, disable the corresponding Airflow tasks.
Expand to remaining catalogs. Each step is independently valuable.

Teams running the control-plane model report far fewer Iceberg incidents — not because the runbook got faster, but because file explosions, manifest storms, and orphan cliffs are resolved while still at Warning. The real end state of a managed data lake is not faster debugging. It is rarely needing to debug at all.

The bottom line

A managed data lake in 2026 is not a storage product, not a single job, and not a platform you must adopt wholesale. It is an operational contract: tables stay healthy, costs stay predictable, queries stay fast, policies apply consistently, and AI agents get the data they need with appropriate guardrails. The question is who upholds that contract — your engineering team writing scripts, or a dedicated system that does it continuously.

Strategy 2 works. It is well-understood, uses open-source tooling, and scales to a moderate number of tables with dedicated platform engineering. The manual playbook in this guide — safe sequence, metadata queries, writer configuration, per-table monitoring — is the same playbook that production teams use every day.

Strategy 1 works better at scale. LakeOps closes the operational loop that manual scripts cannot: continuous health classification across every table and catalog, query-aware compaction on a purpose-built Rust engine, coordinated maintenance that respects the safe sequence, policies that inherit to every new table, cross-engine telemetry that adapts layout to real query patterns, multi-engine routing that sends each query to the right engine, and agentic AI readiness with layered guardrails. It runs on your existing stack without moving data or replacing engines.

The format already gave you ACID and time travel. The operations layer is what determines whether you have a data lake that works — or a data lake that worked once and has been quietly degrading since. Choose the strategy that matches your current scale. Build toward the one that matches where you are headed.

For deeper technical breakdowns: Managed Iceberg in 2026 covers the nine control-plane components. Intelligent Lakehouse: Build Like Netflix shows how Netflix solved these problems internally. Routing Multiple Query Engines with Iceberg covers the multi-engine routing architecture. Automating Table Maintenance walks through the progression from scripts to autonomous operations. Table Health and Maintenance covers every maintenance operation with configuration detail. And From Data Swamp to Modern Lakehouse covers the full architectural transition. For the platform that closes the loop, explore lakeops.dev.