DEV Community: joni sar

Managed Data Lake: A Guide for 2027

joni sar — Sun, 23 Aug 2026 09:05:42 +0000

Managed Data Lake: A Guide for 2027

Apache Iceberg is the standard table format for production data lakes in 2027. Every major engine reads and writes it natively. The catalog ecosystem standardized on REST. You own your data on commodity storage with no lock-in.

But Iceberg deliberately separates the table format from the system that keeps tables healthy. It gives you the primitives for maintenance — rewrite_data_files, expire_snapshots, remove_orphan_files, rewrite_manifests — but not the intelligence to decide when, how, and in what order to run them. Without that operational layer, every Iceberg table degrades over time: small files accumulate, snapshots bloat metadata, sort orders drift from query patterns, orphan files inflate storage costs, and query performance decays silently until something breaks visibly.

This operational gap is the central challenge of running a data lake at production scale. Netflix built four internal services to address it — Autotune for compaction strategy selection, Polaris for catalog management, janitors for garbage collection, Metacat for cross-service observability — each staffed by dedicated teams over multiple years. Google engineered automatic compaction and garbage collection directly into BigLake, so their managed Iceberg tables stay healthy regardless of write volume or query pattern changes.

In 2027, you do not need to replicate that investment. This guide covers what "managed" actually means for a data lake, the degradation mechanics that make it necessary, the control plane architecture that solves it, and the practical paths to getting there — whether you are running 50 tables or 5,000.

Why Lakes Degrade — The Mechanics

The degradation pattern is predictable and present in nearly every Iceberg lake running for more than three months without dedicated maintenance. Understanding these mechanics is necessary regardless of which management approach you choose.

The Small-File Problem

Every streaming writer — Flink, Spark Structured Streaming, Kafka Connect, RisingWave, CDC connectors — produces files sized by checkpoint intervals, not by optimal read patterns. A Flink job committing every 60 seconds across 100 active partitions creates 144,000 new files per day. Each of those files might be 1–5 MB. The target for efficient reads is 128–512 MB.

Every query pays per-file overhead: an S3 GET request ($0.0004 per 1,000), a Parquet footer parse, a task-scheduling slot in the engine. This means query compute scales with file count rather than with actual data volume. You are paying for how the data was written, not for how much data exists.

The insidious part is the rate. A batch table that loads once daily accumulates files slowly — degradation takes months to become visible. A streaming table crosses into problematic territory within days. A high-throughput streaming table (hundreds of partitions, sub-minute commits) can reach a degraded state within hours of going live if no background compaction runs alongside the writer. The longer you wait to compact, the heavier and more disruptive the operation becomes — it conflicts with active writers, consumes significant cluster resources, and takes hours instead of seconds.

Snapshot Accumulation

Every Iceberg commit creates a new snapshot — that is the mechanism that enables time travel and atomic rollback. A streaming table at 5-minute commits accumulates ~288 snapshots per day. Without expiration, counts reach tens of thousands within weeks.

Query planning reads the snapshot chain to resolve the current table state. Once you exceed roughly 1,000–2,000 retained snapshots, planning time degrades measurably. The metadata.json file grows — production environments have seen it reach 400+ MB. The symptom is slow EXPLAIN, slow first-row-time, and users reporting that "the query hangs for a few seconds before anything happens." The data scan itself is fine. It is the planning that has become expensive.

The correct retention window depends on your time-travel requirements. For most analytics workloads, 3–7 days of snapshots is sufficient. Regulated industries (financial, healthcare) may require 30–90 days — but should understand the planning-time trade-off and invest in more aggressive manifest rewriting to compensate. Whatever your retention policy, it must be actively enforced. Iceberg does not expire snapshots on its own.

Manifest Fragmentation

Manifests are Iceberg's lookup structure — they tell engines which data files exist in which partitions and what their column-level statistics are. Every commit appends at least one manifest entry. Over months of streaming writes, you accumulate hundreds or thousands of small manifest files.

The query planner must read all of them to build an execution plan. This adds seconds of latency before any data file is opened. When Iceberg 1.11 introduced server-side scan planning (shifting metadata traversal off query engines entirely), lean manifests became even more valuable — the planning server benefits proportionally from fewer, larger manifest files.

Most teams misdiagnose slow planning as an engine problem — they try tuning memory, scaling coordinators, or upgrading engine versions. It is almost always the manifests. Rewriting them into a smaller number of well-merged files is one of the highest-leverage operations available — and one of the least commonly automated.

Orphan Files

Failed Spark stages, interrupted compaction, aborted writes, schema-evolution retries, OOM kills during large rewrites, and partial commits all leave files on object storage that no Iceberg snapshot references. They are invisible to SELECT queries, invisible to Iceberg metadata, invisible to your data quality checks — but fully visible on your cloud bill.

You cannot find them by querying your tables. You must compare what exists in object storage against what Iceberg metadata references — a set difference operation that requires listing entire prefixes and cross-referencing with the catalog. This is why orphan accumulation goes undetected for months. Without a dedicated process that runs this comparison regularly, you have no way to know how much you are spending on dead data.

Production orphan sweeps routinely reclaim 20–40% of a lake's storage. For a 100 TB lake, that is 20–40 TB of billable storage serving no purpose. At $0.023/GB/month (S3 Standard), that is $460–$920 per month for one lake — just in orphan storage. Multiply by the number of accounts, regions, and environments your organization operates.

Sort-Order Drift

Bin-pack compaction — merging small files into larger ones — solves the file-count problem but does nothing for data organization. If your data is not physically sorted by the columns queries actually filter on, Parquet min/max statistics span wide ranges and engines cannot skip irrelevant row groups.

Consider a transactions table sorted by transaction_id (because it was the natural append key) but queried primarily by customer_id and event_date. Every file contains a wide range of customer IDs, so a predicate like WHERE customer_id = 'abc123' cannot skip any files based on statistics alone — the engine scans everything.

Sort the same data by customer_id, event_date and the statistics become tight. A query for one customer touches one or two files instead of hundreds. The engine eliminates 95%+ of scan volume before reading a single row. This is not an incremental improvement. It is a categorical difference in query performance.

The challenge is that sort orders set at table creation drift. The column that was the primary filter six months ago might not be today. New consumers join. Dashboard patterns shift. ML feature pipelines add access patterns the original designer never anticipated. Across hundreds of tables with different consumers, manual sort-order tuning is impractical — and sort orders that are wrong silently cost you compute on every query, every day, without generating any error or warning.

Delete File Accumulation

Merge-on-read tables accumulate position and equality delete files with every UPDATE or DELETE operation. This is the mechanism that makes updates fast at write time — rather than rewriting the entire file, Iceberg records "this row is deleted" in a separate file.

The cost lands on reads. Every query must reconcile live rows against all accumulated delete files for the partitions it touches. This reconciliation cost grows linearly with each write cycle. At high delete volumes — CDC workloads doing frequent upserts, GDPR compliance deleting records, slowly changing dimensions — the read-time overhead becomes significant. Queries that should complete in seconds spend most of their time reconciling deletes rather than reading actual data.

Iceberg 1.11's deletion vectors (backed by Roaring bitmaps) improve the mechanism by storing deletes more compactly and enabling faster reconciliation. But the fundamental accumulation problem remains: unless compaction regularly applies those deletes (physically rewriting files to exclude deleted rows and then removing the delete files), read performance degrades continuously.

The Organizational Cost

Beyond the technical debt, the team cost scales worst.

Maintenance responsibilities accumulate incrementally: compaction scripts, expiration logic, monitoring, alerting, per-table configuration for heterogeneous workloads, debugging writer conflicts, handling OOM failures, on-call rotation for overnight maintenance failures, documentation, and onboarding. Each table added to the lake adds marginal load to a team that is not growing proportionally.

Platform engineers who should be building data products, self-service tooling, or pipeline reliability improvements are instead maintaining an ever-growing library of operational scripts. The knowledge concentrates in a few heads — when those people leave, the lake degrades within weeks because nobody else knows what the DAGs do or why they are configured that way.

This cost does not show up on cloud bills but manifests in team velocity, retention risk, and the growing gap between what the data platform promises and what it delivers.

What "Managed" Actually Means

The term is overloaded. Vendors use it to mean proprietary platforms you migrate into. Cloud providers use it for single-service table maintenance. Some teams use it to mean a weekly compaction cron job. Others equate it with having a catalog.

A more useful definition: a managed data lake is one where every table stays healthy, every query hits well-organized data, and operational cost does not grow linearly with table count — without requiring a platform team that scales proportionally with the lake.

This definition has three important implications. First, it is about outcomes, not tools. Running compaction nightly does not make your lake managed if half your tables are still degraded. Second, it covers the full lifecycle — not just file compaction, but snapshot management, orphan cleanup, manifest optimization, sort alignment, policy enforcement, and observability. Third, it must scale sublinearly: managing 500 tables should not require 10x the effort of managing 50.

The Architecture

Your data lake already has three layers: commodity object storage (S3, GCS, ADLS), a table format and catalog (Iceberg plus Polaris/Glue/Nessie/Gravitino/Lakekeeper), and compute engines (Spark, Trino, Snowflake, Athena, DuckDB, Flink). These give you storage economics, structural reliability, and query flexibility.

What you are missing is the fourth layer: an operations control plane.

This is not the same as the catalog. The catalog (Polaris, Glue, BigLake Metastore) is a metadata control plane — it resolves table locations, handles access control, and provides the REST interface for multi-engine access. Every major catalog now implements the REST spec: Polaris (Apache top-level project since February 2026), Glue's REST adapter, BigLake Metastore, Nessie, Gravitino, Lakekeeper.

The operations control plane sits above the catalog. It uses metadata as input, but its job is fundamentally different: observe structural health across every table, classify what needs attention, decide what maintenance to run, execute it safely and in the correct sequence, adapt compaction to real query patterns across all engines, enforce governance at lake scale, route queries to optimal engines, and feed outcomes back into future decisions.

LakeOps is this operations control plane — a purpose-built product for Apache Iceberg that does nothing else.

It is not an engine, not storage, not a catalog. It connects to your existing catalogs (AWS Glue, Polaris, Gravitino, Nessie, Lakekeeper, S3 Tables) and engines (Spark, Trino, Flink, Snowflake, Athena, DuckDB), adds autonomous management across the entire lake, and requires no data movement or pipeline changes. Most teams have their first lake-wide health assessment within ten minutes of connecting a catalog.

The key architectural property: it does not replace anything. If you remove the control plane, everything continues working exactly as before — just without the automation. Your storage, your engines, your catalog, your data — all unchanged. The control plane adds intelligence to an existing stack. It does not create dependency.

The Maturity Spectrum

Teams operating Iceberg follow a predictable progression. Identifying where you sit determines what to invest in next.

Stage 1: No maintenance. Iceberg adopted for format-level benefits. Pipelines land data. Nobody explicitly owns table health. Performance degrades and storage costs rise gradually, but the symptoms are not connected to a root cause. Common in organizations that migrated to Iceberg within the last year.

Stage 2: Reactive fixes. After an incident — slow dashboard, cost anomaly, failed SLA — an engineer writes a one-off compaction script or snapshot expiration job. The immediate pain resolves, but only for the tables that broke visibly. The rest of the lake continues degrading silently. The cycle repeats every few months on different tables.

Stage 3: Scheduled automation. The team builds Airflow DAGs or cron jobs running maintenance on a fixed schedule. Compaction nightly. Snapshot expiration weekly. Orphan cleanup monthly. This works for stable batch workloads but has structural limitations at scale:

Wastes compute on healthy tables that do not need maintenance
Misses tables that degrade between cycles (a streaming table needing hourly compaction gets nightly)
Cannot adapt to changing workloads — new query patterns do not trigger sort-order changes
Runs operations as independent jobs without sequencing coordination, creating race conditions
The maintenance infrastructure itself requires ongoing maintenance: updating DAGs, handling failures, adjusting schedules, onboarding new tables

Stage 4: Autonomous management. A dedicated system continuously observes every table, classifies health based on combined structural, storage, and query signals, sequences operations in the correct order, adapts compaction strategy to real query patterns from cross-engine telemetry, enforces policies lake-wide, and feeds outcomes back into decisions. Maintenance is signal-driven rather than schedule-driven. Tables that need attention get it immediately. Healthy tables cost zero compute.

Most production lakes sit between Stage 2 and Stage 3. The gap from Stage 3 to Stage 4 is where teams either build a dedicated platform (6–12 months, 2–3 senior engineers full-time, plus ongoing maintenance of the platform itself) or adopt an existing control plane. Stage 3 is where DIY efforts stall — building cross-engine telemetry collection, health classification logic, sequencing and conflict resolution, adaptive compaction strategy, and policy inheritance is a non-trivial distributed systems project that most teams underestimate.

How the Control Plane Works

A control plane is not a collection of scripts with a dashboard. It is a closed-loop system where each stage feeds the next. The difference between Stage 3 automation (scheduled scripts) and Stage 4 automation (autonomous management) is entirely in this loop: observe → decide → act → optimize → govern → route → enable → learn.

Observation: Unified Telemetry

Everything starts with signals. Three telemetry streams feed every decision the system makes:

First, structural metadata from Iceberg — file counts per partition, average file sizes, manifest depth, snapshot count and velocity, delete file ratios, current sort order configuration, partition layout, and schema history. This tells you the table's physical state — the raw structural facts that indicate health or degradation.

Second, object storage metrics — total bytes per prefix, orphan file counts, access patterns, request rates, storage class distribution. This reveals the cost reality that metadata alone does not capture. A table might look healthy in Iceberg metadata while harboring terabytes of orphaned files in its storage prefix.

Third, and most critically, cross-engine query telemetry — which columns appear in WHERE, JOIN, and GROUP BY clauses from every engine reading your tables. Trino knows its own queries. Spark knows its own. Snowflake knows its own. Athena knows its own. None of them knows what the others are doing on the same table. None of them can tell you what the combined access pattern looks like across your full consumer base.

A control plane collects this telemetry from every connected engine, correlates it per table, and uses the combined signal to drive optimization decisions that no single engine could make alone. This is the fundamental capability that separates an operations control plane from single-engine maintenance tools — it sees the whole picture.

This unified telemetry is what enables the system to classify every table as Healthy, Warning, or Critical — and to surface specific, actionable insights ranked by severity. You know exactly which of your 500 tables needs attention right now, what the problem is, what caused it, and what the fix should be.

Decision: Health-Driven Prioritization

Raw telemetry becomes actionable through health classification. Instead of blindly running compaction on every table overnight (Stage 3), the system targets precisely the tables that need attention — and leaves healthy tables alone.

A table's health score reflects the combined severity of its degradation signals: file count relative to target, manifest fragmentation ratio, snapshot depth relative to retention policy, sort-order alignment with actual query patterns, orphan volume, and delete file accumulation. Each dimension has independent thresholds, and the composite determines the classification.

When a table crosses from Healthy to Warning (file count exceeds target, sort order drifts from query patterns, manifest fragmentation passes threshold), maintenance triggers. When it returns to Healthy, operations pause. Critical tables get immediate attention regardless of scheduling. Warning tables are queued by priority. Healthy tables cost zero compute — no maintenance runs against them until conditions change.

This feedback loop eliminates the two failure modes of Stage 3 automation: wasting compute on healthy tables that do not need attention, and missing tables that degrade between fixed-schedule cycles. A streaming table that crosses into Warning two hours after the last scheduled maintenance run does not wait until tomorrow night. It gets addressed when the signal fires.

Execution: Sequenced Maintenance

The individual operations are well-understood — Iceberg provides the procedures. What matters is the order, the coordination, and the safety.

The correct sequence is: expire snapshots → remove orphan files → compact data files → rewrite manifests → refresh statistics. Each step depends on the previous:

Expiration dereferences files and snapshots that are past their retention window
Orphan cleanup then safely removes those dereferenced files from storage — only possible after expiration has run
Compaction operates on live data only — not on files that are about to be expired — reducing wasted work
Manifest rewriting consolidates the new manifest structure that compaction created
Statistics refresh ensures the newly organized data has accurate min/max values for predicate pushdown

Running these in the wrong order wastes compute (compacting data that will be expired in the next step), creates safety risks (removing files that unexpired snapshots still reference), or produces suboptimal results (rewriting manifests before compaction means they immediately fragment again on the next compact).

At Stage 3, these operations typically run as independent cron jobs — compaction in one DAG, expiration in another, orphan cleanup in a third, each with its own schedule and no awareness of the others. This creates exactly the sequencing failures described above. A control plane runs them as a coordinated pipeline per table, every time, conflict-aware (never compacting partitions with active writers), with OCC-aware retry logic for optimistic concurrency conflicts.

Adaptive maintenance bundles this further: rather than configuring each operation's frequency separately, a single data-driven policy monitors commit frequency, file accumulation rate, query load, and seasonal patterns — then runs the right operations at the right frequency, automatically. A high-velocity streaming table gets frequent bin-pack compaction to keep pace with the write rate. A stable batch table gets occasional deep sort compaction when access patterns shift. A table that received a sudden burst of writes gets immediate attention. One policy toggle replaces dozens of per-table DAG configurations.

Optimization: Query-Aware Compaction

This is where the control plane delivers its highest-impact differentiation from scripts and scheduled jobs.

Basic bin-pack compaction normalizes file sizes — merging many small files into fewer large ones. That is necessary (it solves the per-file overhead problem) but insufficient (it does nothing about data organization). A table with perfectly sized files but random data ordering still forces engines to scan everything because min/max statistics span the full value range.

The real performance leverage is in sort compaction: physically rewriting data so records are ordered by the columns queries actually filter on. When data is correctly sorted, Parquet row-group min/max statistics become tight — each row group contains a narrow range of values for the sort columns. Engines read these statistics during planning and skip entire file groups whose ranges do not match the predicate. The difference is not incremental. It is the difference between scanning 100% of a table and scanning 5–15%.

The challenge at scale is knowing what to sort by — and keeping it current. Static sort orders set at table creation drift within weeks as query patterns evolve. A column that was the primary filter six months ago might be secondary today. New consumers arrive. Dashboard logic changes. ML pipelines add new access patterns. Across hundreds of tables with different consumers across different engines, manual sort-order tuning is impractical.

Cross-engine telemetry solves this. The control plane observes which columns are actually filtered, joined, and grouped across all engines hitting each table — Trino, Spark, Snowflake, Athena, DuckDB — then physically re-sorts data to match the real combined access pattern. As patterns change, the sort order adapts. The optimization is continuous and automatic, not a one-time configuration that decays.

LakeOps executes compaction on a purpose-built Rust/DataFusion engine — no JVM, no garbage collection pauses, no Spark clusters to provision or decommission. Compaction is event-driven (triggered by health signals, not cron schedules) and conflict-aware (never touching partitions with active writers, with built-in OCC retry). The engine completes bin-pack and sort operations in a fraction of the time Spark requires for the same workload, without the cluster management overhead.

Before committing to a sort-order change on a large table — where a wrong decision means rewriting terabytes of data for no benefit — layout simulations let you test strategies safely. Simulations create a real Iceberg branch from the latest snapshot, apply the proposed layout to that branch, replay actual query patterns from cross-engine telemetry against it, and compare results against the current baseline. You see projected file counts, data-skipping improvement percentage, and scan reduction for each candidate strategy — before writing a single byte to production data. The branch is discarded after analysis. This is the verification step that makes autonomous sort optimization trustworthy at scale.

Governance: Policies That Scale

At one table, configuration is a one-time decision. You pick a compaction strategy, set a retention window, choose a target file size, and move on. At 50 tables it becomes repetitive. At hundreds of tables across multiple catalogs and namespaces, it becomes governance entropy.

The symptoms: some tables have compaction configured, others use defaults nobody verified. Retention windows are inconsistent — production tables with 90-day retention next to staging tables with no expiration at all. New tables inherit nothing because nobody templated the configuration. An engineer leaving means their table-specific tuning knowledge leaves with them. Auditors ask "what is your data retention policy?" and nobody can answer precisely because the truth is scattered across table properties, DAG configs, and Terraform state files.

A control plane enforces declarative policies with hierarchical inheritance: compaction strategy, target file size, snapshot retention, cleanup safety windows, and maintenance frequency defined at organization, catalog, namespace, or table level. Everything cascades downward — set a policy at the namespace level and every table in that namespace inherits it. New tables automatically pick up the correct configuration. Exceptions override at the leaf level when specific tables need different treatment. Policies are versioned, auditable, and diff-able — changes are tracked, reviewable, and reversible.

Routing: Multi-Engine Cost Optimization

Running multiple engines on the same Iceberg tables is standard practice in 2027. The REST catalog spec means any engine can read any table. Most production lakehouses use at least three: Trino for interactive analytics, Spark for heavy ETL, and one of Snowflake/Athena/DuckDB for BI or ad-hoc queries. Each has radically different cost and latency profiles.

A point lookup costs fractions of a cent on DuckDB running locally and significantly more on Snowflake's warehouse compute. A multi-table join with complex aggregations completes in seconds on Trino but might take minutes on DuckDB. A 10 TB batch transformation is cheapest on Spark with spot instances. Without routing, applications hardcode engine connections — every query goes to one engine regardless of whether a cheaper or faster option exists for that specific query shape.

A control plane's routing layer classifies query shapes (point lookups, analytical aggregations, complex joins, full-table scans) and sends each to the optimal engine based on cost targets, latency requirements, and current table health. Applications connect to a stable SQL endpoint; backend engine selection adapts as conditions change, as table health improves, and as new engines are added to the pool.

Because the control plane already has cross-engine telemetry and table health data, routing decisions use the same signals that drive compaction — creating a virtuous cycle. Better-compacted tables unlock more routing options (a well-sorted table can serve point lookups from DuckDB that would be too slow on an unsorted table). Better routing data informs compaction priorities (tables with high cross-engine query volume get priority optimization).

AI Readiness: Agent Access at Scale

AI agents are becoming primary consumers of lakehouse data — not a future prediction, but a present reality in many production environments. They issue SQL iteratively, at high frequency, without human review of each query. They explore schemas, join across tables, aggregate at scale, and do it hundreds of times per analysis cycle. This creates requirements that traditional table maintenance does not address: structured schema discovery (agents need to understand table relationships programmatically), cost and safety guardrails (an agent with no limits will issue expensive full-table scans), and storage that self-optimizes for agent access patterns that may differ from human patterns.

LakeOps provides an agent-native MCP interface — any MCP-compatible agent (Claude, GPT, custom models) connects with zero integration code and auto-discovers available tools for schema discovery, health analysis, and query execution. Guardrails stack composably: ReadOnlyGuard prevents DDL and DML, CostEstimateGuard rejects queries that would scan more than a cost threshold, PIIMaskGuard scrubs sensitive columns before results reach the agent, HumanApprovalGuard gates high-stakes operations for human review. These are not all-or-nothing — you compose them per use case.

Agent query telemetry feeds back into the compaction engine alongside human telemetry — the lake self-optimizes for the combined workload. If agents start heavily filtering on a column that humans rarely use, the sort order adapts to serve both access patterns. The lake becomes agent-ready not by adding a separate AI layer, but by extending the same optimization loop to a new class of consumers.

Learning: The Closed Loop

The final stage is what transforms a management tool into a system that improves with time and scale. Every maintenance operation produces measurable outcomes: file count reduction, scan volume improvement, query latency change, storage reclaimed, compute saved. These outcomes are not just metrics to display — they feed back into future decisions.

Sort compaction that improved dashboard latency by 8x on one table informs strategy selection for structurally similar tables. Routing decisions that reduced costs without impacting latency reinforce the model for similar query shapes. Compaction schedules that were too aggressive for a table's write rate — causing frequent OCC conflicts — get automatically dialed back. Target file sizes that produced optimal performance on one storage class get recommended for similar tables.

This is why a control plane gets more effective as the lake grows. More tables under management means more signal about which strategies work for which table shapes. More engines means richer telemetry about cross-engine access patterns. More queries means more precise sort-order recommendations. The operational burden stays flat (or decreases) while the optimization quality increases. The system scales better than a team can — because it learns from the entire lake, not just the tables one engineer happened to look at.

Choosing Your Path

Three approaches to managed data lakes exist in 2027. Each suits different architectural choices and organizational realities. The right one depends on your engine diversity, cloud strategy, and team structure.

Vendor-Managed Platforms

Databricks (Predictive Optimization, Liquid Clustering) and Snowflake (automatic clustering, table maintenance) bundle operational management inside their platforms. If you run exclusively on one vendor's stack — all compute through their engine, all tables in their catalog — this is the lowest-friction path. Maintenance is handled within the ecosystem, tuned for their engine's access patterns, and requires no additional tooling.

The trade-off is architectural scope. Optimization runs only within that vendor's engine. If you later want to query those tables from Trino, DuckDB, or a different vendor — the core value proposition of choosing Iceberg in the first place — maintenance optimization will not account for those external access patterns. The vendor optimizes for its own queries. Cross-engine workloads are invisible to it.

Cloud-Native Services

AWS S3 Tables and Google BigLake provide managed Iceberg maintenance within their respective clouds. Compaction, cleanup, and optimization run automatically for tables registered with that service. No additional tooling needed for those tables.

The trade-off is scope. Maintenance applies to tables within that cloud service only. If your lake spans multiple catalogs, clouds, or includes self-managed Iceberg tables outside S3 Tables or BigLake, you need a separate solution for the rest. Cross-engine query-aware optimization is not typically available because the service lacks visibility into query workloads from external engines — it knows what Athena does but not what Trino or Spark are doing on the same tables.

Dedicated Control Plane

A platform like LakeOps operates across your entire stack — any catalog, any engine, any cloud — because it layers onto existing infrastructure rather than replacing it. It provides the full closed loop (observe → decide → act → optimize → govern → route → enable → learn) without constraining architectural choices.

The trade-off is that you add a service to your stack rather than relying on built-in capabilities. The benefit is that optimization accounts for your actual cross-engine workload, management applies lake-wide regardless of where tables live, policies are consistent across catalogs and clouds, and you retain full engine flexibility — add or remove engines without changing your management layer.

When to Choose Each

The decision framework is practical:

If you run 5–20 tables on a single engine from a single vendor, use that vendor's built-in management. The tooling is sufficient, it is already there, and the additional complexity of a separate service is not justified.

If you run 50+ tables across multiple engines, multiple catalogs, or multiple clouds, you need a layer that spans your stack. No single vendor or cloud service has visibility into your complete workload. Cross-engine telemetry, unified policies, and lake-wide governance require a purpose-built operations layer.

Many organizations run hybrid approaches — vendor-managed tables for workloads that live entirely within one ecosystem, and a dedicated control plane for the open multi-engine portion. The approaches coexist because they solve different scoping problems.

The Economics

Four cost vectors compound in an unmanaged lake. Left unchecked, each grows independently and they reinforce each other. They are also the four that a properly managed lake eliminates:

Storage drops 40–60% as orphan files, stale snapshots, and unreferenced data are continuously cleaned. This is not theoretical — it is the immediate, measurable result of running a proper orphan sweep and snapshot expiration. Production deployments routinely discover that a significant portion of their billable storage is serving no analytical purpose. The savings are recurring: every month you do not clean, the waste accumulates further.

Query compute drops as sort-aware compaction aligns data to real access patterns. When engines skip 80% of row groups via tight min/max statistics instead of scanning everything, compute consumption falls proportionally. Queries that scanned 100 GB to return 100 rows now scan 5 GB. The per-query savings compound across every query, every user, every dashboard refresh, every scheduled report — thousands of times per day across the organization.

Maintenance compute drops when a purpose-built Rust engine replaces Spark clusters for compaction workloads. No JVM startup overhead, no garbage collection pauses, no idle executor cores between compaction runs, no over-provisioning for peak load. Event-driven execution means compute runs only when needed (health signal fires) rather than on a fixed schedule (cluster running overnight regardless of whether any table needs attention).

Engineering time redirects from maintenance firefighting to building data products. Policies replace scripts. Observability replaces guesswork. Health classification replaces manual triage. On-call incidents for table degradation approach zero because degradation is prevented rather than repaired. The platform team stops being reactive (fixing things that broke) and becomes proactive (building capabilities that generate value).

The combined effect for teams running at production scale: the lake becomes cheaper to operate as it grows larger. More data, more tables, more engines — but costs grow sublinearly because the automation handles the marginal complexity without marginal engineering investment. This is the opposite of what happens without management, where cost and complexity grow linearly (or worse) with lake size.

Getting Started

The practical path, regardless of where you sit on the maturity spectrum:

Start with visibility. Connect your catalogs and see the health of every table before any maintenance runs. You will likely be surprised by how many tables are degraded — most teams discover Critical and Warning states on tables they thought were fine. The first insight alone (which tables are Critical, what orphan volume exists, where manifest bloat lives, which sort orders have drifted) changes how you prioritize the next sprint. This step costs nothing and requires no maintenance operations — it is purely diagnostic.

Fix the highest-impact tables first. You do not need to manage everything on day one. Identify the tables with the most file fragmentation, the highest query volume, or the largest orphan accumulation — ideally, tables that score high on all three. Run sequenced maintenance on those: expire, clean, compact, rewrite manifests. The ROI is immediate: dashboards that were slow become fast, storage bills drop measurably, and the team sees what managed looks like on real tables.

Establish policies early. Define your retention windows, target file sizes, compaction strategies, and cleanup schedules — even if you start with manual or semi-automatic execution. These decisions become the governance layer when you move to full automation. A clear, documented policy also prevents the entropy that accumulates when ten engineers configure tables differently based on individual judgment. Consistency across the lake matters more than perfection on any single table.

Expand to full automation. Once you trust the signals and see the impact on your initial tables, enable adaptive maintenance across the lake. Every table gets the right maintenance at the right frequency — coordinated, sequenced, conflict-aware, and fully observable. At this point, your platform team stops maintaining DAGs and scripts and starts building the data products and self-service capabilities that generate value for the organization.

Add routing and AI access last. Multi-engine routing and agent access depend on consistently healthy tables and predictable data layouts. They are the payoff of the operational foundation — not a starting point. A routing layer cannot optimize what it cannot predict. An AI agent cannot reliably query tables whose performance is unpredictable. Get the foundation right first; the advanced capabilities become straightforward once table health is reliable.

The constraint throughout: your data never leaves your storage. Your engines remain yours to choose and change. Your catalogs remain yours. The control plane adds operational intelligence without replacing anything in your stack — and if you ever remove it, everything continues working exactly as before, just without the automation.

Conclusion

Every production data lake faces the same operational reality: without an active system keeping tables healthy, Iceberg's open architecture degrades into slow queries, rising costs, and engineering time spent on maintenance instead of product work.

The problem is not the format. Iceberg provides exactly what it promises — reliable, open, engine-agnostic tables. The operational layer that continuously compacts, cleans, sorts, monitors, governs, and optimizes must exist separately. It must be built or adopted.

Netflix and Google proved the architecture works by building it internally over years with dedicated teams. In 2027, the same architecture is available without that investment. The approaches range from vendor-managed (Databricks, Snowflake) to cloud-native (S3 Tables, BigLake) to dedicated control planes — each fitting a different point on the architectural complexity spectrum.

The question is not whether your lake needs management — the degradation mechanics are inherent to how Iceberg works at scale. The question is which path fits your architecture, engine diversity, and team structure.

For teams running open, multi-engine Iceberg lakehouses at production scale, LakeOps is the control plane that closes the operational loop — across any engine, catalog, and cloud. Connect your existing infrastructure, define your policies, and let the lake manage itself.

Further learning:

Data Lakehouse: How and Why

joni sar — Wed, 19 Aug 2026 14:09:23 +0000

How the data lakehouse works in production — the internal mechanics, the data flow patterns, why tables degrade under real workloads, and the closed-loop control plane that keeps the architecture performing at scale.

What a data lakehouse solves

Every enterprise data platform eventually hits the same wall. Analytical data exists in two places: a data lake (cheap, scalable, ungoverned) and a data warehouse (expensive, fast, governed). Between them sits an ETL pipeline that copies, transforms, and reconciles — creating a permanent tax on engineering time, data freshness, and infrastructure spend.

The symptoms are familiar. Dashboards show yesterday's numbers because the warehouse loads overnight. ML models train on stale copies because the warehouse does not support Python-native access. Storage bills grow because the same curated datasets live in both systems simultaneously. Governance fragments because access controls, lineage, and audit trails cannot span two architecturally different platforms.

The data lakehouse resolves this by collapsing both systems into one: transactional guarantees applied directly to data on object storage. One copy. One governance model. Any engine — SQL, ML, streaming, AI — reads and writes the same tables with full ACID safety. The warehouse is not replaced by a different product; the lake becomes the warehouse through a metadata innovation called a table format.

The practical result: BI dashboards, Spark ML pipelines, Flink streaming jobs, and autonomous AI agents all query the same governed tables — without ETL between them, without stale copies, without paying warehouse markup on storage that costs $0.023/GB/month in its native form.

The internal mechanics: how a lakehouse actually works

A data lakehouse is not a monolithic system. It is four cooperating layers — each handling a specific responsibility, each independently evolvable. Understanding how they interact is the foundation for building one that performs in production.

The storage layer

All data resides on cloud object storage — S3, GCS, or ADLS. This is where the economics originate. Object storage provides eleven-nines durability at $20–25 per TB per year. A petabyte of analytical data costs roughly $20,000/year to persist — compared to $500,000+ when that same data sits inside a traditional warehouse with bundled compute.

Storage is decoupled from everything above it. Shut down every query engine and the data remains. Add a new engine tomorrow — it reads from the same bucket. Move clouds — the files copy byte-for-byte with no format conversion.

Data lands as Apache Parquet files — a columnar format that stores values by column rather than by row. A query touching 5 columns from a 300-column table physically reads only those 5 columns. Parquet also embeds per-column min/max statistics in each file header, enabling query engines to skip entire files whose value ranges cannot satisfy the WHERE clause. This skip-scanning is what makes lakehouse queries competitive with warehouse queries — but only when files are properly sized and organized.

The table format layer

A directory of Parquet files on S3 is a data lake — readable, but not transactional. The table format is the metadata innovation that adds the missing guarantees:

Apache Iceberg — the dominant format as of 2026, supported natively by Spark, Trino, Flink, DuckDB, Snowflake, Databricks, Athena, and StarRocks — provides ACID commits, schema evolution, time travel, partition evolution, and concurrent multi-writer safety through a metadata tree architecture.

How the metadata tree works:

Each Iceberg table maintains a hierarchy of metadata files on the same object storage as the data:

A metadata pointer (tracked by the catalog) — points to the current table version
Snapshot files — each representing the table's complete state at a point in time. Every write creates a new snapshot.
Manifest lists — each snapshot references a manifest list that indexes the active manifests
Manifests — each tracking a batch of data files with per-file statistics (path, partition values, row count, column min/max)
Data files — the actual Parquet files containing rows

This tree enables two properties that make the lakehouse viable:

Atomic commits. Every write produces new data files and a new snapshot. The catalog atomically swaps the pointer from old snapshot to new. Readers always see a consistent state. Failed writes leave no trace — the pointer was never advanced.

Statistical pruning. Query engines read manifests (kilobytes) to determine which data files (gigabytes) are relevant. A query filtering on country = 'DE' checks manifest-level statistics and skips every file whose country column's min/max range excludes 'DE'. No data read. No I/O wasted. This is why lakehouse queries can match warehouse speed — but only when the metadata is current and the data layout aligns with query patterns.

The critical implication: every commit grows the tree. Every streaming write adds files. Every mutation adds delete markers. The tree that enables fast queries also accumulates entropy that degrades query speed over time — unless something actively maintains it.

The catalog layer

The catalog is the coordination service that answers: where is the current metadata pointer for each table, and how do multiple engines safely coordinate writes?

Every engine — Spark, Trino, Flink, DuckDB — consults the catalog before each read (to locate current metadata) and during each write (to commit atomically). Without the catalog, concurrent writers could corrupt table state. With it, they race safely: one commits, the other retries against updated state.

Modern lakehouses use the Iceberg REST Catalog specification — a standard HTTP API that every engine implements. This means adding a new engine is a configuration change (point it at the catalog URL), not an integration project. Catalog options include managed services (AWS Glue), open-source servers (Apache Polaris, Lakekeeper), federated solutions (Apache Gravitino), and Git-branching models (Project Nessie) — all speaking the same REST protocol.

The catalog is also where access control, retention policies, and audit logging materialize. It is the governance anchor for multi-engine environments where every engine must see consistent permissions.

The compute layer

The lakehouse separates compute from storage, enabling multiple specialized engines to operate on the same data concurrently:

Spark — batch ETL, ML training, complex multi-stage transformations. The workhorse for heavy writes and joins.
Trino — interactive SQL with sub-second response times. Excels at dashboard queries and ad-hoc exploration.
Flink — streaming ingestion with exactly-once semantics. Continuous CDC from operational databases into lakehouse tables.
DuckDB — embedded analytics for notebooks, CI/CD pipelines, and single-node processing. Zero infrastructure overhead.
Snowflake / Athena / StarRocks — managed engines for specific workload profiles (high-concurrency BI, serverless ad-hoc, real-time OLAP).

Each engine discovers tables through the same REST catalog and reads/writes through Iceberg's transactional protocol. They coexist safely on the same tables through snapshot isolation. This is the key architectural advantage over the warehouse model: instead of one vendor's engine for everything, you use the best tool for each workload shape.

But multi-engine access multiplies operational pressure. More engines means more write patterns, more file fragmentation profiles, and more divergent query patterns competing for the same table's physical layout — creating the operational challenge that defines production lakehouse management.

How data flows through a production lakehouse

Theory aside — how does data actually move from source systems through the lakehouse to consumers? The medallion architecture (bronze → silver → gold) is the standard pattern for organizing this flow, and each layer creates different operational characteristics that matter for long-term health.

Bronze: the raw capture layer

Source data lands in its original form. CDC events from PostgreSQL (captured by Debezium, written by Flink). Clickstream events from Kafka. Daily batch exports from SaaS APIs. Partner file drops. Everything arrives as append-only, schema-on-read, immutable records.

Bronze is the replay source. When silver-layer transformation logic has a bug discovered months later, you rebuild from bronze. When a new use case requires fields that were previously ignored, they are already preserved in bronze. This immutability is non-negotiable — teams that apply transformations at ingestion lose the ability to retrospectively correct logic errors.

Common bronze ingestion patterns:

Streaming CDC — Debezium captures row-level changes (inserts, updates, deletes) from PostgreSQL, MySQL, or MongoDB. Flink consumes these events and writes them to Iceberg tables with exactly-once semantics at sub-minute commit intervals. Each commit is small (a few thousand rows) but creates a new data file and a new snapshot.
Event streams — Application events (clickstreams, transactions, IoT sensor data, ad impressions) flow from Kafka topics into Flink or Spark Structured Streaming jobs that write to Iceberg. Event volumes are high and bursty — Black Friday traffic spikes create 10x the normal file accumulation rate.
Batch loads — Daily or hourly exports from ERP systems, CRM platforms, and third-party APIs. These produce larger files but less frequently. Batch tables fragment slower but still accumulate snapshot overhead.
File drops — Partners or internal systems deposit CSV/JSON/Parquet files in S3. A detection pipeline validates schema, converts to Iceberg-managed Parquet, and registers the files in the catalog.

What this means operationally: Streaming CDC with 5-minute commits creates approximately 8,600 new files per table per month. A lakehouse ingesting from 20 operational databases at sub-minute latency accumulates hundreds of thousands of files across its bronze layer within weeks. Individual file sizes average 5–20 MB — far below the 256–512 MB target where engines perform optimally. File count — not data volume — is what degrades query planning speed.

Silver: the governed truth layer

Silver transforms bronze into a data model that the organization commits to: one row per entity, resolved foreign keys, deduplicated events, enforced types, quarantined bad records. Silver is not a cleanup step — it is an architectural commitment to a domain model that all downstream consumers depend on. If a table exists because one dashboard needs a specific aggregation, it belongs in gold, not silver.

Key transformations:

Deduplication — Late-arriving records, duplicate events from at-least-once delivery, and replay scenarios resolved to a single authoritative version. Deduplication runs as MERGE INTO with match conditions, producing delete markers for superseded records.
Type enforcement and validation — Strings cast to proper types. Nulls validated against business rules. Referential integrity checked against dimension tables. Records failing validation quarantined to error tables with reason codes — never silently dropped, never propagated downstream.
Slowly changing dimensions (SCD Type 2) — History tracked for entities that evolve over time. When a customer's tier changes, the old row is closed (end_date set) and a new row opened. This produces update operations on every change event.
Reference data enrichment — Raw IDs joined to meaningful attributes. Country codes resolved to names. Product SKUs expanded to categories, brands, and pricing tiers. Enrichment creates wider rows with higher value per query.
Temporal alignment — Events from different sources (orders, payments, shipments) aligned to consistent time zones, granularity, and business calendars.

Silver tables are the enterprise source of truth. Every downstream consumer — dashboards, ML models, AI agents, compliance reports — starts from the same clean, governed layer.

What this means operationally: Silver tables receive MERGE INTO operations — upserts that create delete markers (position-delete files or V3 deletion vectors) that every subsequent read must reconcile. A CDC table receiving 50,000 updates per hour generates thousands of delete markers per day. Query engines must read both the data files and the delete files, reconcile which rows are current, and filter accordingly — adding I/O overhead that increases linearly with delete-file count. Without periodic resolution of these markers, query latency creeps 2–5x higher while the table reports the same row count and appears unchanged to basic monitoring.

Gold: the consumer-optimized layer

Gold tables exist for performance: pre-aggregated metrics for dashboards, denormalized feature tables for ML training, materialized snapshots for compliance reporting. Gold trades storage redundancy for read speed — accepting wider tables and duplicated columns to eliminate joins at query time.

Gold layer patterns:

BI aggregations — Daily/weekly/monthly rollups pre-joined by all relevant dimensions. A dashboard that previously ran a 30-second join across 5 silver tables now reads a single pre-materialized gold table in under 1 second.
ML feature tables — Wide, denormalized tables with one row per entity and all features materialized. Training jobs read a single scan rather than orchestrating multi-table joins. Feature freshness controlled by the gold refresh cadence.
Compliance snapshots — Point-in-time materializations capturing exactly what was known at a specific date. Regulatory filings, board reports, and audit evidence stored as immutable gold snapshots with indefinite retention.
Serving tables — Pre-computed results for user-facing analytics (product recommendations, customer health scores, real-time dashboards) with sub-second SLA requirements.

What this means operationally: Gold tables are typically overwritten on a schedule — daily, hourly, or triggered by upstream freshness signals. Each OVERWRITE operation atomically replaces the table's content: it writes new data files, creates a new snapshot pointing to them, but leaves the previous snapshot's files in storage until explicitly expired. A daily gold refresh creates 365 snapshots per year. At 50 GB per refresh, that is 18 TB of stale data files persisting in storage — even though only the latest 50 GB is logically current. Without snapshot lifecycle management, storage grows linearly forever from data that was replaced weeks or months ago.

The maintenance gradient

The three layers create a gradient of operational need:

Bronze → small-file pressure (compaction priority)
Silver → delete-file pressure (merge resolution priority)
Gold → stale-snapshot pressure (expiration priority)

No uniform maintenance schedule handles this correctly. Each table needs the right operation at the right cadence — determined by its ingestion velocity, mutation pattern, and query load. This is the insight that leads directly to the control plane model.

What the lakehouse unlocks: multiple workloads, one platform

The economic case (10–50x cheaper storage) is straightforward. The architectural case is more powerful: eliminating the boundaries between workload types that the two-system model enforced.

BI and reporting

Dashboards query silver and gold tables directly through Trino or Snowflake. No warehouse sync delay. No overnight ETL window creating stale numbers. Schema changes in silver propagate to consumers without separate migration scripts — because the table format handles schema evolution as a metadata-only operation.

Time travel lets analysts reproduce the exact state of any report at any prior date. In regulated industries (financial services, healthcare, insurance), "what did we report last quarter" must be answerable precisely and auditably. The lakehouse provides this natively through snapshot pinning — no separate archival infrastructure required.

Performance depends on table health: well-compacted tables with accurate column statistics and sort-order alignment with dashboard filter patterns enable engines to prune 90%+ of files on selective queries. When a BI dashboard filters on region = 'EMEA' AND quarter = 'Q2', a properly maintained table skips every file that cannot contain matching rows. A degraded table forces a full scan regardless of how selective the filter is. This is why maintenance directly impacts end-user experience — and why it cannot be deferred.

ML and feature engineering

ML pipelines read training data from the same governed tables that power dashboards — eliminating training-serving skew, the most common source of ML model degradation in production. When the training pipeline and the serving pipeline read from different copies (one from the warehouse, one from the lake), subtle differences in transformation logic cause the model to encounter data distributions in production that it never saw during training. The lakehouse eliminates this by providing one table for both.

Experiment reproducibility comes free: pin to a snapshot ID, retrain on byte-for-byte identical data months later. Feature freshness improves because features and serving data live in the same transactional system — no nightly export, no stale copies, no drift between what the model trained on and what it serves against.

Feature stores built on lakehouse tables inherit versioning, access control, lineage, and time-travel capabilities from the table format. Feature computation runs as standard Spark or Flink jobs writing to Iceberg tables — no separate feature-store infrastructure, no proprietary feature format.

Streaming analytics

Flink and Spark Structured Streaming write directly to lakehouse tables with exactly-once transactional guarantees. Near-real-time dashboards and alerting read from the same tables — no separate streaming infrastructure, no Kafka-to-materialized-view pipeline, no divergent dual-path architecture.

The same transformation logic runs as streaming (for freshness) or batch (for cost) by changing only the trigger configuration. This eliminates the lambda architecture's core problem: maintaining two parallel pipelines (one batch, one stream) with identical logic that inevitably drift apart. In the lakehouse model, there is one pipeline with a tunable latency/cost knob.

AI agents as first-class consumers

In 2026, AI agents autonomously discover schemas, formulate analytical queries, and feed results into multi-step reasoning chains. The lakehouse is their knowledge infrastructure — providing structured, governed, queryable data that agents access through standard protocols.

The requirements for agent access differ from human access: agents generate unpredictable query patterns, cannot diagnose slow responses caused by table degradation, and burn tokens (and money) on retries when they hit poorly-maintained tables. They also need guardrails — limits on what they can read, how much they can scan, and what they can modify.

Why lakehouses degrade: the operational reality

Here is what most guides skip. The four layers give you the architecture. They do not give you a system that stays healthy. Every production lakehouse — without exception — degrades over time unless something actively maintains it. This is not a flaw; it is a consequence of how append-only transactional systems work.

The mechanics of degradation

Iceberg never modifies files in place. Every write creates new files. Every snapshot preserves a complete view. This is what provides transactional safety — but it means the system accumulates structural debt continuously:

File fragmentation. Each commit adds files. Streaming ingestion at 5-minute intervals into a table receiving 2 GB/hour creates ~8,600 files per month. Each file averages 12 MB — well below the 256–512 MB target where query engines perform optimally. The engine must open, plan, and coordinate reads across 100x more files than necessary.

Planning overhead. Engine query planners traverse the manifest tree to identify relevant files. At 1,000 files, planning takes milliseconds. At 50,000 files, planning takes 10–20 seconds — sometimes exceeding the query execution itself. The metadata hierarchy that enables fast pruning becomes a bottleneck when it indexes too many undersized files.

Delete-file accumulation. Every UPDATE and DELETE writes markers that subsequent readers must reconcile against data files at read time. A heavily-mutated table (CDC, SCD, late-arriving corrections) accumulates thousands of delete markers. Query latency increases 2–5x while the table's row count and size appear unchanged — making degradation invisible to basic monitoring.

Storage waste. Expired logical content leaves physical files behind. Orphaned files from failed writes, aborted compaction, and expired snapshots accumulate in storage. Without active cleanup, production lakehouses accumulate terabytes of unreferenced files — pure cost with no analytical value.

Compounding pressure. These four forces are not independent — they compound. More files means larger manifests. Larger manifests means slower planning. Slower planning means longer maintenance operations. Longer maintenance operations means less frequent runs. Less frequent runs means more file accumulation. The system has a natural tendency toward runaway degradation that accelerates once it begins.

Why manual maintenance does not scale

The instinct is to write scripts: a Spark job that compacts every table nightly, a cron that expires old snapshots weekly. This works for 5–10 tables. It fails at production scale for structural reasons:

Tables need different cadences. A streaming table needs compaction hourly. A daily-refresh table needs it weekly. A slowly-growing dimension needs it monthly. One schedule cannot serve all three without either wasting compute on healthy tables or neglecting degraded ones.

Operations have dependencies. Compacting files that will be expired next hour is wasted work. Rewriting manifests before compaction invalidates the rewrite. Computing statistics on pre-compaction layouts produces stale metadata. The five maintenance operations must run in a specific sequence — and re-sequence when conditions change.

Maintenance competes with production. Running compaction on the same Spark cluster that serves production queries creates resource contention. Running it concurrently with active writers on the same table causes commit conflicts. Scheduling around production windows requires coordination that scripts cannot perform.

Static thresholds drift. "Compact when file count exceeds 1,000" is reasonable for one table's current state. But partition cardinality changes, ingestion velocity shifts, query patterns evolve. The threshold that was correct last month is wrong today — and nobody adjusts it until the table is already degraded.

The control plane: a closed-loop system

The answer is not better scripts or more sophisticated cron. It is an architecturally different approach: a control plane that operates the lakehouse as a closed-loop system — continuously sensing table state, reasoning about what each table needs, executing the right operations in the right sequence, and learning from outcomes to improve future decisions.

    ┌──────────────────────────────────────────────────────┐
    │                                                      │
    │   SENSE  →  ASSESS  →  PLAN  →  EXECUTE  →  LEARN   │
    │     ↑                                         │      │
    │     └─────────────────────────────────────────┘      │
    │                (continuous per table)                 │
    │                                                      │
    └──────────────────────────────────────────────────────┘

This is the mental model for LakeOps — an autonomous control plane for data lakehouses that implements this closed loop across every table in your lake, continuously, without cron jobs or manual intervention.

Sense: collecting structural telemetry

The control plane continuously reads metadata from every connected catalog — file counts per partition, file size distributions, manifest depth, snapshot velocity, delete-file ratios, partition-level skew, and cross-engine query patterns. This is metadata inspection, not data access — the control plane never reads or moves your actual data.

The sensing is passive and lightweight. It adds no load to production engines. It works across catalogs (Polaris, Glue, Gravitino, Nessie, Lakekeeper) and across engines (Spark, Trino, Flink, DuckDB, Snowflake, Athena) — aggregating a unified view that no single component can provide alone.

Assess: health classification

Raw telemetry becomes actionable through health scoring. Every table is continuously classified:

Healthy — file sizes within target, manifests compact, delete ratios low, sort order aligned with query patterns. No action needed.
Warning — metrics drifting toward thresholds. File count growing. Delete ratio increasing. Not yet impacting queries — but will if left unaddressed.
Critical — active degradation impacting performance. Planning latency elevated. Read amplification high. Immediate maintenance required.

The thresholds are not static defaults. They account for each table's partition cardinality, ingestion velocity, engine mix, and historical maintenance response — adapting automatically as workloads evolve.

LakeOps observability surfaces this classification as a lake-wide dashboard: how many tables are healthy, warning, critical — with drill-down into specific metrics, proactive insights at severity levels, and cross-engine telemetry that reveals optimization opportunities no single engine can see.

Plan: sequenced maintenance decisions

For each table classified as Warning or Critical, the control plane determines which operations to run and in what order.

The sequencing is not arbitrary — it follows a dependency chain where each step's output becomes the next step's input:

Step 1 → Expire snapshots. Remove snapshots beyond the retention policy. This dereferences files that are no longer needed — reducing the scope for subsequent operations and preventing compaction from processing files about to be garbage-collected.

Step 2 → Remove orphan files. Delete unreferenced files older than the safety window (default: 7 days, configurable). This captures files newly dereferenced by expiration, plus accumulated waste from failed writes and aborted operations.

Step 3 → Compact data files. Merge small files into optimally-sized ones. Resolve delete markers so subsequent reads skip reconciliation overhead. Optionally sort data by query-relevant columns to enable statistical pruning.

Step 4 → Rewrite manifests. Consolidate the manifest tree to reflect the new compacted layout. After compaction reduces file count from 50,000 to 2,000, the manifest tree should reflect 2,000 files — not carry forward metadata from the old layout.

Step 5 → Compute statistics. Refresh Puffin column statistics (distinct values, null counts, extended min/max) on the compacted files. This ensures every engine benefits from accurate metadata for pruning decisions.

LakeOps managed maintenance runs this five-step pipeline as a coordinated sequence per table — respecting dependencies, avoiding conflicts with active writers, and targeting only the partitions that need work.

Execute: fast, conflict-aware action

The control plane executes maintenance on a purpose-built engine rather than borrowing production compute:

LakeOps compaction runs on a dedicated Rust binary powered by Apache DataFusion — not Spark. The practical differences:

Speed: A 1 TB TPC-DS compaction benchmark completes in under 4 minutes — roughly 29x faster than equivalent Spark operations on the same data. This speed means maintenance runs in minutes rather than hours, so tables never accumulate enough debt to degrade between cycles.
Cost: Sub-$5 per TB compacted — roughly 90% cheaper than running equivalent Spark clusters. This economic efficiency is what makes continuous compaction viable rather than forcing teams into overnight-only windows.
Reliability: Bounded memory with graceful spill-to-disk. No JVM garbage collection pauses. No OOM crashes on large tables. No cluster provisioning delay.
Conflict safety: Aware of active writers and in-flight transactions. Never compacts partitions with concurrent writes. Never expires snapshots that active readers depend on.

Intelligent sort: learned from production queries

Beyond file consolidation, LakeOps compaction physically sorts data by the columns that production queries actually filter on — learned automatically from cross-engine telemetry.

This is the highest-leverage optimization in the entire stack. When data within each file spans a narrow value range for the sort columns, statistical pruning becomes surgical: the engine checks file-level min/max metadata and eliminates 90%+ of files before reading any data. The difference between an unsorted table (full scan on every query) and a properly sorted table (targeted reads on 5–10% of files) is typically 8–12x in query speed.

The key insight is that the right sort order is not static. Trino queries might filter on customer_id and order_date. Spark ML pipelines might filter on event_type and region. AI agents might filter on combinations nobody predicted. No single engine has the full picture of how the table is accessed. The control plane aggregates telemetry from all engines and computes the optimal sort strategy for the combined workload — updating it automatically as patterns shift over time.

Learn: outcome-driven improvement

After each maintenance cycle, the control plane measures outcomes against expectations. Did file count reach the target? Did planning latency improve? Did the health score advance? Results feed back into the assessment model:

Tables that responded well to sort compaction on specific columns get those columns reinforced in future cycles
Tables where expiration freed significant storage get more aggressive retention recommendations
Tables where manifest rewriting dramatically improved planning get lower rewrite thresholds

This learning loop means the control plane improves over time — each cycle's outcomes inform the next cycle's decisions. Tables with unusual characteristics (very high partition cardinality, extremely bursty ingestion, mixed engine access) get customized strategies without manual configuration.

Governance: policies that run themselves

Visibility without enforcement is just monitoring. The control plane turns observability into automated governance through declarative policies:

LakeOps governance supports policies at multiple scopes:

Organization-wide defaults — baseline compaction targets, snapshot retention, orphan cleanup schedules applied to every table unless overridden
Namespace-level rules — production namespaces get aggressive maintenance; staging gets relaxed thresholds
Per-table exceptions — compliance tables with 365-day retention; hot tables with 15-minute compaction cadence

Policies inherit downward: new tables automatically receive the governance rules of their namespace. No manual configuration per table. No tables falling through the cracks because someone forgot to add them to the maintenance script.

Every policy execution is logged with full audit trail — what ran, when, what changed, duration, bytes before/after. This is critical for compliance environments where "prove your data lifecycle management" is a regulatory requirement.

Query routing: making multi-engine economically rational

Having multiple engines on the same data is architecturally elegant. But without routing intelligence, it becomes accidentally expensive: queries dispatched to the wrong engine pay the wrong pricing model and get the wrong performance profile.

A point lookup for 100 rows dispatched to Spark burns 30 seconds of cluster startup. That same query on DuckDB resolves in 0.3 seconds. A dashboard query running on Snowflake at $2/credit is 10x more expensive than the same query on a self-hosted Trino cluster. A full-table scan on DuckDB crashes; on Spark, it completes in minutes.

LakeOps query routing provides a single SQL endpoint that dispatches queries to the optimal engine based on:

Workload type — define routing groups (BI/Reporting → cost-optimized, Data Science → performance-optimized, ETL → throughput-optimized) with different backend engine pools
Table health state — well-compacted tables are eligible for lightweight engines (DuckDB, Athena). Fragmented tables route to engines that tolerate planning overhead (Spark).
Cost/latency targets — constraints per group that the router respects (under $0.05/query for BI, under 2s latency for dashboards)

The routing layer and the maintenance layer form a reinforcing loop: as the control plane compacts and sorts tables, more engines become eligible per query shape. More routing options mean lower per-query cost. Lower cost justifies more frequent querying. More frequent querying generates better telemetry for compaction decisions.

AI agent enablement: the lakehouse as knowledge infrastructure

In 2026, AI agents are not just consumers of pre-built reports — they autonomously explore data, formulate queries, and feed results into reasoning chains. The lakehouse becomes the knowledge layer that agents operate on.

This creates specific requirements that a control plane addresses:

Access interface. LakeOps provides an MCP-native interface (Model Context Protocol) with PostgreSQL, MySQL, and Arrow Flight wire compatibility. Any MCP-compatible agent discovers catalogs, browses schemas, executes queries, and receives results — without custom integration per agent framework.

Layered guardrails. Agents need boundaries. The control plane enforces per-session constraints:

ReadOnly — blocks DDL and DML; agents explore but cannot modify
CostEstimate — rejects queries whose estimated scan exceeds configurable thresholds
PIIMask — hashes sensitive columns before results reach the model
HumanApproval — pauses high-stakes operations for human review before execution

Closed-loop optimization. Agent query telemetry feeds back into the control plane's sort and compaction decisions. Tables that agents query heavily get optimized for agent access patterns. The lake adapts to AI workloads without manual intervention — queries get faster as agents use the system more.

Implementing: the practical sequence

You do not need the full architecture on day one. Each step delivers independent value and sets up the next:

1. Start with Iceberg. Choose Apache Iceberg for all new analytical tables. The ecosystem support is universal — every major engine reads and writes it natively. Existing Delta Lake tables interoperate through UniForm.

2. Deploy a REST catalog. This is the coordination point — get it right early. AWS Glue for managed simplicity. Self-hosted Polaris for full control. Gravitino if you need to federate across existing catalogs.

3. Build the ingestion layer (bronze). CDC from operational databases via Flink. Event streams from application telemetry. Batch loads from external sources. Keep it append-only and immutable.

4. Build the semantic layer (silver). Domain modeling, deduplication, quality enforcement. This becomes the organization's governed truth — every downstream consumer starts here.

5. Connect the control plane. Once data flows and tables accumulate, connect LakeOps to your catalog. Ten minutes, no data movement, no infrastructure changes. Instant visibility into every table's health state. Start in manual-approval mode — review what the control plane recommends before enabling autonomous execution. Then enable autopilot and let the closed loop run.

6. Add specialized engines. As workloads diversify, add Trino for interactive SQL, DuckDB for notebooks, Snowflake for governed BI access. The REST catalog makes each addition a configuration step.

7. Build consumer-optimized tables (gold). Pre-aggregations, feature tables, reporting snapshots. Optimize for specific access patterns once you understand what consumers need.

8. Enable routing. With 3+ engines, define routing groups. Let the control plane dispatch each query to the optimal backend automatically.

9. Open AI access. Expose tables to agents via MCP. Configure guardrails per agent type. Agent telemetry feeds back into optimization priorities.

Comparing the options: lake, warehouse, lakehouse

Characteristic	Data lake	Data warehouse	Data lakehouse
Storage model	Open files on object storage	Proprietary format, vendor storage	Open files on object storage
Transactions	None	Full ACID	Full ACID (table format)
Governance	Manual, fragmented	Vendor-managed, centralized	Catalog-unified, multi-engine
Query speed	Depends entirely on maintenance	Vendor-optimized, consistent	Matches warehouse when maintained
Engine flexibility	Any tool reads files	Vendor's engine only	Multiple engines, REST catalog
ML/AI workloads	Native but ungoverned	Export required	Native and governed
Streaming	Separate infrastructure	Separate infrastructure	Same tables, same transactions
Cost at petabyte	~$20K/yr storage	~$500K+/yr storage+compute	~$20K/yr storage + operations
Maintenance burden	None (no transactions)	Zero (vendor manages)	Requires control plane
Vendor lock-in	Low	High	Low (open formats + catalog)

The lakehouse occupies a specific position: warehouse guarantees at lake economics, with the trade-off that operational maintenance is your responsibility rather than a vendor's. The control plane is what makes this trade-off viable at scale — providing the vendor-grade operational ease without the vendor lock-in.

The economic model

The lakehouse's cost advantage operates at four levels that compound:

Level 1: Storage decoupling. Object storage at $20–25/TB/year vs. warehouse storage at $500–2,000/TB/year. For 200 TB of curated data, that is $4,000–5,000/year vs. $100,000–400,000/year. At petabyte scale, the savings fund entire platform teams. But storage cost is only the starting point — the larger savings come from operational efficiency.

Level 2: Elimination of duplication. The lake-to-warehouse ETL that copies, transforms, and reconciles data disappears. This eliminates: the compute cost of the ETL pipeline itself, the engineering hours maintaining it, the storage cost of the warehouse copy, and the reconciliation overhead when copies drift. For a 200 TB estate, duplication elimination saves $100K–400K/year in direct warehouse storage costs plus 1–3 FTEs worth of engineering time maintaining the pipeline.

Level 3: Multi-engine routing efficiency. When every query runs on the cheapest engine that meets its latency requirement, aggregate compute cost drops dramatically. A DuckDB point lookup costs fractions of a cent. A self-hosted Trino dashboard query costs pennies. A Spark batch join costs proportional to actual data processed. Without routing, organizations default to Snowflake pricing for everything — $2+ per credit regardless of query complexity. With routing, simple queries route to lightweight engines and only expensive workloads pay for expensive compute.

Level 4: Maintenance economics. The control plane's Rust-based engine compacts at sub-$5/TB — roughly 90% cheaper than equivalent Spark maintenance clusters. This cost reduction is not incremental — it is structural. It makes continuous maintenance economically viable rather than forcing overnight-only windows where tables degrade for 23 hours between passes. Continuous maintenance means tables never accumulate enough degradation to require expensive recovery operations, queries never slow enough to warrant over-provisioned clusters as compensation, and engineering teams never spend emergency weekends rebuilding collapsed tables.

Level 5: Recovered engineering time. Without autonomous operations, platform teams spend 20–40% of their capacity on table health — writing maintenance scripts, debugging failures, tuning thresholds, responding to "the dashboard is slow" tickets. With a control plane handling the closed loop, that time returns to building data products, onboarding new sources, and serving consumers. For a five-person platform team, that is 1–2 engineers' worth of recovered capacity.

Combined impact: Organizations operating at 500+ TB report 3–5x total cost reduction compared to the warehouse-plus-lake architecture — combining storage savings, duplication elimination, routing-driven compute efficiency, cheaper maintenance, and recovered engineering time.

When to build a data lakehouse

Not every team needs this architecture. A single-engine, single-workload team at moderate scale gets reasonable value from a managed warehouse without the operational complexity. But the lakehouse becomes the correct choice when:

Multiple workload types coexist. BI, ML, streaming, and AI workloads need the same data with different access patterns and different engines.
Scale makes economics matter. Above 50–100 TB of analytical data, the storage cost differential (10–50x) between warehouse and lakehouse becomes significant enough to fund an entire platform team.
Multi-engine is a requirement. Different teams or workloads genuinely need different engines — Spark for ETL, Trino for interactive, Flink for streaming, DuckDB for development. The lakehouse makes this safe; the warehouse makes it impossible.
Vendor independence is a priority. Open formats on your own storage mean no single vendor controls your exit path. Every component decision is reversible.
Data freshness matters. Streaming ingestion directly into governed tables eliminates the ETL delay that warehouses impose between source and dashboard.
AI/ML is a first-class workload. Models and agents need native, governed access to analytical data — not exports, not API wrappers, not stale copies in separate feature stores.

If three or more of these apply, the lakehouse is not a future consideration — it is the architecture to build toward now.

Conclusion

The data lakehouse is not a concept — it is the production standard for teams that need BI, ML, streaming, and AI on the same governed data without warehouse pricing or lake-quality compromises.

Building one is an assembly problem: object storage for economics, a table format (Iceberg) for transactional guarantees, a REST catalog for coordination, and specialized engines for execution.

Operating one is a systems problem: tables degrade through file fragmentation, metadata growth, delete-file accumulation, and storage waste. Bronze layers fragment through streaming commits. Silver layers accumulate delete-file debt through mutations. Gold layers waste storage through stale snapshots. The degradation is inherent to append-only transactional design — not a bug to fix but a force to continuously counteract.

The control plane is the system that counteracts it. LakeOps implements the closed loop — sense structural telemetry across every table and catalog, assess health with adaptive scoring, plan sequenced maintenance respecting operation dependencies, execute on a dedicated Rust engine that is fast enough to run continuously, and learn from outcomes to improve future decisions. Intelligent compaction that sorts by production query patterns across all engines. Sequenced maintenance that runs the right operations in the right order. Observability that classifies every table's health and surfaces degradation before users feel it. Routing that dispatches queries to the optimal engine for each workload. Governance that enforces retention, maintenance, and access policies lake-wide without per-table configuration. And AI enablement that makes the lake agent-ready with discoverable schemas and layered guardrails.

The architecture works because something continuously keeps it working. That something is the control plane.

Thanks for reading!

Further learning:

Autonomous Iceberg Table Maintenance — the closed-loop maintenance system in depth
Iceberg Lakehouse Observability Guide — seven pillars of lake monitoring
Intelligent Lakehouse: Build Like Netflix — how Netflix designed autonomous lakehouse operations
Fixing Small Files in Apache Iceberg — root causes, measurement, and automated resolution
How AI Agents Operate a Data Lake via MCP — the agent-native lakehouse interface
Apache Iceberg Production Readiness Checklist — ten operational dimensions for enterprise Iceberg
Routing Multiple Query Engines with Iceberg — multi-engine dispatch patterns and optimization

Apache Iceberg Query Performance: A Practical Guide

joni sar — Sun, 02 Aug 2026 09:31:14 +0000

Iceberg is the standard. Snowflake, Databricks, AWS, and every major query engine read and write it natively. The format question is settled.

The performance question is not.

Most production Iceberg tables are slower than they need to be — not because of anything wrong with the format, but because the physical state of the table has degraded over time. Thousands of small files from streaming writes. Manifests fragmented across hundreds of snapshots. Data scattered randomly across files with no correlation to how it's actually queried. Delete files piling up from CDC pipelines.

The result: engines scan 5–10x more data than necessary. Query planning takes seconds instead of milliseconds. S3 API costs spike. And every engine — Trino, Spark, Snowflake, Athena, DuckDB — is equally affected because they all read the same physical data layout.

This guide covers what determines Iceberg query performance, how to fix it, and the two paths available: intelligent continuous optimization through a control plane, and the manual approach with SQL and cron. Both paths target the same physical levers — the difference is whether they adapt over time or require ongoing human attention.

How Iceberg scan planning works

Understanding the performance machinery is essential for both approaches. When a query engine receives a SELECT against an Iceberg table, it runs a scan planning process on a single node that determines which files to read. This is one of Iceberg's core advantages over Hive-style directory listing — no distributed scan needed just to find your data.

Planning operates in three progressive levels of elimination:

Level 1: Manifest list pruning. The engine reads the snapshot's manifest list and checks partition value ranges in each manifest's summary. If a manifest's date range doesn't overlap with the query's filter, it's skipped entirely — along with all data files it references. One check eliminates hundreds of manifest files.

Level 2: File-level data skipping. For surviving manifests, the engine reads per-file column statistics (min/max bounds, null counts, row counts) and evaluates predicates against them. A query for amount > 500 skips every file where the amount column's maximum is below 500. This is where sort order determines everything — tight, non-overlapping min/max ranges mean more files eliminated.

Level 3: Row group pruning. Within surviving files, Parquet row group statistics and bloom filters skip sub-file data blocks. If only one of four row groups matches, 75% of the file's I/O is avoided.

The effectiveness of each level depends entirely on the physical state of the table. A detailed walkthrough covers these internals in depth.

Why tables degrade

Iceberg tables don't start slow. They become slow because the physical layout drifts away from what's optimal for the queries hitting them.

Streaming writes create files at every checkpoint interval (every 1–5 minutes). A table receiving continuous events generates hundreds or thousands of files per day — each one tiny, each one adding a manifest entry that must be evaluated during planning.

Static sort orders lose relevance. A sort key chosen at table creation doesn't reflect what queries actually filter on six months later. As access patterns evolve, the sort becomes meaningless and min/max statistics become wide and useless for pruning.

Maintenance runs in isolation. Compaction, snapshot expiration, manifest rewriting, and orphan cleanup each affect each other's effectiveness. Running them independently — or in the wrong order — means wasted work and suboptimal results.

CDC pipelines accumulate delete files. Every update writes a small delete file. After weeks, hundreds of delete files must be reconciled at read time — each query pays the cost.

No feedback loop. Without observing which columns queries actually filter on, which tables are degrading fastest, and which maintenance actions produce the most benefit — optimization is guesswork.

These problems don't happen once and get fixed. They're continuous. They happen every day, on every table, as long as data is being written and queried.

Two paths to query performance

There are fundamentally two approaches to keeping Iceberg tables performant:

Path 1: Intelligent continuous optimization. A system that observes query patterns, understands table state, and takes the right action at the right time — autonomously, adaptively, and in the correct sequence. This is the control plane approach.

Path 2: The manual path. SQL procedures, cron jobs, and configuration tuning. You run the same operations — compaction, manifest rewrites, snapshot expiration — but you decide when, how, and on which tables. You choose the sort order. You monitor for degradation. You sequence the operations correctly.

Both paths target the same physical levers described above. The difference is not what gets done — it's whether it adapts continuously or requires ongoing engineering attention.

Path 1: Intelligent continuous optimization

LakeOps is an autonomous control plane for Apache Iceberg. It connects to your existing catalogs and query engines, observes table state and query patterns, and continuously applies the optimizations that make queries faster — without human intervention, without Spark clusters, and without static configurations that go stale.

Here's what it does and why each piece improves query performance.

Query-aware data layout

This is the highest-impact capability. Sort order is the single most impactful lever for Iceberg query performance — sorted tables scan 51% less data per query than unsorted ones. But choosing the right sort order requires knowing which columns production queries actually filter on. That knowledge changes over time.

LakeOps captures query telemetry from every connected engine — Spark, Trino, Flink, Snowflake, Athena, DuckDB, StarRocks — and identifies which columns appear in WHERE, JOIN, and GROUP BY clauses for each table. During compaction, data is physically re-sorted by those columns.

The result: Parquet row-group min/max statistics become maximally tight. The scan planner eliminates the vast majority of files at Level 2 because each file covers a narrow, non-overlapping value range for the columns queries actually use.

The sort strategy adapts. If a new dashboard starts filtering on customer_segment alongside event_date, the next compaction pass incorporates it. No manual ALTER TABLE SET SORT ORDER needed. On three consecutive runs of a 1.2 TB table, the system improved runtime from 22 min → 18 min → 11 min as it converged on optimal column ordering — zero configuration changes.

Production numbers: a table with 47,000 scattered files, after query-aware sort compaction to 280 files, saw query time drop from 52 seconds to 5.8 seconds. A 9x improvement from layout optimization alone.

Compaction at streaming speed

Sort compaction is the most impactful maintenance operation — it solves both the small-file problem and the sort-order problem in one pass. But on Spark, it's slow and expensive. A 200 GB sort compaction takes ~25 minutes and costs ~$3.50 on EMR. At that speed, running it multiple times per day on streaming tables is impractical. Most teams default to binpack (size-only) because sort is too expensive — leaving the biggest performance lever unused.

LakeOps's compaction engine is built in Rust on Apache DataFusion. No JVM. No garbage collection. No OOM crashes. The speed difference makes sort compaction viable even for streaming tables:

Binpack: 221 seconds vs 1,612s (Spark) vs 6,300s (S3 Tables) on identical 200 GB dataset
Sort: 780 seconds vs estimated 3,000+ for Spark
Cost: $0.21 vs $1.54 for 200 GB binpack
Peak throughput: 2,522 MB/s — TB-scale tables in minutes
Bounded memory: spills to disk gracefully, no OOM regardless of table size

Because it's fast enough to run frequently, tables stay both consolidated and sorted continuously — never degrading between maintenance windows.

Coordinated maintenance pipeline

Individual maintenance operations interact. The order matters:

Expire snapshots — release references to old data files
Remove orphans — delete unreferenced files from storage
Compact — merge remaining files with optimal sort order
Rewrite manifests — consolidate metadata over the new, clean file set

Running these in the wrong order wastes work. Compacting before expiring snapshots means you might rewrite files that should be deleted. Rewriting manifests before compaction means the structure changes again during the compaction commit.

LakeOps sequences operations automatically. Each step produces a clean input for the next. The manifest rewrite runs on an already-consolidated, properly-sorted file set — producing the leanest possible metadata layer. This coordination eliminates the redundant work that independent cron jobs inevitably create.

Event-driven triggers

A streaming table that accumulates 500 files per hour needs compaction multiple times per day. A stable batch table with weekly loads needs compaction monthly. A CDC table with accumulating delete files needs cleanup when the read-time overhead crosses a threshold.

LakeOps triggers operations based on actual table state — file count thresholds, delete-file ratios, manifest fragmentation, partition skew — not arbitrary cron schedules. Tables get exactly the maintenance they need, when they need it. No wasted runs on healthy tables, no missed runs on degrading ones.

Multi-engine query routing and workload optimization

Different engines have different performance profiles: a point lookup takes 0.5s on DuckDB vs 2.3s on Athena. A full-table scan costs $5 on Athena vs $50 on Trino. Without routing, every query goes to the same engine regardless of its shape.

LakeOps includes QueryFlux — an open-source, Rust-based SQL proxy with 0.35ms overhead — for multi-engine routing. Queries route based on shape, latency targets, cost ceilings, engine availability, and table health status. The routing layer learns from execution history: if a query shape consistently runs faster on one engine, future executions route there.

In benchmarking, workload-aware routing reduced total query cost by up to 80%, with individual queries sometimes dropping by 90%.

Observability and self-improvement

You can't optimize what you can't see. LakeOps classifies every table into health tiers (Critical, Warning, Healthy) based on file fragmentation, manifest depth, snapshot velocity, delete ratios, and sort-order staleness. Insights surface at four severity levels — before degradation becomes noticeable in query times.

The system records per-table throughput, partition structure, and memory usage from each compaction run. Subsequent passes execute faster as the planner converges on optimal resource allocation. Every operation feeds back into the next decision.

What setup looks like

Connect your catalogs (AWS Glue, REST catalogs like Polaris/Nessie/Lakekeeper, S3 Tables) and storage. ~10 minutes. No agents to deploy, no data movement, no pipeline changes. Your data stays in your account. The system discovers tables, classifies health, begins autonomous maintenance according to policies you define.

Production results across customers: up to 12x average query acceleration, up to 80% total cost reduction, 786+ tables managed autonomously across 112+ PB.

Path 2: The manual approach

Every optimization LakeOps automates is also achievable with SQL procedures, engine configuration, and scheduling. Here's how to implement each one by hand.

Diagnosing the bottleneck

Before optimizing, use Iceberg's metadata tables to identify what's actually wrong:

-- File health: count, size distribution, small file ratio
SELECT 
  COUNT(*) as file_count,
  AVG(file_size_in_bytes) / 1048576 as avg_mb,
  COUNT(CASE WHEN file_size_in_bytes < 67108864 THEN 1 END) as small_files
FROM prod.db.events.files;

-- Manifest health: fragmentation
SELECT COUNT(*) as manifest_count,
       AVG(added_data_files_count) as avg_files_per_manifest
FROM prod.db.events.manifests;

-- Delete file accumulation (MoR tables)
SELECT COUNT(*) as delete_file_count
FROM prod.db.events.delete_files;

-- Snapshot accumulation
SELECT COUNT(*) as snapshot_count,
       MIN(committed_at) as oldest
FROM prod.db.events.snapshots;

Thresholds: avg file size < 128 MB → compaction needed. Manifest count > 200 → rewrite manifests. Delete files > 50 → compact to apply. Snapshots > 1,000 → expire.

Use EXPLAIN to verify predicates push down to the scan level rather than appearing as residual filters.

Sort order configuration

Choose columns based on your most common query patterns:

-- Set sort order (first column gets tightest clustering)
ALTER TABLE prod.db.events WRITE ORDERED BY event_date, user_id;

-- Sort compaction to apply the order to existing data
CALL system.rewrite_data_files(
  table => 'prod.db.events',
  strategy => 'sort',
  sort_order => 'event_date ASC NULLS LAST, user_id ASC NULLS LAST'
);

-- For tables with diverse multi-column filters, use Z-order
CALL system.rewrite_data_files(
  table => 'prod.db.events',
  strategy => 'sort',
  sort_order => 'zorder(event_date, user_id, region)'
);

Guidelines for choosing sort columns:

First key: Column most frequently in WHERE clauses. Gets tightest min/max ranges
Second key: Diminishing returns, but meaningful for two-column filters
Z-order: When no single column dominates filter patterns

The challenge: you must manually identify which columns queries filter on, and update the sort order when access patterns change. Most teams don't revisit this after initial setup.

File consolidation (compaction)

-- Binpack: merge small files by size (fast, no reordering)
CALL system.rewrite_data_files(
  table => 'prod.db.events',
  strategy => 'binpack',
  options => map('target-file-size-bytes', '268435456', 'min-file-size-bytes', '67108864')
);

-- Sort compaction: merge + apply sort order (slower, better for queries)
CALL system.rewrite_data_files(
  table => 'prod.db.events',
  strategy => 'sort',
  sort_order => 'event_date ASC NULLS LAST, user_id ASC NULLS LAST',
  options => map('target-file-size-bytes', '268435456')
);

Target: 256–512 MB files for analytical workloads. Schedule via Airflow/cron — daily for streaming tables, weekly for batch.

Preventing small files at write time

The cheapest compaction is the one you never need:

ALTER TABLE prod.db.events SET TBLPROPERTIES (
  'write.target-file-size-bytes' = '268435456',    -- 256 MB
  'write.distribution-mode' = 'hash',              -- Group by partition before write
  'write.parquet.compression-codec' = 'zstd'       -- Better compression
);

Critical: write.distribution-mode = 'none' on partitioned tables is the #1 source of small files. With 200 Spark tasks writing to 365 daily partitions, you get 73,000 tiny files per job. Always use hash for partitioned tables.

For Spark, align AQE with your target file size:

spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "true")
spark.conf.set("spark.sql.adaptive.advisoryPartitionSizeInBytes", "268435456")

For Flink streaming writes, increase checkpoint intervals to reduce file frequency:

SET 'execution.checkpointing.interval' = '5min';
SET 'sink.committer.operator.commit-interval' = '5min';

Bloom filters for point lookups

When queries use equality predicates on high-cardinality columns (WHERE user_id = 12345), bloom filters eliminate row groups that provably don't contain the value:

ALTER TABLE prod.db.events SET TBLPROPERTIES (
  'write.parquet.bloom-filter-enabled.column.user_id' = 'true',
  'write.parquet.bloom-filter-enabled.column.session_id' = 'true',
  'write.parquet.bloom-filter-fpp.column.user_id' = '0.05'
);

-- Generate filters on existing data via compaction
CALL system.rewrite_data_files(
  table => 'prod.db.events',
  options => map(
    'write.parquet.bloom-filter-enabled.column.user_id', 'true',
    'write.parquet.bloom-filter-fpp.column.user_id', '0.05'
  )
);

Limit to 2–3 columns per table (each adds ~1 MB to every file footer). Only useful for equality predicates — range queries don't benefit.

Manifest consolidation

-- Check if needed
SELECT COUNT(*) FROM prod.db.events.manifests;

-- Consolidate (run AFTER compaction)
CALL system.rewrite_manifests('prod.db.events');

Target: 10–50 manifests. Run after compaction stabilizes the file set — not before.

Snapshot expiration and orphan cleanup

-- Expire snapshots older than 7 days, keep at least 5
CALL system.expire_snapshots(
  table => 'prod.db.events',
  older_than => TIMESTAMP '2026-07-26 00:00:00',
  retain_last => 5
);

-- Remove orphan files from failed writes
CALL system.remove_orphan_files(
  table => 'prod.db.events',
  older_than => TIMESTAMP '2026-07-26 00:00:00'
);

Guidelines: 3–7 days retention for streaming tables, 30–90 days for batch. Always retain_last >= 2 to avoid breaking concurrent readers.

Delete file cleanup (MoR tables)

-- Compact to apply pending deletes
CALL system.rewrite_data_files(
  table => 'prod.db.events',
  options => map('delete-file-threshold', '3')
);

Monitor regularly. A table accumulating 200+ delete files/day will see read times increase from 3s → 22s over two weeks.

Partition strategy

-- Set partition (at table creation or via evolution)
CREATE TABLE prod.db.events (
  event_id BIGINT,
  event_time TIMESTAMP,
  user_id BIGINT,
  region STRING
) USING iceberg
PARTITIONED BY (day(event_time));

-- Evolve without data rewrite (metadata-only, instant)
ALTER TABLE prod.db.events SET PARTITION SPEC (day(event_time));

Rules: each partition should hold at least 128 MB. Daily partitioning on a 10 MB/day table → use monthly. bucket(col, 1024) on a 50 GB table → try 16–64 buckets instead.

Puffin statistics for join planning

-- Generate NDV statistics for cost-based optimization
CALL system.compute_table_stats(table => 'prod.db.events');

-- Or via ANALYZE TABLE (Spark)
ANALYZE TABLE prod.db.events COMPUTE STATISTICS FOR COLUMNS
  user_id, event_type, region;

Recompute after major data changes. For details: Puffin Statistics Guide.

Parquet-level tuning

ALTER TABLE prod.db.events SET TBLPROPERTIES (
  'write.parquet.row-group-size-bytes' = '134217728',   -- 128 MB row groups
  'write.parquet.page-size-bytes' = '1048576',          -- 1 MB pages
  'write.parquet.dict-size-bytes' = '2097152',          -- 2 MB dict threshold

  -- Disable stats for columns never filtered on (reduces manifest bloat)
  'write.metadata.metrics.column.raw_payload' = 'none',
  'write.metadata.metrics.column.debug_info' = 'none'
);

For read-side tuning in Spark:

spark.conf.set("spark.sql.files.maxPartitionBytes", "268435456")  # 256 MB per task
spark.conf.set("spark.sql.files.openCostInBytes", "4194304")      # 4 MB open cost

Copy-on-Write vs. Merge-on-Read

Choose your update strategy based on your read/write ratio:

Workload	Strategy	Why
Batch reporting (updated nightly, read all day)	Copy-on-Write	Zero read overhead, clean files always
Streaming CDC (Debezium/Kafka)	Merge-on-Read + compaction	Write speed critical, compact to restore reads
Infrequent GDPR/compliance deletes	Merge-on-Read + periodic cleanup	Fast compliance, clean up later
High-frequency MERGE INTO	Deletion Vectors (V3)	Best write + read performance

The correct maintenance sequence

Whether automated or manual, the order of operations matters:

Expire snapshots → releases references to dead files
Remove orphans → deletes unreferenced files from storage (cheaper subsequent compaction)
Compact (with sort) → merges files, applies sort order, resolves delete files
Rewrite manifests → consolidates metadata over the clean, final file set

Running these out of order wastes compute. Compacting before expiring rewrites files that should be deleted. Manifest rewrites before compaction get invalidated immediately.

Performance troubleshooting checklist

Symptom	Likely cause	Fix
Slow range queries	Unsorted data, many small files	Sort compaction on filter columns
Slow point lookups	No bloom filters	Enable per-column bloom filters + compact
Slow query planning (>5s)	Too many manifests	`rewrite_manifests`
Full table scans	Wrong partition scheme	Partition evolution
Gradually slowing reads	Accumulating delete files	Compact to apply deletes
High S3 API costs	Thousands of small files	Compaction + write-side tuning
Dashboard timeouts	Multiple degradation factors	Full maintenance pipeline + routing

Choosing your path

The manual path works. Every SQL procedure shown above is production-proven. The challenge is sustaining it:

Choosing the right sort order requires analyzing query logs across all engines — and updating when patterns change
Scheduling compaction requires different frequencies for different tables — streaming tables need it hourly, batch tables weekly
Sequencing operations correctly requires understanding their dependencies
Monitoring 50+ tables for degradation signals requires dashboard infrastructure
Scaling to hundreds of tables means the maintenance burden grows linearly with your data estate

For teams with fewer than 10–20 tables and stable access patterns, the manual path is manageable. Beyond that, the operational burden compounds — and the sort order inevitably goes stale because nobody has time to analyze query patterns per table per quarter.

The control plane approach eliminates this scaling problem. It handles the continuous observation, adaptation, and execution at any table count — and because its compaction engine is 95% faster than Spark, it can apply sort compaction at frequencies that would be cost-prohibitive manually.

Both paths lead to the same destination: tables where queries are 12x faster because the physical layout matches how data is actually accessed. The question is whether you want that as a continuous property of your lake, or as a point-in-time optimization that needs periodic human re-tuning.

Summary

Iceberg query performance is determined by the physical state of your tables — not by the format spec, not by the engine, and not by hardware. A well-maintained table is fast on every engine. A degraded table is slow on all of them.

The physical levers that matter:

Sort order aligned to query patterns — 51% less data scanned, up to 12x faster queries
File consolidation — 5–10x faster planning, reduced I/O
Bloom filters — near-instant elimination for equality lookups
Manifest optimization — 2–5x faster planning
Write-side configuration — prevent degradation at source
Delete file cleanup — restore read performance on MoR tables
Snapshot lifecycle — lean metadata, lower storage costs
Partition strategy — coarse-grained elimination at the foundation
Multi-engine routing — right engine per query, up to 56% cost reduction

The intelligent path keeps all of these optimal continuously. The manual path gives you the tools to do it yourself. Either way — the performance ceiling is the physical state of your tables, and now you know how to raise it.

Further reading:

AI Data Lake and Lakehouse

joni sar — Fri, 24 Jul 2026 14:07:16 +0000

The word "AI" appears in every data platform pitch now. Most of the time, it means nothing — a chatbot bolted onto a dashboard, an LLM that generates SQL, or a marketing slide that says "AI-powered" next to a cron job.

This article is about something different: what happens when you put actual intelligence — machine learning, adaptive optimization, closed-loop feedback systems — at the core of how a data lakehouse operates. Not AI as a feature. AI as the operating model.

The result is a lakehouse that understands its own workloads, optimizes its own storage layouts, routes its own queries, heals its own tables, and continuously improves — without human intervention. That is what an AI data lake actually is. Everything else is automation with better branding.

Automation Is Not Intelligence
The AI Control Plane
How AI Transforms Each Layer of the Lakehouse
AI-Driven Compaction: Learning How Data Is Read
AI-Driven Maintenance: Predicting What Breaks
AI-Driven Query Routing: Matching Queries to Engines
AI-Driven Observability: From Dashboards to Diagnosis
AI-Driven Governance: Policies That Adapt
The Closed Loop: How Every Query Makes the Lake Smarter
AI Agents as Lake Citizens
AI-Native vs. AI-Washed: How to Tell the Difference
The Compounding Effect
Where This Is Heading

Automation Is Not Intelligence

Most data lake "automation" today works like this: a cron job runs compaction every night at 2 AM. It merges small files into bigger ones. It does not know if the table needs compaction. It does not know which partitions are degraded. It does not know what columns queries filter on. It does not know whether last night's compaction actually helped.

That is automation. It follows a fixed rule, regardless of context.

Intelligence is different. Intelligence observes, learns, decides, acts, measures the outcome, and adjusts. A system that compacts a table because its file health crossed a threshold, sorts by the columns that real queries actually filter on, sequences the operation after snapshot expiration to avoid wasted work, and then verifies that query latency improved — that is intelligent.

The distinction matters because data lakes are not static systems. Workloads shift. New dashboards appear. AI agents start querying tables that were designed for nightly batch. Streaming pipelines change write patterns. A schema evolves.

Fixed rules cannot keep up. Intelligence can.

The AI Control Plane

An AI data lake needs a brain — a system that continuously ingests telemetry from every table, every engine, and every query, and uses that signal to drive optimization decisions across the entire lake.

LakeOps is that system. It is an autonomous control plane for Apache Iceberg that puts AI at the center of lakehouse operations — compaction, maintenance, query routing, observability, governance, and agent enablement, all driven by continuous learning rather than static rules.

What makes it a control plane and not another tool: LakeOps is not an engine, not a catalog, and not storage. It connects to whatever you already run — AWS Glue, Polaris, Nessie, Gravitino, Lakekeeper, or S3 Tables as catalogs; Trino, Spark, Snowflake, Athena, DuckDB, Flink as engines; S3, GCS, or ADLS as storage — and adds the intelligence layer on top. No data movement. No code changes. No vendor lock-in. Setup takes about ten minutes.

What makes it AI-native: every decision LakeOps makes is informed by learned behavior, not hardcoded rules. It observes query patterns across all engines, predicts table degradation from structural signals, adapts data layouts as workloads shift, learns which engine performs best for each query shape, and measures the outcome of every optimization to improve the next one. It also exposes a native MCP (Model Context Protocol) server so AI agents can query the lake directly — with schema discovery, intelligent routing, and layered guardrails built in. The more the lake is used — by humans and agents alike — the smarter the control plane gets.

Three operating modes let teams choose how much autonomy the AI gets:

Autopilot — the AI decides and executes. Define policies and thresholds; the system handles everything.
Recommendations — the AI analyzes and recommends with full context (what, why, expected impact). A human approves.
Policy-bounded — the AI operates within declared boundaries, escalating edge cases.

Most teams start with recommendations to build trust, then move tables to autopilot as confidence grows. The intelligence is the same in all modes — the only difference is who clicks "execute."

With that foundation, here is how AI transforms each layer of lakehouse operations.

How AI Transforms Each Layer of the Lakehouse

When we talk about an "AI data lake," we are talking about specific capabilities that require learning, prediction, and adaptation — not features that could be implemented with a shell script.

Layer	What AI Does	What Scripts Cannot
Compaction	Learns query patterns across engines, sorts data by the columns actually used in WHERE/JOIN/GROUP BY, adapts as patterns change	Static sort keys, fixed file-size targets, blind to query patterns
Maintenance	Predicts which tables will degrade next, sequences operations by dependency, prioritizes by business impact	Fixed schedules, independent jobs, no awareness of health trajectory
Query routing	Learns which engine performs best for each query shape, uses LLMs to reason about new templates, adapts as tables improve	Static rules, manual engine assignment, no cost optimization
Observability	Correlates structural signals with query performance, predicts degradation before user impact	Threshold alerts, no predictive capability, no cross-signal correlation
Governance	Adapts policy enforcement to workload patterns, recommends changes from observed behavior	Static policies, manual updates, no feedback from operations
Agent integration	Feeds agent telemetry back into optimization, shapes the lake around AI access patterns	No awareness of agents, no closed-loop optimization

Each of these deserves a deeper look.

AI-Driven Compaction: Learning How Data Is Read

Traditional compaction is a janitor: it sweeps up small files and merges them. It does not know why the files exist or how they are read. An intelligent compaction engine does.

Query Pattern Analysis

LakeOps continuously observes every query that hits every table — across Trino, Spark, Snowflake, Athena, DuckDB, Flink, and any other connected engine. It extracts which columns appear in WHERE, JOIN, and GROUP BY clauses. It tracks how these patterns change over time. It weights recent queries more heavily than old ones.

This is not a one-time analysis. It is a continuous learning loop.

From this analysis, the AI knows — per table — which columns matter most for data layout. A table where 80% of queries filter on event_date and region should have its data physically sorted by those columns. A table where queries mostly join on customer_id should sort differently.

From Analysis to Action

During compaction, data files are physically re-sorted by the learned columns. The result: Parquet row-group min/max statistics become effective, and engines skip irrelevant data without reading it. Predicate pushdown and column pruning actually work — not because someone manually configured sort keys, but because the AI figured out what the right sort order is.

On production tables, this AI-driven sorting reduces data scanned by 51% and delivers 12x faster queries compared to unsorted compaction. The numbers are large because the difference between "sort by whatever was configured at table creation" and "sort by what queries actually need" is enormous.

Adaptive Sort Evolution

Here is where intelligence really separates from automation. When query patterns shift — a new dashboard starts filtering on product_category instead of region, a data science team changes their join keys, AI agents start accessing different columns — the sort strategy evolves on the next compaction pass. No configuration change. No ticket. No human intervention.

A table whose workload stabilizes gets compacted less frequently — the AI recognizes that the current layout is already optimal and avoids redundant work. A table whose workload spikes gets prioritized. This is resource allocation driven by learned behavior, not fixed schedules.

Speed as an AI Enabler

The AI's compaction decisions need an engine fast enough to execute them continuously — not queue them for a nightly batch window. LakeOps runs compaction on a purpose-built Rust engine powered by Apache DataFusion: 221 seconds where Spark takes 1,612 seconds (95% faster), at roughly $5/TB versus $50/TB (90% cheaper). Peak throughput reaches 2,522 MB/s. Tables that OOM Spark — a 1.2 TB table with 12,000 files, for example — complete without special configuration because memory is bounded, not heap-dependent.

That speed is not just a performance metric. It is the enabler for AI-driven compaction: when compaction takes minutes instead of hours, the system can respond to workload changes on the same day they happen rather than falling behind. The AI can iterate. Fast execution makes continuous learning practical.

AI-Driven Maintenance: Predicting What Breaks

Iceberg tables require six maintenance operations: data compaction, snapshot expiration, manifest consolidation, orphan cleanup, position delete resolution, and statistics computation. Every platform team knows this. The question is when and in what order to run them — and that is where AI changes the game.

Health Prediction, Not Health Reaction

Traditional monitoring tells you a table is degraded after queries slow down. By then, users have already filed tickets and data pipelines are late.

LakeOps predicts degradation before it hits. The AI continuously scores every table on structural signals — file count trajectory, partition balance trends, manifest growth rate, snapshot accumulation velocity, delete-file ratios. It sees the pattern: "this table's small-file ratio is climbing at 4% per day; at current trajectory it will hit Critical in 72 hours."

This is a classification and prediction problem that humans cannot solve manually at scale. A platform team with 800 tables across multiple catalogs cannot eyeball metadata trends for each one. An AI system that ingests every commit, every query, and every structural change can.

Intelligent Sequencing

The order of maintenance operations matters. Running orphan cleanup before snapshot expiration risks deleting referenced files. Running manifest rewrites before compaction wastes the rewrite since compaction will invalidate file references.

The AI learns the correct dependency graph and sequences operations per table:

Compaction first (merge small files, apply learned sort order)
Snapshot expiration after (remove old references safely, respecting active readers)
Manifest consolidation next (consolidate after file references stabilize)
Orphan cleanup last (reclaim newly unreferenced storage)

This is not just a fixed ordering — the AI evaluates whether each step is needed based on current table state. A table with healthy manifests skips manifest consolidation. A table with no orphans skips cleanup. The maintenance plan is tailored per table, per cycle, based on real signals.

Operation Outcome Learning

After each operation completes, the AI measures the outcome: did compaction actually reduce query latency? Did snapshot expiration free the expected storage? Did manifest consolidation improve planning time?

These outcomes feed back into future decisions. If compacting a particular table's hot partition consistently delivers a 5x query speedup, the system learns to prioritize that partition. If expiring snapshots on a rarely-queried table has minimal impact, it deprioritizes that work. Over time, the maintenance engine gets better at deciding what to do — not just executing a predefined list.

AI-Driven Query Routing: Matching Queries to Engines

Most production lakehouses use multiple query engines. Trino for interactive analytics. Spark for heavy ETL. Snowflake for BI dashboards. DuckDB for ad-hoc exploration. Athena for pay-per-query workloads. The problem is deciding which query goes where.

Without intelligence, this is a manual decision — or worse, no decision at all, and every query hits the same engine regardless of whether it is a 50 MB selective filter or a 500 GB full table scan.

The Three-Router AI Stack

LakeOps implements intelligent routing through QueryFlux, an open-source Rust-based SQL proxy, extended with a three-layer AI stack:

1. Adaptive routing (learned behavior). The AI tracks historical execution data for every query template — which engine ran it, how long it took, how much it cost, whether it succeeded. For previously seen query shapes, routing is instant: the system already knows which engine delivers the best cost-performance ratio. Cached decisions take 0ms.

2. LLM-based routing (reasoning). For new query templates the system has not seen before, an LLM reasons over the query structure combined with live table statistics — file count, data size, sort order, partition layout, health score. It predicts which engine is the best fit. This handles the cold-start problem that pure historical routing cannot.

3. Semantic routing (intent matching). When neither historical data nor structural analysis alone is sufficient, semantic matching infers the query's intent — exploratory analysis, dashboard refresh, ETL pipeline, or AI agent retrieval — and routes accordingly.

The three routers compose. Over time, more query templates enter the adaptive cache and routing gets faster and more accurate.

Data-Quality-Aware Routing

This is where AI routing connects to AI compaction and AI observability. The routing layer knows — from LakeOps's health scoring — whether a table is well-compacted, fragmented, or degraded. It adjusts routing decisions accordingly:

A well-compacted table with good sort order? Route to a lightweight, cheaper engine that benefits from efficient data skipping.
A degraded table with 40,000 small files? Route to an engine that can handle the overhead while compaction runs in the background.
A table that was just compacted and is now healthy? Update the routing weights so future queries use cheaper engines.

As the AI compaction engine improves table health, the AI routing engine shifts queries to cheaper engines. The cost savings compound. Production benchmarks show up to 56% workload cost reduction from intelligent routing alone.

AI-Driven Observability: From Dashboards to Diagnosis

Traditional observability is a dashboard: it shows metrics, you interpret them. AI-driven observability is a diagnostic system: it interprets the metrics for you, tells you what is wrong, explains why, and either fixes it or recommends a fix.

Cross-Signal Correlation

LakeOps does not monitor file counts and query latency independently. The AI correlates signals across layers:

Storage signals — file count, size distribution, access patterns, orphan volume
Metadata signals — manifest count and depth, snapshot velocity, partition skew, statistics coverage
Engine signals — query latency, scan volume, CPU utilization, per-engine field access
Operation signals — maintenance duration, compaction throughput, health score changes

From these, it produces actionable insights with causal chains. Example: "Query latency on raw_clickstream increased 8x because 312 partitions exceed the file count threshold — scan amplification is 8x. Compaction is recommended and will reduce query time to baseline."

That is not a threshold alert. That is a diagnosis.

Severity-Ranked Insights

The AI surfaces per-table findings at four severity levels — CRITICAL, HIGH, WARNING, LOW — each with the specific metric, current value, threshold violated, and a remediation action. Critical and High Insights trigger automated remediation. Warning and Low Insights surface for human review.

The key AI element: the system does not just check "is file count above X?" It evaluates the combination of signals. A table with many small files but no queries gets a lower severity than a table with fewer small files but high query frequency. Business impact is inferred from workload patterns, not just structural metrics.

Predictive Degradation

Instead of alerting after a table degrades, the AI predicts when it will degrade based on trajectory. A table whose snapshot accumulation rate will hit its retention limit in 48 hours triggers an alert now — with enough lead time to act before users notice.

AI-Driven Governance: Policies That Adapt

Static governance policies are set-and-forget — which usually means set-and-break. Workloads change, new tables appear, query patterns shift, and the policies that made sense six months ago no longer match reality.

Intelligent Policy Recommendations

LakeOps's AI observes actual workload behavior and recommends policy adjustments:

"Table orders_archive has not been queried in 90 days. Current retention policy keeps 30 days of snapshots. Recommend reducing to 7 days to reclaim 18 GB of metadata."
"Namespace analytics.* consistently produces small files from streaming writes. Recommend adding an auto-compaction policy with a 1-hour trigger window."
"Table customer_events is queried by 3 AI agents with PII columns. Recommend adding PIIMaskGuard policy to agent-facing routing groups."

These are not generic suggestions. They are derived from observed behavior — query patterns, write patterns, access patterns, and health trajectories specific to each table.

Policy Inheritance with Workload Awareness

Policies cascade from organization to catalog to namespace to table — but the AI ensures inheritance makes sense. A compaction policy designed for batch tables is not blindly applied to streaming tables in the same namespace. The system flags conflicts between inherited policies and actual workload patterns, and either adjusts automatically (in autopilot mode) or recommends changes (in manual mode).

Policies apply uniformly across catalogs — AWS Glue, Polaris, Nessie, Gravitino, Lakekeeper — without per-catalog scripts or engine-specific logic. Every policy is auditable, versioned, and controllable with a single toggle.

The Closed Loop: How Every Query Makes the Lake Smarter

The most important concept in an AI data lake is the closed-loop feedback system. This is what separates an intelligent system from a collection of automated scripts — and it is the core of how LakeOps operates.

Here is the loop:

    Queries hit the lake
    (engines, agents, dashboards, pipelines)
            │
            ▼
    Telemetry captures patterns
    (columns, engines, latency, cost, frequency)
            │
            ▼
    AI analyzes and learns
    (sort order, routing weights, health scores,
     maintenance priorities, policy fitness)
            │
            ▼
    Optimizer acts
    (compact with new sort, reroute queries,
     adjust policies, trigger maintenance)
            │
            ▼
    Outcomes measured
    (query faster? cost lower? health improved?)
            │
            ▼
    Learnings feed back ──────► next cycle

Every query — whether from a BI dashboard, a Spark ETL job, or an AI agent — feeds signal into this loop. The more the lake is used, the better it gets at organizing itself. This is the defining characteristic of an AI data lake: usage improves performance, rather than degrading it.

In a traditional lake, usage degrades performance — more queries means more load, more writes means more small files, more engines means more fragmentation. In an AI data lake, usage is signal. More queries means better sort-order decisions. More engines means richer routing data. More writes means more opportunities for the optimizer to learn and act.

AI Agents as Lake Citizens

AI agents introduce a fundamentally new workload pattern. They issue SQL iteratively — dozens of queries per reasoning step — at high frequency, without human review, with expectations of sub-second latency. A single customer-support agent answering one ticket can issue 30–50 SQL statements across user history, orders, product metadata, and support logs. Multiply that across hundreds of concurrent agent sessions and the workload profile looks nothing like the BI traffic your lake was designed for.

Read: Routing Multiple Query Engines with Iceberg

The Infrastructure Problem

Agents querying uncompacted tables pay a 5–10x latency penalty. When an agent runs 12 sequential queries in a reasoning loop, each 3-second query delay adds up to 36 seconds of wall-clock time for a single interaction. Users perceive this as the agent being broken. And unlike a human analyst who learns to avoid expensive queries, an agent in a loop repeats the same mistake indefinitely — compounding costs every iteration.

The LakeOps MCP: Agent-Native Connectivity

The Model Context Protocol (MCP) is the emerging standard for how AI agents interact with data infrastructure — filling the same role for agent-to-lakehouse communication that REST catalogs fill for engine-to-catalog communication.

Read: Agentic AI MCP for Iceberg

LakeOps exposes a native MCP server that any compatible agent — Claude, LangChain, LlamaIndex, or custom — can connect to with zero integration code. The MCP server provides four purpose-built tools rather than a generic "run anything" endpoint (agents that get a single execute_sql tool tend to skip schema discovery and hallucinate table names):

list_schemas — returns all available schemas and databases across connected engines. Agents call this before constructing queries to avoid guessing at namespace structures.
describe_table — returns column names, types, partition specs, sort orders, and optionally sample rows in a single call. Eliminates the multiple round-trips agents typically make to understand a table's shape.
execute_query — runs SQL through the full routing and guardrail pipeline, with optional engine_hint and enforced max_rows to prevent unbounded result sets from flooding the LLM context window.
explain_query — returns estimated cost, row count, and execution plan without running the query. Agents use this for pre-flight cost checks.

Each routing endpoint gets a stable URL — for example, storefront-analytics.lakeops.dev. Agents connect to that URL and inherit the full policy stack (guardrails, routing rules, cost limits) automatically. No per-agent configuration. Wire compatibility spans PostgreSQL, MySQL, and Arrow Flight SQL, plus async query support with SSE streaming so agents in tool-use loops can handle long-running queries gracefully.

Intelligent Guardrails

Unsupervised agents need enforcement at the query level, not at the application level. LakeOps provides a composable guard chain — every query passes through it regardless of which agent or frontend issued it:

ReadOnlyGuard — blocks DDL and DML from agent sessions using SQL parsing (not string matching), catching CTEs that wrap mutations and function calls with side effects.
RowLimitGuard — injects LIMIT N when a query lacks one. Agents in reasoning loops often forget to limit results, leading to multi-gigabyte result sets that overflow LLM context.
CostEstimateGuard — issues EXPLAIN before execution and rejects queries where estimated scanned bytes exceed a threshold. This is the primary defense against the cartesian-join and missing-WHERE patterns that agents frequently generate.
PIIMaskGuard — rewrites queries to exclude, hash, or null-out columns tagged as sensitive. PII never enters the LLM context window.
HumanApprovalGuard — pauses high-stakes queries and sends a webhook (Slack, email, or custom) for manual review.

Guards stack per routing group. A typical agent-facing configuration: read_only → row_limit → cost_estimate → pii_mask. A data engineering agent gets: cost_estimate → human_approval for DDL. Every guard action is recorded with full auditability — what fired, what was rewritten, what was rejected.

AI Solving the AI Problem

This is where the AI data lake concept becomes recursive. The same intelligence that optimizes the lake for human workloads also optimizes it for AI workloads — and agents produce particularly rich signal:

Higher query frequency — more data points for the adaptive optimizer per unit of time
Predictable patterns — roughly 80% of agent queries are repeated templates (same SQL structure, different literal values), making sort-order learning fast
Agent context propagation — every query carries structured metadata (agent_id, conversation_id, step_index, tool_call_id) that enables per-agent cost attribution, session replay for debugging, and workload-specific routing optimization

The feedback loop tightens: agents query the lake → telemetry captures their patterns → the AI reshapes data to match → agents get faster results → the agents produce even better signal → the cycle accelerates. Production deployments show agent query p95 latency dropping from 5–10 seconds to under 500ms as tables are compacted and sorted for observed access patterns, with per-query compute cost dropping 65% through intelligent routing.

AI-Native vs. AI-Washed: How to Tell the Difference

Not every product that claims "AI-powered" actually uses intelligence. Here is how to evaluate:

Signal	AI-Native	AI-Washed
Compaction	Learns sort order from cross-engine query patterns; adapts as patterns change	Fixed sort keys configured manually; schedule-based execution
Maintenance	Predicts degradation; sequences by dependency; learns from outcomes	Runs on cron; independent jobs; no feedback loop
Routing	Learns cost-performance per query shape; uses LLM for cold-start; adapts weights	Static rules; manual engine assignment
Observability	Correlates signals across layers; predicts issues; diagnoses root cause	Threshold alerts; dashboard without interpretation
Governance	Recommends policy changes from observed behavior; adapts inheritance	Static policies; manual updates
Feedback loop	Closed-loop: every query improves the system	Open-loop: no learning from outcomes
Agent support	Native MCP with intelligent guardrails and self-optimization	REST API bolt-on; no workload-aware optimization

The litmus test: does the system get better over time without human configuration changes? If yes, there is actual intelligence. If no, it is automation with a marketing label.

The Compounding Effect

The individual components are valuable. The compound effect is transformational.

AI-driven compaction reduces data scanned by 51%. That means every query across every engine benefits. Routing then shifts those improved queries to cheaper engines, because well-compacted tables do not need heavyweight compute. Storage costs drop because compaction reduces total file count and orphan cleanup reclaims unreferenced data. Observability improves because the AI has better signal to work with. Agents respond faster, which means fewer retries, lower token costs, and better AI application quality.

Each AI component amplifies the others. Better compaction → better routing → lower cost → more headroom for AI agents → richer signal → better compaction. The loop runs continuously.

Production numbers from LakeOps deployments:

80% reduction in compute and storage costs
12x faster queries after AI-driven compaction and sorting
95% faster compaction than Spark (221s vs. 1,612s on 200 GB)
90% cheaper compaction ($5/TB vs. $50/TB for Spark)
56% workload cost reduction from intelligent routing
100% of tables continuously monitored and maintained
~10 minutes to connect and start optimizing

These numbers compound because the system is a closed loop, not a collection of independent tools.

Where This Is Heading

The trajectory is clear. Data lakehouses are moving from manually operated infrastructure to self-managing systems where intelligence handles the operational layer end-to-end.

Today, LakeOps's AI handles compaction, maintenance sequencing, query routing, health prediction, and policy adaptation. The direction extends to:

Automatic schema evolution — the AI detects that a new column is consistently added to queries and recommends schema changes or materialized views
Predictive capacity planning — forecasting storage and compute needs based on workload growth trends and seasonal patterns
Cross-lake optimization — intelligence that spans multiple lakes, regions, and clouds, optimizing data placement and replication based on global access patterns
Agent-native data contracts — AI agents negotiate data freshness, latency, and cost SLAs directly with the control plane, and the system optimizes to meet them

The foundation for all of this is the closed-loop feedback system. A lake that learns from every query is a lake that can evolve to meet requirements that do not yet exist.

Summary

An AI data lake is not a data lake with an AI chatbot. It is a data lake where intelligence drives operations:

AI decides how to compact — learning from cross-engine query patterns, not static configuration
AI decides when to maintain — predicting degradation, sequencing by dependency, learning from outcomes
AI decides where to route — matching query shapes to engines using learned cost-performance models
AI diagnoses problems — correlating signals across storage, metadata, engines, and operations
AI adapts governance — recommending policy changes based on observed behavior
AI optimizes for agents — feeding agent telemetry back into the optimization loop

The closed-loop feedback system is the defining characteristic. Usage improves performance. Every query makes the lake smarter.

LakeOps is the autonomous control plane that implements this intelligence stack for Apache Iceberg — connecting to your existing catalogs, engines, and storage in minutes, with no data movement and no vendor lock-in.

Explore the platform · Read the docs · Get a demo

Managed Data Lake: A Guide for 2026

joni sar — Tue, 21 Jul 2026 11:32:48 +0000

There are two ways to run a data lake. You can manage every table by hand — writing Airflow DAGs, scheduling Spark compaction jobs, building monitoring dashboards, and staffing a platform team that grows linearly with table count. Or you can deploy a control plane that continuously observes every table in the lake, classifies health, sequences maintenance operations, optimizes data layout for real query patterns, enforces policies at lake scale, and adapts as workloads change — while your data stays in your storage account and your engines stay yours. This guide covers both strategies and explains why teams that start with the first keep arriving at the second.

The data lake format question is settled. Apache Iceberg is the default table format for the modern lakehouse. Every major engine — Spark, Trino, Snowflake, Athena, DuckDB, Flink, Databricks — reads and writes Iceberg natively. Adoption is near-universal among data platform teams building on open storage.

The operations question is not settled. Iceberg gives you ACID semantics, schema evolution, time travel, hidden partitioning, and partition evolution on your own object storage. What it does not give you is a system that keeps tables healthy. Compaction, snapshot expiration, orphan cleanup, manifest optimization, sort-order alignment, governance, and lake-wide observability are left to you — the official maintenance guide is explicit about that. At a handful of tables, scripts work. At hundreds of tables across multiple engines, catalogs, and write patterns, the lake quietly degrades into a swamp — and the engineering hours spent holding it together scale faster than the team.

"Managed data lake" is an overloaded term. Some products use it to mean a closed platform. Others mean one automated job. This guide defines what managed actually requires in 2026, then walks through two strategies to get there. First, the control-plane approach — using LakeOps, a dedicated autonomous control plane for Apache Iceberg that handles observability, compaction, snapshot lifecycle, orphan cleanup, manifest optimization, governance policies, multi-engine query routing, and agentic AI readiness across your existing stack without moving data or replacing engines. Then, the manual approach using open-source tooling and custom scripts. Both paths are real. The difference is in what scales.

What "managed" actually means in 2026

Iceberg on S3 is not a managed data lake. A nightly Spark rewrite_data_files job is not either. "Managed" means the operational contract is closed continuously, safely, and at lake scale — without engineering time growing linearly with table count.

An open data lake has four architectural layers:

Storage — S3, GCS, ADLS, or on-premises object storage. Commodity infrastructure you own.
Table format + catalog — Iceberg plus a catalog (Glue, Polaris, Nessie, Gravitino, Lakekeeper, S3 Tables, Unity Catalog). The structural contract every engine speaks.
Compute engines — Spark, Trino, Snowflake, Athena, DuckDB, Flink, Databricks, ClickHouse. Chosen per workload, not locked to a single vendor.
Operations control plane — the layer that observes table health, decides what maintenance to run, executes it safely, enforces policy, and learns from query patterns across engines. This is the layer that most lakes are missing.

Closed platforms like Databricks and Snowflake bundle the last three layers. Open lakes decouple them on purpose — you pick the best engine for each workload, keep data in your own storage, and avoid vendor lock-in. The trade-off: you inherit the operations that the closed platform handled for you. That trade-off is the subject of this guide.

In practice, a managed data lake means: tables stay compact and sorted for real query patterns. Snapshots and orphans do not inflate storage. Manifests stay lean enough that planning never dominates query time. Policies apply lake-wide and new tables inherit them automatically. Health is visible in one place. Correct results are the floor. Predictable cost and latency are the product.

Why data lakes degrade — the mechanics of silent failure

The failure mode is predictable, well-documented, and present in nearly every Iceberg lake that has been running for more than a few months without dedicated maintenance. The financial impact compounds from four directions at once: storage costs balloon from orphan files and stale snapshots, compute costs spike as engines scan thousands of small files instead of a few optimally-sized ones, query latency degrades as tables that returned results in seconds now take minutes, and engineering time evaporates as someone has to write, maintain, debug, and be on-call for the scripts that hold it all together. Understanding the mechanics is necessary for both strategies.

Small file accumulation. Streaming writers (Flink, Spark Structured Streaming, Kafka Connect Iceberg sink, CDC connectors) produce files sized by checkpoint throughput, not by optimal read size. A streaming pipeline at 1 commit per minute across 10 partitions creates approximately 14,400 files per day. Each query pays an object-store GET request, a Parquet footer parse, and task setup overhead per file. Cost and latency scale with file count, not with data volume.

Snapshot accumulation. Every Iceberg commit creates a new snapshot. A streaming table at 5-minute commits creates ~288 snapshots per day. Without expiration, snapshot counts reach tens of thousands within weeks. Query planning reads the snapshot chain; above ~1,000–2,000 retained snapshots, planning time degrades measurably — from sub-second to 30+ seconds. The mechanism is detailed in how Iceberg query planning works.

Manifest fragmentation. Every commit adds at least one manifest file to the manifest list. Streaming tables accumulate thousands of manifests per month. Each manifest must be read during planning; at 500+ manifests, planner latency alone can exceed multiple seconds. This is the layer most teams diagnose last because the symptom (slow EXPLAIN, slow first row) does not point at data files.

Orphan file growth. Failed writes, aborted Spark jobs, interrupted compaction, and schema-evolution retries leave data files on object storage that no snapshot references. These orphans are invisible to Iceberg but billable by the cloud provider. Production orphan sweeps routinely reclaim a significant share of billable storage on affected prefixes. A detailed breakdown of these nine degradation vectors and how to address each is in the Managed Iceberg in 2026 guide.

Delete file amplification. Merge-on-read tables accumulate position and equality delete files with every UPDATE or DELETE. Without compaction that physically applies deletes, every read must reconcile live rows against all accumulated delete files. Above a certain delete-to-data file ratio, reads pay measurable reconciliation tax and users feel it on every query.

Layout drift. Binpack compaction normalizes file sizes but does not sort data. Engines cannot skip irrelevant file groups because Parquet min/max statistics span wide ranges. A table compacted by size but not sorted by its most-filtered columns still forces full scans on common predicates. And the sort order that accelerated one workload six months ago may not match today's query patterns — layout needs to adapt as usage evolves.

Write amplification in metadata. Every Iceberg write — even a tiny append — produces a new metadata.json, a new manifest list, and one or more new manifest files. For a batch job running hourly, this overhead is invisible. For a streaming job committing every few seconds, metadata writing dominates I/O. Object storage starts throttling. Query planning suffers badly because every new metadata version adds to the chain the planner must traverse.

Governance entropy. Some tables have compaction configured; others do not. Some have snapshot retention; others use defaults that never expire. Retention windows are too aggressive for long-running queries, or too conservative for GDPR. No single view shows the health of the entire lake across catalogs and engines. Every new table starts from scratch unless someone remembers to configure it.

These failure modes do not trigger alarms. They compound silently over weeks and months — and the first real symptom is usually an analyst asking why their dashboard takes 40 seconds instead of 3. By the time someone investigates, the debt has accumulated across dozens of tables and unwinding it is a project, not a task.

Strategy 1: Smart automation with a control plane

A control plane is an architectural layer that sits between your storage, catalogs, and engines — observing the state of every table, understanding cross-engine query patterns, and applying the right maintenance at the right time. It does not replace anything in your stack. It adds the operational intelligence that open-source components do not provide on their own. Netflix built this layer in-house over years — Autotune, janitors, Metacat, Polaris. That story is in Intelligent Lakehouse: Build Like Netflix. Most teams cannot staff that.

LakeOps is a dedicated control plane for Apache Iceberg — the only product that exists purely as this ops layer. Not an engine, not storage, not a catalog. It connects to your existing catalogs (AWS Glue, Polaris, Nessie, Gravitino, Lakekeeper, S3 Tables) and query engines (Trino, Spark, Flink, Snowflake, Athena, DuckDB) without moving data or changing pipelines. Setup takes roughly ten minutes. Your data stays in your storage account.

What you get is a closed-loop system covering:

Lake-wide observability — continuous health scoring (Critical / Warning / Healthy) across every table and catalog, proactive insights ranked by severity, and cross-engine telemetry in one dashboard

Query-aware compaction — a purpose-built Rust/DataFusion engine that sorts data by the columns your queries actually filter on, orders of magnitude faster and cheaper than Spark-based compaction
Snapshot lifecycle management — automated, concurrency-safe expiration with configurable retention policies that respect active readers

Orphan file cleanup — safe detection and removal with age-threshold protection, sequenced after expiration

Manifest and metadata optimization — consolidation, position delete rewrites, and Puffin statistics computation, triggered automatically after compaction

Declarative policies and governance — compaction, retention, cleanup, and configuration rules scoped at organization, catalog, namespace, or table level with cascade inheritance

Multi-engine query routing — route queries across engines optimized for cost, latency, or throughput via QueryFlux

Agentic AI readiness — native MCP interface with schema discovery, layered guardrails (ReadOnly, CostEstimate, PIIMask, HumanApproval), and agent query telemetry feeding back into optimization

Layout simulations — preview compaction impact on Iceberg branches before committing to production

The operating model repeats continuously in four phases: connect and collect telemetry, classify table health and raise insights, execute maintenance in the correct sequence (expire snapshots → remove orphans → compact data files → rewrite manifests), and observe outcomes while enforcing governance. Each cycle the system learns — sort orders adapt as query patterns evolve, the compaction planner improves throughput on repeated runs, and policies self-enforce as new tables are created.

Observability and health classification

After connecting a catalog, every table is continuously scored and classified. The dashboard shows aggregate health across catalogs, recent operations, storage trends, and a summary of proactive insights. Platform teams can triage the entire lake in one screen without writing a single query.

Health is computed from the same Iceberg signals that matter in manual debugging: file count and size distribution per partition, manifest count and depth, snapshot accumulation, delete-file ratio, partition skew, and sort-order alignment with real query patterns. The difference is that the control plane evaluates these signals continuously across every table — not one table at a time when someone asks "why is this slow?"

Proactive insights go beyond classification to tell you why a table is degraded and what to do about it. Each insight is tied to a specific table, a severity level (CRITICAL, HIGH, WARNING, LOW), and a recommended action. In a manual workflow, you discover these conditions after a user files a ticket. With Insights, you discover them while they are still at Warning — before they escalate to a production incident. Details on the observability surface.

Intelligent compaction

Compaction is the highest-impact maintenance operation. LakeOps approaches it differently from generic Spark-based compaction in four ways.

Query-aware sorting. LakeOps tracks which columns appear in WHERE, JOIN, and GROUP BY clauses across every connected engine for every table. During compaction, data is physically sorted by those columns — so Parquet row-group statistics enable engines to skip irrelevant data without reading it. When query patterns change, the sort order adapts on subsequent compaction passes without manual configuration.

Rust/DataFusion engine. Compaction runs on a purpose-built engine written in Rust with Apache DataFusion. Arrow columnar buffers, bounded memory, zero garbage collection, lock-free parallelism, native Parquet I/O with predicate pushdown. Orders of magnitude faster and cheaper than Spark-based compaction on identical datasets. Partitions that OOM Spark complete in minutes because memory is bounded, not heap-dependent. Full benchmark details are in the compaction product page.

This cost reduction is what makes frequent compaction economically viable. Instead of one nightly job that lets streaming tables degrade for 24 hours between passes, LakeOps runs compaction multiple times per day when file health signals warrant it. Tables stay continuously healthy instead of oscillating between degradation and cleanup.

Conflict-aware execution. The system inspects active writer state and targets only cold partitions — partitions without active streaming appends. If a boundary-case conflict occurs, only the affected file group retries on the next cycle. No data loss, no full-job restarts, no CommitFailedException storms.

Self-improving planner. The compaction planner learns from workload telemetry across consecutive runs. Same table, zero config changes — throughput and runtime improve with each pass as the planner adapts its internal parallelism, file-group sizing, and sort strategy.

Snapshot, orphan, and manifest lifecycle

LakeOps automates the full maintenance lifecycle with correct sequencing:

Snapshot expiration. Policy-based retention (time and count) enforced continuously. Concurrency-safe — expiration respects active readers and in-flight queries, preventing the most common production failure in teams that automate snapshot expiration without accounting for concurrent readers. Version history, snapshot comparison, and one-click rollback remain available within the retention window.

Orphan cleanup. Age-threshold safety (7+ days by default) protects in-flight writes. Sequenced after expiration so newly dereferenced files are captured in the same sweep. Full audit trail of every removed file.

Manifest and metadata optimization. Three operations triggered automatically after compaction: manifest consolidation (fewer manifests = faster planning), position delete file rewrite (clean reads for merge-on-read tables), and Puffin statistics computation (column-level NDV, min/max, null counts for aggressive file-level pruning). Manifests are rewritten after compaction, not before, so the index reflects the final file set.

Every maintenance operation is logged lake-wide and per table with duration, impact, and status — the audit trail that answers "what ran, when, and did it succeed?" during incident response, without parsing executor logs across multiple systems.

Policies, routing, and AI readiness

Declarative policies scope compaction targets, snapshot retention, orphan cleanup, and manifest optimization at four levels with cascade inheritance: organization → catalog → namespace → table. Adaptive Maintenance bundles the full lifecycle into one data-driven policy — the system decides which operations each table needs from structural signals and runs them in the correct sequence. Cross-catalog enforcement works uniformly across AWS Glue, Polaris, Nessie, and Gravitino.

Multi-engine routing via QueryFlux connects all engines to a unified routing layer that makes per-query decisions based on cost, latency, table health, and historical patterns. Without routing, teams either standardize on one engine (losing optimization) or let each user choose (creating cost chaos). Routing decisions are enriched by table health — if a table is currently degraded, queries can be directed to engines that handle fragmentation better until compaction resolves the issue.

Agentic AI readiness provides a native MCP (Model Context Protocol) interface for AI agents with schema discovery, wire compatibility (PostgreSQL, MySQL, Arrow Flight), and layered guardrails configurable per agent session. Agent query telemetry feeds back into compaction and sort-order decisions — when agents start filtering on columns that no human dashboard uses, the compaction planner detects the new pattern and adjusts accordingly. Details in the AI agents guide.

The compound effect

The capabilities above are not independent features. They form a closed loop where each component's output feeds the next: observability detects degradation, the system runs expiration to unpin files, orphan cleanup captures newly dereferenced objects, compaction runs sorted by the columns that real queries filter on, manifests are rewritten against the final file set, Puffin statistics are recomputed, the table moves from Warning to Healthy, and the event is logged with before/after metrics. Next cycle, the compaction planner runs faster because it learned from the previous pass.

The engineering hours that used to go into writing, scheduling, monitoring, and debugging maintenance scripts are returned to product work. Policies are defined once and the system executes them continuously — every action logged, auditable, and reversible. Data stays in your storage account. No pipeline rewrites. Minutes to install.

Strategy 2: Manual management with open-source tooling

Not every team needs a control plane on day one. If you have a handful of Iceberg tables, one or two engines, batch-only writes, and a platform engineer who owns maintenance, manual management works. This section is the complete playbook — the same methodology that works whether you have a control plane or not.

The manual stack

The typical setup: an orchestrator (Airflow, Dagster, or Prefect) scheduling maintenance as DAG tasks, Spark (on EMR, Dataproc, or Kubernetes) running Iceberg maintenance procedures, a catalog (Glue, Polaris, Nessie), custom SQL queries against metadata tables piped into Grafana or Datadog, and threshold-based alerting. This stack is well-understood and runs at many organizations today. The constraint is not capability but coverage: every new table, every new write pattern, and every new engine requires incremental engineering work to bring under management.

Compaction by hand

Use Spark's rewrite_data_files procedure:

CALL catalog.system.rewrite_data_files(
  table => 'db.events',
  strategy => 'binpack',
  where => 'event_date >= current_date() - INTERVAL 7 DAYS',
  options => map(
    'target-file-size-bytes', '268435456',
    'min-input-files', '3',
    'partial-progress.enabled', 'true',
    'partial-progress.max-commits', '20',
    'max-concurrent-file-group-rewrites', '15'
  )
);

Key decisions: binpack for file-size normalization, sort for query-pattern optimization, zorder for multi-column predicates. Target 256–512 MB files. Always enable partial-progress. Never compact the active write partition on a streaming table — compacting the hot partition while Flink is appending is the primary cause of CommitFailedException storms. Run compaction on a separate cluster. Schedule cadence must match write cadence — if you write 14,400 files per day and compact once at midnight, the table spends 23 hours degraded. Decision frameworks are in the compaction strategies guide.

For sort compaction, you must determine sort columns yourself by analyzing query patterns across engines. When patterns change, you must notice and update the sort order manually. This is the gap a control plane closes automatically with cross-engine telemetry.

CALL catalog.system.rewrite_data_files(
  table => 'db.events',
  strategy => 'sort',
  sort_order => 'event_date ASC, region ASC, user_id ASC',
  where => 'event_date >= current_date() - INTERVAL 7 DAYS',
  options => map(
    'target-file-size-bytes', '268435456',
    'partial-progress.enabled', 'true',
    'partial-progress.max-commits', '20',
    'max-file-group-size-bytes', '10737418240'
  )
);

Snapshot expiration, orphan cleanup, and manifests by hand

Expire snapshots regularly. Retention window must exceed your longest-running query — a Spark job that runs for 3 hours will fail if its snapshot is expired mid-flight. This is one of the most common production failures in teams that automate aggressively.

CALL catalog.system.expire_snapshots(
  table => 'db.events',
  older_than => TIMESTAMP '2026-07-14 00:00:00',
  retain_last => 50
);

ALTER TABLE db.events SET TBLPROPERTIES (
  'write.metadata.delete-after-commit.enabled' = 'true',
  'write.metadata.previous-versions-max' = '100'
);

Orphan cleanup runs after expiration, never before. Always dry-run first. The older_than threshold must be far enough in the past to protect in-flight writes — production teams use 7+ days. Verify URI schemes match (s3:// vs s3a:// vs s3n://) before your first run.

CALL catalog.system.remove_orphan_files(
  table => 'db.events',
  older_than => TIMESTAMP '2026-07-07 00:00:00',
  dry_run => true
);

Rewrite manifests after every compaction cycle. This is the step most teams skip. Without rewriting, planning time remains elevated even though the data files are healthy. On extreme fragmentation, rewriting alone can drop planning from 30+ seconds to under a second. Full methodology in the orphan cleanup guide and the table health guide.

CALL catalog.system.rewrite_manifests(
  table => 'db.events'
);

Health monitoring and the safe sequence

Manual monitoring means querying Iceberg metadata tables and setting alert thresholds. Start with four numbers per table: file count and average size per partition, snapshot count, manifest count, and delete-file ratio for MOR/CDC tables.

SELECT partition, COUNT(*) AS file_count,
  ROUND(AVG(file_size_in_bytes) / 1048576, 1) AS avg_size_mb
FROM catalog.db.events.files
GROUP BY partition ORDER BY file_count DESC LIMIT 20;

SELECT COUNT(*) AS snapshot_count
FROM catalog.db.events.snapshots;

SELECT COUNT(*) AS manifest_count
FROM catalog.db.events.manifests;

The correct maintenance order — consistent across the official Iceberg docs and the automating table maintenance guide — is:

Expire snapshots — release metadata references to files nobody needs for time travel
Remove orphan files — delete unreferenced objects from storage (with 7+ day safety window)
Compact data files — merge small files and physically apply deletes
Rewrite manifests — consolidate the metadata index against the final file set
Compute statistics (optional) — refresh Puffin stats so planners see current NDVs

Compacting before expiration rewrites files about to be discarded — wasted compute. Orphan cleanup before expiration misses the largest reclaimable set. Manifest rewrite before compaction produces an index stale minutes later. Never invert the order under pressure.

Where manual management breaks down

Manual management does not fail because the tools are inadequate. Spark procedures work. Airflow can schedule them. Grafana can chart health. The breakdown happens at the system level.

Coverage gap. Every new table must be added to the DAG, configured, monitored, and validated. New tables slip through. Policies do not inherit automatically.

Workload blindness. Manual compaction sorts by columns you choose at configuration time. When a new dashboard goes live, when a new team starts querying with different filters, when AI agents start accessing columns nobody filtered before — the sort order becomes stale.

Sequence violations under pressure. At 3 AM during an incident, the instinct is to compact first. Compacting before expiration wastes compute. Running orphan cleanup with a too-short safety window corrupts data.

Concurrency danger. Snapshot expiration that does not respect active readers kills long-running queries. Compaction that targets active write partitions causes commit conflicts. Manual scripts do not know what other processes are doing.

Cost of execution. Spark compaction means spinning up JVM clusters, allocating executors, and paying for compute that is orders of magnitude more expensive than a purpose-built engine. Most teams run on over-provisioned clusters because under-provisioning causes failures and 2 AM pages.

Linear engineering scaling. Maintenance time grows linearly with table count. The team size does not. The data swamp to modern lakehouse guide walks through the transition in detail.

Fix the writers, not just the table

This section applies to both strategies. Many data lake incidents are writer-configuration debt — the pipeline creates structural damage faster than maintenance can repair it.

Checkpoint interval. The single most impactful setting for streaming Iceberg tables. Flink at 30-second checkpoints with 16 write tasks produces ~46,000 files per day per table. Move to 3–5 minute checkpoints and file creation drops dramatically. For staging tables, feature stores, and compliance archives, 10–15 minute intervals are often appropriate. Details in the Flink Iceberg optimization guide.

SET 'execution.checkpointing.interval' = '5min';
SET 'execution.checkpointing.mode' = 'EXACTLY_ONCE';
SET 'execution.checkpointing.timeout' = '10min';

Write distribution mode. Set write.distribution-mode=hash for partitioned streaming tables. Hash distribution shuffles records by partition key before writing, cutting file count by the parallelism factor.

ALTER TABLE catalog.db.events SET TBLPROPERTIES (
  'write.distribution-mode' = 'hash',
  'write.target-file-size-bytes' = '268435456'
);

Sink parallelism. Decouple source parallelism from sink parallelism. A Kafka topic with 64 partitions does not need 64 writer subtasks. Setting source at 64 and sink at 8–16 with hash distribution yields far fewer files while maintaining consumption throughput.

Partition grain. Over-partitioning is the leading cause of small files at scale. Prefer days(ts) over hours(ts). Use bucket(N, col) instead of identity transforms on high-cardinality columns. Decision frameworks in the partitioning best practices guide.

Delete mode vs mutation rate. Merge-on-read (MOR) is right for high-churn CDC where write latency matters more than read performance. Copy-on-write (COW) is right when read latency is sacred. Mixing MOR with no compaction policy is how delete-ratio incidents start.

Fixing writers halves the maintenance burden for both strategies. Skipping this step turns your control plane into a very expensive mop — and makes your manual scripts a full-time job.

Choosing your strategy

Both strategies work. The question is scale, cost, and where engineering time delivers the most value.

Strategy 2 (manual) fits when: you have fewer than ~50 tables, one or two engines, batch-only or low-frequency writes, a platform engineer who owns maintenance, and predictable query patterns that do not change frequently. The metadata queries and safe sequence in this guide are your on-call playbook.

Strategy 1 (control plane) fits when: table count crosses ~50, you run streaming or CDC pipelines, multiple engines hit the same tables, engineering time spent on maintenance scripts exceeds the cost of a purpose-built system, GDPR or compliance requirements need policy-based enforcement, or query patterns change and layout optimization must adapt without manual intervention.

The strategies compose. Many teams start with manual management, adopt a control plane for the highest-traffic catalogs first, and expand as value is proven.

The decision framework reduces to a few questions: Does the system see all my catalogs in one place? Is compaction query-aware across engines? Are operations coordinated in the correct sequence? Do policies inherit to new tables automatically? Does maintenance compete with my query clusters for compute? Can I simulate layouts before committing? Are agent guardrails first-class? Do I have to move data?

If most answers are "no" on an open multi-engine Iceberg stack, you do not have a managed data lake yet — you have Iceberg plus hope.

The migration path: manual to control plane

The transition from manual management to a control plane is incremental, not disruptive:

Connect a catalog. Discovery and health classification begin immediately without touching existing maintenance infrastructure.
Inspect and learn. Use the observability dashboard to understand the current state of your lake. Teams often discover tables they did not know were degraded.
Run maintenance on the worst tables. Trigger compaction, expiration, or orphan cleanup on the most Critical tables. Watch the before/after metrics.
Enable policies per namespace. Define compaction targets, retention windows, and cleanup thresholds. Every table in scope inherits automatically.
Enable Adaptive Maintenance for streaming tables. Let the system decide when to compact based on file health signals rather than cron schedules.
Retire manual scripts. As each namespace is covered by policies, disable the corresponding Airflow tasks.
Expand to remaining catalogs. Each step is independently valuable.

Teams running the control-plane model report far fewer Iceberg incidents — not because the runbook got faster, but because file explosions, manifest storms, and orphan cliffs are resolved while still at Warning. The real end state of a managed data lake is not faster debugging. It is rarely needing to debug at all.

The bottom line

A managed data lake in 2026 is not a storage product, not a single job, and not a platform you must adopt wholesale. It is an operational contract: tables stay healthy, costs stay predictable, queries stay fast, policies apply consistently, and AI agents get the data they need with appropriate guardrails. The question is who upholds that contract — your engineering team writing scripts, or a dedicated system that does it continuously.

Strategy 2 works. It is well-understood, uses open-source tooling, and scales to a moderate number of tables with dedicated platform engineering. The manual playbook in this guide — safe sequence, metadata queries, writer configuration, per-table monitoring — is the same playbook that production teams use every day.

Strategy 1 works better at scale. LakeOps closes the operational loop that manual scripts cannot: continuous health classification across every table and catalog, query-aware compaction on a purpose-built Rust engine, coordinated maintenance that respects the safe sequence, policies that inherit to every new table, cross-engine telemetry that adapts layout to real query patterns, multi-engine routing that sends each query to the right engine, and agentic AI readiness with layered guardrails. It runs on your existing stack without moving data or replacing engines.

The format already gave you ACID and time travel. The operations layer is what determines whether you have a data lake that works — or a data lake that worked once and has been quietly degrading since. Choose the strategy that matches your current scale. Build toward the one that matches where you are headed.

For deeper technical breakdowns: Managed Iceberg in 2026 covers the nine control-plane components. Intelligent Lakehouse: Build Like Netflix shows how Netflix solved these problems internally. Routing Multiple Query Engines with Iceberg covers the multi-engine routing architecture. Automating Table Maintenance walks through the progression from scripts to autonomous operations. Table Health and Maintenance covers every maintenance operation with configuration detail. And From Data Swamp to Modern Lakehouse covers the full architectural transition. For the platform that closes the loop, explore lakeops.dev.

Netflix Intelligent Lakehouse Solves Iceberg Maintenance — You Can Easily Too

joni sar — Fri, 29 May 2026 13:40:00 +0000

Every production Iceberg data lake eventually hits the same wall: tables that looked fast at 10 GB start crawling at 10 TB. Small files pile up from streaming ingestion, snapshots accumulate because nobody set expiration, orphaned data lingers from failed Spark jobs, and manifest lists grow until planning a simple SELECT takes longer than running it.

Netflix hit this wall years ago — and their solution shaped how the industry thinks about lakehouse architecture. At AWS re:Invent, their engineers walked through the ecosystem they assembled around Iceberg: Polaris for catalog management, Autotune for automated compaction, janitors for continuous cleanup, and Metacat for observability. The outcome was a 25% cost reduction and tables that stayed healthy without manual intervention.

But Netflix had something most teams don't: a dedicated platform organization building custom distributed services backed by CockroachDB, Kafka, and fleets of Spark clusters.

Today, a lakehouse control plane — just as good as Netflix's or better — is available for everyone to install on their Iceberg lakehouse.

The industry's favorite solution is LakeOps.

This article breaks down what actually makes a lakehouse "intelligent" — component by component — and shows how each piece maps to tooling that exists today.

Today with LakeOps, every team in the world is 10 minutes away from an intelligent lakehouse. And yes, it includes autonomous snapshot optimization as well as orphan files, metadata, manifests, and more.

The maintenance gap nobody talks about

Apache Iceberg solved the table format problem. Schema evolution, hidden partitioning, time travel, snapshot isolation — these features are why every major engine from Snowflake to DuckDB now speaks Iceberg natively.

What Iceberg intentionally left unsolved is who runs the maintenance. The format gives you powerful primitives. Keeping those primitives performing well at scale is your responsibility.

In practice, this creates a silent degradation cycle:

Streaming writes produce small files — a pipeline appending every 5 minutes to 100 partitions creates 100 new files per commit. After a week, some partitions contain thousands of sub-megabyte Parquet files.
Snapshots grow unbounded — without explicit expiration, every commit adds a snapshot. A table with hourly writes accumulates 8,760 snapshots per year, each referencing its own manifest list.
Orphan files accumulate — aborted Spark jobs, failed compaction runs, and expired snapshots leave behind data files that no snapshot references. These files cost storage but serve nothing.
Manifests fragment — as files are added and removed, the manifest layer becomes a web of small manifest files. Query planning reads every one of them before scanning a single data file.

The financial impact compounds from four directions: storage waste (orphans + snapshots), compute waste (scanning small files), metadata overhead (fragmented manifests), and engineering time (maintaining cron scripts that break silently).

Netflix's insight was that solving these problems one at a time, with isolated scripts, doesn't scale. You need an integrated system — a control plane that sees the full picture and acts on it continuously.

The six components of an intelligent lakehouse

Looking across Netflix's published architecture and detailed breakdowns of their Iceberg ecosystem, six capabilities separate an intelligent lakehouse from a collection of Iceberg tables with maintenance scripts taped to the side:

1. Universal catalog connectivity

Netflix built Polaris to replace the Hive Metastore with a catalog purpose-built for Iceberg — scalable, CockroachDB-backed, and supporting the Iceberg REST catalog specification for multi-engine access.

Most teams aren't replacing their catalog. They're running AWS Glue, or they adopted a REST catalog like Nessie or Lakekeeper early on, or they have tables spread across multiple catalogs in different regions.

An intelligent lakehouse connects to existing catalogs — Glue, DynamoDB, REST (Polaris, Nessie, Lakekeeper, Gravitino), S3 Tables, or custom implementations — discovers every namespace and table, and normalizes metadata into a single operational view. No catalog migration required.

LakeOps does exactly this: point it at your catalog credentials, and within minutes it inventories every table and starts collecting metadata signals.

2. Intelligent and efficient compaction that actually works

Netflix's Autotune watches for table write events through SQS and spins up Spark jobs to compact small files in the background. It's the core of their self-maintaining architecture.

The Spark-based approach works but carries significant overhead. You need provisioned compute clusters, JVM tuning, IAM roles for each cluster, and someone on-call for job failures. Spark compaction typically costs around $50 per TB processed.

A Rust-based alternative changes the economics entirely. LakeOps runs compaction with a native engine built on Apache DataFusion — no JVM, no cluster provisioning, no shuffle stages.

It reads Iceberg metadata, plans optimal merges, and writes compacted Parquet directly to your storage. Production benchmarks show roughly $5/TB — a 10x cost reduction over Spark.

On top, LakeOps runs compaction based on actual query patterns — so the way your files are organized is optimized to minimize I/O, cutting CPU costs as well as storage by up to 80% compared to Spark or S3 Tables.

It also coordinates all operations and events with Adaptive Maintenance to maximize results and cut time and costs. The sequence matters, and event- or trigger-driven ops are much smarter and more efficient than cron-based jobs.

Two strategies cover every workload:

Binpack — combines small files targeting optimal file sizes (~512 MB). Handles most tables well with minimal configuration.
Sort — reorders data by query-relevant columns so engines skip irrelevant row groups through predicate pushdown. Dramatic speedups for tables with clear access patterns.

Each table can be run manually first (configure → Execute → review results) and then switched to automated scheduling with a cron expression. No all-or-nothing commitment.

3. Metadata lifecycle automation

Netflix runs dedicated "janitor" services for orphan cleanup and snapshot expiration. Without them, their exabyte-scale lake would drown in stale metadata and unreferenced files.

The same operations — snapshot retention, orphan removal, manifest consolidation — need to run continuously on any production Iceberg lake. LakeOps provides all four as per-table operations with independent configuration:

Operation	What it does	Why it matters
Snapshot retention	Expires snapshots beyond a retention period, respecting min counts	Reclaims metadata, enables cleanup
Orphan file cleanup	Removes files unreferenced by any snapshot (with age threshold)	Recovers wasted storage
Manifest optimization	Consolidates fragmented manifests	Speeds up query planning
File compaction	Merges small files (Binpack or Sort)	Reduces scan overhead and S3 API costs

The execution order matters: expire first, then clean orphans, then compact, then consolidate manifests. Running them out of sequence wastes compute or risks removing files still in use.

When you want all four automated together, Adaptive Maintenance bundles them into a single data-driven policy that reacts to table activity — the closest equivalent to Netflix's integrated approach.

4. Full-stack observability without building a pipeline

Netflix's Metacat provides unified metadata access across all datasets, backed by Kafka event streams for real-time operational visibility. Building this took years and a dedicated team.

Out-of-the-box observability should include:

Table health classification — every table scored as Healthy, Warning, or Critical based on file counts, size distributions, snapshot accumulation, and metadata fragmentation
AI-generated insights — ranked recommendations that flag small-file hotspots, excessive snapshots, and missing retention before they become incidents
Event audit trail — every maintenance operation recorded with before/after metrics, timestamps, and status — per-table or lake-wide, filterable by catalog and operation type
Dashboard — total operations, query speed gains, cost savings, and resource reduction in a single view

And table health dashboards:

The difference from building it yourself is time-to-value: connect a catalog and immediately see what's degraded, what's wasting money, and what to fix first.

5. Policy-driven governance that scales with the lake

Configuring maintenance table-by-table stops working somewhere between 50 and 100 tables. Netflix needed organization-wide rules; so does every team that's past the proof-of-concept stage.

A policy engine lets you define maintenance rules at the catalog or namespace level — snapshot retention every hour, orphan cleanup daily, compaction at 2 AM — and every table in scope inherits them automatically. New tables that appear in a governed catalog get the right configuration without anyone touching them.

Two policy categories cover the ground:

Maintenance policies — schedule and configure any operation (or all of them via Adaptive Maintenance) across a scope
Configuration policies — enforce table settings like Iceberg format version, file format, and write distribution mode

Per-table overrides always take precedence, so you set sensible defaults broadly and customize only where needed.

6. Multi-engine query routing

Netflix connects engines through the REST catalog endpoint, but routing decisions — which engine handles which query — remain manual architecture choices in most organizations.

An intelligent routing layer dispatches queries to the best engine based on the workload:

Cost-optimized — sends queries to the cheapest engine that meets your latency SLA
Latency-optimized — picks the fastest engine for the query shape
Throughput-optimized — distributes load for maximum concurrency

Applications connect to a single SQL endpoint (Postgres wire, MySQL wire, or Arrow Flight). When an engine goes down, failover reroutes automatically. When you add or remove engines, application code doesn't change.

LakeOps handles this through QueryFlux, an open-source Rust SQL proxy that translates SQL dialects with sqlglot and adds ~0.35ms of overhead.

What this adds up to

Each component is useful independently. Together, they compound:

Storage costs drop 40–55% from continuous orphan removal, bounded snapshots, and compaction
Compute costs drop up to 75% from Rust-native compaction replacing Spark clusters, sort-order optimization reducing scan volume, and routing hitting the cheapest viable engine
Query latency improves up to 12x through optimized file sizes, sorted layouts, consolidated manifests, and Puffin column statistics
Engineering hours shift from maintaining scripts and debugging overnight failures to building data products

Beyond human users: AI agent access

The next layer of intelligence is enabling AI agents to interact with lakehouse data programmatically. LakeOps provides an MCP (Model Context Protocol) interface that gives agents structured access — table discovery, SQL execution through the routing layer, column statistics without scanning, and maintenance triggers — all within configurable guardrails.

You can enforce read-only access, row limits, PII masking, cost caps, and human approval per agent. As agent usage grows, their query telemetry feeds back into compaction decisions — tables agents query most get optimized first, with sort orders aligned to the predicates agents actually use. The lake self-optimizes as AI adoption scales.

Getting started

Netflix took years to build their intelligent lakehouse with dedicated teams.

The same architecture is now accessible in about ten minutes.

Visit lakeops.dev:

Connect your catalog — Glue, DynamoDB, REST, S3 Tables, or Custom. Every table is discovered automatically.
Optimize a few tables — run compaction or snapshot expiration manually, review results, then flip to automated scheduling.
Scale with policies — define rules at the catalog or namespace level. New tables inherit everything.
Monitor — the dashboard shows real-time impact, insights flag what needs attention, events provide the audit trail.

Your data never leaves your account. No agents to install, no pipelines to change, no infrastructure to provision.

The intelligent lakehouse is no longer reserved for companies that can build Netflix-scale infrastructure. The building blocks are here. The question is whether your tables are maintained — or quietly degrading while you read this.

Managed Iceberg: Optimizing a Modern Lakehouse

joni sar — Sun, 10 May 2026 13:52:17 +0000

A modern lakehouse looks simple from the outside.

Data lands in object storage. Apache Iceberg gives you tables, snapshots, schema evolution, time travel, and multi-engine access. Spark writes. Trino queries. Flink streams. Snowflake or Athena may read the same data. Everyone is happy.

Then the lakehouse starts growing.

Small files pile up. Snapshots never expire. Manifest metadata gets heavier. Delete files slow down reads. Failed jobs leave orphan files behind. Query planning becomes slower. Storage cost grows in places nobody is watching. Every engine has its own behavior, its own tuning, and its own operational gaps.

This is the part that gets underestimated.

Iceberg solves the table format problem. It does not magically solve lakehouse operations.

The same pattern already happened in compute infrastructure: once systems grew large enough, manual tuning stopped scaling and platforms like LakeOps became useful because they continuously optimized resources instead of relying on people to chase every inefficiency.

Iceberg lakehouses need the same shift. Not more scripts. Not more periodic cleanup jobs. A real control plane.

That is the idea behind LakeOps: autonomous lakehouse management for Apache Iceberg. It sits above your lake, watches table and engine behavior, and continuously manages the operational work that keeps Iceberg fast, clean, and cost-efficient.

This article is a practical guide to what that means.

The real job of running an Iceberg lakehouse

When teams first adopt Iceberg, the focus is usually on features.

ACID transactions on object storage.

Time travel.

Schema evolution.

Partition evolution.

Hidden partitioning.

Multiple engines reading the same tables.

Those are important. They are why Iceberg became popular.

But after the first production workloads move in, the job changes. You are no longer just “using Iceberg.” You are operating a lakehouse.

That means you are responsible for table layout, file size, metadata growth, snapshot retention, stale data, query planning, engine behavior, storage waste, and workload safety.

A healthy lakehouse needs constant maintenance.

A production Iceberg table is not static. Every ingest, append, merge, delete, compaction, schema change, or streaming write changes its physical and metadata shape. Even when the logical table looks clean, the underlying table may be drifting.

That drift is the problem.

A table can still return correct results while becoming slower and more expensive every week.

Where lakehouse maintenance gets painful

The pain usually appears in a few predictable places.

Small files are the first one. Streaming jobs, CDC pipelines, frequent appends, and micro-batches create many small files. Query engines then spend too much time opening files, planning splits, and scanning inefficiently. A table that should be read in seconds starts feeling heavy.

Snapshots are the second. Iceberg creates snapshots so readers can see consistent table versions and users can time travel or roll back. That is useful, but old snapshots accumulate unless someone expires them. Over time, they keep metadata and old data references alive.

Manifests are the third. Iceberg tracks data files through metadata files. That is what makes Iceberg reliable and engine-independent, but metadata also needs maintenance. If manifests grow or fragment, query planning slows down before the engine even starts scanning data.

Orphan files are the fourth. Failed writes, aborted jobs, migrations, dropped tables, and imperfect cleanup flows can leave files in object storage that are no longer referenced by Iceberg metadata. Queries do not read them, but storage still bills for them.

Delete files are another common issue, especially with merge-on-read workloads. If they accumulate, every query may pay the cost of applying deletes at read time.

Then there is the engine layer. Spark, Trino, Flink, Athena, Snowflake, Databricks, DuckDB, and other engines do not behave the same way. They have different cost models, latency profiles, concurrency limits, and operational strengths. Managing Iceberg across engines is not the same as managing one Spark pipeline.

This is why “we have Iceberg” is not the same as “we have a managed lakehouse.”

The manual way most teams start

Most teams start with scripts.

A Spark job for compaction.

A scheduled job for snapshot expiration.

A cleanup script for orphan files.

A few dashboards.

Some alerts.

A runbook in a wiki.

A Slack channel where someone asks, “Why is this table slow again?”

There is nothing wrong with this as a starting point. It is how most platforms mature.

The problem is that static maintenance does not understand table behavior.

A daily compaction job does not know whether a table had a quiet day or a massive ingestion spike.

A weekly snapshot cleanup job does not know whether the table has long-running readers, branch retention requirements, or compliance rules.

A manifest rewrite schedule does not know which tables are suffering from planning latency.

A generic orphan cleanup script may be too conservative to reclaim meaningful storage or too aggressive to be safe.

And none of this naturally connects to cost, query performance, engine behavior, or table-level business importance.

Manual maintenance works when you have a small number of tables and a small number of workloads. At lakehouse scale, it becomes operational debt.

What managed Iceberg means

Managed Iceberg means the maintenance loop becomes part of the platform.

Not a one-off script.

Not a quarterly cleanup project.

Not a few jobs someone hopes are still running.

A managed Iceberg layer continuously observes the lakehouse, decides what needs attention, runs the right operation, and records the result.

It should manage the core lifecycle of Iceberg tables:

Compaction.

Snapshot expiration.

Manifest optimization.

Orphan file cleanup.

Delete file handling.

Statistics and metadata optimization.

Table health monitoring.

Policy enforcement.

Engine visibility.

Cost and performance tracking.

The key point is that management should be table-aware and workload-aware.

A hot BI table is not the same as a streaming staging table. A CDC table is not the same as a cold archive table. A table queried by Trino all day is not the same as a table used by a nightly Spark job. A table exposed to AI agents has different risk and cost patterns than an internal batch table.

A managed lakehouse should understand those differences.

The control plane model

A lakehouse control plane is the layer that coordinates operations across storage, Iceberg metadata, catalogs, engines, policies, and observability.

and

It does not replace Iceberg.

It does not replace your object storage.

It does not force all teams into one query engine.

It gives you one operating layer for the lakehouse.

LakeOps describes this as a control plane for your data lake: end-to-end optimization for tables and metadata across storage and query engines, with telemetry-driven orchestration and visibility in one place.

That distinction matters.

The goal is not to make Iceberg proprietary. The goal is to make open lakehouse operations manageable.

A good control plane should answer questions like:

Which tables are unhealthy?

Which tables are wasting the most storage?

Which tables have the worst small-file problem?

Which tables have metadata planning issues?

Which tables should be compacted now?

Which tables should not be touched because active workloads are running?

Which compaction strategy should be used?

Which snapshots can safely expire?

Which files are safe to delete?

Which engine is best for this workload?

Did the optimization actually improve cost or performance?

If you cannot answer these questions quickly, the lakehouse is being managed manually, even if it has automation scripts.

Solving the small-file problem

Small files are the most visible Iceberg maintenance issue.

They usually come from streaming ingestion, frequent appends, CDC, micro-batches, partition skew, and multi-writer workloads. The result is predictable: more files, more metadata, more object-store requests, more planning work, and slower queries.

The normal fix is compaction.

But not all compaction is equal.

The simple version is bin-packing: combine many small files into fewer larger files. This is often the right first step because it quickly reduces file count and improves scan efficiency.

The more advanced version is sort-based compaction: rewrite files according to the columns that queries filter or join on most often. This can improve data skipping and reduce scanned data, but it is more workload-sensitive. Sorting everything blindly can waste compute.

This is where autonomous management becomes useful.

LakeOps includes compaction for Apache Iceberg that uses table metadata and query patterns to decide which files to rewrite and how. The useful part is not only that it compacts. The useful part is that compaction becomes part of a continuous feedback loop.

A practical operating model looks like this:

Start with bin-pack compaction on tables with severe small-file pressure.

Use query-aware sort compaction only where query patterns justify it.

Avoid compacting cold tables just because a schedule says so.

Prioritize tables where compaction will reduce real query cost or latency.

Track before-and-after impact: file count, data scanned, planning time, runtime, and cost.

That is the difference between maintenance and optimization.

Managing snapshots safely

Snapshots are one of the best things about Iceberg.

They enable time travel, rollback, auditability, and consistent reads. But they also create retention work.

Every write creates a new table version. If snapshots are never expired, metadata grows and old data can remain retained longer than needed. On busy tables, this becomes a real cost and performance issue.

The hard part is not running expire_snapshots.

The hard part is knowing the right policy.

Some tables need long time-travel windows because they support audits, debugging, or recovery. Some tables only need short retention. Some tables may have branches or tags that must be protected. Some workloads may have long-running readers. Some environments need different rules for production, staging, and development.

A managed layer should make this explicit.

LakeOps provides snapshot management as part of table optimization, so retention can be controlled through policies rather than remembered manually per table.

For platform teams, this is a major shift.

Instead of asking, “Did someone remember to clean up snapshots on this table?”

You define retention behavior once, apply it at the right scope, and let the platform enforce it continuously.

A good default might be:

Keep enough snapshots for rollback and debugging.

Retain a minimum number of recent snapshots.

Use longer retention for critical regulated tables.

Use shorter retention for temporary or staging data.

Monitor how much storage is blocked by old snapshots.

Run expiration before orphan cleanup.

The exact values depend on the organization. The important thing is that snapshot retention becomes intentional.

Keeping metadata lean with manifest optimization

Iceberg query performance is not only about data files.

It is also about planning.

Before an engine scans data, it reads Iceberg metadata to understand which files belong to the snapshot and which files can be skipped. Manifest files are part of this metadata layer. They are essential, but they can also become fragmented over time.

When manifests grow poorly, planning time grows. Users experience this as “the query is slow,” but the engine may be spending too much time before meaningful scanning even begins.

This is easy to miss if you only look at execution time.

A managed Iceberg system should monitor metadata health directly.

LakeOps includes manifest optimization so teams can consolidate and optimize metadata as part of the same table health loop.

The principle is simple: metadata is part of performance.

If you only compact data files but ignore manifests, you are only managing half the table.

Cleaning orphan files without breaking things

Orphan files are a storage leak.

They sit in object storage but are not referenced by Iceberg metadata. Queries do not use them, but the cloud provider still charges for them.

They can appear after failed jobs, aborted commits, manual migrations, dropped tables, incorrect cleanup flows, or maintenance operations that leave old data behind.

The dangerous part is cleanup.

Deleting files from a data lake is easy. Deleting the right files safely is hard.

A safe orphan cleanup process must compare files in storage against Iceberg metadata, apply a conservative age threshold, avoid active write windows, and usually run after snapshot expiration. The age threshold matters because a file that looks unreferenced during an in-progress write may still be committed later.

LakeOps documents orphan file cleanup as a managed operation with metadata awareness and safety controls.

In practice, this is one of the strongest arguments for a control plane.

Nobody wants platform engineers manually reviewing millions of object-store paths. Nobody wants an unsafe script deleting files from production. And nobody wants to keep paying for dead data because cleanup feels risky.

Managed orphan cleanup should be boring, visible, and conservative.

Run a dry run.

Show candidates.

Apply retention thresholds.

Delete only when safe.

Record what was removed.

Measure storage reclaimed.

That is how cleanup becomes an operational capability instead of a dangerous maintenance task.

Policies are what make this scale

Manual tuning does not scale across hundreds or thousands of tables.

You need policies.

Policies let you define how table maintenance should behave at different scopes: organization, catalog, namespace, table, environment, or workload class.

For example:

Production BI tables get frequent compaction and manifest optimization.

Streaming tables get aggressive small-file management.

CDC tables get delete-file-aware compaction.

Staging tables get short snapshot retention.

Archive tables get minimal compute-heavy optimization but regular storage cleanup.

Critical tables require approvals or simulations before major rewrites.

Development tables get cheaper, more aggressive cleanup.

LakeOps supports policies for maintenance automation, allowing teams to define behavior for compaction, snapshots, manifests, orphan cleanup, and governance across the lakehouse.

This is important because the platform team should not be in the business of hand-tuning every table forever.

Good policies give teams defaults, guardrails, and exceptions.

That is how Iceberg operations become manageable.

Observability turns maintenance into engineering

If maintenance runs but nobody can measure the effect, it is not really managed.

A lakehouse control plane needs observability at the table, engine, and operation level.

You need to see:

Table health.

File count.

Average file size.

Small-file pressure.

Snapshot count.

Manifest count.

Delete file pressure.

Storage waste.

Query latency.

Planning time.

Data scanned.

Engine cost.

Operation history.

Before-and-after optimization impact.

Failed or skipped operations.

Policy coverage.

LakeOps includes lakehouse observability so platform teams can see table health, engine metrics, cross-system telemetry, and maintenance history from one place.

This changes how you operate.

Instead of waiting for users to complain that dashboards are slow, you can see which tables are drifting.

Instead of guessing whether compaction helped, you can measure the before and after.

Instead of discovering storage waste in a cloud bill, you can identify stale data and orphan files directly.

Good observability turns Iceberg maintenance from reactive firefighting into normal platform engineering.

Multi-engine lakehouses need engine-aware management

The whole point of Iceberg is that many engines can work over the same tables.

That is also what makes operations harder.

Spark may be good for heavy rewrites.

Trino may be better for interactive analytics.

Athena may be useful for serverless access.

Snowflake may serve BI workloads.

Flink may write continuously.

DuckDB may support local or embedded analytics.

Each engine has a different performance model. Each workload has a different latency and cost profile.

A managed lakehouse should not pretend all engines are the same.

LakeOps supports engine management and query routing, giving teams a unified view of engine health, cost, usage, and routing behavior.

This matters because optimization is not only about the table. It is also about where and how workloads run.

Sometimes the best optimization is a better file layout.

Sometimes it is a better engine choice.

Sometimes it is avoiding an expensive engine for simple queries.

Sometimes it is routing a workload away from an unhealthy engine.

A modern lakehouse control plane should see the full system, not just the storage layer.

Why continuous optimization beats scheduled maintenance

The old model is schedule-based.

Run compaction every night.

Expire snapshots every Sunday.

Clean orphan files once a month.

Rewrite manifests when someone remembers.

That is better than nothing, but it is not how real workloads behave.

A high-volume table may need attention multiple times a day.

A cold table may not need compaction for months.

A table may become hot because a new dashboard launched.

A backfill may create temporary file pressure.

A failed ingestion job may create orphan files.

A new AI agent may generate many new query patterns.

A static schedule cannot react to that.

Continuous optimization uses telemetry to decide what should happen next.

That is the core value of autonomous lakehouse management.

The lakehouse is not optimized because a cron job ran. It is optimized because the platform understands table state, workload behavior, and cost impact.

This is where LakeOps is useful in practice. It continuously analyzes the lakehouse, recommends or runs the right operation, and keeps optimizing as workloads change.

For a platform team, this reduces the amount of manual judgment required for routine operations.

You still set policies.

You still define guardrails.

You still decide what level of autonomy is acceptable.

But you are no longer manually chasing every unhealthy table.

A practical rollout plan

The best way to adopt managed Iceberg is not to turn everything on everywhere.

Start with visibility.

Connect the catalogs and engines. Let the platform observe table health, file layout, metadata, snapshots, and query behavior. Identify the worst tables by cost, latency, file count, metadata weight, and storage waste.

Then start with a small set of high-impact tables.

Good candidates are usually:

Large tables with many small files.

Hot BI tables with growing latency.

Streaming or CDC tables with constant writes.

Tables with high object-store request cost.

Tables with many snapshots.

Tables where users already complain about performance.

Apply conservative policies first.

Use bin-pack compaction before sort compaction.

Use dry runs for cleanup operations.

Set safe snapshot retention.

Run orphan cleanup with conservative age thresholds.

Measure everything.

Only after you see stable improvements should you expand to more tables, more aggressive compaction, sort optimization, and autonomous mode.

The point is not to give control away. The point is to move from manual table-by-table work to policy-driven operations with visibility.

What LakeOps solves, problem by problem

If you maintain Iceberg yourself, you eventually build pieces of a control plane internally.

You build table health checks.

You build compaction jobs.

You build snapshot cleanup.

You build orphan cleanup.

You build dashboards.

You build job orchestration.

You build policy conventions.

You build alerts.

You build runbooks.

You build engine-specific scripts.

You build cost reports.

Then you maintain all of that.

LakeOps packages that operating layer into one platform.

For small files, it provides autonomous compaction and layout optimization.

For slow queries, it optimizes file sizes, sort layout, manifests, and routing decisions.

For snapshot bloat, it manages retention policies.

For orphan files, it performs safe metadata-aware cleanup.

For fragmented metadata, it rewrites and optimizes manifests.

For multi-engine complexity, it gives one view of engines and can route workloads based on cost, latency, or throughput.

For operational visibility, it surfaces table health, engine metrics, events, recommendations, and optimization history.

For governance, it gives policies, auditability, and controlled automation.

For adoption risk, it works with the existing lakehouse stack instead of requiring pipeline rewrites or data movement.

That last point is important.

A control plane should reduce operational burden without becoming a migration project.

What to keep managing yourself

Autonomous management does not mean the platform team disappears.

You still own architecture.

You still own data modeling.

You still decide retention requirements.

You still define governance boundaries.

You still choose which engines belong in the platform.

You still control policies and exceptions.

You still review critical workloads.

The difference is where your time goes.

Instead of manually compacting tables, you define compaction policies.

Instead of hunting stale files, you monitor cleanup impact.

Instead of guessing why queries slowed down, you inspect table and engine telemetry.

Instead of writing one-off Spark jobs, you operate the lakehouse as a managed platform.

That is a better use of senior data platform engineering time.

The benefits beyond cost and performance

Cost and performance are the obvious wins.

Fewer small files means less scan overhead.

Cleaner metadata means faster planning.

Expired snapshots and orphan cleanup reduce storage waste.

Better layout reduces data scanned.

Better routing reduces unnecessary compute.

But there are other benefits that matter just as much.

Reliability improves because maintenance is consistent instead of ad hoc.

Governance improves because policies are explicit.

Debugging improves because every operation is visible.

Onboarding improves because new tables inherit sane defaults.

Security improves when access and actions are auditable.

Capacity planning improves because table growth and engine behavior are observable.

AI readiness improves because agents query cleaner, faster, better-governed tables.

Team focus improves because engineers stop spending so much time on repetitive maintenance.

These benefits compound.

A lakehouse that is continuously maintained becomes easier to trust.

Common mistakes when managing Iceberg manually

The first mistake is treating compaction as the whole problem. Compaction is important, but it does not replace snapshot expiration, manifest optimization, orphan cleanup, delete-file handling, or observability.

The second mistake is applying the same policy to every table. A staging table and a production revenue table should not have the same retention and optimization strategy.

The third mistake is running cleanup without a safety model. Orphan cleanup especially needs conservative thresholds and visibility.

The fourth mistake is ignoring metadata. Data files get attention because they are visible, but manifests and snapshots often explain planning latency and storage drift.

The fifth mistake is optimizing for one engine while the table is used by many engines.

The sixth mistake is not measuring impact. If you cannot show what changed after maintenance, you cannot tune the lakehouse intelligently.

The seventh mistake is waiting for incidents. Iceberg degradation is often gradual. By the time users complain, the table may have been unhealthy for weeks.

The target operating model

A modern Iceberg lakehouse should operate like this:

Tables are continuously monitored.

Health is measured at the file, metadata, snapshot, storage, and query level.

Policies define maintenance behavior.

Compaction runs when it has measurable value.

Snapshot expiration follows retention rules.

Orphan cleanup is safe and auditable.

Manifest optimization keeps planning fast.

Engine behavior is visible.

Query routing can account for cost and latency.

Optimization history is recorded.

Engineers can override, approve, or inspect operations.

The system improves continuously.

That is managed Iceberg.

Not a hosted table format.

Not a black box.

Not a replacement for engineering judgment.

A control plane that takes the repetitive, error-prone, high-volume operational work and turns it into policy-driven automation.

Final thoughts

Iceberg is a strong foundation for the modern lakehouse, but it is not the whole platform.

Once Iceberg becomes production infrastructure, the work shifts from “how do we create tables?” to “how do we keep hundreds or thousands of tables healthy while many engines and workloads use them?”

That is where many teams feel the pain.

Manual maintenance is fine at the beginning. Scripts are fine at the beginning. But as the lakehouse grows, entropy wins unless something is continuously managing the system.

Managed Iceberg is the next layer.

It means compaction, snapshots, manifests, orphan files, engines, policies, observability, and cost optimization are handled as one operating system for the lakehouse.

LakeOps is built around that idea: autonomous lakehouse management for Apache Iceberg, running on top of the stack teams already use.

For data platform engineers, the value is simple.

You keep Iceberg open.

You keep your storage.

You keep your engines.

You keep control.

But you stop managing the lakehouse one table, one script, and one incident at a time. Thanks for reading! :)

Managed Iceberg Data Lakes: A Guide

joni sar — Thu, 07 May 2026 16:47:06 +0000

Apache Iceberg has become the default table format for open data lakes. The 2025 State of the Apache Iceberg Ecosystem survey found 96.4% Spark adoption, 60.7% Trino, and growing DuckDB and Flink usage. Ryft's 2026 enterprise study reports that 58% of organizations now use Iceberg for business-critical analytics, and 79% plan to move their remaining data to it within 12 months.

Adoption is no longer the question. The question is: who maintains all of this?

Iceberg gives you snapshot isolation, schema evolution, hidden partitioning, and time travel. It does not give you someone to compact your files, expire your snapshots, clean up orphans, rewrite your manifests, or tell you which of your 800 tables is about to make your morning dashboards unusable. That is your job — and at scale, it is a job that breaks.

This guide covers what it actually takes to run an Iceberg data lake in production: the maintenance operations, the failure modes, and how LakeOps — an autonomous control plane for Apache Iceberg — addresses each of them.

What "managed" means in the Iceberg context

The word "managed" gets overloaded. In the Iceberg world, it refers to the ongoing operational work required to keep tables healthy after data is written. Iceberg handles transactional correctness — atomic commits, snapshot isolation, optimistic concurrency control. But it intentionally leaves maintenance to the operator. The format's spec defines what snapshots, manifests, and data files are. It does not define when they should be cleaned up, how files should be reorganized, or which tables need attention first.

A managed Iceberg data lake is one where these operations are handled continuously, automatically, and with awareness of the broader system — not just individual tables.

There are roughly six categories of maintenance work:

Compaction — merging small files into optimally-sized ones
Snapshot lifecycle management — expiring old snapshots and reclaiming storage
Manifest maintenance — rewriting manifests, consolidating position deletes, refreshing statistics
Orphan file cleanup — removing data files that are no longer referenced by any snapshot
Observability — knowing which tables are healthy, which are degrading, and why
Policy and governance — enforcing consistent maintenance rules across catalogs, namespaces, and teams

Each of these is straightforward in isolation. The difficulty is doing all of them, for every table, at the right frequency, in the right order, without breaking anything. Most teams start with custom Spark scripts scheduled in Airflow. That works at 10 tables. At 200 tables across multiple catalogs and engines, the scripts become the problem — brittle, uncoordinated, and blind to the interactions between operations.

LakeOps treats these six categories as a single coordinated system. You connect your catalog and storage — typically in under 10 minutes, with no agents installed, no data movement, and no pipeline changes — and the platform continuously manages all six across your entire fleet.

The small file problem and why compaction matters

Every Iceberg write produces new data files. Streaming pipelines that commit every few seconds can generate thousands of files per hour. Batch jobs with high partition cardinality scatter data across many small files. Even well-designed pipelines accumulate file fragmentation over time as tables evolve.

The consequences compound:

Query planning overhead: the query engine must open and parse every manifest entry to build a scan plan. More files means more manifest entries means slower planning. A table with 100,000+ files can push planning time from milliseconds to tens of seconds before a single byte of data is read.
I/O amplification: each file requires a separate object storage GET request. Object stores are optimized for throughput on large sequential reads, not for opening thousands of small files. A query that should scan 10 GB across 20 files instead scans the same 10 GB across 2,000 files — same data volume, dramatically worse latency.
Metadata bloat: more files means larger manifests. Larger manifests means more data for the coordinator to load into memory. At extreme scale, this can cause OOM failures during query planning.

The standard solution is compaction: periodically rewriting groups of small files into fewer, larger files targeting 256–512 MB each. Iceberg supports this natively through the RewriteDataFiles action in Spark, or through engine-specific SQL commands. But how you compact — the engine, the strategy, the awareness of actual query patterns — makes a dramatic difference.

How LakeOps handles compaction

LakeOps replaces Spark-based compaction with a purpose-built Rust engine built on Apache Arrow and DataFusion. The difference is architectural: no JVM, no garbage collection pauses, bounded memory, vectorized execution. In production benchmarks on a 5.5 TB dataset across 10 tables, the engine achieved up to 99.8% file reduction and peak throughput of 2,522 MB/s — completing jobs that caused Spark OOM on identical hardware.

But raw speed is only part of it. LakeOps compaction is query-aware: it analyzes actual query patterns against each table to determine which file groups to prioritize and which sort orders will produce the best scan pruning. This means compaction cycles directly reduce I/O for the queries your team actually runs, rather than blindly reorganizing data no one reads.

Two strategies are available:

Binpack consolidates small files into optimally-sized files (~512 MB) without changing sort order. It is the default, and it resolves the small file problem with minimal compute.
Sort compaction rewrites data in a column order derived from query filter and join patterns — enabling Iceberg's min/max metadata pruning to skip entire files. On TPC-H benchmarks, sorted layouts reduced bytes scanned by 51% with an additional ~9% compression improvement.

LakeOps also supports branch-based simulations: you can test a layout change on an Iceberg branch, see the projected impact on scan volume and query latency, and promote only if the results justify it — without touching the production snapshot.

Snapshot lifecycle management

Every Iceberg commit creates a new snapshot. Snapshots enable time travel, but they are not free:

Each snapshot references a manifest list, which references manifests, which reference data files. Old snapshots keep references alive to data files that may have been logically deleted or replaced by compaction.
A table with 120 days of hourly commits accumulates ~2,880 snapshots. Each snapshot's metadata must be tracked, and the data files it references cannot be garbage collected until the snapshot is expired.
In production, unexpired snapshots are one of the largest sources of storage waste. Teams routinely discover that 30–50% of their object storage bill is data referenced only by old snapshots that should have been expired weeks ago.

Snapshot expiration removes old snapshots and makes their exclusively-referenced data files eligible for deletion. The parameters are straightforward — retention window, minimum snapshot count, schedule — but the execution at fleet scale is where things break down. Different tables need different retention windows. Streaming tables commit far more frequently than batch tables. And expiration must happen before orphan cleanup to avoid deleting files still referenced by unexpired snapshots.

LakeOps automates snapshot lifecycle per table, respecting configurable retention policies at every scope level. You set the rules once — the platform handles scheduling, sequencing, and cleanup continuously. It also provides a snapshot explorer in the UI: browse snapshots, compare states, tag versions, and roll back — with full visibility into how much storage each retention window is costing you.

Manifest maintenance

Manifests are Iceberg's indexing layer. Each manifest file tracks a subset of data files along with their partition values, column-level min/max statistics, and file sizes. Over time, they accumulate problems that silently degrade query performance.

Manifest fragmentation: as tables evolve through many small commits, manifests proliferate. A table with 500+ manifests forces the query planner to open and parse each one during scan planning. Rewriting manifests consolidates them — reducing the number of files the planner must read and cutting planning latency.

Position delete files: Iceberg v2 supports row-level deletes via position delete files (merge-on-read). At query time, the engine reads both data files and their associated delete files, filtering out deleted rows. As delete files accumulate, read amplification grows. Rewriting position deletes physically merges them into the data files, converting merge-on-read to copy-on-write and eliminating per-query overhead.

Puffin statistics: Puffin files store column-level statistics (NDV, histograms, null counts) that enable cost-based query optimization. Stale Puffin stats lead to suboptimal join ordering and filter selectivity estimates. They need to be refreshed after significant data changes.

These operations interact with each other and with compaction. Manifest rewrites should happen after compaction (since compaction changes the file layout). Position delete merges should be coordinated with compaction to avoid redundant rewrites. Puffin refresh should follow any significant layout change.

LakeOps treats all of these as coordinated operations within a single autonomous maintenance loop. The platform understands the dependency graph — expire snapshots first, then clean orphans, then compact, then rewrite manifests and refresh statistics — and executes the full sequence at the right cadence for each table based on its ingestion rate and query patterns. No manual scheduling, no sequencing bugs, no forgotten tables.

Orphan file cleanup

Orphan files are data files that exist in storage but are not referenced by any current snapshot or metadata file. They accumulate from failed writes, completed compaction (old files remain until explicitly deleted), snapshot expiration (references removed, files left behind), and table drops or partition deletes.

Left unchecked, orphan files consume enormous storage. Production lakes regularly accumulate hundreds of terabytes of dead data — files that cost money every month but serve no purpose. One team discovered ~200 TB of orphan files across their lake, costing roughly $4,000/month in pure storage waste.

Cleanup involves listing all files in the table's storage location, comparing against all file references in current metadata, and deleting files that are unreferenced and older than a configurable age threshold. The age threshold is critical — set it shorter than your longest-running write job and you risk deleting files that are part of an in-progress transaction.

LakeOps runs orphan cleanup as part of its coordinated maintenance sequence, after snapshot expiration has properly released references. The age threshold is configurable per policy (default 7 days), and the platform ensures cleanup never runs ahead of expiration. For teams that have never cleaned orphans, the first run often reclaims a surprising amount of storage — one documented case saw a 350 TB lake shrink to 230 TB in 10 minutes.

Observability: knowing what is broken before users notice

The most dangerous state for a data lake is one where tables are degrading but no one knows. Queries get slower by 5% per week. Storage costs creep up. A table that used to plan in 200ms now takes 4 seconds. No single commit is the cause — it is the accumulation of hundreds of small writes without maintenance.

Effective Iceberg observability requires monitoring at multiple levels:

Table-level health metrics: file count, average file size, manifest count, snapshot count and age, position delete file count, orphan file volume. A table with 50,000 files averaging 2 MB is in trouble. A table with 500+ manifests has a planning bottleneck. These signals are available via Iceberg metadata tables ($files, $snapshots, $manifests), but querying them across hundreds of tables and correlating the results into actionable classifications is a significant engineering effort on its own.

Query-level signals: planning time vs. execution time (if planning exceeds 30% of total query time, metadata or file count is the bottleneck), data scanned vs. data returned (high ratios mean poor file layout), and engine-specific metrics like Trino split counts or Spark task counts.

Fleet-level views: you need to see across all tables in all catalogs to identify systemic patterns — which namespaces have the most critical tables, whether streaming tables degrade faster than batch tables, and where storage cost is concentrating.

LakeOps provides this observability out of the box. Every table gets a health classification (critical, warning, healthy) based on file count, file size distribution, manifest fragmentation, snapshot depth, and orphan accumulation. The platform dashboard shows fleet-wide health, engine metrics, cost signals, and query performance — correlated across engines so you see the full picture rather than stitching together engine-specific monitoring. Insights surface specific issues (e.g., "12 tables have small-file hotspots," "3 tables have snapshot bloat") with severity and recommended action, so you know exactly where your lake needs attention and what to do about it.

Policy and governance at fleet scale

When you have 10 tables, you configure maintenance manually. When you have 800 tables across 5 catalogs, you need policies.

A policy defines what maintenance operations should run, on which scope, at what schedule, with what parameters. Policies should cascade — an organization-wide default applies everywhere, but a namespace-level policy can override it for streaming tables that need more aggressive compaction, and a table-level policy can override that for a specific high-priority table.

Common policy dimensions:

Compaction: target file size, strategy (binpack/sort), schedule, concurrency limits
Snapshot expiration: retention period, minimum snapshots to retain, schedule
Manifest rewrite: schedule, trigger threshold (e.g., rewrite when manifest count exceeds 50)
Orphan cleanup: age threshold, schedule, scope
Table configuration standards: enforce Iceberg v2, default Parquet compression, write distribution modes, commit retry settings

LakeOps implements a hierarchical policy engine with four scopes: table → namespace → catalog → organization. More specific policies override broader ones. You define the rules once, and the platform enforces them continuously across your entire fleet. Every action is logged, auditable, and reversible — which matters when you are explaining to compliance why a table's data was reorganized at 2 AM.

For reference, a reasonable manual scheduling baseline looks like this:

Operation	Schedule	Rationale
Snapshot expiration	Hourly	High-frequency commits need frequent cleanup
Compaction	Daily, 2 AM	Off-peak, after expiration has freed references
Orphan cleanup	Daily, 3 AM	After expiration, before compaction
Manifest rewrite	Daily, 4 AM	After compaction produces new file layout

With LakeOps, you do not need to manage these schedules yourself — the platform's autonomous loop handles sequencing and adapts frequency to each table's ingestion rate — but the table above is useful for understanding the dependency order.

Multi-engine considerations

One of Iceberg's core value propositions is engine independence. A single table can be read and written by Spark, queried by Trino, accessed by Snowflake's external tables, and analyzed by DuckDB — all through the same catalog.

This creates two challenges: maintenance operations must be compatible with all engines accessing the table, and routing decisions affect both cost and performance.

Catalog as the coordination point

The Iceberg REST Catalog API has emerged as the standard for multi-engine access. LakeOps connects to your existing catalog — AWS Glue, Apache Polaris, Gravitino, Nessie, Lakekeeper, or REST-compatible catalogs — and operates through it. All maintenance commits go through the same catalog path your query engines use, ensuring atomic commits and proper concurrency handling. No separate catalog, no sidecar processes, no data movement.

Query routing

With multiple engines available, the question becomes: which engine should run which query? A small lookup query costs $0.01 on DuckDB and $0.08 on Snowflake. A complex analytical join runs in 2 seconds on Trino and 45 seconds on DuckDB. Sending every query to the same engine wastes money or wastes time.

LakeOps includes a multi-engine query routing layer that classifies queries and routes them to the optimal engine based on cost, latency, or throughput targets. Routing groups present a single SQL endpoint to downstream consumers (via PostgreSQL, MySQL, or Arrow Flight wire protocols) while the platform handles backend engine selection. Applications, BI tools, and agents connect to one endpoint — the routing layer optimizes which engine actually executes each query, and adapts as table health and engine load change.

Agentic AI and the data lake

A newer challenge: AI agents that issue SQL queries autonomously. LLM-powered agents, feature store pipelines, and autonomous data workflows increasingly query Iceberg tables directly — often at unpredictable volumes, with unpredictable query shapes, and without the human judgment that would avoid scanning a 500 TB table with SELECT *.

This changes the maintenance equation:

Latency sensitivity increases: agents expect sub-second responses. A table with a 4-second planning time because of small file accumulation is unusable for agent workflows, even if it is "fine" for batch analytics. Benchmarks show uncompacted tables impose a 5–10x latency penalty on agent queries.
Query volume increases: agents can issue 50 queries per interaction, multiplied across thousands of concurrent agents. Poorly laid-out tables amplify costs multiplicatively.
Safety becomes critical: an agent with write access to production tables can issue DDL, trigger full table scans on petabyte tables, or expose PII. Without guardrails, autonomous SQL access is a liability.

LakeOps addresses this through a dedicated agentic AI enablement layer. Agents connect via Model Context Protocol (MCP) with structured tools for schema discovery, query execution, and statistics retrieval — no raw JDBC connections. A layered guardrail chain enforces read-only access, row limits, cost estimation (via EXPLAIN), PII masking, and optional human approval for high-risk operations. The routing layer classifies agent queries and sends them to the right engine automatically. And critically, the maintenance system monitors agent query patterns and adjusts sort orders, compaction priority, and file sizes accordingly — so the lake self-optimizes for the workloads actually hitting it.

Common anti-patterns

Running compaction without expiring snapshots first. Compaction creates new files and commits a new snapshot, but the old files remain referenced by old snapshots. If you compact aggressively without expiring, you increase storage consumption — you now have both the old files and the new compacted files.

Setting orphan cleanup age threshold too low. If your longest-running write job takes 3 hours, and you clean orphans older than 1 hour, you will delete files that are part of an in-progress transaction. Set the threshold to at least 2x your longest write job.

Compacting everything on the same schedule. Not all tables need the same compaction frequency. A streaming table committing every 10 seconds needs hourly compaction. A daily batch table needs weekly compaction. One-size-fits-all scheduling either wastes compute on cold tables or under-maintains hot tables. LakeOps adapts compaction frequency per table based on ingestion rate and health signals, eliminating this tradeoff entirely.

Ignoring manifest count. Teams focus on file count and miss manifest fragmentation. A table with 500 manifests and well-sized data files will still have slow query planning because the planner must parse every manifest. Manifest rewriting is cheap — do it regularly.

Sort compaction without query analysis. Sorting by arbitrary columns wastes compute and can actually make queries slower if the sort order does not match dominant filter patterns. Always analyze query patterns before choosing sort columns — or let a system that observes actual query traffic make the decision for you.

No observability before automation. Automating maintenance without visibility into table health is flying blind. You will not know if your automation is effective, if it is missing tables, or if it is making things worse. This is why LakeOps starts with observability — classifying every table's health before taking any action.

Getting started

If you are standing up or improving a managed Iceberg data lake, here is a practical path:

Audit your current state. Query the $files, $snapshots, and $manifests metadata tables for your most important tables. How many files? What is the average file size? How many snapshots? This gives you a baseline — and usually a few surprises.
Connect LakeOps to your catalog. The platform connects to your existing catalog and storage in under 10 minutes — no agents, no data movement, no pipeline changes. Once connected, it automatically classifies every table as healthy, warning, or critical, giving you immediate visibility into your fleet's health.
Let autonomous maintenance run. LakeOps handles snapshot expiration, orphan cleanup, compaction, and manifest maintenance in the correct sequence, at the right frequency for each table. Start in observation mode if you want to review actions before they execute, then switch to full autopilot once you are confident.
Define policies for your fleet. Set organization-wide defaults for retention windows, compaction targets, and cleanup thresholds. Override at the namespace or table level where specific workloads need different treatment.
Enable query routing. If you run multiple engines, configure routing groups to optimize cost and latency across your engine mix. Present a single endpoint to consumers and let the routing layer handle engine selection.
Evaluate sort compaction selectively. For your most-queried tables, use LakeOps simulations to test sort strategies against actual query patterns. Promote layouts that show significant scan reduction.
Plan for agents. If AI agents will query your lake, enable the MCP interface and guardrail chain. Monitor agent workload patterns and let the platform optimize table layouts accordingly.

Closing

The work of managing an Iceberg data lake is not glamorous. It is compaction schedules, orphan cleanup thresholds, manifest fragmentation, and snapshot retention policies. It is the kind of work that, when done well, is invisible — queries are fast, storage is efficient, and engineers are not firefighting.

When done poorly — or not done at all — it is the reason your dashboards are slow, your cloud bill is unexplainable, and your platform team spends their time writing one-off maintenance scripts instead of building the things that actually matter.

Iceberg gives you a powerful foundation. LakeOps is what turns that foundation into a production-grade data platform — autonomously managed, fully observable, and ready for whatever workloads come next.

Introducing QueryFlux: Open-Source Universal Multi-Engine Query Router and SQL Proxy

joni sar — Mon, 06 Apr 2026 09:07:13 +0000

Efficiently routing multiple query engines is a critical challenge.

QueryFlux is a universal SQL proxy and multi-engine query router written in Rust. It sits between clients and query engines. Clients connect to QueryFlux using a protocol they already know. QueryFlux routes each query to the right backend, translates SQL dialects when needed, enforces concurrency limits, and gives you a unified observability surface.

Open table formats unified the data. QueryFlux unifies the access.

If you already run more than one query engine, you know the problem is not only where data lives. The harder part is how query access works in practice.

Which engine should this run on?
Which client should connect where?
How do you protect low-latency traffic from batch workloads?
What happens when one cluster is saturated?
How much routing logic ends up hardcoded across the stack?

That is the problem QueryFlux is built to solve.

Why QueryFlux exists

Modern data platforms are multi-engine by design.

A team may use Trino for federated queries, DuckDB for embedded analytics, StarRocks for low-latency serving, and Athena for pay-per-scan workloads on cold data. That mix is not a sign of architectural drift. In many cases, it is the right shape of the system.

Open table formats made this possible. With Apache Iceberg, Delta Lake, or Hudi, multiple engines can read the same data in object storage without duplicating it. That solved storage interoperability.

What it did not solve is compute access.

Each engine still comes with its own protocol, its own SQL dialect, its own connection handling, and its own operational behavior. Clients still need to know where to connect. Routing logic still leaks into notebooks, applications, dashboards, and team conventions. Capacity management is still fragmented across backends.

QueryFlux adds the missing layer above the table format: one access layer in front of the engine fleet.

What QueryFlux does

At a high level, QueryFlux handles three things:

protocol ingestion
routing
dispatch and dialect translation

Clients connect using protocols they already speak. QueryFlux supports:

Trino HTTP
PostgreSQL wire
MySQL wire
Arrow Flight SQL
Admin REST API

On the backend side, it already supports:

Trino
DuckDB
StarRocks
Athena

That gives it a very specific place in the stack. It is not trying to replace engines, and it is not introducing a custom client model. It is making a heterogeneous engine fleet look coherent from the access layer.

How a query flows through the system

A client connects to QueryFlux using a native protocol.

The query is evaluated against an ordered routing chain.

The first matching rule selects the cluster group that should handle the query.

From there, QueryFlux selects a healthy cluster in that group, optionally rewrites the SQL into the target dialect using sqlglot, and dispatches the query.

If the group is already at its concurrency limit, the query can queue at the proxy instead of failing immediately.

That is the important design move. QueryFlux is not just a forwarder. It is the runtime layer where access, routing, translation, and capacity handling meet.

Architecture

Client (psql / Trino CLI / mysql / BI tool)
    │
    │ native protocol
    ▼
┌─────────────────────────────────────────────┐
│                 QueryFlux                   │
│                                             │
│  Frontend ──► Router ──► Dialect translation│
│                    │                        │
│              Cluster group                  │
│         (concurrency limit + queue)         │
└──────────────────┬──────────────────────────┘
                   │
      ┌────────────┼────────────┐
      ▼            ▼            ▼
   Trino       StarRocks      Athena
      └────────────┴────────────┘
          Apache Iceberg / Delta / Hudi

The architecture is simple enough to understand quickly, but deep enough to be useful in real environments.

The simplicity is at the edge. Clients keep using the protocols they already know.

The depth is inside the routing and dispatch path, where QueryFlux can apply routing policy, translation, concurrency limits, queueing, health-aware selection, and load balancing without pushing that complexity back into every client.

Routing is where the value becomes obvious

QueryFlux evaluates each query against an ordered router chain.

Routing can be based on:

protocol
HTTP headers
SQL text using regex
client tags
Python script logic
compound rules
fallback routing

That matters because real routing logic is rarely a single condition. In practice, you may want to steer PostgreSQL wire traffic to a low-latency group, send ETL-tagged traffic to a batch-oriented cluster, and use query patterns to catch common fast-path cases.

A simple example looks like this:

routes:
  - name: fast_queries
    match:
      query_regex: "SELECT .* LIMIT \\d+"
    target: duckdb_group

  - name: dashboard_queries
    match:
      protocol: mysql
    target: starrocks_group

  - name: heavy_analytics
    match:
      query_regex: "JOIN|GROUP BY|WINDOW"
    target: trino_group

  - name: fallback
    fallback: true
    target: athena_group

The point is not that every deployment should use this exact policy. The point is that the policy becomes explicit, traceable, and shared.

That alone removes a surprising amount of hidden operational drag.

Cluster groups make routing operational

Once a route resolves to a cluster group, QueryFlux handles execution there.

It supports these load-balancing strategies:

roundRobin
leastLoaded
failover
engineAffinity
weighted

It also supports:

per-group concurrency limits
proxy-side queueing when groups are full
health-aware cluster selection
background health checks

This is where QueryFlux starts to feel deeper than a typical proxy.

It is not only deciding where a query should go. It is also giving operators a place to control how traffic behaves when systems are under load, how overflow is absorbed, and how healthy capacity is chosen.

That is the part that makes the system practical.

SQL translation is built into the path

Multi-engine routing is much more useful when SQL dialect differences do not immediately get in the way.

QueryFlux integrates dialect-only translation through sqlglot. When needed, it can rewrite SQL into the target engine’s dialect during dispatch.

That means:

clients can keep speaking the SQL they naturally emit
QueryFlux can normalize for the backend that will actually execute the query
teams do not need to maintain multiple versions of the same query only because engines differ

The current design is disciplined here. What is implemented today is dialect-only translation. Schema-aware translation is explicitly on the roadmap.

That is a good balance: the system is already useful now, and the path to deeper translation is clear.

Observability is part of the product, not an add-on

A routing layer only works if operators can see what it is doing.

QueryFlux includes:

Prometheus metrics
Grafana dashboard
Admin REST API
QueryFlux Studio

The current observability surface covers:

query counts
query duration
translation metrics
running queries
queued queries

It also supports routing traces, which matters in practice. When you introduce a routing layer, one of the first questions engineers ask is: why did this query land there? QueryFlux has a real answer to that.

Where this becomes useful quickly

The value of QueryFlux is easier to see in real scenarios than in abstract feature lists.

A multi-engine platform

BI tools connect through one access layer
different workloads are routed to the engines they fit best
backend topology becomes configuration instead of client code

Dashboard SLA protection

low-latency groups can be protected with concurrency limits
overflow can queue or spill instead of degrading the serving path

Incremental engine migration

weighted routing makes gradual traffic shifts possible
clients do not need to change while the migration happens

Mixed workloads on shared data

batch, interactive, and exploratory traffic can be separated by policy
routing intent lives in one place instead of being spread across the stack

These are practical benefits. They show up immediately once a platform becomes multi-engine.

Getting started is intentionally simple

One of the nice things about the project is that the first run experience is straightforward.

A minimal setup looks like this:

git clone https://github.com/lakeops-org/queryflux.git
cd queryflux/examples/minimal-trino
docker compose up -d --wait

That gives you:

QueryFlux on http://localhost:8080
Trino direct on http://localhost:8081
Admin API on http://localhost:9000
Studio on http://localhost:3000
Postgres on localhost:5433

You can then send a simple query through the Trino HTTP frontend:

curl -X POST http://localhost:8080/v1/statement \
  -H "X-Trino-User: dev" \
  -d "SELECT 42"

There are also examples for:

a minimal in-memory setup
a Prometheus + Grafana stack
a full stack with Trino, StarRocks, and Iceberg-related services

That combination is important. The system is conceptually ambitious, but the on-ramp is short.

It feels like deep infrastructure without feeling heavy to try.

What is already shipped

QueryFlux already includes a substantial set of capabilities on the main branch:

Trino HTTP frontend
PostgreSQL wire frontend
MySQL wire frontend
Arrow Flight SQL frontend
Admin REST API
Trino backend
DuckDB backend
StarRocks backend
Athena backend
ordered router chains and routing fallback
route tracing support
per-group concurrency limits
proxy-side queueing
multiple load-balancing strategies
health-aware cluster selection
dialect-only translation through sqlglot
in-memory persistence
PostgreSQL persistence
authentication providers including none, static, OIDC, and LDAP
authorization modes including allow-all, simple policy, and OpenFGA
Prometheus metrics
Grafana dashboard
QueryFlux Studio
dynamic config reload from Postgres

This matters because the project already feels like infrastructure, not just an idea.

What comes next

The roadmap extends the same core design.

Near-term work includes:

schema-aware SQL translation
ClickHouse backend and HTTP frontend
richer routing telemetry in Studio

Medium-term work includes:

cost- and performance-aware routing
Snowflake backend
BigQuery backend
Redis persistence

That roadmap makes sense. It deepens the same access layer instead of changing the project’s center of gravity.

Solving the data ccess side

The most interesting thing about QueryFlux is not that it is a proxy.

It is that it is a carefully placed layer in a part of the modern data stack that is still surprisingly underbuilt.

Open table formats solved the data side.

QueryFlux is solving the access side.

That creates an appealing combination:

conceptually clean architecture
obvious operational benefits
room for sophisticated policy and routing logic
low-friction adoption path

It feels like the kind of infrastructure that becomes more valuable as the rest of the stack becomes more heterogeneous.

Getting started

Once a data platform becomes multi-engine, the missing piece is usually not another engine.

It is the access layer.

Clients still need to know where to connect. Routing still leaks into tools and applications. SQL dialect differences still show up at the edges. Capacity handling is still fragmented.

QueryFlux gives that layer a shape.

It makes multi-engine access easier to reason about, easier to operate, and easier to evolve.

That is why it is a compelling project: the idea is deep, the benefits are immediate, and the first experience is simple.

To try it out visit: https://queryflux.dev/

11 Compaction Optimizations for Iceberg Data Lakes

joni sar — Mon, 16 Feb 2026 12:55:54 +0000

Compaction should provide an easy solution to a very difficult problem: controlling file count, minimizing the cost of delete operations over read costs, and keeping metadata growth from turning every query plan into a deep, time-consuming walk through snapshots and manifests.

The data layer has compaction as its mechanism for solving these issues, but only if compaction is run using some defined set of rules governing the scope of compactions, the thresholds above which compactions are run, and the synchronization of compaction runs with the timing of snapshot expirations and the maintenance of manifests.

Compaction can be run manually via scripts and schedules, or automatically by a control plane.

When manual scripts are used to run compaction, they can effectively manage compaction for a small number of tables and one engine. However, as soon as there are many tables, and/or multiple engines, the manual process becomes guesswork; scripts may rewrite too much, may run too infrequently, may interfere with ongoing ingestions, and will cause churn in both the snapshots and manifests.

A control plane flips this model completely around.

Instead of rewriting everything all the time, a control plane continuously monitors the health and workload characteristics of tables. Then, only when necessary, a control plane spends rewrite budget on the parts of the tables that actually change performance or cost, while also managing the entire lifecycle of maintaining the table.

This article will teach you how to run compaction like the production lakes do it: how to choose your base line strategy (bin-packing vs sorting) for compaction, how to prevent rewrites of healthy partitioning, how to limit the scope of each compaction so maintenance remains invisible, how to focus on hot and delete-heavy areas first, how to prevent the continuous commit cadence of streaming data from creating a factory of snapshot partitions, and how to synchronize compaction with the metadata cleanup that is needed to maintain stable query planning.

For further reading on other aspects of optimizing and maintaining Iceberg, please also refer to:

Let's move on to the compaction strategies that actually work in real production lakes.

1. Add a Control Plane for 20x Faster Compaction and Optimized Table Maintenance

If you add a control plane to your lake, LakeOps comes with the most powerful and intelligent compaction engine that exists today, and will also manage and optimize table maintenance and lake operations for you.

Snowflake-like experience for Iceberg with 10x performance (source: lakeops.dev)

Instead of being confined to fixed schedules, LakeOps operates as a control plane for Iceberg tables and can know when and how to compact what. It treats compaction as a continuous operational problem rather than a periodic batch job, and optimizes it in real time.

It analyzes telemetry data from query engines and Iceberg catalogs and uses that data to decide when compaction is actually needed, what to compact, and how. It takes actual usage patterns into account as well.

Under the hood, LakeOps uses a dedicated Rust-based compaction engine that is designed specifically for Iceberg layouts and metadata behavior. Compaction is coordinated with snapshot expiration, manifest rewrites, orphan cleanup, and statistics maintenance so these operations reinforce each other instead of fighting.

The results are ~20x faster compaction, ~15x faster queries, and ~80% CPU/Storage cost saving.

🚢 Apache Iceberg compaction is not “background maintenance.” It’s a time-critical optimization problem that directly impacts query latency, metadata growth, and infrastructure cost.

Learn more about it here:

Apache Iceberg Compaction: Time-Critical Optimization | Amit Gilad

In addition to compaction LakeOps gives you control with manual and autopilot modes for all maintenance operations in youe tables and coordinates them with compaction. That includes expiring snapshots, manifest rewrites, orphan file cleanups and more.

Real-Time comoaction and maintenance optimization with a control plane (source: lakeops.dev/)

You can choose between manual mode and auto-pilot per table or for groups of tables to control compaction and maintenance proccesses.

LakeOps also lets you define policies across the lake to enforce your standards, and provides you with dashboards to see and manage all compaction and maintenance processes per table and for the entire lake.

Learn more: https://lakeops.dev

2. Use bin-pack as the baseline correction

Most iceberg tables do not require complex layout schemes. However, they all suffer from file fragmentation.

Before attempting to fix the problem using sort-based layouts, clustering, or partitioning, take a closer look at the most obvious source of file fragmentation: the writing process itself. In almost all cases, the initial performance decline caused by file fragmentation is due to the streaming ingestion of very small batches; micro-batch commits occur very frequently; and backfill data will always come in very unevenly-sized chunks.

As a result of the write process, many small Parquet files exist within each partition. None of these files are "broken," and queries will still continue to provide accurate answers. However, as more and more small Parquet files exist within each partition, planning time will increase, the overhead associated with task scheduling will grow, and the number of object store calls will grow.

This is not a layout issue; it is a file count issue.

The easiest and most reliable method to solve the file count issue is to use bin-pack compaction. Bin-pack compaction combines small data files into a smaller number of larger, properly sized files. Bin-pack compaction does not alter the existing file sort order, nor does it re-cluster data; it merely normalizes the file size and decreases the metadata overhead associated with having a high number of files.

In practice, this is usually sufficient.

Why This Approach to Compaction Really Does Improve Performance

Iceberg engines operate at the file level. The more files you add to an engine, the more the engine must plan:

More manifest entries must be read

More file footers must be inspected

More scan tasks must be scheduled

More file references to delete must be tracked

As the number of files grows exponentially, so too does the planning time. Bin-pack compaction eliminates the number of files physically on disk, while maintaining the existing logical layout. Therefore, there are fewer planning reads, and fewer tasks to schedule without requiring additional shuffling.

A good rule of thumb for most production tables is to target file sizes ranging from 128 MB to 512 MB. The specific size range will depend on the engine, and the workload. What is important is consistency.

Start with the Default Rewriting Method

Unless you specify otherwise, Iceberg uses the bin-pack rewriting method:

CALL prod.system.rewrite_data_files( 
  table => 'db.events'
);

Verify the effectiveness of the rewrite:

SELECT
  count(*) AS file_count, 
  sum(file_size_in_bytes)/1024/1024/1024 AS total_size_gb
FROM prod.db.events.files;

You want to see fewer files and the same total size. If the total size is substantially changed, something other than file fragmentation is occurring.

Specify Your Target File Size

If file fragmentation continues, specify a target file size at the table level:

ALTER TABLE prod.db.events SET TBLPROPERTIES ( 
  'write.target-file-size-bytes'='536870912' -- 512MB
);

And then perform the rewrite with the same target file size:

CALL prod.system.rewrite_data_files( 
  table => 'db.events', 
  options => map( 
    'target-file-size-bytes','536870912' 
  ) 
);

Without specifying a target file size, the engine and writer will create files of varying sizes, which will require subsequent compactions to restore the target file size.

Define Thresholds to Prevent Unnecessary Rewrites

At scale, performing rewrites on healthy data wastes compute resources. Define the following thresholds to prevent unnecessary rewrites:

CALL prod.system.rewrite_data_files( 
  table => 'db.events', 
  options => map( 
    'min-input-files','5', 
    'min-file-size-bytes','134217728' -- 128MB 
  ) 
);

This will ensure that only those partitions with a minimum of five input files are rewritten. Partitions with one or two files of reasonable sizes are ignored.

These thresholds can be made tighter by defining multiple conditions:

CALL prod.system.rewrite_data_files( 
  table => 'db.events', 
  options => map( 
    'min-input-files','5', 
    'min-file-size-bytes','134217728', -- 128MB 
    'target-file-size-bytes','536870912' -- 512MB 
  ) 
);

Therefore, compaction will now only run under the following conditions:

There are enough small files to warrant consolidation

Files are currently below a reasonable size threshold

There is a clear target to normalize to

This transforms compaction from an automated rewrite operation to a targeted repair operation.

Prior to executing a compaction operation, review the distribution of files:

SELECT 
  partition.event_date, 
  count(*) AS file_count, 
  avg(file_size_in_bytes)/1024/1024 AS avg_mb 
FROM prod.db.events.files 
GROUP BY partition.event_date 
ORDER BY file_count DESC 
LIMIT 20;

If a partition contains four files averaging 480 MB and your target file size is 512 MB, rewriting the partition will not significantly affect either planning time or scan time.

However, if another partition contains 180 files averaging 25 MB, that partition clearly requires compaction.

Decisions regarding compaction operations should be based on this type of signal. Not a schedule.

3. Conditional Compaction

Another way to waste compute cycles in an Iceberg lake is to blindly compact data on a regular basis.

It usually begins innocently. A periodic rewrite job is created to "keep things tidy." For a period of time, it appears to help. Eventually, however, it begins to rewrite partitions that were previously healthy. Each rewrite generates new files, new snapshots, and updates the manifest list. Although none of the files are "broken," the system is spending compute cycles on processing data that does not improve performance.

Compaction should be used as a corrective measure, not as a routine activity.

Why Unconditional Rewrites Cause Churn

Each time you rewrite data files, Iceberg:

Creates new data files

Generates a new snapshot

Updates manifest lists

Increases metadata history

If the files being rewritten are already close to their target size, you are essentially cycling the data through the system. The added churn causes increased metadata depth and longer planning times over time.

At scale, this overhead becomes noticeable.

Your objective is not to compact frequently. It is to compact when the layout is measurably unhealthy.

Implement Gateways to Control Rewrites

Iceberg provides a rewrite_data_files procedure that allows you to implement gateway conditions. The most effective condition is min_input_files.

CALL prod.system.rewrite_data_files( 
  table => 'db.events', 
  options => map( 
    'min-input-files','3' 
  ) 
);

Using this condition, Iceberg will only rewrite file groups that contain at least three files. Partitions that already have one or two files that are of reasonable size are excluded.

This is a relatively small change to make, but in large lakes, it will significantly reduce unnecessary compaction.

You can implement additional conditions to make this gateway even tighter:

CALL prod.system.rewrite_data_files( 
  table => 'db.events', 
  options => map( 
    'min-input-files','5', 
    'min-file-size-bytes','134217728', -- 128MB 
    'target-file-size-bytes','536870912' -- 512MB 
  ) 
);

Compaction will now only run when the following conditions are met:

There are sufficient small files to warrant consolidation

Files are currently below a reasonable size threshold

There is a valid target to normalize to

This converts compaction from an automatic rewrite into a targeted repair operation.

Let the Table State Drive the Decision

Prior to initiating a compaction operation, review the distribution of files:

SELECT 
  partition.event_date, 
  count(*) AS file_count, 
  avg(file_size_in_bytes)/1024/1024 AS avg_mb 
FROM prod.db.events.files 
GROUP BY partition.event_date 
ORDER BY file_count DESC 
LIMIT 20;

If a partition has four files averaging 480 MB and your target file size is 512 MB, rewriting the partition will not materialy affect either planning or scanning time.

However, if another partition has 180 files averaging 25 MB, that partition is a prime candidate for compaction.

Compaction decisions should be based on signals such as this. Not a schedule.

4. Limit Rewrite Scope Per Run

Big compaction jobs appear great on paper. In reality, they are among the easiest ways to create instability in a production lake.

Backfills, partition evolutions, or long stretches of time without maintenance can quickly turn terabytes of data into rewrite candidates. If you don't specify any boundaries, Iceberg will rewrite everything that meets its criteria. The outcome is well understood: long-running jobs, significant shuffles, large amounts of object store I/O, significant increases in snapshot sizes, and sometimes even cluster contention with users' queries.

Compaction operates at its best when it is incremental.

Why "Rewrite Everything" Is A Risk

When you rewrite a large section of a table in a single pass, you are executing a number of costly operations simultaneously:

Reading numerous data files

Shuffle and rewrite them

Create a lot of new files

Create a new large snapshot

Possibly rewrite manifests

Regardless of whether it is successful, you have produced a major maintenance event. Even if it fails in the middle of its execution, you will lose some resources and extend the length of your maintenance period.

Operationally, smaller, and more frequent corrections are safer than infrequent large-scale rewrites.

Restrict rewrite size explicitly

Rewrite_data_files is a method that allows Iceberg to provide parameters that can help limit the amount of effort that is put into a single run of rewriting data.

To illustrate, you may desire to restrict the maximum number of file group rewrites:

CALL prod.system.rewrite_data_files(
  table => 'db.events',
  options => map(
    'max-file-group-rewrites','20'
  )
);

This restricts the number of rewrite groups executed in a single pass of rewrite_data_files. Instead of rewriting hundreds of partitions in one pass, you will execute a controlled sequence of batch passes.

You can also utilize these with eligibility thresholds:

CALL prod.system.rewrite_data_files(
  table => 'db.events',
  options => map(
    'min-input-files','5',
    'max-file-group-rewrites','20',
    'target-file-size-bytes','536870912'
  )
);

Compaction now becomes predictable. Each pass of compaction corrects a finite amount of drift.

Disperse correction over cycles

There is rarely a good reason to try to "Fix everything tonight" if a table contains months of small files due to a large number of small writes.

Instead, a more sustainable and steady state approach would be:

Compact data with limited rewrite scope.

Permit normal operation of user workloads.

Repeat on the subsequent maintenance cycle.

Within a couple of cycles, fragmentation will decrease significantly, without generating a maintenance peak.

This also produces less "snapshot shock." Instead of a large, single-pass rewrite snapshot replacing nearly half of the table, you generate a series of smaller, incremental snapshots.

Prioritize rather than rewriting randomly

Limiting rewrite scope, prioritizing becomes essential.

Practically speaking, you want to rewrite:

Partition groups with the greatest number of files.

Partition groups with the least average file size.

Partition groups with the largest numbers of deletions.

Partition groups with the most user queries.

You can find the worst offending partition groups using the following query:

SELECT
  partition.event_date,
  count(*) AS file_count,
  avg(file_size_in_bytes)/1024/1024 AS avg_mb
FROM prod.db.events.files
GROUP BY partition.event_date
ORDER BY file_count DESC
LIMIT 50;

Then either target the offending partition groups directly using a WHERE statement or permit your orchestration layer to determine the highest impact groups to rewrite first.

For example:

CALL prod.system.rewrite_data_files(
  table => 'db.events',
  WHERE => 'event_date >= DATE ''2026-01-01''',
  options => map(
    'max-file-group-rewrites','10'
  )
);

This limits both the logical scope (only recent partitions) and the physical rewrite volume.

Maintain Maintenance Invisible To Users

The ultimate objective of limiting rewrite scope is not merely cluster stability. It is predictability.

When compaction is done in a manner such that each run is relatively small and bounded:

Maintenance windows are brief.

Resource spikes are under control.

Snapshots grow gradually.

Query performance improves incrementally, rather than suddenly.

In production lakes, stability is usually more important than the rate of correction. Incremental correction is generally preferred to dramatic restructuring.

5. Focus On Hot Partition Groups

In the majority of Iceberg tables, compaction impact is not uniformly distributed.

A small segment of partitions is responsible for most of the pain: They receive the most writes (thus they tend to fragment the quickest) and they receive the most reads (thus every additional file appears as both planning + scan overhead). If you rewrite the partitions that are "hot", you will typically gain 80% of the benefits with only a fraction of the rewrite volume.

The simplest approach to achieve this is to treat compaction as a rolling window problem.

"Hot" Generally Means Two Things

Hot partitions generally represent partitions that are still active:

They are still receiving new files from streaming or micro-batch systems

They are the partitions that your analysts / dashboards / downstream jobs are accessing constantly.

This results in two operational principles:

Compact relatively recent partitions regularly, since they will accumulate the most small files.

Do not compact the actively written partitions unless you know you can tolerate collisions with writers.

AWS describes this for Iceberg compaction: Use a where predicate to exclude actively written partitions, so that you do not encounter data conflicts with writers and leave only metadata conflicts that Iceberg can normally resolve.

Identify Your Rolling Window With Where

Iceberg's Spark procedure provides a where predicate for filtering which files (and hence which partitions) qualify for rewriting.

An extremely common use case is "Compact everything older than the current ingest window":

CALL prod.system.rewrite_data_files(
  table => 'db.events',
  WHERE => 'event_date < DATE ''2026-02-10''',
  options => map(
    'target-file-size-bytes', '536870912',
    'min-input-files', '5'
  )
);

This maintains compaction away from the partition(s) that are currently being mutated, while continually cleaning up yesterday's and previous data.

If your table is partitioned hourly, perform the same concept at the hour granularity:

CALL prod.system.rewrite_data_files(
  table => 'db.events',
  WHERE => 'event_hour < TIMESTAMP ''2026-02-11 12:00:00''',
  options => map(
    'target-file-size-bytes', '536870912',
    'min-input-files', '5'
  )
);

The main principle here is not the specific cut-off. It is maintaining a buffer so that compaction does not conflict with ingestion.

Locate the Worst Partition Groups First

Even within the "hot-ish" window, not all partition groups are equally bad. You can typically find the worst offender simply by examining the file count and average size:

SELECT
  partition.event_date,
  count(*) AS file_count,
  avg(file_size_in_bytes)/1024/1024 AS avg_mb
FROM prod.db.events.files
GROUP BY partition.event_date
ORDER BY file_count DESC
LIMIT 30;

If you plan to compact only a limited portion of the data per pass (as you probably should), this query will indicate where you will obtain the largest initial benefit.

If You Must Compact Partition Groups That Are Still Receiving Late Data

There are certain types of workloads that receive late-arriving events, updates, or merges that keep older partition groups "active." If you compact them regardless, you may periodically collide with writers.

Iceberg includes a partial progress mode that commits compaction in smaller portions, rather than committing a large block, which reduces the collision risk associated with retries when conflicts occur.

CALL prod.system.rewrite_data_files(
  table => 'db.events',
  WHERE => 'event_date >= DATE ''2026-02-01'' AND event_date < DATE ''2026-02-10''',
  options => map(
    'partial-progress.enabled', 'true',
    'partial-progress.max-commits', '10',
    'max-concurrent-file-group-rewrites', '10'
  )
);

You are making a tradeoff of "one clean commit" versus "multiple smaller commits that fail at lower expense." In actual production lakes with continuous write activity, that trade is typically worthwhile.

Where a Control Plane Helps

Once you have numerous tables, "hot partition groups" ceases to be something you determine through experience. You require a loop that continuously determines the "hot partition groups" based on the actual read/write usage of your system and then applies the rolling window concept to those partition groups.

That is the point at which a control plane such as LakeOps becomes useful: It is not adding a new compaction algorithm as much as it is determining where to expend your rewrite budget based on real workload telemetry and applying that determination in a consistent manner across hundreds of tables.

6. Sort or Z-Order When Scan Efficiency Is The Bottleneck

Bin-packing compaction decreases the number of files. However, it does not affect how the data is organized internally within those files.

If partitions are appropriately sized and file counts are reasonable, but selectivity-based queries are scanning a disproportionately larger number of data files than anticipated, the cause is likely clustering. You will commonly observe the following pattern: Planning times are consistent, partition pruning is functioning, but filtered queries are reading a large percent of files within a given partition. This is the point where sorting-based compaction becomes relevant.

Data engines depend on file-based statistical information (such as min and max values) to determine whether a file can be excluded from processing. When data is written in a random fashion, the value ranges between files overlap greatly. Therefore, even selective predicates typically cannot exclude a large number of files. Sorting changes this. When data is ordered by a frequently used filter column, each file will generally contain a narrower value range than previously existed, thus allowing more files to be skipped based upon the predicate and reducing the number of bytes to be scanned.

If there is one column that dominates your predicates (e.g., event_time within a date-partitioned table), a simple sort-based rewrite is typically sufficient:

CALL prod.system.rewrite_data_files(
  table => 'db.events',
  strategy => 'sort',
  sort_order => 'event_time ASC NULLS LAST',
  options => map(
    'target-file-size-bytes','536870912',
    'min-input-files','5'
  )
);

This rearranges the rows of data within files so that time-based predicates can remove more files early in the process. The effect is evident not only in terms of runtime, but also in the reduction in scanned bytes and the number of splits generated.

If your workload filters on multiple columns (e.g., user_id, event_type, and occasionally device_type), Z-order is typically a superior choice:

CALL prod.system.rewrite_data_files(
  table => 'db.events',
  strategy => 'sort',
  sort_order => 'zorder(user_id, event_type)',
  options => map(
    'target-file-size-bytes','536870912',
    'min-input-files','5'
  )
);

Z-ordering enhances locality across multiple dimensions. While Z-ordering will never perfectly optimize any individual column, it typically minimizes overall scan expansion when filter patterns vary.

Important Note: Defining a sort order at the table level does not rewrite historical data. It only affects newly-written data. All existing files will remain unmodified until a rewrite occurs.

It is typical to define a sort order and then expect improvements, only to discover that no changes occurred to the physical layout of the data.

After performing a sort-based compaction, verify it correctly. Examine the number of files that are scanned for common predicates. Compare the total bytes scanned before and after. Runtime can be difficult to quantify in shared environments; however, file count and total bytes scanned are more reliable metrics.

Sorting is more resource-intensive than bin-packing. Sorting involves additional shuffle and CPU overhead during compaction. If you were to blindly apply sorting to all partitions, the costs of maintenance could potentially exceed the query performance improvements. In general, sorting works best when applied selectively: Target high-traffic partitions; Align the sort with real filter patterns; Apply rewrites incrementally.

When scan efficiency is the primary bottleneck rather than file count, sorting or Z-order is one of the few techniques that will reliably enhance pruning. The key is to apply sorting or Z-order in a manner that aligns with the characteristics of your workload.

7. Compact delete-heavy partitions deliberately

You can view a table, note that file sizes are healthy, and believe that compaction is being handled - yet the queries are slowing down.

A common cause is delete files.

Iceberg does not immediately write out data files when rows are modified or deleted. Rather, it will store the row position deletes or equality deletes with the data. At read time, the engine will combine the data files with their associated delete files. Although this provides an efficient mechanism for writing, as delete files become numerous, every query will incur additional cost.

The affect is subtle. File sizes appear fine. Bin-pack has already normalized fragmentation. However, scan CPU increases, and partitions that are update-heavy begin to perform poorer than append-only partitions.

You may usually verify this by examining the delete file distribution:

SELECT
  partition.event_date,
  count(*) AS delete_file_count
FROM prod.db.events.delete_files
GROUP BY partition.event_date
ORDER BY delete_file_count DESC
LIMIT 20;

If certain partitions contain a high concentration of delete files, that is an indication that reads are performing more work than necessary.

Iceberg supports delete aware compaction. Instead of rewriting files solely based upon their size, you may specify a threshold for the ratio of deleted rows and have Iceberg rewrite data files that meet this criteria. For example:

CALL prod.system.rewrite_data_files(
  table => 'db.events',
  options => map(
    'delete-ratio-threshold','0.3',
    'remove-dangling-deletes','true',
    'target-file-size-bytes','536870912'
  )
);

In this case, Iceberg will rewrite data files that are severely impacted by deletes and will remove the physical representation of deleted rows as well as the associated delete files.

The practical result is that queries will no longer require the merging of as many delete files at runtime; CPU decreases; scan cost stabilizes; and planning becomes easier since there are fewer auxiliary files to track.

This has a significant impact primarily on tables that have been subjected to upserts, CDC pipelines, and/or frequent merges. Event tables that are used exclusively for appending data do not typically exhibit this behavior. Similarly, dimension tables and slowly changing datasets do.

Just like with any other compaction strategy, maintain focus. Use the combination of delete thresholds, partition filters, and rewrite limits to optimize the compaction strategy. It is unnecessary to rewrite the entire table simply because a handful of partitions contain a high amount of delete activity.

Healthy file size does not ensure healthy performance. If delete files comprise the majority of a partition, the compaction strategy must specifically target these delete files - otherwise, read cost will continue to increase regardless of whether the layout appears to be "correct".

8. Rewrite position delete files individually when needed

Rewriting data files does not necessarily resolve all delete related issues.

In many update-intensive workloads, position delete files accumulate more quickly than data files are rewritten. Even if you execute a delete aware compaction strategy, you can still find yourself with a large number of position delete files attached to otherwise healthy data files.

Even after executing a compaction strategy that reduces the number of data files, the engine still has to open and apply the delete files during a read operation. Therefore, as delete files accumulate, the scan overhead will remain greater than it should be.

This is particularly prevalent in tables that receive regular upserts or merges. Tables that are subject to append-only inserts do not typically exhibit this type of behavior. However, CDC pipelines and dimension tables do.

Iceberg permits the explicit rewriting of position delete files as follows:

CALL prod.system.rewrite_position_delete_files('db.events');

Rewrite smaller delete files into fewer, larger ones and attempt to eliminate obsolete entries wherever possible. The objective of this approach is not merely to reduce the total number of files, but rather to minimize the number of files that the engine has to open during a read operation and thus improve performance.

You may also limit the scope of this rewrite operation, similar to how you limit the scope of data file rewrites, by specifying the partition(s) to be rewrote:

CALL prod.system.rewrite_position_delete_files(
  table => 'db.events',
  where => 'event_date >= DATE ''2026-02-01'''
);

If you have already rewritten the data files and the performance of your application has not improved, inspect the distribution of delete files. In some cases, rewriting data files will reduce fragmentation, but will leave a heavy delete layer behind. In such cases, the separate rewriting of delete files will be required.

Similar to all of the strategies described throughout this guide, keep the rewriting of delete files focused. There is little to be gained by rewriting delete files across the entire table if only a limited number of partitions are subject to upserts.

In addition, treating the maintenance of delete files as a separate entity maintains the predictability of read cost. Otherwise, even though the data files themselves are sized appropriately, the accumulated overhead that is generated by the engine processing the delete files can slow over time.

9. Lower the commit frequency for streaming workloads

When writing to Iceberg from streaming or micro-batch applications, the commit frequency is one of the largest factors contributing to the overall cost multiplier in the system.

Each commit generates a new snapshot and produces new metadata work. This includes updating the manifest and creating new, small data files. As you commit every few seconds, you don't simply create small files; you create a long chain of snapshots and a continuous flow of metadata churn. While nothing "breaks," the planning is slowed and maintenance must continually struggle to keep pace with the increasing overhead.

The frustrating aspect is that teams typically attempt to resolve this issue by applying more compaction, while the true solution lies upstream: stop committing as often.

The benefits of modifying the commit frequency

When you increase the interval between commits, you generally gain three tangible benefits at once.

First, you generate fewer snapshots, resulting in fewer pieces of metadata that the engine has to evaluate during planning.

Second, you generate fewer manifests / manifest updates overall.

Third, each commit contains more data, resulting in larger files (or fewer small files) and therefore reduced compaction pressure.

You're essentially lowering entropy at the source.

Structured Streaming using Spark: Set a valid trigger interval

A common antipattern is to configure structured streaming to run "as fast as possible" or with a very short trigger. If you are writing to Iceberg tables, avoid this practice unless you have a true requirement for sub-minute freshness.

The following shows the configuration for setting a reasonable commit interval in PySpark:

(
  df.writeStream
  .format("iceberg")
  .outputMode("append")
  .option("checkpointLocation", "s3://prod-checkpoints/events/")
  .trigger(processingTime="1 minute")
  .toTable("prod.db.events")
)

If your service level agreement (SLA) allows it, increase the commit interval to 2–5 minutes. In most analytics lakes, this tradeoff is worthwhile: slightly increased data freshness lag in exchange for significantly decreased metadata churn and less maintenance overhead.

Flink: Commit frequency follows checkpointing

For Flink, Iceberg commits typically follow the checkpoint intervals. If you checkpoint every 30 seconds, you are essentially committing every 30 seconds. That's a lot.

A more reasonable interval would be minutes, not seconds, unless you are operating a low-latency serving pipeline.

Example:

// 5 minutes
env.enableCheckpointing(300_000);

Ultimately, the best value will depend on recovery requirements and end-to-end latency needs. However, the underlying premise is the same: do not checkpoint so frequently that you turn your Iceberg table into a snapshot factory.

A simple method to select the interval

Do not overcomplicate things. Ask yourself: what is the longest delay that your downstream consumers can tolerate for data freshness?

If the response is "near real-time", you may still be fine at 1 minute. If the response is "a few minutes", take advantage of the situation and commit every few minutes.

If the response is "we run dashboards hourly", then committing every 10 seconds is just self-imposed suffering.

Sanity check

If you observe thousands of snapshots being created daily for a single table, this is typically an indication that your commit cadence is too aggressive for an analytics lake. You can certainly use Iceberg as a means of generating data in this manner - it is designed to be correct - but you will pay for it in terms of planning overhead and ongoing maintenance.

Lowering the commit frequency is one of the few optimization techniques that will decrease costs and improve stability, independent of whether you adjust the compaction strategy. Fixing this earlier is beneficial because once you have multiple dozen or hundred of streaming-written tables, this behavior will dictate your operational overhead.

10. Stop the repair loop; fix the write path

Write paths that produce too many small files or heavily skewed partitions will make compaction a never-ending battle. As long as you continue to re-write the same issues, the lake will always drift back into an unhealthy condition.

The majority of "we need more compaction" situations are actually "our write path is poorly configured."

Begin with Distribution Mode

Small file generation is a common result of poor data distribution during the write process. A common scenario is when one writer (task) has the majority of the data for a given partition, it emits a couple of large files, while the remaining writers (tasks) emit a large number of smaller files. Worse, if the data distribution is unstable between batches, you will experience fragmentation regardless of how often you compact.

Iceberg allows you to configure the way data is written across multiple writers. A good baseline configuration for many workloads is hash, as it generally spreads rows out more evenly:

ALTER TABLE prod.db.events SET TBLPROPERTIES (
  'write.distribution-mode' = 'hash'
);

This does not completely remove the necessity for compaction, but it helps slow down how quickly fragmentation occurs again.

Establish a Target File Size for Writers at the Table Level

When writers do not have a target file size, you will see variability in the file sizes produced by writers across the engine and job. Some will produce 16MB files, some will produce 1GB files, etc., and compaction will continually attempt to normalize the mess.

Create a target file size for writers at the table level and maintain it consistent:

ALTER TABLE prod.db.events SET TBLPROPERTIES (
  'write.target-file-size-bytes' = '536870912' -- 512MB
);

Once you have established a target file size for writers at the table level, compaction will transition from "fix everything" to "fix the outliers."

Optimize the Writer and Not Just the Table

Another common reason writers produce small files is due to the number of tasks that are utilized when writing versus the amount of data in each micro-batch or partition. The easiest method to optimize this is to adjust the degree of parallelism at the point of write.

If you are experiencing hundreds of files per partition per batch, consider reducing the number of output partitions prior to writing:

(
  df
  .repartition(200) // pick a number that corresponds to your cluster and batch size
  .writeTo("prod.db.events")
  .append()
)

You don't have to find the optimal number. All you need to do is stop creating 1000 small files because your job happened to run with 1000 tasks.

Do Not Create "Hot Partitions" by Design

Some datasets inherently skew towards certain items, such as one customer producing 70% of the events, or a specific date receiving a massive backfill. When a partitioning scheme directs a large amount of data into a single partition, you will continually be fighting it with compaction.

This is one of the few instances where adjusting the partitioning scheme to provide less skewness can greatly reduce compaction load. A common strategy is to add another dimension to the partitioning scheme (or create a derived shard key) to ensure that a single logical partition does not become a physical hotspot.

You do not need to re-design the entire table. One additional dimension may be sufficient to prevent the worst skewness.

11. Maintain Metadata With Compaction

You can obtain the desired file sizes, reduce the number of files, and yet still end up with a table whose performance and cost characteristics degrade over time. This is typically a metadata issue and not a physical layout issue.

Every time you run compaction, you create a new snapshot. Every snapshot adds to the table's history. Each manifestation accumulates. The old metadata remains until something removes the history. If nothing removes the history, the table becomes deeper and more expensive to reason through, even if the physical data files appear to be clean.

This is the most common trap: Teams focus on rewrite_data_files and neglect what happens to the snapshots and manifestations after the fact.

In general, compaction should be run immediately followed by snapshot expiration. If you keep thousands of historical snapshots around "just in case," the engine still has to traverse the lineage when performing planning. Over time, this shows up as slower metadata reads and longer planning times.

Typically, a snapshot expiration would look something like this:

CALL prod.system.expire_snapshots(
  table => 'db.events',
  retain_last => 10
);

The actual number of retained snapshots will vary based on your rollback and time-travel policies, however, it is critical that you establish a clear retention policy. Retention policies greater than infinite are rarely what you truly need.

After expiring snapshots, it is also beneficial to perform manifest consolidation:

CALL prod.system.rewrite_manifests('db.events');

Even if your file sizes are acceptable, fragmented manifests will require the planner to open and evaluate numerous small metadata files. Manifest consolidation will reduce the fan-out and stabilize the planner costs.

Then there is the removal of orphans. Failed jobs, speculative tasks, and partial re-writes will leave files in object storage that are no longer referenced by the table. Over months, this is a lot of money. Removing these orphans will help to predictably manage the lake.

CALL prod.system.remove_orphan_files(
  table => 'db.events',
  older_than => TIMESTAMP '2026-02-10 00:00:00'
);

The older_than guardrail is essential. You cannot afford to be racing with active writers in a production environment. Safety is more important than aggression.

What complicates this is that the above actions are not independent. How you retain snapshots impacts what you can remove. How frequently the table is updated determines how frequently you need to rewrite manifests. Removing orphans is related to when commits occur and streaming jobs.

Therefore, compaction is not simply a single maintenance action. It is part of a life cycle. Physical data files, snapshots, manifests, and physical storage move in tandem.

At small scales, you can run these actions manually and get away with it. At larger scales, you need to be able to enforce consistency. Tables drift in various ways at varying rates. Without coordinated metadata maintenance, you will continue to repair file layout, while the metadata layer quietly continues to grow.

Your goal is not simply to minimize the number of small files. Your goal is to maintain a table whose performance and cost characteristics remain stable over time. Compaction addresses the data layer. The three above actions address the metadata layer to prevent it from becoming the next bottleneck.

Recap and Conclusion

Compaction in Iceberg is not about scheduling rewrite_data_files. It is about maintaining the alignment between the layout, deletes, and metadata of a table with its actual usage.

Compaction plus table maintenance is now a coordination problem. Therefore, having a control plane to continuously assess the health of tables, prioritize the top partitions requiring compaction, and coordinate compaction with metadata maintenance rather than treat these as separate jobs is a home run on the first step:

Manual work and scripting are typically the alternatives.

We previously reviewed the practical aspects of this:

Use bin-pack to control the file count

Escalate to sorting only when the scan efficiency is the limiting factor

Use gates to restrict the re-writing of healthy data

Limit the scope to make the maintenance predictable

Proactively resolve delete-heavy partitions

Reduce the commit entropy in streaming jobs

Correct the write path to prevent constant repair of the same issues

Connect compaction to snapshot expiration, manifest rewrites, and orphan removal

Most importantly, we connected compaction to snapshot expiration, manifest rewrites, and orphan removal - because the physical data layout and the metadata health are interdependent.

Your goal is not to simply minimize the number of small files. Your goal is to maintain a lake that remains predictable - in terms of performance, cost, and operational overhead - as it grows.

If you are operating Iceberg in production, I would appreciate your feedback regarding what has worked (and failed) for you. Real world patterns are always more interesting than theoretical ones.

Thank you for reading 🍺

Iceberg Rewrite Manifest Files: A Guide

joni sar — Sun, 08 Feb 2026 15:31:17 +0000

Iceberg Rewrite Manifest Files: A Guide

A data engineer happily runing into the quicksand

Manifest rewrites are a critical ongoing operation in Iceberg table maintenance.

Data keeps landing, queries stay correct, and nothing looks obviously wrong. But over time, query planning takes longer, metadata reads increase, and latency creeps up even though the amount of data scanned hasn’t really changed. In most production systems, the root cause is not data layout — it’s metadata, and specifically how manifest files accumulate and degrade over time.

Manifest files are central to how Iceberg works. They’re what allow engines to plan efficiently without listing object storage. But with frequent commits, streaming writes, deletes, and long snapshot histories, manifests naturally fragment. Iceberg doesn’t reorganize them automatically, so planning cost quietly grows until it starts to matter.

This guide focuses on rewrite manifests: what they actually do, when they help, and how to run them correctly in production. You’ll learn how to detect when manifest rewrites are needed, how they interact with snapshot expiration and compaction, and why running them in isolation often delivers disappointing results.

We’ll also contrast two operational models: managing all of this manually with scripts and schedules, and handling it continuously through a Control Plane like LakeOps, which optimizes table maintenance based on real workload behavior instead of fixed timers.

The rest of the article guides you through mannual optimization with practical examples. No spec theory, no generic advice — just what actually works when Iceberg tables grow, change, and age in real systems.

Let’s begin then 🙂

Smart Automation vs Manual Scripts

Before getting into mechanics, it’s important to understand the two main ways teams approach manifest management: Automated maintenance optimization with a Control Plane like LakeOps, and doing this operation manually and ongoingly by hand, and using generic scripts.

Continuous Optimization with a Control Plane

A control plane is a layer that sits above your data lake, catalogs, and query engines and takes responsibility for operating and optimizing tables over time. Iceberg defines table structure and guarantees correctness, but it intentionally does not decide when, where, or how aggressively maintenance should run. That operational and optimization gap is exactly what a control plane fills.

Instead of running maintenance because a schedule says it’s time, a control plane continuously optimizes tables based on what is actually happening in the system. Operations run only when and where they are needed, or according to explicit policies you define, rather than blindly across all tables.

LakeOps acts as a control plane for Iceberg by continuously analyzing telemetry from Iceberg catalogs and query engines. Using this data, LakeOps builds a live understanding of how each table behaves in practice.

Cotinious Optimization and Maintenance with an Iceberg Control Plane (source: lakeops.dev)

From that telemetry, LakeOps continuously optimizes table maintenance. Manifest rewrites are triggered only when metadata fragmentation begins to impact planning or cost. Snapshot expiration runs only when retained history no longer provides real value. Compaction is optimized continuously to reduce small files before they create downstream metadata pressure. Orphan cleanup runs when metadata and data files are no longer referenced and can safely be removed.

Coordination is central to optimization. Rewrite manifests, snapshot expiration, compaction, and cleanup are not independent jobs. They are executed as part of a single, continuous optimization loop that ensures only the required operations run, only on the tables that need them, and only at the point where they actually improve performance or cost.

Automating Smart Rewrite manfiest operations (source: lakeops.dev)

Engineers don’t tune per-table schedules or chase drifting thresholds. They decide what should be optimized and within what constraints, and the control plane decides when and where to run each operation. The result is stable metadata, predictable performance, and far less work.

Learn more:

9 Apache Iceberg Table Maintenance Tools You Should Know

Start at the beginning: What Manifest Files Are

Manifest files are the core metadata units Iceberg uses to describe which data files exist and what is inside them. They sit between snapshots and actual data files and are the reason Iceberg can plan queries efficiently without scanning directories or listing objects in storage.

Each manifest file is essentially a list of data file entries. For every data file, the manifest records:

the partition values for that file
record count
per-column statistics such as min and max values
file size and other low-level attributes

When a query runs, the engine does not discover data files by walking object storage. Instead, it reads manifests referenced by the current snapshot and uses the stored statistics to decide which data files can be skipped entirely. This is how Iceberg enables predicate and partition pruning at planning time.

Manifests are immutable. Every commit creates new metadata. When a write happens, Iceberg typically creates one or more new manifest files describing the files added or removed by that commit. Over time, a snapshot references many manifests, some created recently and some carried forward from older snapshots.

This design is powerful, but it has predictable operational consequences.

Frequent small commits, especially from streaming or micro-batch ingestion, tend to produce many small manifest files. For example, a streaming job that commits every minute may generate hundreds or thousands of manifests per day, each describing only a handful of data files. From Iceberg’s point of view this is correct, but for the query engine it means more metadata to read and evaluate during planning.

Another issue is manifest clustering. Manifests are not automatically reorganized around how tables are queried. If files are appended over time with mixed partitions or evolving data distributions, manifests may contain entries that are poorly aligned with common filters. The engine still prunes correctly, but it has to examine more metadata to do so.

Snapshots make this worse if they are not expired. Each snapshot retains references to the manifests that describe its table state. Even if newer snapshots supersede old ones, the metadata remains live as long as those snapshots are kept. This means manifests that are no longer useful for active queries still participate in metadata reads and storage costs.

The net effect is subtle but significant. Query planning time increases even though data size stays flat. Metadata I/O grows quietly. Storage costs creep up due to retained metadata. None of this breaks correctness, which is why it often goes unnoticed until performance degrades.

Manifest rewrites exist specifically to address these issues. They allow Iceberg to reorganize and consolidate manifests so that the metadata layer reflects the current table state and access patterns, rather than the historical accident of how data arrived over time.

What Rewrite Manifests Does

A rewrite manifests operation restructures the metadata layer of an Iceberg table without touching the data itself.

At a high level, Iceberg takes the manifest files referenced by the current snapshot, reads the data-file entries inside them, and writes a new set of manifest files that describe the exact same live data files, just in a layout that’s cheaper for engines to plan against. The commit updates table metadata to point the current snapshot at the new manifests. The old manifests become obsolete once nothing references them anymore (usually after snapshot expiration and cleanup).

This is a metadata rewrite, not a data rewrite. No Parquet/ORC/Avro files are rewritten.

What actually improves

Rewrite manifests helps in three very concrete ways.

It reduces manifest fan-out. When you have many small commits (streaming, micro-batch), you often end up with lots of tiny manifest files. Each query has to open and evaluate those manifests during planning. Rewriting consolidates many small manifests into fewer, larger ones, which reduces metadata I/O and planning latency.

It aligns manifest layout with partitioning. Iceberg sorts data-file entries in manifests by fields in the partition spec. In practice, this tends to make partition pruning cheaper because related entries are adjacent and engines do less work to decide what to skip.

It removes “historical write shape” from the current snapshot. Without rewrites, manifests reflect how data arrived over time, not how it’s queried. Rewriting reorganizes metadata around the current state, which is usually what you actually care about for planning.

What rewrite manifests does not do

It does not compact data files. Tiny data files stay tiny. It does not change partitioning or rewrite records.

It does not delete old manifests by itself. If old snapshots still reference them, they’ll remain. Cleanup is a separate step.

Practical code examples

Below are a few examples that are actually useful in day-to-day operations, not just “hello world”.

1) Measure the problem before you touch anything

Start by inspecting the metadata table that lists manifests. Don’t assume column names — Iceberg versions and engines can differ — so first look at the schema:

\-- Spark: inspect the manifests metadata table schema  
DESCRIBE TABLE EXTENDED prod.db.my\_table.manifests;

Then get a baseline count:

\-- How many manifests does the current snapshot reference?  
SELECT COUNT(\*) AS manifest\_count  
FROM prod.db.my\_table.manifests;

If this number grows steadily week over week while the table isn’t exploding in size, planning overhead is usually creeping up.

2) Rewrite manifests via Spark SQL procedure (the most common operational path)

This runs the rewrite in parallel using Spark:

CALL prod.system.rewrite_manifests('db.my_table');

In Spark, this returns a small result set with counters (how many manifests were rewritten, how many were added). In practice, you run the call, note the counters, and then re-check my_table.manifests to see the manifest count drop.

3) Rewrite manifests for a specific partition spec (when you’ve done partition evolution)

If your table has evolved partition specs over time, you may want to rewrite manifests for a particular spec id:

CALL prod.system.rewrite\_manifests(  
  table   => 'db.my\_table',  
  spec\_id => 1  
);

This is useful when an older spec still contributes a lot of manifest fragmentation and you want to target it instead of doing everything blindly.

4) Disable Spark caching if executors get memory pressure during rewrites

Some environments prefer to avoid caching during maintenance to reduce executor memory footprint:

CALL prod.system.rewrite_manifests('db.my_table', false);

If you’ve ever seen maintenance jobs destabilize executor memory, this is one of the first knobs to reach for.

5) Validate the effect (simple but important)

After the rewrite, validate that you actually improved the metadata shape:

SELECT COUNT(\*) AS manifest\_count\_after  
FROM prod.db.my\_table.manifests;

If the count doesn’t drop (or drops only slightly), the usual causes are that snapshots weren’t expired (old manifests still referenced), or the table’s write pattern keeps producing fragmentation faster than your maintenance cadence.

That’s the point where you either tighten the full maintenance loop (expire snapshots, rewrite manifests, remove orphans, and revisit compaction) or stop doing this manually and let a control plane keep it stable continuously.

When You Should Rewrite Manifests

Manifest rewrites are not something you run on a fixed schedule “just in case”. They are most effective when there is a clear signal that metadata, not data, is becoming the bottleneck.

The most common trigger is planning getting slower while data size stays flat. If query runtimes increase but the amount of data scanned is roughly the same, the extra time is often spent in planning and metadata evaluation. This is especially visible in engines that log planning or analysis time separately.

Another strong signal is manifest growth that outpaces data growth. If storage size grows slowly but the number of manifests keeps climbing, you are accumulating metadata fragmentation. This usually happens in tables with frequent commits, even if each commit is small.

Tables that receive streaming or micro-batch writes are prime candidates. Frequent commits tend to generate many small manifests. Even if data files are reasonably sized, the metadata layer becomes increasingly expensive to process.

A very common real-world pattern is a table that “looks healthy” in storage metrics but becomes steadily slower to query over weeks. Nothing is broken, nothing obvious changed, but planning time creeps up. That is almost always a manifest problem.

Managing Manifest Rewrites Manually

If you don’t use a control plane, the following sequence reflects what works well in production if done right and in context.

Step 1: Inspect Metadata Health

Before you decide to rewrite manifests, you need visibility into the live metadata — not guesswork, not periodic dashboards, but concrete numbers that reflect how fragmented the metadata has become.

Iceberg exposes metadata tables that you can query just like regular tables. These include tables like …$manifests, …$files, …$snapshots, etc. You can use these directly in SQL to inspect current state and spot trouble early.

Iceberg stores metadata in layers:

-   A **manifest list** per snapshot points to all manifests for that snapshot.
-   Each **manifest file** lists a subset of data files, partition values, and column statistics (min/max/null counts).
-   Manifests may be reused across snapshots.

As a result:

Lots of small commits → many small manifests.
Old snapshots hold onto old manifests.
Query engines read manifests during plan time to prune partitions/files.

If manifests are fragmented or numerous, query planning becomes slow because engines read and evaluate many metadata files before they touch actual data.

This is why metadata health matters early, not late.

What to Look At

Here are the core checks you should be doing regularly — ideally automated — to monitor manifest health.

🔍 1) Count the Current Manifests

Run a live count of manifests referenced by the current snapshot:

SELECT COUNT(\*) AS active\_manifest\_count  
FROM prod.db.my\_table$manifests;

A sudden jump in this number relative to data size usually correlates with planning overhead.

A steady climb over time, without data volume growth, is a strong indicator your metadata is fragmenting.

2) Look at Files per Manifest

Iceberg metadata stores statistics such as file counts per manifest. Pull a distribution:

SELECT  
  CASE  
    WHEN record\_count < 10 THEN '<10 rows'  
    WHEN record\_count BETWEEN 10 AND 100 THEN '10–100 rows'  
    ELSE '100+ rows'  
  END AS manifest\_size\_bucket,  
  COUNT(\*) AS manifests  
FROM prod.db.my\_table$manifests  
GROUP BY 1  
ORDER BY 1;

If you see lots of manifests with very few rows/files, that means fragmentation. It means many small manifests (from tiny commits) that blow up planning work.

You can also look at larger manifests: if lots of manifests hold small amounts of data, it’s a sign that maintenance will be valuable.

3) Compare Manifests to Data Growth

If you track how data size and manifest count change together, you can spot divergence:

\-- number of data files  
SELECT COUNT(\*) AS data\_file\_count  
FROM prod.db.my\_table$files;

\-- number of manifests  
SELECT COUNT(\*) AS manifest\_count  
FROM prod.db.my\_table$manifests;

If manifest_count grows faster than data_file_count, that’s another sign of metadata inefficiency.

4) Look at Snapshots (Optional but Useful)

Snapshots tell you how many historical versions you’re retaining, which impacts how many manifests persist:

SELECT  
  committed\_at,  
  snapshot\_id  
FROM prod.db.my\_table$snapshots  
ORDER BY committed\_at DESC  
LIMIT 10;

Long snapshot histories mean old manifests may still be referenced and not cleaned up until expiration happens.

Interpreting the Results

Here are practical heuristics data engineers use:

High manifest count with small average manifest size → metadata fragmentation (good candidate for rewrite).
Stable manifest count but slow query planning → the problem might be clustering, not count; manifest rewrites can help.
Lots of snapshots older than retention needs → metadata is being kept too long; expire them first so rewrites can be effective.
Manifest growth outpacing data file growth → metadata is drifting away from the current shape of data.

Example Scenario

Imagine a streaming table ingesting updates every minute. You might see:

5,000 data files
2,000 manifests
70% of manifests contain <10 files

That’s a classic candidate for consolidated manifests: smaller number of larger manifests will cut planning time dramatically, especially if queries filter on partitions that aren’t well clustered yet.

Step 2: Expire Snapshots First

Always expire snapshots before rewriting manifests. This is not a best-practice nicety — it directly determines whether a manifest rewrite will actually do anything useful.

The easiest way to achieve this is using a Control Plane for automated and optimized maintenance operations that include snapshot expirations in addition to manifest rewrites. Learn more:

Automated and optimized snapshot expiration with a Control Plane (source: lakeops.dev)

Her’es a deep dive into the topic and practical solutions:

11 Expire Snapshots Optimizations for Apache Iceberg

Snapshots are what keep manifests alive. Every snapshot references a specific set of manifest files that describe the table state at that point in time. As long as a snapshot exists, all of its manifests must remain reachable, even if they describe data that is no longer relevant for current queries.

If you run a manifest rewrite while old snapshots are still retained, Iceberg can only optimize the manifests referenced by the current snapshot. Older snapshots will continue to reference older manifests, which means:

old manifests stay in storage,
metadata fan-out remains higher than expected,
storage costs don’t drop,
and in some engines, planning still touches more metadata than necessary.

This is the most common reason teams say “we ran rewrite manifests and it didn’t really help”.

Why snapshot expiration comes first

Think of snapshot expiration as pruning the metadata graph.

Until you expire snapshots, Iceberg is obligated to preserve historical metadata for correctness and time travel. A rewrite cannot remove or consolidate manifests that are still referenced by retained snapshots. Expiring snapshots reduces the metadata surface area first, so the rewrite can actually consolidate what remains.

In practice, snapshot expiration is what turns a rewrite from “cosmetic” into “effective”.

Inspect snapshot history before expiring

Before expiring anything, look at what you’re retaining:

SELECT  
  snapshot\_id,  
  committed\_at,  
  operation  
FROM prod.db.my\_table$snapshots  
ORDER BY committed\_at DESC;

In many production systems, you’ll find snapshots going back weeks or months, even though nobody ever queries historical versions beyond a few days.

That’s usually accidental, not intentional.

Expire snapshots based on real needs

Snapshot retention should reflect actual recovery and audit requirements, not defaults or copy-pasted examples.

If you only need:

a few days of rollback for operational safety, or
short-term auditability,

then retaining dozens or hundreds of snapshots actively hurts metadata efficiency with no upside.

A common pattern is:

retain snapshots newer than a time threshold, and
always keep the last N snapshots as a safety net.

Example: expire old snapshots in Spark

Here’s a practical Spark SQL example:

CALL prod.system.expire\_snapshots(  
  table => 'db.my\_table',  
  older\_than => TIMESTAMP '2024-01-01',  
  retain\_last => 2  
);

This removes snapshots older than the specified timestamp while keeping the most recent snapshots for safety.

After this runs, many old manifests will become unreferenced — which is exactly what you want before rewriting manifests.

Validate the effect

After expiring snapshots, re-check your metadata:

SELECT COUNT(\*) AS remaining\_snapshots  
FROM prod.db.my\_table$snapshots;

You should see a much smaller snapshot set. At this point:

old manifests are no longer protected,
rewrite manifests can actually consolidate metadata,
and orphan cleanup will be able to reclaim storage.

Step 3: Run Rewrite Manifests

At this point, the current snapshot references only the metadata that still matters. That gives Iceberg room to consolidate and reorganize manifests instead of carrying forward historical baggage.

What this step actually does now

After snapshot expiration, rewrite manifests can:

merge many small manifests into fewer, larger ones,
reorganize data-file entries so they’re better clustered by partition and statistics,
reduce the amount of metadata the engine has to read during planning.

If you skip snapshot expiration, most of these benefits are muted. After expiration, they show up immediately in planning time and metadata size.

Running rewrite manifests (Spark example)

In Spark-based environments, this is usually done via a system procedure:

CALL prod.system.rewrite\_manifests('db.my\_table');

This executes the rewrite in parallel across the cluster. Spark will read the existing manifests, generate a new optimized set, and commit a new snapshot that references them.

The command itself is simple. The impact depends entirely on whether you prepared the table correctly in the earlier steps.

Validate that it actually worked

After the rewrite, always check the result:

SELECT COUNT(\*) AS manifest\_count\_after  
FROM prod.db.my\_table$manifests;

You should see a noticeable drop in manifest count or, at the very least, fewer very small manifests. If nothing changes, the usual reasons are:

snapshots were not expired, so old manifests are still referenced,
the table’s write pattern is fragmenting metadata faster than maintenance runs,
or the table is already in a reasonably healthy state.

Why this step is cheap — but not free

Rewrite manifests does not rewrite data files, so it’s much cheaper than compaction. However, it still:

reads all manifests referenced by the current snapshot,
writes new manifest files,
and commits new metadata.

On large tables with many manifests, this can still consume noticeable CPU, memory, and I/O. That’s why you should not run it blindly across hundreds of tables at once.

Practical scheduling guidance

If you’re doing this manually:

avoid peak query hours,
stagger rewrites across tables,
and gate execution on actual metadata health signals rather than time alone.

Running rewrite manifests selectively, when metadata drift is real, is what keeps it a high-ROI operation instead of background noise.

Step 4: Remove Orphan Files

Once snapshots are expired and manifests are rewritten, you need to clean up what is no longer referenced.

Orphan files are data or metadata files that exist in storage but are no longer referenced by any snapshot. They typically appear after snapshot expiration, manifest rewrites, failed jobs, or aborted commits. Iceberg does not delete these files automatically, because doing so without coordination would risk correctness.

If you stop after rewriting manifests, those unreferenced files will remain in object storage indefinitely.

From Iceberg’s point of view, everything is correct after a rewrite. From your cloud bill’s point of view, nothing changed.

Skipping orphan cleanup is one of the most common reasons teams see storage costs grow even though they “ran all the maintenance jobs.” The metadata graph is clean, but the physical files are still sitting in S3, GCS, or ADLS.

This step is what turns logical cleanup into actual cost reduction.

What orphan cleanup actually removes

Orphan cleanup removes:

old manifest files no longer referenced by any snapshot,
metadata files left behind by rewrites and expired snapshots,
data files created by failed or rolled-back writes.

It does not remove any file that is reachable from a live snapshot. If a file is still referenced, it stays.

Running orphan cleanup (Spark example)

In Spark environments, orphan cleanup is typically done with a system procedure:

CALL prod.system.remove\_orphan\_files(  
  table \=\> 'db.my\_table',  
  older\_than \=\> TIMESTAMP '2024-01-01'  
);

The older_than guard is critical. It ensures Iceberg only deletes files older than a safe cutoff, protecting against races with in-flight or recently committed jobs.

Never run orphan cleanup without a time threshold.

Validate the effect

After cleanup, you should see:

a reduction in storage usage over time,
fewer unreferenced metadata files,
and no change in query results or correctness.

Storage metrics won’t always drop instantly due to object store reporting delays, but the trend should flatten instead of creeping upward.

The key takeaway

Snapshot expiration and manifest rewrites clean up logical metadata. Orphan cleanup is what turns that into physical cleanup.

If you skip this step, maintenance looks successful on paper but storage costs keep rising. If you include it consistently, metadata maintenance finally translates into real, measurable savings.

Step 5: Coordinate with Compaction

Manifest rewrites optimize how files are described. Compaction optimizes how many files exist. If you ignore compaction, manifest rewrites will help briefly — then fragmentation will return.

Small data files are the main upstream cause of manifest churn. Every time a write job produces many small files, Iceberg must record them in metadata. Even if you rewrite manifests perfectly, frequent small-file writes will recreate fragmentation within days.

The optimal solution is to use an Iceberg Control Plane. LakeOps for example, compacts data 95% faster and cheaper than alternatives thanks to a rust based engine and analyzed cross-system data. Compaction is also smarter, so query times go up ans costs go down dramatically.

![](https://cdn-images-1.medium.com/max/1600/1*7w1IT-CzuDQRQsVuCionDA.png)

Compaction optimizagtion with a dcontrol plane (source:lakeops/dev)

In LakeOps, compaction processes are also synchronized with maintenance processes like manifest rewrites, so everything runs smoothly and you don’t have to connect and coordinate it yourself. Results are optimized and are usually far better than a home-made solution.

Example: why rewrites alone don’t hold

Let’s go back to the core problem for a second, and then see how to manually address it if you don’t use a control plane.

Consider a table with streaming ingestion committing every few minutes:

Each commit writes 20–50 small Parquet files.
Each commit creates one or more new manifests.
After a week, the table has thousands of data files and hundreds of manifests.

You run snapshot expiration and rewrite manifests. Planning time improves.

Two days later, the table is slow again.

Nothing is broken. The write pattern simply recreated the same metadata pressure. This is what happens when compaction is missing or misaligned.

Use metadata to confirm compaction pressure

Before scheduling more manifest rewrites, check whether small files are the real problem:

SELECT  
  COUNT(\*) AS file\_count,  
  AVG(file\_size\_in\_bytes) / 1024 / 1024 AS avg\_file\_mb  
FROM prod.db.my\_table$files;

If the average file size is far below your engine’s sweet spot, manifest rewrites are treating symptoms, not the cause.

Another useful signal:

SELECT  
  COUNT(\*) AS manifests,  
  SUM(added\_data\_files\_count) AS total\_files\_tracked  
FROM prod.db.my\_table$manifests;

If file counts are high and keep growing quickly, metadata pressure will return unless compaction slows it down.

How compaction stabilizes manifest rewrites

When compaction is running correctly:

fewer data files are created per write cycle,
manifests grow more slowly and stay denser,
rewrite manifests becomes an occasional cleanup, not a recurring firefight.

In stable tables, teams often find that:

compaction runs frequently (or continuously),
manifest rewrites run infrequently,
snapshot expiration runs regularly.

That balance is what keeps planning predictable.

Practical coordination rule

In production, a simple rule holds up well: If manifest rewrites are needed often, compaction is not doing enough.

If you find yourself rewriting manifests weekly or daily on the same tables, it’s usually a sign that upstream file layout is unstable.

Should you add compaction code here?

At this point in the guide, full compaction code examples are usually not helpful. Compaction is engine-specific, workload-specific, and already well-covered elsewhere. What matters here is understanding the dependency:

compaction reduces metadata churn,
reduced churn makes manifest rewrites effective,
without compaction, rewrites are temporary relief.

That mental model is more valuable than a generic compaction snippet.

Step 6: Add policies and guardrails

If you manage maintenance with scripts or schedulers, automation needs guardrails or it will drift out of alignment with reality.

The simpest way is to add a Control Plane.

Then you can define policies per table or for your entire lake.

Define maintenance policies with a control plane (source:lakeops.dev/)

Start by skipping inactive tables. Tables that are rarely queried or written to don’t need aggressive maintenance. Running rewrites on them just burns cluster resources without benefit.

Avoid peak query hours. Even though manifest rewrites are cheaper than data compaction, they still consume CPU, memory, and I/O. Running them during high query load increases contention and hurts user-facing performance.

Trigger maintenance based on observed metadata health, not time alone. Manifest count, average files per manifest, snapshot growth, and planning time trends are far better signals than “once a day” or “once a week”.

Finally, expect thresholds to change. Write patterns evolve, query behavior shifts, and what worked six months ago may be wrong today. Scripts that never get revisited slowly turn into background noise or, worse, a source of instability.

This is the point where many teams decide that maintaining guardrails manually is more work than it’s worth and move metadata maintenance into a control plane.

Practical steps to take

Rewrite manifests is one of those Iceberg operations that looks optional until it isn’t. When metadata is healthy, planning is fast and predictable. When it drifts, everything still works — just slower and more expensively. That’s why manifest issues often go unnoticed for a long time.

In this guide, we walked through what manifest files actually are, why they fragment in real systems, and what rewrite manifests really does under the hood. We covered when rewrites are worth running, why snapshot expiration has to come first, how orphan cleanup turns logical cleanup into real cost savings, and why compaction is the long-term stabilizer that keeps metadata from degrading again.

If you’re managing this manually, the step-by-step approach will get you there. If you want this handled continuously and optimzied, a Control Plane exists to do exactly that — operating and optimizing Iceberg tables based on real workload behavior instead of fixed schedules.

The big takeaway is that rewrite manifests only works well as part of a coordinated maintenance loop. Run it in isolation and the benefits are usually temporary. Pair it with snapshot expiration, compaction, and cleanup, and it becomes one of the highest-ROI metadata optimizations Iceberg offers.

If you’ve run into edge cases, different patterns, or lessons learned the hard way, feel free to share them in the comments.

Thanks for reading, and hope this helps keep your Iceberg tables healthy, fast, predictable, and boring in all the right ways.

Cheers 🍺

Learn more

11 Iceberg Performance Optimizations You Should Know.

13 Apache Iceberg Optimizations You Should Know.

11 Apache Iceberg Cost Reduction Strategies You Should Know.

9 Data Lake Cost Optimization Tools You Should Know.

11 Must-Know FrontEnd Trends for 2020

joni sar — Sun, 29 Dec 2019 12:30:59 +0000

Or- how to sound smart in frontEnd lunch conversations!

Sounding smart at your team's lunch talks is obviously a great reason to stay updated with the latest frontend trends. It might even help you become a better developer, build better technology and better products. Maybe.

So, please allow me to make this honorable quest easier by pointing you in a few interesting directions. I will not explain every concept A-Z, but will introduce the concept, how it’s useful and direct to further resources.

For example, we’ll shortly cover an introduction to Micro Fontends, Atmoic Design, Web components TS take-over, ESM CDN and even Design tokens. Feel free to scroll through and mark the topics you’d like to learn more about. For any questions or more suggestions, just drop a comment below.

Short disclaimer: I'm on the team building Bit. This doesn't make any of the following less true though. Enjoy!

1. Micro frontends

Micro Frontends are the buzziest frontend topic for lunch conversations.
Ironically, while frontend development enjoys the modular advantages of components, it is still largely more monolithic than backend microservices.

Micro frontends bring the promise of splitting your frontend architecture into different frontends for different teams working on different parts of your app. Each team can gain autonomy over the end-to-end lifecycle of their micro frontend, which can be developed, versioned, tested, built, rendered, updated and deployed independently (using tools like Bit for example).
Instead of explaining the whole concept here, **read this great post** by @thecamjackson published at the @martinfowler blog. It’s really good and should cover everything you need to start digging into this concept.

However, there are still certain shortages in today’s ecosystem. Mostly, people are worried by issues like the deployments of separate frontends, bundling, environment differences etc. Bit already lets you isolate, version, build, test and update individual frontends/components. For now, this is mainly useful when working with multiple applications (though It’s already commonly used for gradually refactoring parts of existing apps via components).
When Bit will introduce deployments in 2020, independent teams will get the power to develop, compose, version, deploy and update standalone frontends. It will let you compose UI apps together and let teams create simple decoupled codebases with independent continuous deployments and incremental upgrades. The composition of these frontends will end up creating your application. Here's how a UI app composed with Bit looks like:

Learn more:
Micro Frontends - Martin Fowler

2. Atomic Design

Atomic Design is yet another super interesting topic for lunch talks, which I like to think about more of as a philosophy than a pure methodology.
Simply put, the theory introduced by Brad Frost compares the composition of web applications to the natural composition of Atoms, Molecules, Organisms and so on- ending with concrete web pages. Atoms compose molecules (e.g. text-input + button + label atoms = search molecule). Molecules compose an organism. Organisms live in a layout template, which can be concretized into a page delivered to your users.
Here’s a *detailed 30-seconds explanation with visual examples. *It includes very impressive drawings I made with great artistic talent, which you can copy-paste to your office board 😆
The advantages of Atomic components go beyond building modular UI applications through modular and reusable components. This paradigm forces you to think in composition so you better understand the role and API of every component, their hierarchy, and how to abstract the building process of your application in an effective and efficient way. Take a look.

3. Encapsulated Styling and Shadow Dom

Source: developer.mozzila.org
An important aspect of components is encapsulation — being able to keep the markup structure, style, and behavior hidden and separate from other code on the page so that different parts do not clash, and the code can be kept nice and clean. The Shadow DOM API is a key part of this, providing a way to attach a hidden separated DOM to an element.
Shadow DOM is actually used by browsers for a long time now. You can think of the shadow DOM as a “DOM within a DOM”. It is its own isolated DOM tree with its own elements and styles, completely isolated from the original DOM.
It allows hidden DOM trees to be attached to elements in the regular DOM tree — this shadow DOM tree starts with a shadow root, underneath which can be attached to any elements you want, in the same way as the normal DOM. The main implication of this is that we have no need for a namespace for our classes, as there’s no risk of name clashing or style spilling. There also additional advantages. It is often referred to as the long-promised solution to a true encapsulation of styles for web components. Learn more:
Using shadow DOM

4. The TypeScript take over

So lately every conversation makes it sound like TS is taking over frontend development. It is reported that **80% of developers admit they would like to use or learn TypeScript in their next project**.
Although it has it’s shortcomings, TS code is easier to understand, faster to implement, it produces less bugs and requires less boilerplate. Want to refactor your React app to work with TS? Go for it. Want to start gradually? Use tools like Bit to gradually refactor components in your app to TS and use the React-Typescript compiler to build them independently from your app. This way to can gradually upgrade your code one component at a time.
learn more:
Why TypeScript is the best way to write Front-end in 2019And why you should convince everybody to use it.
TypeScript at Lyft
TypeScript at Slack

5. Web components- Stencil, Svelte, Lit & friends!

So basically, this is the future. Why? because these pure web components are framework agnostic and can work without a framework or with any framework- spelling standardization. Because they are free from JS fatigue and are supported by modern browsers. Because their bundle size and consumption will be optimal, and VDOM rendering is mind-blowing.
These components provide Custom Element, a Javascript API that allows you to define a new kind of html tag, HTML templates to specify layouts, and of course the Shadow DOM which is component-specific by nature.
Prominent tools to know in this space are **Lit-html (and Lit-element), **StencilJS, **SvelteJS and of course Bit**, for reusable modular components which can be directly shared, consumed and developed anywhere.
When thinking of the future of our UI development, and of how principles of modularity, reusability, encapsulation, and standardization should look like in the era of components, web components are the answer. Learn more:

6. From component libraries to dynamic collections

Organize components in dynamic collections; reuse, compose, stay independent
The emergence of component-driven development gave birth to a verity of tools. One prominent tool is Bit, alongside it’s hosting platform Bit.dev.
Instead of working hard to build a cumbersome and highly-coupled component-library, use Bit to continuously isolate and export existing components into a dynamically reusable shared-collection.
Using Bit (GitHub) you can independently isolate, version, build, test and update UI components. It streamlines the process of isolating a component in an existing app, harvesting it to a remote collection, and using it anywhere. Every component can build, test, and render outside of any project. You can update a single component (and it’s dependants) and not the whole app.

In the bit.dev platform (or on your own server) your components can be remotely hosted and organized for different teams, so that every team can control the development of their own components. Every team can share and reuse components but keep their independence and control.
The platform also provides the all-in-one ecosystem for a shared components out-of-the-box: It auto-documents UI components, renders components in an interactive playground, and even provides a built-in registry to install components using npm/yarn. In addition, you can bit import components for modifications in any repository.

In the short run, this revolutionizes the process of sharing and composing components in a similar way to how Spotify/iTunes changed the process of previously sharing Music through static CD Music Albums. It’s a dynamic and modular solution that lets everyone share and use components together.
In the long run, Bit helps pave the way to micro-frontends. Why? Because it already lets you independently version, test, build and update parts of your UI application. In 2020 it will introduce independent deployments, which will finally allow different teams to own parts of your apps end-to-end: keep decoupled and simple codebases, let teams cautiously and continuously build and deploy incremental UI upgrades, and compose frontends together.

7. State management: Bye Bye Redux? (Not….)

Redux is a hard beast to kill. While the pains of globally managing states in your app are becoming more clear as frontend becomes more modular, the sheer usefulness of Redux makes it a go-to solution for many teams.
So will we say bye-bye to Redux in 2020? Probably not entirely 😄
However, the uprising of new features within frameworks that handle states (React hooks, Context-API etc) are painting the way to a future without a global store. Tools like Mobx, which only a year ago were rather scarcely adopted, are becoming more popular every day thanks to their component-oriented and scalable nature. You can explore more alternatives here.
Read: *Making Sense of React Hooks* — by Dan Abramov

8. ESM CDN

ES Modules is the standard for working with modules in the browser, standardized by ECMAScript. Using ES modules you can easily encapsulate functionalities into modules which can be consumed via CDN etc. With the release of Firefox 60, all major browsers will support ES modules, and the Node mteam is working on adding ES module support to Node.js. Also, ES module integration for WebAssembly is coming in the next few years. Just imagine modular Bit UI components composed in your app via CDN…

9. Progressive web apps. Still growing.

Progressive web applications take advantage of the latest technologies to combine the best of web and mobile apps. Think of it as a website built using web technologies but that acts and feels like an app. Recent advancements in the browser and in the availability of service workers and in the Cache and Push APIs have enabled web developers to allow users to install web apps to their home screen, receive push notifications and even work offline.
Since PWAs provide an intimate user experience and because all network requests can be intercepted through service workers, it is imperative that the app be hosted over HTTPS to prevent man-in-the-middle attacks, which also spells better security. Here’s a great talk by Facebook developer Omer Goldberg outlining best practices for PWAs.

10. Designer-developer integrations

With the uprise of component-driven design systems to enable a consistent UI across products and teams, new tools have emerged to bridge the gap between designers and developers. This is no simple task however; While code itself is really the only source of truth (this is what your user really gets), most tools try to bridge the gap from the designer’s end. In this category you can find Framer, Figma, Invision DSM and more.
From the developer’s end you can see how platforms like Bit.dev, which host your next-gen component library and helps create adoption for shared components. The platform provides rendered visualization for your actual source-code so that designers can collaborate wit developers and create discussions over the source-code itself, in a visual way.
Another promising idea to take note of is design-tokens. Placing tokens in your code through which designers can really control simple styling aspects (e.g. colors) directly through external collaboration tools. Integrated with platforms like Bit.dev, this can create a tighter workflow than ever before.

UI Component Design System: A Developer’s Guide

JoniSar ・ Oct 23 '19 ・ 10 min read

#design #ui #javascript #frontend

UI Design System and Component Library: Where Things Break | by Jonathan Saring | codeburst

Jonathan Saring ・ Aug 22, 2019 ・ 8 min read
Medium

7 Tools for Building Your Design System in 2020 | by Jonathan Saring | Bits and Pieces

Jonathan Saring ・ Dec 4, 2019 ・ 11 min read
Medium

11. Web assembly — into the future?

Web assembly brings language diversity into web development to cover gaps created by JavaScript. It is defined as a “a binary instruction format for a stack-based virtual machine. Wasm is designed as a portable target for compilation of high-level languages like C/C++/Rust, enabling deployment on the web for client and server applications”.
In his post, Eric Elliott elegantly outlines the concept’s benefits:

An improvement to JavaScript: Implement your performance critical stuff in wasm and import it like a standard JavaScript module.
A new language: WebAssembly code defines an AST (Abstract Syntax Tree) represented in a binary format. You can author and debug in a text format so it’s readable.
A browser improvement: Browsers will understand the binary format, which means we’ll be able to compile binary bundles that compress smaller than the text JavaScript we use today. Smaller payloads mean faster delivery. Depending on compile-time optimization opportunities, WebAssembly bundles may run faster than JavaScript, too!
A Compile Target: A way for other languages to get first-class binary support across the entire web platform stack To learn more about this concept, why it’s useful, where it will be used and why it’s not here yet, I suggest this great post and this great video. Why We Need WebAssembly: An Interview with Brendan Eich *Brendan Eich & Eric Elliott Discuss WebAssembly Details*medium.com