DEV Community: joni sar

Apache Iceberg Query Performance: A Practical Guide

joni sar — Sun, 02 Aug 2026 09:31:14 +0000

Iceberg is the standard. Snowflake, Databricks, AWS, and every major query engine read and write it natively. The format question is settled.

The performance question is not.

Most production Iceberg tables are slower than they need to be — not because of anything wrong with the format, but because the physical state of the table has degraded over time. Thousands of small files from streaming writes. Manifests fragmented across hundreds of snapshots. Data scattered randomly across files with no correlation to how it's actually queried. Delete files piling up from CDC pipelines.

The result: engines scan 5–10x more data than necessary. Query planning takes seconds instead of milliseconds. S3 API costs spike. And every engine — Trino, Spark, Snowflake, Athena, DuckDB — is equally affected because they all read the same physical data layout.

This guide covers what determines Iceberg query performance, how to fix it, and the two paths available: intelligent continuous optimization through a control plane, and the manual approach with SQL and cron. Both paths target the same physical levers — the difference is whether they adapt over time or require ongoing human attention.

How Iceberg scan planning works

Understanding the performance machinery is essential for both approaches. When a query engine receives a SELECT against an Iceberg table, it runs a scan planning process on a single node that determines which files to read. This is one of Iceberg's core advantages over Hive-style directory listing — no distributed scan needed just to find your data.

Planning operates in three progressive levels of elimination:

Level 1: Manifest list pruning. The engine reads the snapshot's manifest list and checks partition value ranges in each manifest's summary. If a manifest's date range doesn't overlap with the query's filter, it's skipped entirely — along with all data files it references. One check eliminates hundreds of manifest files.

Level 2: File-level data skipping. For surviving manifests, the engine reads per-file column statistics (min/max bounds, null counts, row counts) and evaluates predicates against them. A query for amount > 500 skips every file where the amount column's maximum is below 500. This is where sort order determines everything — tight, non-overlapping min/max ranges mean more files eliminated.

Level 3: Row group pruning. Within surviving files, Parquet row group statistics and bloom filters skip sub-file data blocks. If only one of four row groups matches, 75% of the file's I/O is avoided.

The effectiveness of each level depends entirely on the physical state of the table. A detailed walkthrough covers these internals in depth.

Why tables degrade

Iceberg tables don't start slow. They become slow because the physical layout drifts away from what's optimal for the queries hitting them.

Streaming writes create files at every checkpoint interval (every 1–5 minutes). A table receiving continuous events generates hundreds or thousands of files per day — each one tiny, each one adding a manifest entry that must be evaluated during planning.

Static sort orders lose relevance. A sort key chosen at table creation doesn't reflect what queries actually filter on six months later. As access patterns evolve, the sort becomes meaningless and min/max statistics become wide and useless for pruning.

Maintenance runs in isolation. Compaction, snapshot expiration, manifest rewriting, and orphan cleanup each affect each other's effectiveness. Running them independently — or in the wrong order — means wasted work and suboptimal results.

CDC pipelines accumulate delete files. Every update writes a small delete file. After weeks, hundreds of delete files must be reconciled at read time — each query pays the cost.

No feedback loop. Without observing which columns queries actually filter on, which tables are degrading fastest, and which maintenance actions produce the most benefit — optimization is guesswork.

These problems don't happen once and get fixed. They're continuous. They happen every day, on every table, as long as data is being written and queried.

Two paths to query performance

There are fundamentally two approaches to keeping Iceberg tables performant:

Path 1: Intelligent continuous optimization. A system that observes query patterns, understands table state, and takes the right action at the right time — autonomously, adaptively, and in the correct sequence. This is the control plane approach.

Path 2: The manual path. SQL procedures, cron jobs, and configuration tuning. You run the same operations — compaction, manifest rewrites, snapshot expiration — but you decide when, how, and on which tables. You choose the sort order. You monitor for degradation. You sequence the operations correctly.

Both paths target the same physical levers described above. The difference is not what gets done — it's whether it adapts continuously or requires ongoing engineering attention.

Path 1: Intelligent continuous optimization

LakeOps is an autonomous control plane for Apache Iceberg. It connects to your existing catalogs and query engines, observes table state and query patterns, and continuously applies the optimizations that make queries faster — without human intervention, without Spark clusters, and without static configurations that go stale.

Here's what it does and why each piece improves query performance.

Query-aware data layout

This is the highest-impact capability. Sort order is the single most impactful lever for Iceberg query performance — sorted tables scan 51% less data per query than unsorted ones. But choosing the right sort order requires knowing which columns production queries actually filter on. That knowledge changes over time.

LakeOps captures query telemetry from every connected engine — Spark, Trino, Flink, Snowflake, Athena, DuckDB, StarRocks — and identifies which columns appear in WHERE, JOIN, and GROUP BY clauses for each table. During compaction, data is physically re-sorted by those columns.

The result: Parquet row-group min/max statistics become maximally tight. The scan planner eliminates the vast majority of files at Level 2 because each file covers a narrow, non-overlapping value range for the columns queries actually use.

The sort strategy adapts. If a new dashboard starts filtering on customer_segment alongside event_date, the next compaction pass incorporates it. No manual ALTER TABLE SET SORT ORDER needed. On three consecutive runs of a 1.2 TB table, the system improved runtime from 22 min → 18 min → 11 min as it converged on optimal column ordering — zero configuration changes.

Production numbers: a table with 47,000 scattered files, after query-aware sort compaction to 280 files, saw query time drop from 52 seconds to 5.8 seconds. A 9x improvement from layout optimization alone.

Compaction at streaming speed

Sort compaction is the most impactful maintenance operation — it solves both the small-file problem and the sort-order problem in one pass. But on Spark, it's slow and expensive. A 200 GB sort compaction takes ~25 minutes and costs ~$3.50 on EMR. At that speed, running it multiple times per day on streaming tables is impractical. Most teams default to binpack (size-only) because sort is too expensive — leaving the biggest performance lever unused.

LakeOps's compaction engine is built in Rust on Apache DataFusion. No JVM. No garbage collection. No OOM crashes. The speed difference makes sort compaction viable even for streaming tables:

Binpack: 221 seconds vs 1,612s (Spark) vs 6,300s (S3 Tables) on identical 200 GB dataset
Sort: 780 seconds vs estimated 3,000+ for Spark
Cost: $0.21 vs $1.54 for 200 GB binpack
Peak throughput: 2,522 MB/s — TB-scale tables in minutes
Bounded memory: spills to disk gracefully, no OOM regardless of table size

Because it's fast enough to run frequently, tables stay both consolidated and sorted continuously — never degrading between maintenance windows.

Coordinated maintenance pipeline

Individual maintenance operations interact. The order matters:

Expire snapshots — release references to old data files
Remove orphans — delete unreferenced files from storage
Compact — merge remaining files with optimal sort order
Rewrite manifests — consolidate metadata over the new, clean file set

Running these in the wrong order wastes work. Compacting before expiring snapshots means you might rewrite files that should be deleted. Rewriting manifests before compaction means the structure changes again during the compaction commit.

LakeOps sequences operations automatically. Each step produces a clean input for the next. The manifest rewrite runs on an already-consolidated, properly-sorted file set — producing the leanest possible metadata layer. This coordination eliminates the redundant work that independent cron jobs inevitably create.

Event-driven triggers

A streaming table that accumulates 500 files per hour needs compaction multiple times per day. A stable batch table with weekly loads needs compaction monthly. A CDC table with accumulating delete files needs cleanup when the read-time overhead crosses a threshold.

LakeOps triggers operations based on actual table state — file count thresholds, delete-file ratios, manifest fragmentation, partition skew — not arbitrary cron schedules. Tables get exactly the maintenance they need, when they need it. No wasted runs on healthy tables, no missed runs on degrading ones.

Multi-engine query routing and workload optimization

Different engines have different performance profiles: a point lookup takes 0.5s on DuckDB vs 2.3s on Athena. A full-table scan costs $5 on Athena vs $50 on Trino. Without routing, every query goes to the same engine regardless of its shape.

LakeOps includes QueryFlux — an open-source, Rust-based SQL proxy with 0.35ms overhead — for multi-engine routing. Queries route based on shape, latency targets, cost ceilings, engine availability, and table health status. The routing layer learns from execution history: if a query shape consistently runs faster on one engine, future executions route there.

In benchmarking, workload-aware routing reduced total query cost by up to 80%, with individual queries sometimes dropping by 90%.

Observability and self-improvement

You can't optimize what you can't see. LakeOps classifies every table into health tiers (Critical, Warning, Healthy) based on file fragmentation, manifest depth, snapshot velocity, delete ratios, and sort-order staleness. Insights surface at four severity levels — before degradation becomes noticeable in query times.

The system records per-table throughput, partition structure, and memory usage from each compaction run. Subsequent passes execute faster as the planner converges on optimal resource allocation. Every operation feeds back into the next decision.

What setup looks like

Connect your catalogs (AWS Glue, REST catalogs like Polaris/Nessie/Lakekeeper, S3 Tables) and storage. ~10 minutes. No agents to deploy, no data movement, no pipeline changes. Your data stays in your account. The system discovers tables, classifies health, begins autonomous maintenance according to policies you define.

Production results across customers: up to 12x average query acceleration, up to 80% total cost reduction, 786+ tables managed autonomously across 112+ PB.

Path 2: The manual approach

Every optimization LakeOps automates is also achievable with SQL procedures, engine configuration, and scheduling. Here's how to implement each one by hand.

Diagnosing the bottleneck

Before optimizing, use Iceberg's metadata tables to identify what's actually wrong:

-- File health: count, size distribution, small file ratio
SELECT 
  COUNT(*) as file_count,
  AVG(file_size_in_bytes) / 1048576 as avg_mb,
  COUNT(CASE WHEN file_size_in_bytes < 67108864 THEN 1 END) as small_files
FROM prod.db.events.files;

-- Manifest health: fragmentation
SELECT COUNT(*) as manifest_count,
       AVG(added_data_files_count) as avg_files_per_manifest
FROM prod.db.events.manifests;

-- Delete file accumulation (MoR tables)
SELECT COUNT(*) as delete_file_count
FROM prod.db.events.delete_files;

-- Snapshot accumulation
SELECT COUNT(*) as snapshot_count,
       MIN(committed_at) as oldest
FROM prod.db.events.snapshots;

Thresholds: avg file size < 128 MB → compaction needed. Manifest count > 200 → rewrite manifests. Delete files > 50 → compact to apply. Snapshots > 1,000 → expire.

Use EXPLAIN to verify predicates push down to the scan level rather than appearing as residual filters.

Sort order configuration

Choose columns based on your most common query patterns:

-- Set sort order (first column gets tightest clustering)
ALTER TABLE prod.db.events WRITE ORDERED BY event_date, user_id;

-- Sort compaction to apply the order to existing data
CALL system.rewrite_data_files(
  table => 'prod.db.events',
  strategy => 'sort',
  sort_order => 'event_date ASC NULLS LAST, user_id ASC NULLS LAST'
);

-- For tables with diverse multi-column filters, use Z-order
CALL system.rewrite_data_files(
  table => 'prod.db.events',
  strategy => 'sort',
  sort_order => 'zorder(event_date, user_id, region)'
);

Guidelines for choosing sort columns:

First key: Column most frequently in WHERE clauses. Gets tightest min/max ranges
Second key: Diminishing returns, but meaningful for two-column filters
Z-order: When no single column dominates filter patterns

The challenge: you must manually identify which columns queries filter on, and update the sort order when access patterns change. Most teams don't revisit this after initial setup.

File consolidation (compaction)

-- Binpack: merge small files by size (fast, no reordering)
CALL system.rewrite_data_files(
  table => 'prod.db.events',
  strategy => 'binpack',
  options => map('target-file-size-bytes', '268435456', 'min-file-size-bytes', '67108864')
);

-- Sort compaction: merge + apply sort order (slower, better for queries)
CALL system.rewrite_data_files(
  table => 'prod.db.events',
  strategy => 'sort',
  sort_order => 'event_date ASC NULLS LAST, user_id ASC NULLS LAST',
  options => map('target-file-size-bytes', '268435456')
);

Target: 256–512 MB files for analytical workloads. Schedule via Airflow/cron — daily for streaming tables, weekly for batch.

Preventing small files at write time

The cheapest compaction is the one you never need:

ALTER TABLE prod.db.events SET TBLPROPERTIES (
  'write.target-file-size-bytes' = '268435456',    -- 256 MB
  'write.distribution-mode' = 'hash',              -- Group by partition before write
  'write.parquet.compression-codec' = 'zstd'       -- Better compression
);

Critical: write.distribution-mode = 'none' on partitioned tables is the #1 source of small files. With 200 Spark tasks writing to 365 daily partitions, you get 73,000 tiny files per job. Always use hash for partitioned tables.

For Spark, align AQE with your target file size:

spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "true")
spark.conf.set("spark.sql.adaptive.advisoryPartitionSizeInBytes", "268435456")

For Flink streaming writes, increase checkpoint intervals to reduce file frequency:

SET 'execution.checkpointing.interval' = '5min';
SET 'sink.committer.operator.commit-interval' = '5min';

Bloom filters for point lookups

When queries use equality predicates on high-cardinality columns (WHERE user_id = 12345), bloom filters eliminate row groups that provably don't contain the value:

ALTER TABLE prod.db.events SET TBLPROPERTIES (
  'write.parquet.bloom-filter-enabled.column.user_id' = 'true',
  'write.parquet.bloom-filter-enabled.column.session_id' = 'true',
  'write.parquet.bloom-filter-fpp.column.user_id' = '0.05'
);

-- Generate filters on existing data via compaction
CALL system.rewrite_data_files(
  table => 'prod.db.events',
  options => map(
    'write.parquet.bloom-filter-enabled.column.user_id', 'true',
    'write.parquet.bloom-filter-fpp.column.user_id', '0.05'
  )
);

Limit to 2–3 columns per table (each adds ~1 MB to every file footer). Only useful for equality predicates — range queries don't benefit.

Manifest consolidation

-- Check if needed
SELECT COUNT(*) FROM prod.db.events.manifests;

-- Consolidate (run AFTER compaction)
CALL system.rewrite_manifests('prod.db.events');

Target: 10–50 manifests. Run after compaction stabilizes the file set — not before.

Snapshot expiration and orphan cleanup

-- Expire snapshots older than 7 days, keep at least 5
CALL system.expire_snapshots(
  table => 'prod.db.events',
  older_than => TIMESTAMP '2026-07-26 00:00:00',
  retain_last => 5
);

-- Remove orphan files from failed writes
CALL system.remove_orphan_files(
  table => 'prod.db.events',
  older_than => TIMESTAMP '2026-07-26 00:00:00'
);

Guidelines: 3–7 days retention for streaming tables, 30–90 days for batch. Always retain_last >= 2 to avoid breaking concurrent readers.

Delete file cleanup (MoR tables)

-- Compact to apply pending deletes
CALL system.rewrite_data_files(
  table => 'prod.db.events',
  options => map('delete-file-threshold', '3')
);

Monitor regularly. A table accumulating 200+ delete files/day will see read times increase from 3s → 22s over two weeks.

Partition strategy

-- Set partition (at table creation or via evolution)
CREATE TABLE prod.db.events (
  event_id BIGINT,
  event_time TIMESTAMP,
  user_id BIGINT,
  region STRING
) USING iceberg
PARTITIONED BY (day(event_time));

-- Evolve without data rewrite (metadata-only, instant)
ALTER TABLE prod.db.events SET PARTITION SPEC (day(event_time));

Rules: each partition should hold at least 128 MB. Daily partitioning on a 10 MB/day table → use monthly. bucket(col, 1024) on a 50 GB table → try 16–64 buckets instead.

Puffin statistics for join planning

-- Generate NDV statistics for cost-based optimization
CALL system.compute_table_stats(table => 'prod.db.events');

-- Or via ANALYZE TABLE (Spark)
ANALYZE TABLE prod.db.events COMPUTE STATISTICS FOR COLUMNS
  user_id, event_type, region;

Recompute after major data changes. For details: Puffin Statistics Guide.

Parquet-level tuning

ALTER TABLE prod.db.events SET TBLPROPERTIES (
  'write.parquet.row-group-size-bytes' = '134217728',   -- 128 MB row groups
  'write.parquet.page-size-bytes' = '1048576',          -- 1 MB pages
  'write.parquet.dict-size-bytes' = '2097152',          -- 2 MB dict threshold

  -- Disable stats for columns never filtered on (reduces manifest bloat)
  'write.metadata.metrics.column.raw_payload' = 'none',
  'write.metadata.metrics.column.debug_info' = 'none'
);

For read-side tuning in Spark:

spark.conf.set("spark.sql.files.maxPartitionBytes", "268435456")  # 256 MB per task
spark.conf.set("spark.sql.files.openCostInBytes", "4194304")      # 4 MB open cost

Copy-on-Write vs. Merge-on-Read

Choose your update strategy based on your read/write ratio:

Workload	Strategy	Why
Batch reporting (updated nightly, read all day)	Copy-on-Write	Zero read overhead, clean files always
Streaming CDC (Debezium/Kafka)	Merge-on-Read + compaction	Write speed critical, compact to restore reads
Infrequent GDPR/compliance deletes	Merge-on-Read + periodic cleanup	Fast compliance, clean up later
High-frequency MERGE INTO	Deletion Vectors (V3)	Best write + read performance

The correct maintenance sequence

Whether automated or manual, the order of operations matters:

Expire snapshots → releases references to dead files
Remove orphans → deletes unreferenced files from storage (cheaper subsequent compaction)
Compact (with sort) → merges files, applies sort order, resolves delete files
Rewrite manifests → consolidates metadata over the clean, final file set

Running these out of order wastes compute. Compacting before expiring rewrites files that should be deleted. Manifest rewrites before compaction get invalidated immediately.

Performance troubleshooting checklist

Symptom	Likely cause	Fix
Slow range queries	Unsorted data, many small files	Sort compaction on filter columns
Slow point lookups	No bloom filters	Enable per-column bloom filters + compact
Slow query planning (>5s)	Too many manifests	`rewrite_manifests`
Full table scans	Wrong partition scheme	Partition evolution
Gradually slowing reads	Accumulating delete files	Compact to apply deletes
High S3 API costs	Thousands of small files	Compaction + write-side tuning
Dashboard timeouts	Multiple degradation factors	Full maintenance pipeline + routing

Choosing your path

The manual path works. Every SQL procedure shown above is production-proven. The challenge is sustaining it:

Choosing the right sort order requires analyzing query logs across all engines — and updating when patterns change
Scheduling compaction requires different frequencies for different tables — streaming tables need it hourly, batch tables weekly
Sequencing operations correctly requires understanding their dependencies
Monitoring 50+ tables for degradation signals requires dashboard infrastructure
Scaling to hundreds of tables means the maintenance burden grows linearly with your data estate

For teams with fewer than 10–20 tables and stable access patterns, the manual path is manageable. Beyond that, the operational burden compounds — and the sort order inevitably goes stale because nobody has time to analyze query patterns per table per quarter.

The control plane approach eliminates this scaling problem. It handles the continuous observation, adaptation, and execution at any table count — and because its compaction engine is 95% faster than Spark, it can apply sort compaction at frequencies that would be cost-prohibitive manually.

Both paths lead to the same destination: tables where queries are 12x faster because the physical layout matches how data is actually accessed. The question is whether you want that as a continuous property of your lake, or as a point-in-time optimization that needs periodic human re-tuning.

Summary

Iceberg query performance is determined by the physical state of your tables — not by the format spec, not by the engine, and not by hardware. A well-maintained table is fast on every engine. A degraded table is slow on all of them.

The physical levers that matter:

Sort order aligned to query patterns — 51% less data scanned, up to 12x faster queries
File consolidation — 5–10x faster planning, reduced I/O
Bloom filters — near-instant elimination for equality lookups
Manifest optimization — 2–5x faster planning
Write-side configuration — prevent degradation at source
Delete file cleanup — restore read performance on MoR tables
Snapshot lifecycle — lean metadata, lower storage costs
Partition strategy — coarse-grained elimination at the foundation
Multi-engine routing — right engine per query, up to 56% cost reduction

The intelligent path keeps all of these optimal continuously. The manual path gives you the tools to do it yourself. Either way — the performance ceiling is the physical state of your tables, and now you know how to raise it.

Further reading:

AI Data Lake and Lakehouse

joni sar — Fri, 24 Jul 2026 14:07:16 +0000

The word "AI" appears in every data platform pitch now. Most of the time, it means nothing — a chatbot bolted onto a dashboard, an LLM that generates SQL, or a marketing slide that says "AI-powered" next to a cron job.

This article is about something different: what happens when you put actual intelligence — machine learning, adaptive optimization, closed-loop feedback systems — at the core of how a data lakehouse operates. Not AI as a feature. AI as the operating model.

The result is a lakehouse that understands its own workloads, optimizes its own storage layouts, routes its own queries, heals its own tables, and continuously improves — without human intervention. That is what an AI data lake actually is. Everything else is automation with better branding.

Automation Is Not Intelligence
The AI Control Plane
How AI Transforms Each Layer of the Lakehouse
AI-Driven Compaction: Learning How Data Is Read
AI-Driven Maintenance: Predicting What Breaks
AI-Driven Query Routing: Matching Queries to Engines
AI-Driven Observability: From Dashboards to Diagnosis
AI-Driven Governance: Policies That Adapt
The Closed Loop: How Every Query Makes the Lake Smarter
AI Agents as Lake Citizens
AI-Native vs. AI-Washed: How to Tell the Difference
The Compounding Effect
Where This Is Heading

Automation Is Not Intelligence

Most data lake "automation" today works like this: a cron job runs compaction every night at 2 AM. It merges small files into bigger ones. It does not know if the table needs compaction. It does not know which partitions are degraded. It does not know what columns queries filter on. It does not know whether last night's compaction actually helped.

That is automation. It follows a fixed rule, regardless of context.

Intelligence is different. Intelligence observes, learns, decides, acts, measures the outcome, and adjusts. A system that compacts a table because its file health crossed a threshold, sorts by the columns that real queries actually filter on, sequences the operation after snapshot expiration to avoid wasted work, and then verifies that query latency improved — that is intelligent.

The distinction matters because data lakes are not static systems. Workloads shift. New dashboards appear. AI agents start querying tables that were designed for nightly batch. Streaming pipelines change write patterns. A schema evolves.

Fixed rules cannot keep up. Intelligence can.

The AI Control Plane

An AI data lake needs a brain — a system that continuously ingests telemetry from every table, every engine, and every query, and uses that signal to drive optimization decisions across the entire lake.

LakeOps is that system. It is an autonomous control plane for Apache Iceberg that puts AI at the center of lakehouse operations — compaction, maintenance, query routing, observability, governance, and agent enablement, all driven by continuous learning rather than static rules.

What makes it a control plane and not another tool: LakeOps is not an engine, not a catalog, and not storage. It connects to whatever you already run — AWS Glue, Polaris, Nessie, Gravitino, Lakekeeper, or S3 Tables as catalogs; Trino, Spark, Snowflake, Athena, DuckDB, Flink as engines; S3, GCS, or ADLS as storage — and adds the intelligence layer on top. No data movement. No code changes. No vendor lock-in. Setup takes about ten minutes.

What makes it AI-native: every decision LakeOps makes is informed by learned behavior, not hardcoded rules. It observes query patterns across all engines, predicts table degradation from structural signals, adapts data layouts as workloads shift, learns which engine performs best for each query shape, and measures the outcome of every optimization to improve the next one. It also exposes a native MCP (Model Context Protocol) server so AI agents can query the lake directly — with schema discovery, intelligent routing, and layered guardrails built in. The more the lake is used — by humans and agents alike — the smarter the control plane gets.

Three operating modes let teams choose how much autonomy the AI gets:

Autopilot — the AI decides and executes. Define policies and thresholds; the system handles everything.
Recommendations — the AI analyzes and recommends with full context (what, why, expected impact). A human approves.
Policy-bounded — the AI operates within declared boundaries, escalating edge cases.

Most teams start with recommendations to build trust, then move tables to autopilot as confidence grows. The intelligence is the same in all modes — the only difference is who clicks "execute."

With that foundation, here is how AI transforms each layer of lakehouse operations.

How AI Transforms Each Layer of the Lakehouse

When we talk about an "AI data lake," we are talking about specific capabilities that require learning, prediction, and adaptation — not features that could be implemented with a shell script.

Layer	What AI Does	What Scripts Cannot
Compaction	Learns query patterns across engines, sorts data by the columns actually used in WHERE/JOIN/GROUP BY, adapts as patterns change	Static sort keys, fixed file-size targets, blind to query patterns
Maintenance	Predicts which tables will degrade next, sequences operations by dependency, prioritizes by business impact	Fixed schedules, independent jobs, no awareness of health trajectory
Query routing	Learns which engine performs best for each query shape, uses LLMs to reason about new templates, adapts as tables improve	Static rules, manual engine assignment, no cost optimization
Observability	Correlates structural signals with query performance, predicts degradation before user impact	Threshold alerts, no predictive capability, no cross-signal correlation
Governance	Adapts policy enforcement to workload patterns, recommends changes from observed behavior	Static policies, manual updates, no feedback from operations
Agent integration	Feeds agent telemetry back into optimization, shapes the lake around AI access patterns	No awareness of agents, no closed-loop optimization

Each of these deserves a deeper look.

AI-Driven Compaction: Learning How Data Is Read

Traditional compaction is a janitor: it sweeps up small files and merges them. It does not know why the files exist or how they are read. An intelligent compaction engine does.

Query Pattern Analysis

LakeOps continuously observes every query that hits every table — across Trino, Spark, Snowflake, Athena, DuckDB, Flink, and any other connected engine. It extracts which columns appear in WHERE, JOIN, and GROUP BY clauses. It tracks how these patterns change over time. It weights recent queries more heavily than old ones.

This is not a one-time analysis. It is a continuous learning loop.

From this analysis, the AI knows — per table — which columns matter most for data layout. A table where 80% of queries filter on event_date and region should have its data physically sorted by those columns. A table where queries mostly join on customer_id should sort differently.

From Analysis to Action

During compaction, data files are physically re-sorted by the learned columns. The result: Parquet row-group min/max statistics become effective, and engines skip irrelevant data without reading it. Predicate pushdown and column pruning actually work — not because someone manually configured sort keys, but because the AI figured out what the right sort order is.

On production tables, this AI-driven sorting reduces data scanned by 51% and delivers 12x faster queries compared to unsorted compaction. The numbers are large because the difference between "sort by whatever was configured at table creation" and "sort by what queries actually need" is enormous.

Adaptive Sort Evolution

Here is where intelligence really separates from automation. When query patterns shift — a new dashboard starts filtering on product_category instead of region, a data science team changes their join keys, AI agents start accessing different columns — the sort strategy evolves on the next compaction pass. No configuration change. No ticket. No human intervention.

A table whose workload stabilizes gets compacted less frequently — the AI recognizes that the current layout is already optimal and avoids redundant work. A table whose workload spikes gets prioritized. This is resource allocation driven by learned behavior, not fixed schedules.

Speed as an AI Enabler

The AI's compaction decisions need an engine fast enough to execute them continuously — not queue them for a nightly batch window. LakeOps runs compaction on a purpose-built Rust engine powered by Apache DataFusion: 221 seconds where Spark takes 1,612 seconds (95% faster), at roughly $5/TB versus $50/TB (90% cheaper). Peak throughput reaches 2,522 MB/s. Tables that OOM Spark — a 1.2 TB table with 12,000 files, for example — complete without special configuration because memory is bounded, not heap-dependent.

That speed is not just a performance metric. It is the enabler for AI-driven compaction: when compaction takes minutes instead of hours, the system can respond to workload changes on the same day they happen rather than falling behind. The AI can iterate. Fast execution makes continuous learning practical.

AI-Driven Maintenance: Predicting What Breaks

Iceberg tables require six maintenance operations: data compaction, snapshot expiration, manifest consolidation, orphan cleanup, position delete resolution, and statistics computation. Every platform team knows this. The question is when and in what order to run them — and that is where AI changes the game.

Health Prediction, Not Health Reaction

Traditional monitoring tells you a table is degraded after queries slow down. By then, users have already filed tickets and data pipelines are late.

LakeOps predicts degradation before it hits. The AI continuously scores every table on structural signals — file count trajectory, partition balance trends, manifest growth rate, snapshot accumulation velocity, delete-file ratios. It sees the pattern: "this table's small-file ratio is climbing at 4% per day; at current trajectory it will hit Critical in 72 hours."

This is a classification and prediction problem that humans cannot solve manually at scale. A platform team with 800 tables across multiple catalogs cannot eyeball metadata trends for each one. An AI system that ingests every commit, every query, and every structural change can.

Intelligent Sequencing

The order of maintenance operations matters. Running orphan cleanup before snapshot expiration risks deleting referenced files. Running manifest rewrites before compaction wastes the rewrite since compaction will invalidate file references.

The AI learns the correct dependency graph and sequences operations per table:

Compaction first (merge small files, apply learned sort order)
Snapshot expiration after (remove old references safely, respecting active readers)
Manifest consolidation next (consolidate after file references stabilize)
Orphan cleanup last (reclaim newly unreferenced storage)

This is not just a fixed ordering — the AI evaluates whether each step is needed based on current table state. A table with healthy manifests skips manifest consolidation. A table with no orphans skips cleanup. The maintenance plan is tailored per table, per cycle, based on real signals.

Operation Outcome Learning

After each operation completes, the AI measures the outcome: did compaction actually reduce query latency? Did snapshot expiration free the expected storage? Did manifest consolidation improve planning time?

These outcomes feed back into future decisions. If compacting a particular table's hot partition consistently delivers a 5x query speedup, the system learns to prioritize that partition. If expiring snapshots on a rarely-queried table has minimal impact, it deprioritizes that work. Over time, the maintenance engine gets better at deciding what to do — not just executing a predefined list.

AI-Driven Query Routing: Matching Queries to Engines

Most production lakehouses use multiple query engines. Trino for interactive analytics. Spark for heavy ETL. Snowflake for BI dashboards. DuckDB for ad-hoc exploration. Athena for pay-per-query workloads. The problem is deciding which query goes where.

Without intelligence, this is a manual decision — or worse, no decision at all, and every query hits the same engine regardless of whether it is a 50 MB selective filter or a 500 GB full table scan.

The Three-Router AI Stack

LakeOps implements intelligent routing through QueryFlux, an open-source Rust-based SQL proxy, extended with a three-layer AI stack:

1. Adaptive routing (learned behavior). The AI tracks historical execution data for every query template — which engine ran it, how long it took, how much it cost, whether it succeeded. For previously seen query shapes, routing is instant: the system already knows which engine delivers the best cost-performance ratio. Cached decisions take 0ms.

2. LLM-based routing (reasoning). For new query templates the system has not seen before, an LLM reasons over the query structure combined with live table statistics — file count, data size, sort order, partition layout, health score. It predicts which engine is the best fit. This handles the cold-start problem that pure historical routing cannot.

3. Semantic routing (intent matching). When neither historical data nor structural analysis alone is sufficient, semantic matching infers the query's intent — exploratory analysis, dashboard refresh, ETL pipeline, or AI agent retrieval — and routes accordingly.

The three routers compose. Over time, more query templates enter the adaptive cache and routing gets faster and more accurate.

Data-Quality-Aware Routing

This is where AI routing connects to AI compaction and AI observability. The routing layer knows — from LakeOps's health scoring — whether a table is well-compacted, fragmented, or degraded. It adjusts routing decisions accordingly:

A well-compacted table with good sort order? Route to a lightweight, cheaper engine that benefits from efficient data skipping.
A degraded table with 40,000 small files? Route to an engine that can handle the overhead while compaction runs in the background.
A table that was just compacted and is now healthy? Update the routing weights so future queries use cheaper engines.

As the AI compaction engine improves table health, the AI routing engine shifts queries to cheaper engines. The cost savings compound. Production benchmarks show up to 56% workload cost reduction from intelligent routing alone.

AI-Driven Observability: From Dashboards to Diagnosis

Traditional observability is a dashboard: it shows metrics, you interpret them. AI-driven observability is a diagnostic system: it interprets the metrics for you, tells you what is wrong, explains why, and either fixes it or recommends a fix.

Cross-Signal Correlation

LakeOps does not monitor file counts and query latency independently. The AI correlates signals across layers:

Storage signals — file count, size distribution, access patterns, orphan volume
Metadata signals — manifest count and depth, snapshot velocity, partition skew, statistics coverage
Engine signals — query latency, scan volume, CPU utilization, per-engine field access
Operation signals — maintenance duration, compaction throughput, health score changes

From these, it produces actionable insights with causal chains. Example: "Query latency on raw_clickstream increased 8x because 312 partitions exceed the file count threshold — scan amplification is 8x. Compaction is recommended and will reduce query time to baseline."

That is not a threshold alert. That is a diagnosis.

Severity-Ranked Insights

The AI surfaces per-table findings at four severity levels — CRITICAL, HIGH, WARNING, LOW — each with the specific metric, current value, threshold violated, and a remediation action. Critical and High Insights trigger automated remediation. Warning and Low Insights surface for human review.

The key AI element: the system does not just check "is file count above X?" It evaluates the combination of signals. A table with many small files but no queries gets a lower severity than a table with fewer small files but high query frequency. Business impact is inferred from workload patterns, not just structural metrics.

Predictive Degradation

Instead of alerting after a table degrades, the AI predicts when it will degrade based on trajectory. A table whose snapshot accumulation rate will hit its retention limit in 48 hours triggers an alert now — with enough lead time to act before users notice.

AI-Driven Governance: Policies That Adapt

Static governance policies are set-and-forget — which usually means set-and-break. Workloads change, new tables appear, query patterns shift, and the policies that made sense six months ago no longer match reality.

Intelligent Policy Recommendations

LakeOps's AI observes actual workload behavior and recommends policy adjustments:

"Table orders_archive has not been queried in 90 days. Current retention policy keeps 30 days of snapshots. Recommend reducing to 7 days to reclaim 18 GB of metadata."
"Namespace analytics.* consistently produces small files from streaming writes. Recommend adding an auto-compaction policy with a 1-hour trigger window."
"Table customer_events is queried by 3 AI agents with PII columns. Recommend adding PIIMaskGuard policy to agent-facing routing groups."

These are not generic suggestions. They are derived from observed behavior — query patterns, write patterns, access patterns, and health trajectories specific to each table.

Policy Inheritance with Workload Awareness

Policies cascade from organization to catalog to namespace to table — but the AI ensures inheritance makes sense. A compaction policy designed for batch tables is not blindly applied to streaming tables in the same namespace. The system flags conflicts between inherited policies and actual workload patterns, and either adjusts automatically (in autopilot mode) or recommends changes (in manual mode).

Policies apply uniformly across catalogs — AWS Glue, Polaris, Nessie, Gravitino, Lakekeeper — without per-catalog scripts or engine-specific logic. Every policy is auditable, versioned, and controllable with a single toggle.

The Closed Loop: How Every Query Makes the Lake Smarter

The most important concept in an AI data lake is the closed-loop feedback system. This is what separates an intelligent system from a collection of automated scripts — and it is the core of how LakeOps operates.

Here is the loop:

    Queries hit the lake
    (engines, agents, dashboards, pipelines)
            │
            ▼
    Telemetry captures patterns
    (columns, engines, latency, cost, frequency)
            │
            ▼
    AI analyzes and learns
    (sort order, routing weights, health scores,
     maintenance priorities, policy fitness)
            │
            ▼
    Optimizer acts
    (compact with new sort, reroute queries,
     adjust policies, trigger maintenance)
            │
            ▼
    Outcomes measured
    (query faster? cost lower? health improved?)
            │
            ▼
    Learnings feed back ──────► next cycle

Every query — whether from a BI dashboard, a Spark ETL job, or an AI agent — feeds signal into this loop. The more the lake is used, the better it gets at organizing itself. This is the defining characteristic of an AI data lake: usage improves performance, rather than degrading it.

In a traditional lake, usage degrades performance — more queries means more load, more writes means more small files, more engines means more fragmentation. In an AI data lake, usage is signal. More queries means better sort-order decisions. More engines means richer routing data. More writes means more opportunities for the optimizer to learn and act.

AI Agents as Lake Citizens

AI agents introduce a fundamentally new workload pattern. They issue SQL iteratively — dozens of queries per reasoning step — at high frequency, without human review, with expectations of sub-second latency. A single customer-support agent answering one ticket can issue 30–50 SQL statements across user history, orders, product metadata, and support logs. Multiply that across hundreds of concurrent agent sessions and the workload profile looks nothing like the BI traffic your lake was designed for.

Read: Routing Multiple Query Engines with Iceberg

The Infrastructure Problem

Agents querying uncompacted tables pay a 5–10x latency penalty. When an agent runs 12 sequential queries in a reasoning loop, each 3-second query delay adds up to 36 seconds of wall-clock time for a single interaction. Users perceive this as the agent being broken. And unlike a human analyst who learns to avoid expensive queries, an agent in a loop repeats the same mistake indefinitely — compounding costs every iteration.

The LakeOps MCP: Agent-Native Connectivity

The Model Context Protocol (MCP) is the emerging standard for how AI agents interact with data infrastructure — filling the same role for agent-to-lakehouse communication that REST catalogs fill for engine-to-catalog communication.

Read: Agentic AI MCP for Iceberg

LakeOps exposes a native MCP server that any compatible agent — Claude, LangChain, LlamaIndex, or custom — can connect to with zero integration code. The MCP server provides four purpose-built tools rather than a generic "run anything" endpoint (agents that get a single execute_sql tool tend to skip schema discovery and hallucinate table names):

list_schemas — returns all available schemas and databases across connected engines. Agents call this before constructing queries to avoid guessing at namespace structures.
describe_table — returns column names, types, partition specs, sort orders, and optionally sample rows in a single call. Eliminates the multiple round-trips agents typically make to understand a table's shape.
execute_query — runs SQL through the full routing and guardrail pipeline, with optional engine_hint and enforced max_rows to prevent unbounded result sets from flooding the LLM context window.
explain_query — returns estimated cost, row count, and execution plan without running the query. Agents use this for pre-flight cost checks.

Each routing endpoint gets a stable URL — for example, storefront-analytics.lakeops.dev. Agents connect to that URL and inherit the full policy stack (guardrails, routing rules, cost limits) automatically. No per-agent configuration. Wire compatibility spans PostgreSQL, MySQL, and Arrow Flight SQL, plus async query support with SSE streaming so agents in tool-use loops can handle long-running queries gracefully.

Intelligent Guardrails

Unsupervised agents need enforcement at the query level, not at the application level. LakeOps provides a composable guard chain — every query passes through it regardless of which agent or frontend issued it:

ReadOnlyGuard — blocks DDL and DML from agent sessions using SQL parsing (not string matching), catching CTEs that wrap mutations and function calls with side effects.
RowLimitGuard — injects LIMIT N when a query lacks one. Agents in reasoning loops often forget to limit results, leading to multi-gigabyte result sets that overflow LLM context.
CostEstimateGuard — issues EXPLAIN before execution and rejects queries where estimated scanned bytes exceed a threshold. This is the primary defense against the cartesian-join and missing-WHERE patterns that agents frequently generate.
PIIMaskGuard — rewrites queries to exclude, hash, or null-out columns tagged as sensitive. PII never enters the LLM context window.
HumanApprovalGuard — pauses high-stakes queries and sends a webhook (Slack, email, or custom) for manual review.

Guards stack per routing group. A typical agent-facing configuration: read_only → row_limit → cost_estimate → pii_mask. A data engineering agent gets: cost_estimate → human_approval for DDL. Every guard action is recorded with full auditability — what fired, what was rewritten, what was rejected.

AI Solving the AI Problem

This is where the AI data lake concept becomes recursive. The same intelligence that optimizes the lake for human workloads also optimizes it for AI workloads — and agents produce particularly rich signal:

Higher query frequency — more data points for the adaptive optimizer per unit of time
Predictable patterns — roughly 80% of agent queries are repeated templates (same SQL structure, different literal values), making sort-order learning fast
Agent context propagation — every query carries structured metadata (agent_id, conversation_id, step_index, tool_call_id) that enables per-agent cost attribution, session replay for debugging, and workload-specific routing optimization

The feedback loop tightens: agents query the lake → telemetry captures their patterns → the AI reshapes data to match → agents get faster results → the agents produce even better signal → the cycle accelerates. Production deployments show agent query p95 latency dropping from 5–10 seconds to under 500ms as tables are compacted and sorted for observed access patterns, with per-query compute cost dropping 65% through intelligent routing.

AI-Native vs. AI-Washed: How to Tell the Difference

Not every product that claims "AI-powered" actually uses intelligence. Here is how to evaluate:

Signal	AI-Native	AI-Washed
Compaction	Learns sort order from cross-engine query patterns; adapts as patterns change	Fixed sort keys configured manually; schedule-based execution
Maintenance	Predicts degradation; sequences by dependency; learns from outcomes	Runs on cron; independent jobs; no feedback loop
Routing	Learns cost-performance per query shape; uses LLM for cold-start; adapts weights	Static rules; manual engine assignment
Observability	Correlates signals across layers; predicts issues; diagnoses root cause	Threshold alerts; dashboard without interpretation
Governance	Recommends policy changes from observed behavior; adapts inheritance	Static policies; manual updates
Feedback loop	Closed-loop: every query improves the system	Open-loop: no learning from outcomes
Agent support	Native MCP with intelligent guardrails and self-optimization	REST API bolt-on; no workload-aware optimization

The litmus test: does the system get better over time without human configuration changes? If yes, there is actual intelligence. If no, it is automation with a marketing label.

The Compounding Effect

The individual components are valuable. The compound effect is transformational.

AI-driven compaction reduces data scanned by 51%. That means every query across every engine benefits. Routing then shifts those improved queries to cheaper engines, because well-compacted tables do not need heavyweight compute. Storage costs drop because compaction reduces total file count and orphan cleanup reclaims unreferenced data. Observability improves because the AI has better signal to work with. Agents respond faster, which means fewer retries, lower token costs, and better AI application quality.

Each AI component amplifies the others. Better compaction → better routing → lower cost → more headroom for AI agents → richer signal → better compaction. The loop runs continuously.

Production numbers from LakeOps deployments:

80% reduction in compute and storage costs
12x faster queries after AI-driven compaction and sorting
95% faster compaction than Spark (221s vs. 1,612s on 200 GB)
90% cheaper compaction ($5/TB vs. $50/TB for Spark)
56% workload cost reduction from intelligent routing
100% of tables continuously monitored and maintained
~10 minutes to connect and start optimizing

These numbers compound because the system is a closed loop, not a collection of independent tools.

Where This Is Heading

The trajectory is clear. Data lakehouses are moving from manually operated infrastructure to self-managing systems where intelligence handles the operational layer end-to-end.

Today, LakeOps's AI handles compaction, maintenance sequencing, query routing, health prediction, and policy adaptation. The direction extends to:

Automatic schema evolution — the AI detects that a new column is consistently added to queries and recommends schema changes or materialized views
Predictive capacity planning — forecasting storage and compute needs based on workload growth trends and seasonal patterns
Cross-lake optimization — intelligence that spans multiple lakes, regions, and clouds, optimizing data placement and replication based on global access patterns
Agent-native data contracts — AI agents negotiate data freshness, latency, and cost SLAs directly with the control plane, and the system optimizes to meet them

The foundation for all of this is the closed-loop feedback system. A lake that learns from every query is a lake that can evolve to meet requirements that do not yet exist.

Summary

An AI data lake is not a data lake with an AI chatbot. It is a data lake where intelligence drives operations:

AI decides how to compact — learning from cross-engine query patterns, not static configuration
AI decides when to maintain — predicting degradation, sequencing by dependency, learning from outcomes
AI decides where to route — matching query shapes to engines using learned cost-performance models
AI diagnoses problems — correlating signals across storage, metadata, engines, and operations
AI adapts governance — recommending policy changes based on observed behavior
AI optimizes for agents — feeding agent telemetry back into the optimization loop

The closed-loop feedback system is the defining characteristic. Usage improves performance. Every query makes the lake smarter.

LakeOps is the autonomous control plane that implements this intelligence stack for Apache Iceberg — connecting to your existing catalogs, engines, and storage in minutes, with no data movement and no vendor lock-in.

Explore the platform · Read the docs · Get a demo

Managed Data Lake: A Guide for 2026

joni sar — Tue, 21 Jul 2026 11:32:48 +0000

There are two ways to run a data lake. You can manage every table by hand — writing Airflow DAGs, scheduling Spark compaction jobs, building monitoring dashboards, and staffing a platform team that grows linearly with table count. Or you can deploy a control plane that continuously observes every table in the lake, classifies health, sequences maintenance operations, optimizes data layout for real query patterns, enforces policies at lake scale, and adapts as workloads change — while your data stays in your storage account and your engines stay yours. This guide covers both strategies and explains why teams that start with the first keep arriving at the second.

The data lake format question is settled. Apache Iceberg is the default table format for the modern lakehouse. Every major engine — Spark, Trino, Snowflake, Athena, DuckDB, Flink, Databricks — reads and writes Iceberg natively. Adoption is near-universal among data platform teams building on open storage.

The operations question is not settled. Iceberg gives you ACID semantics, schema evolution, time travel, hidden partitioning, and partition evolution on your own object storage. What it does not give you is a system that keeps tables healthy. Compaction, snapshot expiration, orphan cleanup, manifest optimization, sort-order alignment, governance, and lake-wide observability are left to you — the official maintenance guide is explicit about that. At a handful of tables, scripts work. At hundreds of tables across multiple engines, catalogs, and write patterns, the lake quietly degrades into a swamp — and the engineering hours spent holding it together scale faster than the team.

"Managed data lake" is an overloaded term. Some products use it to mean a closed platform. Others mean one automated job. This guide defines what managed actually requires in 2026, then walks through two strategies to get there. First, the control-plane approach — using LakeOps, a dedicated autonomous control plane for Apache Iceberg that handles observability, compaction, snapshot lifecycle, orphan cleanup, manifest optimization, governance policies, multi-engine query routing, and agentic AI readiness across your existing stack without moving data or replacing engines. Then, the manual approach using open-source tooling and custom scripts. Both paths are real. The difference is in what scales.

What "managed" actually means in 2026

Iceberg on S3 is not a managed data lake. A nightly Spark rewrite_data_files job is not either. "Managed" means the operational contract is closed continuously, safely, and at lake scale — without engineering time growing linearly with table count.

An open data lake has four architectural layers:

Storage — S3, GCS, ADLS, or on-premises object storage. Commodity infrastructure you own.
Table format + catalog — Iceberg plus a catalog (Glue, Polaris, Nessie, Gravitino, Lakekeeper, S3 Tables, Unity Catalog). The structural contract every engine speaks.
Compute engines — Spark, Trino, Snowflake, Athena, DuckDB, Flink, Databricks, ClickHouse. Chosen per workload, not locked to a single vendor.
Operations control plane — the layer that observes table health, decides what maintenance to run, executes it safely, enforces policy, and learns from query patterns across engines. This is the layer that most lakes are missing.

Closed platforms like Databricks and Snowflake bundle the last three layers. Open lakes decouple them on purpose — you pick the best engine for each workload, keep data in your own storage, and avoid vendor lock-in. The trade-off: you inherit the operations that the closed platform handled for you. That trade-off is the subject of this guide.

In practice, a managed data lake means: tables stay compact and sorted for real query patterns. Snapshots and orphans do not inflate storage. Manifests stay lean enough that planning never dominates query time. Policies apply lake-wide and new tables inherit them automatically. Health is visible in one place. Correct results are the floor. Predictable cost and latency are the product.

Why data lakes degrade — the mechanics of silent failure

The failure mode is predictable, well-documented, and present in nearly every Iceberg lake that has been running for more than a few months without dedicated maintenance. The financial impact compounds from four directions at once: storage costs balloon from orphan files and stale snapshots, compute costs spike as engines scan thousands of small files instead of a few optimally-sized ones, query latency degrades as tables that returned results in seconds now take minutes, and engineering time evaporates as someone has to write, maintain, debug, and be on-call for the scripts that hold it all together. Understanding the mechanics is necessary for both strategies.

Small file accumulation. Streaming writers (Flink, Spark Structured Streaming, Kafka Connect Iceberg sink, CDC connectors) produce files sized by checkpoint throughput, not by optimal read size. A streaming pipeline at 1 commit per minute across 10 partitions creates approximately 14,400 files per day. Each query pays an object-store GET request, a Parquet footer parse, and task setup overhead per file. Cost and latency scale with file count, not with data volume.

Snapshot accumulation. Every Iceberg commit creates a new snapshot. A streaming table at 5-minute commits creates ~288 snapshots per day. Without expiration, snapshot counts reach tens of thousands within weeks. Query planning reads the snapshot chain; above ~1,000–2,000 retained snapshots, planning time degrades measurably — from sub-second to 30+ seconds. The mechanism is detailed in how Iceberg query planning works.

Manifest fragmentation. Every commit adds at least one manifest file to the manifest list. Streaming tables accumulate thousands of manifests per month. Each manifest must be read during planning; at 500+ manifests, planner latency alone can exceed multiple seconds. This is the layer most teams diagnose last because the symptom (slow EXPLAIN, slow first row) does not point at data files.

Orphan file growth. Failed writes, aborted Spark jobs, interrupted compaction, and schema-evolution retries leave data files on object storage that no snapshot references. These orphans are invisible to Iceberg but billable by the cloud provider. Production orphan sweeps routinely reclaim a significant share of billable storage on affected prefixes. A detailed breakdown of these nine degradation vectors and how to address each is in the Managed Iceberg in 2026 guide.

Delete file amplification. Merge-on-read tables accumulate position and equality delete files with every UPDATE or DELETE. Without compaction that physically applies deletes, every read must reconcile live rows against all accumulated delete files. Above a certain delete-to-data file ratio, reads pay measurable reconciliation tax and users feel it on every query.

Layout drift. Binpack compaction normalizes file sizes but does not sort data. Engines cannot skip irrelevant file groups because Parquet min/max statistics span wide ranges. A table compacted by size but not sorted by its most-filtered columns still forces full scans on common predicates. And the sort order that accelerated one workload six months ago may not match today's query patterns — layout needs to adapt as usage evolves.

Write amplification in metadata. Every Iceberg write — even a tiny append — produces a new metadata.json, a new manifest list, and one or more new manifest files. For a batch job running hourly, this overhead is invisible. For a streaming job committing every few seconds, metadata writing dominates I/O. Object storage starts throttling. Query planning suffers badly because every new metadata version adds to the chain the planner must traverse.

Governance entropy. Some tables have compaction configured; others do not. Some have snapshot retention; others use defaults that never expire. Retention windows are too aggressive for long-running queries, or too conservative for GDPR. No single view shows the health of the entire lake across catalogs and engines. Every new table starts from scratch unless someone remembers to configure it.

These failure modes do not trigger alarms. They compound silently over weeks and months — and the first real symptom is usually an analyst asking why their dashboard takes 40 seconds instead of 3. By the time someone investigates, the debt has accumulated across dozens of tables and unwinding it is a project, not a task.

Strategy 1: Smart automation with a control plane

A control plane is an architectural layer that sits between your storage, catalogs, and engines — observing the state of every table, understanding cross-engine query patterns, and applying the right maintenance at the right time. It does not replace anything in your stack. It adds the operational intelligence that open-source components do not provide on their own. Netflix built this layer in-house over years — Autotune, janitors, Metacat, Polaris. That story is in Intelligent Lakehouse: Build Like Netflix. Most teams cannot staff that.

LakeOps is a dedicated control plane for Apache Iceberg — the only product that exists purely as this ops layer. Not an engine, not storage, not a catalog. It connects to your existing catalogs (AWS Glue, Polaris, Nessie, Gravitino, Lakekeeper, S3 Tables) and query engines (Trino, Spark, Flink, Snowflake, Athena, DuckDB) without moving data or changing pipelines. Setup takes roughly ten minutes. Your data stays in your storage account.

What you get is a closed-loop system covering:

Lake-wide observability — continuous health scoring (Critical / Warning / Healthy) across every table and catalog, proactive insights ranked by severity, and cross-engine telemetry in one dashboard

Query-aware compaction — a purpose-built Rust/DataFusion engine that sorts data by the columns your queries actually filter on, orders of magnitude faster and cheaper than Spark-based compaction
Snapshot lifecycle management — automated, concurrency-safe expiration with configurable retention policies that respect active readers

Orphan file cleanup — safe detection and removal with age-threshold protection, sequenced after expiration

Manifest and metadata optimization — consolidation, position delete rewrites, and Puffin statistics computation, triggered automatically after compaction

Declarative policies and governance — compaction, retention, cleanup, and configuration rules scoped at organization, catalog, namespace, or table level with cascade inheritance

Multi-engine query routing — route queries across engines optimized for cost, latency, or throughput via QueryFlux

Agentic AI readiness — native MCP interface with schema discovery, layered guardrails (ReadOnly, CostEstimate, PIIMask, HumanApproval), and agent query telemetry feeding back into optimization

Layout simulations — preview compaction impact on Iceberg branches before committing to production

The operating model repeats continuously in four phases: connect and collect telemetry, classify table health and raise insights, execute maintenance in the correct sequence (expire snapshots → remove orphans → compact data files → rewrite manifests), and observe outcomes while enforcing governance. Each cycle the system learns — sort orders adapt as query patterns evolve, the compaction planner improves throughput on repeated runs, and policies self-enforce as new tables are created.

Observability and health classification

After connecting a catalog, every table is continuously scored and classified. The dashboard shows aggregate health across catalogs, recent operations, storage trends, and a summary of proactive insights. Platform teams can triage the entire lake in one screen without writing a single query.

Health is computed from the same Iceberg signals that matter in manual debugging: file count and size distribution per partition, manifest count and depth, snapshot accumulation, delete-file ratio, partition skew, and sort-order alignment with real query patterns. The difference is that the control plane evaluates these signals continuously across every table — not one table at a time when someone asks "why is this slow?"

Proactive insights go beyond classification to tell you why a table is degraded and what to do about it. Each insight is tied to a specific table, a severity level (CRITICAL, HIGH, WARNING, LOW), and a recommended action. In a manual workflow, you discover these conditions after a user files a ticket. With Insights, you discover them while they are still at Warning — before they escalate to a production incident. Details on the observability surface.

Intelligent compaction

Compaction is the highest-impact maintenance operation. LakeOps approaches it differently from generic Spark-based compaction in four ways.

Query-aware sorting. LakeOps tracks which columns appear in WHERE, JOIN, and GROUP BY clauses across every connected engine for every table. During compaction, data is physically sorted by those columns — so Parquet row-group statistics enable engines to skip irrelevant data without reading it. When query patterns change, the sort order adapts on subsequent compaction passes without manual configuration.

Rust/DataFusion engine. Compaction runs on a purpose-built engine written in Rust with Apache DataFusion. Arrow columnar buffers, bounded memory, zero garbage collection, lock-free parallelism, native Parquet I/O with predicate pushdown. Orders of magnitude faster and cheaper than Spark-based compaction on identical datasets. Partitions that OOM Spark complete in minutes because memory is bounded, not heap-dependent. Full benchmark details are in the compaction product page.

This cost reduction is what makes frequent compaction economically viable. Instead of one nightly job that lets streaming tables degrade for 24 hours between passes, LakeOps runs compaction multiple times per day when file health signals warrant it. Tables stay continuously healthy instead of oscillating between degradation and cleanup.

Conflict-aware execution. The system inspects active writer state and targets only cold partitions — partitions without active streaming appends. If a boundary-case conflict occurs, only the affected file group retries on the next cycle. No data loss, no full-job restarts, no CommitFailedException storms.

Self-improving planner. The compaction planner learns from workload telemetry across consecutive runs. Same table, zero config changes — throughput and runtime improve with each pass as the planner adapts its internal parallelism, file-group sizing, and sort strategy.

Snapshot, orphan, and manifest lifecycle

LakeOps automates the full maintenance lifecycle with correct sequencing:

Snapshot expiration. Policy-based retention (time and count) enforced continuously. Concurrency-safe — expiration respects active readers and in-flight queries, preventing the most common production failure in teams that automate snapshot expiration without accounting for concurrent readers. Version history, snapshot comparison, and one-click rollback remain available within the retention window.

Orphan cleanup. Age-threshold safety (7+ days by default) protects in-flight writes. Sequenced after expiration so newly dereferenced files are captured in the same sweep. Full audit trail of every removed file.

Manifest and metadata optimization. Three operations triggered automatically after compaction: manifest consolidation (fewer manifests = faster planning), position delete file rewrite (clean reads for merge-on-read tables), and Puffin statistics computation (column-level NDV, min/max, null counts for aggressive file-level pruning). Manifests are rewritten after compaction, not before, so the index reflects the final file set.

Every maintenance operation is logged lake-wide and per table with duration, impact, and status — the audit trail that answers "what ran, when, and did it succeed?" during incident response, without parsing executor logs across multiple systems.

Policies, routing, and AI readiness

Declarative policies scope compaction targets, snapshot retention, orphan cleanup, and manifest optimization at four levels with cascade inheritance: organization → catalog → namespace → table. Adaptive Maintenance bundles the full lifecycle into one data-driven policy — the system decides which operations each table needs from structural signals and runs them in the correct sequence. Cross-catalog enforcement works uniformly across AWS Glue, Polaris, Nessie, and Gravitino.

Multi-engine routing via QueryFlux connects all engines to a unified routing layer that makes per-query decisions based on cost, latency, table health, and historical patterns. Without routing, teams either standardize on one engine (losing optimization) or let each user choose (creating cost chaos). Routing decisions are enriched by table health — if a table is currently degraded, queries can be directed to engines that handle fragmentation better until compaction resolves the issue.

Agentic AI readiness provides a native MCP (Model Context Protocol) interface for AI agents with schema discovery, wire compatibility (PostgreSQL, MySQL, Arrow Flight), and layered guardrails configurable per agent session. Agent query telemetry feeds back into compaction and sort-order decisions — when agents start filtering on columns that no human dashboard uses, the compaction planner detects the new pattern and adjusts accordingly. Details in the AI agents guide.

The compound effect

The capabilities above are not independent features. They form a closed loop where each component's output feeds the next: observability detects degradation, the system runs expiration to unpin files, orphan cleanup captures newly dereferenced objects, compaction runs sorted by the columns that real queries filter on, manifests are rewritten against the final file set, Puffin statistics are recomputed, the table moves from Warning to Healthy, and the event is logged with before/after metrics. Next cycle, the compaction planner runs faster because it learned from the previous pass.

The engineering hours that used to go into writing, scheduling, monitoring, and debugging maintenance scripts are returned to product work. Policies are defined once and the system executes them continuously — every action logged, auditable, and reversible. Data stays in your storage account. No pipeline rewrites. Minutes to install.

Strategy 2: Manual management with open-source tooling

Not every team needs a control plane on day one. If you have a handful of Iceberg tables, one or two engines, batch-only writes, and a platform engineer who owns maintenance, manual management works. This section is the complete playbook — the same methodology that works whether you have a control plane or not.

The manual stack

The typical setup: an orchestrator (Airflow, Dagster, or Prefect) scheduling maintenance as DAG tasks, Spark (on EMR, Dataproc, or Kubernetes) running Iceberg maintenance procedures, a catalog (Glue, Polaris, Nessie), custom SQL queries against metadata tables piped into Grafana or Datadog, and threshold-based alerting. This stack is well-understood and runs at many organizations today. The constraint is not capability but coverage: every new table, every new write pattern, and every new engine requires incremental engineering work to bring under management.

Compaction by hand

Use Spark's rewrite_data_files procedure:

CALL catalog.system.rewrite_data_files(
  table => 'db.events',
  strategy => 'binpack',
  where => 'event_date >= current_date() - INTERVAL 7 DAYS',
  options => map(
    'target-file-size-bytes', '268435456',
    'min-input-files', '3',
    'partial-progress.enabled', 'true',
    'partial-progress.max-commits', '20',
    'max-concurrent-file-group-rewrites', '15'
  )
);

Key decisions: binpack for file-size normalization, sort for query-pattern optimization, zorder for multi-column predicates. Target 256–512 MB files. Always enable partial-progress. Never compact the active write partition on a streaming table — compacting the hot partition while Flink is appending is the primary cause of CommitFailedException storms. Run compaction on a separate cluster. Schedule cadence must match write cadence — if you write 14,400 files per day and compact once at midnight, the table spends 23 hours degraded. Decision frameworks are in the compaction strategies guide.

For sort compaction, you must determine sort columns yourself by analyzing query patterns across engines. When patterns change, you must notice and update the sort order manually. This is the gap a control plane closes automatically with cross-engine telemetry.

CALL catalog.system.rewrite_data_files(
  table => 'db.events',
  strategy => 'sort',
  sort_order => 'event_date ASC, region ASC, user_id ASC',
  where => 'event_date >= current_date() - INTERVAL 7 DAYS',
  options => map(
    'target-file-size-bytes', '268435456',
    'partial-progress.enabled', 'true',
    'partial-progress.max-commits', '20',
    'max-file-group-size-bytes', '10737418240'
  )
);

Snapshot expiration, orphan cleanup, and manifests by hand

Expire snapshots regularly. Retention window must exceed your longest-running query — a Spark job that runs for 3 hours will fail if its snapshot is expired mid-flight. This is one of the most common production failures in teams that automate aggressively.

CALL catalog.system.expire_snapshots(
  table => 'db.events',
  older_than => TIMESTAMP '2026-07-14 00:00:00',
  retain_last => 50
);

ALTER TABLE db.events SET TBLPROPERTIES (
  'write.metadata.delete-after-commit.enabled' = 'true',
  'write.metadata.previous-versions-max' = '100'
);

Orphan cleanup runs after expiration, never before. Always dry-run first. The older_than threshold must be far enough in the past to protect in-flight writes — production teams use 7+ days. Verify URI schemes match (s3:// vs s3a:// vs s3n://) before your first run.

CALL catalog.system.remove_orphan_files(
  table => 'db.events',
  older_than => TIMESTAMP '2026-07-07 00:00:00',
  dry_run => true
);

Rewrite manifests after every compaction cycle. This is the step most teams skip. Without rewriting, planning time remains elevated even though the data files are healthy. On extreme fragmentation, rewriting alone can drop planning from 30+ seconds to under a second. Full methodology in the orphan cleanup guide and the table health guide.

CALL catalog.system.rewrite_manifests(
  table => 'db.events'
);

Health monitoring and the safe sequence

Manual monitoring means querying Iceberg metadata tables and setting alert thresholds. Start with four numbers per table: file count and average size per partition, snapshot count, manifest count, and delete-file ratio for MOR/CDC tables.

SELECT partition, COUNT(*) AS file_count,
  ROUND(AVG(file_size_in_bytes) / 1048576, 1) AS avg_size_mb
FROM catalog.db.events.files
GROUP BY partition ORDER BY file_count DESC LIMIT 20;

SELECT COUNT(*) AS snapshot_count
FROM catalog.db.events.snapshots;

SELECT COUNT(*) AS manifest_count
FROM catalog.db.events.manifests;

The correct maintenance order — consistent across the official Iceberg docs and the automating table maintenance guide — is:

Expire snapshots — release metadata references to files nobody needs for time travel
Remove orphan files — delete unreferenced objects from storage (with 7+ day safety window)
Compact data files — merge small files and physically apply deletes
Rewrite manifests — consolidate the metadata index against the final file set
Compute statistics (optional) — refresh Puffin stats so planners see current NDVs

Compacting before expiration rewrites files about to be discarded — wasted compute. Orphan cleanup before expiration misses the largest reclaimable set. Manifest rewrite before compaction produces an index stale minutes later. Never invert the order under pressure.

Where manual management breaks down

Manual management does not fail because the tools are inadequate. Spark procedures work. Airflow can schedule them. Grafana can chart health. The breakdown happens at the system level.

Coverage gap. Every new table must be added to the DAG, configured, monitored, and validated. New tables slip through. Policies do not inherit automatically.

Workload blindness. Manual compaction sorts by columns you choose at configuration time. When a new dashboard goes live, when a new team starts querying with different filters, when AI agents start accessing columns nobody filtered before — the sort order becomes stale.

Sequence violations under pressure. At 3 AM during an incident, the instinct is to compact first. Compacting before expiration wastes compute. Running orphan cleanup with a too-short safety window corrupts data.

Concurrency danger. Snapshot expiration that does not respect active readers kills long-running queries. Compaction that targets active write partitions causes commit conflicts. Manual scripts do not know what other processes are doing.

Cost of execution. Spark compaction means spinning up JVM clusters, allocating executors, and paying for compute that is orders of magnitude more expensive than a purpose-built engine. Most teams run on over-provisioned clusters because under-provisioning causes failures and 2 AM pages.

Linear engineering scaling. Maintenance time grows linearly with table count. The team size does not. The data swamp to modern lakehouse guide walks through the transition in detail.

Fix the writers, not just the table

This section applies to both strategies. Many data lake incidents are writer-configuration debt — the pipeline creates structural damage faster than maintenance can repair it.

Checkpoint interval. The single most impactful setting for streaming Iceberg tables. Flink at 30-second checkpoints with 16 write tasks produces ~46,000 files per day per table. Move to 3–5 minute checkpoints and file creation drops dramatically. For staging tables, feature stores, and compliance archives, 10–15 minute intervals are often appropriate. Details in the Flink Iceberg optimization guide.

SET 'execution.checkpointing.interval' = '5min';
SET 'execution.checkpointing.mode' = 'EXACTLY_ONCE';
SET 'execution.checkpointing.timeout' = '10min';

Write distribution mode. Set write.distribution-mode=hash for partitioned streaming tables. Hash distribution shuffles records by partition key before writing, cutting file count by the parallelism factor.

ALTER TABLE catalog.db.events SET TBLPROPERTIES (
  'write.distribution-mode' = 'hash',
  'write.target-file-size-bytes' = '268435456'
);

Sink parallelism. Decouple source parallelism from sink parallelism. A Kafka topic with 64 partitions does not need 64 writer subtasks. Setting source at 64 and sink at 8–16 with hash distribution yields far fewer files while maintaining consumption throughput.

Partition grain. Over-partitioning is the leading cause of small files at scale. Prefer days(ts) over hours(ts). Use bucket(N, col) instead of identity transforms on high-cardinality columns. Decision frameworks in the partitioning best practices guide.

Delete mode vs mutation rate. Merge-on-read (MOR) is right for high-churn CDC where write latency matters more than read performance. Copy-on-write (COW) is right when read latency is sacred. Mixing MOR with no compaction policy is how delete-ratio incidents start.

Fixing writers halves the maintenance burden for both strategies. Skipping this step turns your control plane into a very expensive mop — and makes your manual scripts a full-time job.

Choosing your strategy

Both strategies work. The question is scale, cost, and where engineering time delivers the most value.

Strategy 2 (manual) fits when: you have fewer than ~50 tables, one or two engines, batch-only or low-frequency writes, a platform engineer who owns maintenance, and predictable query patterns that do not change frequently. The metadata queries and safe sequence in this guide are your on-call playbook.

Strategy 1 (control plane) fits when: table count crosses ~50, you run streaming or CDC pipelines, multiple engines hit the same tables, engineering time spent on maintenance scripts exceeds the cost of a purpose-built system, GDPR or compliance requirements need policy-based enforcement, or query patterns change and layout optimization must adapt without manual intervention.

The strategies compose. Many teams start with manual management, adopt a control plane for the highest-traffic catalogs first, and expand as value is proven.

The decision framework reduces to a few questions: Does the system see all my catalogs in one place? Is compaction query-aware across engines? Are operations coordinated in the correct sequence? Do policies inherit to new tables automatically? Does maintenance compete with my query clusters for compute? Can I simulate layouts before committing? Are agent guardrails first-class? Do I have to move data?

If most answers are "no" on an open multi-engine Iceberg stack, you do not have a managed data lake yet — you have Iceberg plus hope.

The migration path: manual to control plane

The transition from manual management to a control plane is incremental, not disruptive:

Connect a catalog. Discovery and health classification begin immediately without touching existing maintenance infrastructure.
Inspect and learn. Use the observability dashboard to understand the current state of your lake. Teams often discover tables they did not know were degraded.
Run maintenance on the worst tables. Trigger compaction, expiration, or orphan cleanup on the most Critical tables. Watch the before/after metrics.
Enable policies per namespace. Define compaction targets, retention windows, and cleanup thresholds. Every table in scope inherits automatically.
Enable Adaptive Maintenance for streaming tables. Let the system decide when to compact based on file health signals rather than cron schedules.
Retire manual scripts. As each namespace is covered by policies, disable the corresponding Airflow tasks.
Expand to remaining catalogs. Each step is independently valuable.

Teams running the control-plane model report far fewer Iceberg incidents — not because the runbook got faster, but because file explosions, manifest storms, and orphan cliffs are resolved while still at Warning. The real end state of a managed data lake is not faster debugging. It is rarely needing to debug at all.

The bottom line

A managed data lake in 2026 is not a storage product, not a single job, and not a platform you must adopt wholesale. It is an operational contract: tables stay healthy, costs stay predictable, queries stay fast, policies apply consistently, and AI agents get the data they need with appropriate guardrails. The question is who upholds that contract — your engineering team writing scripts, or a dedicated system that does it continuously.

Strategy 2 works. It is well-understood, uses open-source tooling, and scales to a moderate number of tables with dedicated platform engineering. The manual playbook in this guide — safe sequence, metadata queries, writer configuration, per-table monitoring — is the same playbook that production teams use every day.

Strategy 1 works better at scale. LakeOps closes the operational loop that manual scripts cannot: continuous health classification across every table and catalog, query-aware compaction on a purpose-built Rust engine, coordinated maintenance that respects the safe sequence, policies that inherit to every new table, cross-engine telemetry that adapts layout to real query patterns, multi-engine routing that sends each query to the right engine, and agentic AI readiness with layered guardrails. It runs on your existing stack without moving data or replacing engines.

The format already gave you ACID and time travel. The operations layer is what determines whether you have a data lake that works — or a data lake that worked once and has been quietly degrading since. Choose the strategy that matches your current scale. Build toward the one that matches where you are headed.

For deeper technical breakdowns: Managed Iceberg in 2026 covers the nine control-plane components. Intelligent Lakehouse: Build Like Netflix shows how Netflix solved these problems internally. Routing Multiple Query Engines with Iceberg covers the multi-engine routing architecture. Automating Table Maintenance walks through the progression from scripts to autonomous operations. Table Health and Maintenance covers every maintenance operation with configuration detail. And From Data Swamp to Modern Lakehouse covers the full architectural transition. For the platform that closes the loop, explore lakeops.dev.

Netflix Intelligent Lakehouse Solves Iceberg Maintenance — You Can Easily Too

joni sar — Fri, 29 May 2026 13:40:00 +0000

Every production Iceberg data lake eventually hits the same wall: tables that looked fast at 10 GB start crawling at 10 TB. Small files pile up from streaming ingestion, snapshots accumulate because nobody set expiration, orphaned data lingers from failed Spark jobs, and manifest lists grow until planning a simple SELECT takes longer than running it.

Netflix hit this wall years ago — and their solution shaped how the industry thinks about lakehouse architecture. At AWS re:Invent, their engineers walked through the ecosystem they assembled around Iceberg: Polaris for catalog management, Autotune for automated compaction, janitors for continuous cleanup, and Metacat for observability. The outcome was a 25% cost reduction and tables that stayed healthy without manual intervention.

But Netflix had something most teams don't: a dedicated platform organization building custom distributed services backed by CockroachDB, Kafka, and fleets of Spark clusters.

Today, a lakehouse control plane — just as good as Netflix's or better — is available for everyone to install on their Iceberg lakehouse.

The industry's favorite solution is LakeOps.

This article breaks down what actually makes a lakehouse "intelligent" — component by component — and shows how each piece maps to tooling that exists today.

Today with LakeOps, every team in the world is 10 minutes away from an intelligent lakehouse. And yes, it includes autonomous snapshot optimization as well as orphan files, metadata, manifests, and more.

The maintenance gap nobody talks about

Apache Iceberg solved the table format problem. Schema evolution, hidden partitioning, time travel, snapshot isolation — these features are why every major engine from Snowflake to DuckDB now speaks Iceberg natively.

What Iceberg intentionally left unsolved is who runs the maintenance. The format gives you powerful primitives. Keeping those primitives performing well at scale is your responsibility.

In practice, this creates a silent degradation cycle:

Streaming writes produce small files — a pipeline appending every 5 minutes to 100 partitions creates 100 new files per commit. After a week, some partitions contain thousands of sub-megabyte Parquet files.
Snapshots grow unbounded — without explicit expiration, every commit adds a snapshot. A table with hourly writes accumulates 8,760 snapshots per year, each referencing its own manifest list.
Orphan files accumulate — aborted Spark jobs, failed compaction runs, and expired snapshots leave behind data files that no snapshot references. These files cost storage but serve nothing.
Manifests fragment — as files are added and removed, the manifest layer becomes a web of small manifest files. Query planning reads every one of them before scanning a single data file.

The financial impact compounds from four directions: storage waste (orphans + snapshots), compute waste (scanning small files), metadata overhead (fragmented manifests), and engineering time (maintaining cron scripts that break silently).

Netflix's insight was that solving these problems one at a time, with isolated scripts, doesn't scale. You need an integrated system — a control plane that sees the full picture and acts on it continuously.

The six components of an intelligent lakehouse

Looking across Netflix's published architecture and detailed breakdowns of their Iceberg ecosystem, six capabilities separate an intelligent lakehouse from a collection of Iceberg tables with maintenance scripts taped to the side:

1. Universal catalog connectivity

Netflix built Polaris to replace the Hive Metastore with a catalog purpose-built for Iceberg — scalable, CockroachDB-backed, and supporting the Iceberg REST catalog specification for multi-engine access.

Most teams aren't replacing their catalog. They're running AWS Glue, or they adopted a REST catalog like Nessie or Lakekeeper early on, or they have tables spread across multiple catalogs in different regions.

An intelligent lakehouse connects to existing catalogs — Glue, DynamoDB, REST (Polaris, Nessie, Lakekeeper, Gravitino), S3 Tables, or custom implementations — discovers every namespace and table, and normalizes metadata into a single operational view. No catalog migration required.

LakeOps does exactly this: point it at your catalog credentials, and within minutes it inventories every table and starts collecting metadata signals.

2. Intelligent and efficient compaction that actually works

Netflix's Autotune watches for table write events through SQS and spins up Spark jobs to compact small files in the background. It's the core of their self-maintaining architecture.

The Spark-based approach works but carries significant overhead. You need provisioned compute clusters, JVM tuning, IAM roles for each cluster, and someone on-call for job failures. Spark compaction typically costs around $50 per TB processed.

A Rust-based alternative changes the economics entirely. LakeOps runs compaction with a native engine built on Apache DataFusion — no JVM, no cluster provisioning, no shuffle stages.

It reads Iceberg metadata, plans optimal merges, and writes compacted Parquet directly to your storage. Production benchmarks show roughly $5/TB — a 10x cost reduction over Spark.

On top, LakeOps runs compaction based on actual query patterns — so the way your files are organized is optimized to minimize I/O, cutting CPU costs as well as storage by up to 80% compared to Spark or S3 Tables.

It also coordinates all operations and events with Adaptive Maintenance to maximize results and cut time and costs. The sequence matters, and event- or trigger-driven ops are much smarter and more efficient than cron-based jobs.

Two strategies cover every workload:

Binpack — combines small files targeting optimal file sizes (~512 MB). Handles most tables well with minimal configuration.
Sort — reorders data by query-relevant columns so engines skip irrelevant row groups through predicate pushdown. Dramatic speedups for tables with clear access patterns.

Each table can be run manually first (configure → Execute → review results) and then switched to automated scheduling with a cron expression. No all-or-nothing commitment.

3. Metadata lifecycle automation

Netflix runs dedicated "janitor" services for orphan cleanup and snapshot expiration. Without them, their exabyte-scale lake would drown in stale metadata and unreferenced files.

The same operations — snapshot retention, orphan removal, manifest consolidation — need to run continuously on any production Iceberg lake. LakeOps provides all four as per-table operations with independent configuration:

Operation	What it does	Why it matters
Snapshot retention	Expires snapshots beyond a retention period, respecting min counts	Reclaims metadata, enables cleanup
Orphan file cleanup	Removes files unreferenced by any snapshot (with age threshold)	Recovers wasted storage
Manifest optimization	Consolidates fragmented manifests	Speeds up query planning
File compaction	Merges small files (Binpack or Sort)	Reduces scan overhead and S3 API costs

The execution order matters: expire first, then clean orphans, then compact, then consolidate manifests. Running them out of sequence wastes compute or risks removing files still in use.

When you want all four automated together, Adaptive Maintenance bundles them into a single data-driven policy that reacts to table activity — the closest equivalent to Netflix's integrated approach.

4. Full-stack observability without building a pipeline

Netflix's Metacat provides unified metadata access across all datasets, backed by Kafka event streams for real-time operational visibility. Building this took years and a dedicated team.

Out-of-the-box observability should include:

Table health classification — every table scored as Healthy, Warning, or Critical based on file counts, size distributions, snapshot accumulation, and metadata fragmentation
AI-generated insights — ranked recommendations that flag small-file hotspots, excessive snapshots, and missing retention before they become incidents
Event audit trail — every maintenance operation recorded with before/after metrics, timestamps, and status — per-table or lake-wide, filterable by catalog and operation type
Dashboard — total operations, query speed gains, cost savings, and resource reduction in a single view

And table health dashboards:

The difference from building it yourself is time-to-value: connect a catalog and immediately see what's degraded, what's wasting money, and what to fix first.

5. Policy-driven governance that scales with the lake

Configuring maintenance table-by-table stops working somewhere between 50 and 100 tables. Netflix needed organization-wide rules; so does every team that's past the proof-of-concept stage.

A policy engine lets you define maintenance rules at the catalog or namespace level — snapshot retention every hour, orphan cleanup daily, compaction at 2 AM — and every table in scope inherits them automatically. New tables that appear in a governed catalog get the right configuration without anyone touching them.

Two policy categories cover the ground:

Maintenance policies — schedule and configure any operation (or all of them via Adaptive Maintenance) across a scope
Configuration policies — enforce table settings like Iceberg format version, file format, and write distribution mode

Per-table overrides always take precedence, so you set sensible defaults broadly and customize only where needed.

6. Multi-engine query routing

Netflix connects engines through the REST catalog endpoint, but routing decisions — which engine handles which query — remain manual architecture choices in most organizations.

An intelligent routing layer dispatches queries to the best engine based on the workload:

Cost-optimized — sends queries to the cheapest engine that meets your latency SLA
Latency-optimized — picks the fastest engine for the query shape
Throughput-optimized — distributes load for maximum concurrency

Applications connect to a single SQL endpoint (Postgres wire, MySQL wire, or Arrow Flight). When an engine goes down, failover reroutes automatically. When you add or remove engines, application code doesn't change.

LakeOps handles this through QueryFlux, an open-source Rust SQL proxy that translates SQL dialects with sqlglot and adds ~0.35ms of overhead.

What this adds up to

Each component is useful independently. Together, they compound:

Storage costs drop 40–55% from continuous orphan removal, bounded snapshots, and compaction
Compute costs drop up to 75% from Rust-native compaction replacing Spark clusters, sort-order optimization reducing scan volume, and routing hitting the cheapest viable engine
Query latency improves up to 12x through optimized file sizes, sorted layouts, consolidated manifests, and Puffin column statistics
Engineering hours shift from maintaining scripts and debugging overnight failures to building data products

Beyond human users: AI agent access

The next layer of intelligence is enabling AI agents to interact with lakehouse data programmatically. LakeOps provides an MCP (Model Context Protocol) interface that gives agents structured access — table discovery, SQL execution through the routing layer, column statistics without scanning, and maintenance triggers — all within configurable guardrails.

You can enforce read-only access, row limits, PII masking, cost caps, and human approval per agent. As agent usage grows, their query telemetry feeds back into compaction decisions — tables agents query most get optimized first, with sort orders aligned to the predicates agents actually use. The lake self-optimizes as AI adoption scales.

Getting started

Netflix took years to build their intelligent lakehouse with dedicated teams.

The same architecture is now accessible in about ten minutes.

Visit lakeops.dev:

Connect your catalog — Glue, DynamoDB, REST, S3 Tables, or Custom. Every table is discovered automatically.
Optimize a few tables — run compaction or snapshot expiration manually, review results, then flip to automated scheduling.
Scale with policies — define rules at the catalog or namespace level. New tables inherit everything.
Monitor — the dashboard shows real-time impact, insights flag what needs attention, events provide the audit trail.

Your data never leaves your account. No agents to install, no pipelines to change, no infrastructure to provision.

The intelligent lakehouse is no longer reserved for companies that can build Netflix-scale infrastructure. The building blocks are here. The question is whether your tables are maintained — or quietly degrading while you read this.

Managed Iceberg: Optimizing a Modern Lakehouse

joni sar — Sun, 10 May 2026 13:52:17 +0000

A modern lakehouse looks simple from the outside.

Data lands in object storage. Apache Iceberg gives you tables, snapshots, schema evolution, time travel, and multi-engine access. Spark writes. Trino queries. Flink streams. Snowflake or Athena may read the same data. Everyone is happy.

Then the lakehouse starts growing.

Small files pile up. Snapshots never expire. Manifest metadata gets heavier. Delete files slow down reads. Failed jobs leave orphan files behind. Query planning becomes slower. Storage cost grows in places nobody is watching. Every engine has its own behavior, its own tuning, and its own operational gaps.

This is the part that gets underestimated.

Iceberg solves the table format problem. It does not magically solve lakehouse operations.

The same pattern already happened in compute infrastructure: once systems grew large enough, manual tuning stopped scaling and platforms like LakeOps became useful because they continuously optimized resources instead of relying on people to chase every inefficiency.

Iceberg lakehouses need the same shift. Not more scripts. Not more periodic cleanup jobs. A real control plane.

That is the idea behind LakeOps: autonomous lakehouse management for Apache Iceberg. It sits above your lake, watches table and engine behavior, and continuously manages the operational work that keeps Iceberg fast, clean, and cost-efficient.

This article is a practical guide to what that means.

The real job of running an Iceberg lakehouse

When teams first adopt Iceberg, the focus is usually on features.

ACID transactions on object storage.

Time travel.

Schema evolution.

Partition evolution.

Hidden partitioning.

Multiple engines reading the same tables.

Those are important. They are why Iceberg became popular.

But after the first production workloads move in, the job changes. You are no longer just “using Iceberg.” You are operating a lakehouse.

That means you are responsible for table layout, file size, metadata growth, snapshot retention, stale data, query planning, engine behavior, storage waste, and workload safety.

A healthy lakehouse needs constant maintenance.

A production Iceberg table is not static. Every ingest, append, merge, delete, compaction, schema change, or streaming write changes its physical and metadata shape. Even when the logical table looks clean, the underlying table may be drifting.

That drift is the problem.

A table can still return correct results while becoming slower and more expensive every week.

Where lakehouse maintenance gets painful

The pain usually appears in a few predictable places.

Small files are the first one. Streaming jobs, CDC pipelines, frequent appends, and micro-batches create many small files. Query engines then spend too much time opening files, planning splits, and scanning inefficiently. A table that should be read in seconds starts feeling heavy.

Snapshots are the second. Iceberg creates snapshots so readers can see consistent table versions and users can time travel or roll back. That is useful, but old snapshots accumulate unless someone expires them. Over time, they keep metadata and old data references alive.

Manifests are the third. Iceberg tracks data files through metadata files. That is what makes Iceberg reliable and engine-independent, but metadata also needs maintenance. If manifests grow or fragment, query planning slows down before the engine even starts scanning data.

Orphan files are the fourth. Failed writes, aborted jobs, migrations, dropped tables, and imperfect cleanup flows can leave files in object storage that are no longer referenced by Iceberg metadata. Queries do not read them, but storage still bills for them.

Delete files are another common issue, especially with merge-on-read workloads. If they accumulate, every query may pay the cost of applying deletes at read time.

Then there is the engine layer. Spark, Trino, Flink, Athena, Snowflake, Databricks, DuckDB, and other engines do not behave the same way. They have different cost models, latency profiles, concurrency limits, and operational strengths. Managing Iceberg across engines is not the same as managing one Spark pipeline.

This is why “we have Iceberg” is not the same as “we have a managed lakehouse.”

The manual way most teams start

Most teams start with scripts.

A Spark job for compaction.

A scheduled job for snapshot expiration.

A cleanup script for orphan files.

A few dashboards.

Some alerts.

A runbook in a wiki.

A Slack channel where someone asks, “Why is this table slow again?”

There is nothing wrong with this as a starting point. It is how most platforms mature.

The problem is that static maintenance does not understand table behavior.

A daily compaction job does not know whether a table had a quiet day or a massive ingestion spike.

A weekly snapshot cleanup job does not know whether the table has long-running readers, branch retention requirements, or compliance rules.

A manifest rewrite schedule does not know which tables are suffering from planning latency.

A generic orphan cleanup script may be too conservative to reclaim meaningful storage or too aggressive to be safe.

And none of this naturally connects to cost, query performance, engine behavior, or table-level business importance.

Manual maintenance works when you have a small number of tables and a small number of workloads. At lakehouse scale, it becomes operational debt.

What managed Iceberg means

Managed Iceberg means the maintenance loop becomes part of the platform.

Not a one-off script.

Not a quarterly cleanup project.

Not a few jobs someone hopes are still running.

A managed Iceberg layer continuously observes the lakehouse, decides what needs attention, runs the right operation, and records the result.

It should manage the core lifecycle of Iceberg tables:

Compaction.

Snapshot expiration.

Manifest optimization.

Orphan file cleanup.

Delete file handling.

Statistics and metadata optimization.

Table health monitoring.

Policy enforcement.

Engine visibility.

Cost and performance tracking.

The key point is that management should be table-aware and workload-aware.

A hot BI table is not the same as a streaming staging table. A CDC table is not the same as a cold archive table. A table queried by Trino all day is not the same as a table used by a nightly Spark job. A table exposed to AI agents has different risk and cost patterns than an internal batch table.

A managed lakehouse should understand those differences.

The control plane model

A lakehouse control plane is the layer that coordinates operations across storage, Iceberg metadata, catalogs, engines, policies, and observability.

and

It does not replace Iceberg.

It does not replace your object storage.

It does not force all teams into one query engine.

It gives you one operating layer for the lakehouse.

LakeOps describes this as a control plane for your data lake: end-to-end optimization for tables and metadata across storage and query engines, with telemetry-driven orchestration and visibility in one place.

That distinction matters.

The goal is not to make Iceberg proprietary. The goal is to make open lakehouse operations manageable.

A good control plane should answer questions like:

Which tables are unhealthy?

Which tables are wasting the most storage?

Which tables have the worst small-file problem?

Which tables have metadata planning issues?

Which tables should be compacted now?

Which tables should not be touched because active workloads are running?

Which compaction strategy should be used?

Which snapshots can safely expire?

Which files are safe to delete?

Which engine is best for this workload?

Did the optimization actually improve cost or performance?

If you cannot answer these questions quickly, the lakehouse is being managed manually, even if it has automation scripts.

Solving the small-file problem

Small files are the most visible Iceberg maintenance issue.

They usually come from streaming ingestion, frequent appends, CDC, micro-batches, partition skew, and multi-writer workloads. The result is predictable: more files, more metadata, more object-store requests, more planning work, and slower queries.

The normal fix is compaction.

But not all compaction is equal.

The simple version is bin-packing: combine many small files into fewer larger files. This is often the right first step because it quickly reduces file count and improves scan efficiency.

The more advanced version is sort-based compaction: rewrite files according to the columns that queries filter or join on most often. This can improve data skipping and reduce scanned data, but it is more workload-sensitive. Sorting everything blindly can waste compute.

This is where autonomous management becomes useful.

LakeOps includes compaction for Apache Iceberg that uses table metadata and query patterns to decide which files to rewrite and how. The useful part is not only that it compacts. The useful part is that compaction becomes part of a continuous feedback loop.

A practical operating model looks like this:

Start with bin-pack compaction on tables with severe small-file pressure.

Use query-aware sort compaction only where query patterns justify it.

Avoid compacting cold tables just because a schedule says so.

Prioritize tables where compaction will reduce real query cost or latency.

Track before-and-after impact: file count, data scanned, planning time, runtime, and cost.

That is the difference between maintenance and optimization.

Managing snapshots safely

Snapshots are one of the best things about Iceberg.

They enable time travel, rollback, auditability, and consistent reads. But they also create retention work.

Every write creates a new table version. If snapshots are never expired, metadata grows and old data can remain retained longer than needed. On busy tables, this becomes a real cost and performance issue.

The hard part is not running expire_snapshots.

The hard part is knowing the right policy.

Some tables need long time-travel windows because they support audits, debugging, or recovery. Some tables only need short retention. Some tables may have branches or tags that must be protected. Some workloads may have long-running readers. Some environments need different rules for production, staging, and development.

A managed layer should make this explicit.

LakeOps provides snapshot management as part of table optimization, so retention can be controlled through policies rather than remembered manually per table.

For platform teams, this is a major shift.

Instead of asking, “Did someone remember to clean up snapshots on this table?”

You define retention behavior once, apply it at the right scope, and let the platform enforce it continuously.

A good default might be:

Keep enough snapshots for rollback and debugging.

Retain a minimum number of recent snapshots.

Use longer retention for critical regulated tables.

Use shorter retention for temporary or staging data.

Monitor how much storage is blocked by old snapshots.

Run expiration before orphan cleanup.

The exact values depend on the organization. The important thing is that snapshot retention becomes intentional.

Keeping metadata lean with manifest optimization

Iceberg query performance is not only about data files.

It is also about planning.

Before an engine scans data, it reads Iceberg metadata to understand which files belong to the snapshot and which files can be skipped. Manifest files are part of this metadata layer. They are essential, but they can also become fragmented over time.

When manifests grow poorly, planning time grows. Users experience this as “the query is slow,” but the engine may be spending too much time before meaningful scanning even begins.

This is easy to miss if you only look at execution time.

A managed Iceberg system should monitor metadata health directly.

LakeOps includes manifest optimization so teams can consolidate and optimize metadata as part of the same table health loop.

The principle is simple: metadata is part of performance.

If you only compact data files but ignore manifests, you are only managing half the table.

Cleaning orphan files without breaking things

Orphan files are a storage leak.

They sit in object storage but are not referenced by Iceberg metadata. Queries do not use them, but the cloud provider still charges for them.

They can appear after failed jobs, aborted commits, manual migrations, dropped tables, incorrect cleanup flows, or maintenance operations that leave old data behind.

The dangerous part is cleanup.

Deleting files from a data lake is easy. Deleting the right files safely is hard.

A safe orphan cleanup process must compare files in storage against Iceberg metadata, apply a conservative age threshold, avoid active write windows, and usually run after snapshot expiration. The age threshold matters because a file that looks unreferenced during an in-progress write may still be committed later.

LakeOps documents orphan file cleanup as a managed operation with metadata awareness and safety controls.

In practice, this is one of the strongest arguments for a control plane.

Nobody wants platform engineers manually reviewing millions of object-store paths. Nobody wants an unsafe script deleting files from production. And nobody wants to keep paying for dead data because cleanup feels risky.

Managed orphan cleanup should be boring, visible, and conservative.

Run a dry run.

Show candidates.

Apply retention thresholds.

Delete only when safe.

Record what was removed.

Measure storage reclaimed.

That is how cleanup becomes an operational capability instead of a dangerous maintenance task.

Policies are what make this scale

Manual tuning does not scale across hundreds or thousands of tables.

You need policies.

Policies let you define how table maintenance should behave at different scopes: organization, catalog, namespace, table, environment, or workload class.

For example:

Production BI tables get frequent compaction and manifest optimization.

Streaming tables get aggressive small-file management.

CDC tables get delete-file-aware compaction.

Staging tables get short snapshot retention.

Archive tables get minimal compute-heavy optimization but regular storage cleanup.

Critical tables require approvals or simulations before major rewrites.

Development tables get cheaper, more aggressive cleanup.

LakeOps supports policies for maintenance automation, allowing teams to define behavior for compaction, snapshots, manifests, orphan cleanup, and governance across the lakehouse.

This is important because the platform team should not be in the business of hand-tuning every table forever.

Good policies give teams defaults, guardrails, and exceptions.

That is how Iceberg operations become manageable.

Observability turns maintenance into engineering

If maintenance runs but nobody can measure the effect, it is not really managed.

A lakehouse control plane needs observability at the table, engine, and operation level.

You need to see:

Table health.

File count.

Average file size.

Small-file pressure.

Snapshot count.

Manifest count.

Delete file pressure.

Storage waste.

Query latency.

Planning time.

Data scanned.

Engine cost.

Operation history.

Before-and-after optimization impact.

Failed or skipped operations.

Policy coverage.

LakeOps includes lakehouse observability so platform teams can see table health, engine metrics, cross-system telemetry, and maintenance history from one place.

This changes how you operate.

Instead of waiting for users to complain that dashboards are slow, you can see which tables are drifting.

Instead of guessing whether compaction helped, you can measure the before and after.

Instead of discovering storage waste in a cloud bill, you can identify stale data and orphan files directly.

Good observability turns Iceberg maintenance from reactive firefighting into normal platform engineering.

Multi-engine lakehouses need engine-aware management

The whole point of Iceberg is that many engines can work over the same tables.

That is also what makes operations harder.

Spark may be good for heavy rewrites.

Trino may be better for interactive analytics.

Athena may be useful for serverless access.

Snowflake may serve BI workloads.

Flink may write continuously.

DuckDB may support local or embedded analytics.

Each engine has a different performance model. Each workload has a different latency and cost profile.

A managed lakehouse should not pretend all engines are the same.

LakeOps supports engine management and query routing, giving teams a unified view of engine health, cost, usage, and routing behavior.

This matters because optimization is not only about the table. It is also about where and how workloads run.

Sometimes the best optimization is a better file layout.

Sometimes it is a better engine choice.

Sometimes it is avoiding an expensive engine for simple queries.

Sometimes it is routing a workload away from an unhealthy engine.

A modern lakehouse control plane should see the full system, not just the storage layer.

Why continuous optimization beats scheduled maintenance

The old model is schedule-based.

Run compaction every night.

Expire snapshots every Sunday.

Clean orphan files once a month.

Rewrite manifests when someone remembers.

That is better than nothing, but it is not how real workloads behave.

A high-volume table may need attention multiple times a day.

A cold table may not need compaction for months.

A table may become hot because a new dashboard launched.

A backfill may create temporary file pressure.

A failed ingestion job may create orphan files.

A new AI agent may generate many new query patterns.

A static schedule cannot react to that.

Continuous optimization uses telemetry to decide what should happen next.

That is the core value of autonomous lakehouse management.

The lakehouse is not optimized because a cron job ran. It is optimized because the platform understands table state, workload behavior, and cost impact.

This is where LakeOps is useful in practice. It continuously analyzes the lakehouse, recommends or runs the right operation, and keeps optimizing as workloads change.

For a platform team, this reduces the amount of manual judgment required for routine operations.

You still set policies.

You still define guardrails.

You still decide what level of autonomy is acceptable.

But you are no longer manually chasing every unhealthy table.

A practical rollout plan

The best way to adopt managed Iceberg is not to turn everything on everywhere.

Start with visibility.

Connect the catalogs and engines. Let the platform observe table health, file layout, metadata, snapshots, and query behavior. Identify the worst tables by cost, latency, file count, metadata weight, and storage waste.

Then start with a small set of high-impact tables.

Good candidates are usually:

Large tables with many small files.

Hot BI tables with growing latency.

Streaming or CDC tables with constant writes.

Tables with high object-store request cost.

Tables with many snapshots.

Tables where users already complain about performance.

Apply conservative policies first.

Use bin-pack compaction before sort compaction.

Use dry runs for cleanup operations.

Set safe snapshot retention.

Run orphan cleanup with conservative age thresholds.

Measure everything.

Only after you see stable improvements should you expand to more tables, more aggressive compaction, sort optimization, and autonomous mode.

The point is not to give control away. The point is to move from manual table-by-table work to policy-driven operations with visibility.

What LakeOps solves, problem by problem

If you maintain Iceberg yourself, you eventually build pieces of a control plane internally.

You build table health checks.

You build compaction jobs.

You build snapshot cleanup.

You build orphan cleanup.

You build dashboards.

You build job orchestration.

You build policy conventions.

You build alerts.

You build runbooks.

You build engine-specific scripts.

You build cost reports.

Then you maintain all of that.

LakeOps packages that operating layer into one platform.

For small files, it provides autonomous compaction and layout optimization.

For slow queries, it optimizes file sizes, sort layout, manifests, and routing decisions.

For snapshot bloat, it manages retention policies.

For orphan files, it performs safe metadata-aware cleanup.

For fragmented metadata, it rewrites and optimizes manifests.

For multi-engine complexity, it gives one view of engines and can route workloads based on cost, latency, or throughput.

For operational visibility, it surfaces table health, engine metrics, events, recommendations, and optimization history.

For governance, it gives policies, auditability, and controlled automation.

For adoption risk, it works with the existing lakehouse stack instead of requiring pipeline rewrites or data movement.

That last point is important.

A control plane should reduce operational burden without becoming a migration project.

What to keep managing yourself

Autonomous management does not mean the platform team disappears.

You still own architecture.

You still own data modeling.

You still decide retention requirements.

You still define governance boundaries.

You still choose which engines belong in the platform.

You still control policies and exceptions.

You still review critical workloads.

The difference is where your time goes.

Instead of manually compacting tables, you define compaction policies.

Instead of hunting stale files, you monitor cleanup impact.

Instead of guessing why queries slowed down, you inspect table and engine telemetry.

Instead of writing one-off Spark jobs, you operate the lakehouse as a managed platform.

That is a better use of senior data platform engineering time.

The benefits beyond cost and performance

Cost and performance are the obvious wins.

Fewer small files means less scan overhead.

Cleaner metadata means faster planning.

Expired snapshots and orphan cleanup reduce storage waste.

Better layout reduces data scanned.

Better routing reduces unnecessary compute.

But there are other benefits that matter just as much.

Reliability improves because maintenance is consistent instead of ad hoc.

Governance improves because policies are explicit.

Debugging improves because every operation is visible.

Onboarding improves because new tables inherit sane defaults.

Security improves when access and actions are auditable.

Capacity planning improves because table growth and engine behavior are observable.

AI readiness improves because agents query cleaner, faster, better-governed tables.

Team focus improves because engineers stop spending so much time on repetitive maintenance.

These benefits compound.

A lakehouse that is continuously maintained becomes easier to trust.

Common mistakes when managing Iceberg manually

The first mistake is treating compaction as the whole problem. Compaction is important, but it does not replace snapshot expiration, manifest optimization, orphan cleanup, delete-file handling, or observability.

The second mistake is applying the same policy to every table. A staging table and a production revenue table should not have the same retention and optimization strategy.

The third mistake is running cleanup without a safety model. Orphan cleanup especially needs conservative thresholds and visibility.

The fourth mistake is ignoring metadata. Data files get attention because they are visible, but manifests and snapshots often explain planning latency and storage drift.

The fifth mistake is optimizing for one engine while the table is used by many engines.

The sixth mistake is not measuring impact. If you cannot show what changed after maintenance, you cannot tune the lakehouse intelligently.

The seventh mistake is waiting for incidents. Iceberg degradation is often gradual. By the time users complain, the table may have been unhealthy for weeks.

The target operating model

A modern Iceberg lakehouse should operate like this:

Tables are continuously monitored.

Health is measured at the file, metadata, snapshot, storage, and query level.

Policies define maintenance behavior.

Compaction runs when it has measurable value.

Snapshot expiration follows retention rules.

Orphan cleanup is safe and auditable.

Manifest optimization keeps planning fast.

Engine behavior is visible.

Query routing can account for cost and latency.

Optimization history is recorded.

Engineers can override, approve, or inspect operations.

The system improves continuously.

That is managed Iceberg.

Not a hosted table format.

Not a black box.

Not a replacement for engineering judgment.

A control plane that takes the repetitive, error-prone, high-volume operational work and turns it into policy-driven automation.

Final thoughts

Iceberg is a strong foundation for the modern lakehouse, but it is not the whole platform.

Once Iceberg becomes production infrastructure, the work shifts from “how do we create tables?” to “how do we keep hundreds or thousands of tables healthy while many engines and workloads use them?”

That is where many teams feel the pain.

Manual maintenance is fine at the beginning. Scripts are fine at the beginning. But as the lakehouse grows, entropy wins unless something is continuously managing the system.

Managed Iceberg is the next layer.

It means compaction, snapshots, manifests, orphan files, engines, policies, observability, and cost optimization are handled as one operating system for the lakehouse.

LakeOps is built around that idea: autonomous lakehouse management for Apache Iceberg, running on top of the stack teams already use.

For data platform engineers, the value is simple.

You keep Iceberg open.

You keep your storage.

You keep your engines.

You keep control.

But you stop managing the lakehouse one table, one script, and one incident at a time. Thanks for reading! :)

Managed Iceberg Data Lakes: A Guide

joni sar — Thu, 07 May 2026 16:47:06 +0000

Apache Iceberg has become the default table format for open data lakes. The 2025 State of the Apache Iceberg Ecosystem survey found 96.4% Spark adoption, 60.7% Trino, and growing DuckDB and Flink usage. Ryft's 2026 enterprise study reports that 58% of organizations now use Iceberg for business-critical analytics, and 79% plan to move their remaining data to it within 12 months.

Adoption is no longer the question. The question is: who maintains all of this?

Iceberg gives you snapshot isolation, schema evolution, hidden partitioning, and time travel. It does not give you someone to compact your files, expire your snapshots, clean up orphans, rewrite your manifests, or tell you which of your 800 tables is about to make your morning dashboards unusable. That is your job — and at scale, it is a job that breaks.

This guide covers what it actually takes to run an Iceberg data lake in production: the maintenance operations, the failure modes, and how LakeOps — an autonomous control plane for Apache Iceberg — addresses each of them.

What "managed" means in the Iceberg context

The word "managed" gets overloaded. In the Iceberg world, it refers to the ongoing operational work required to keep tables healthy after data is written. Iceberg handles transactional correctness — atomic commits, snapshot isolation, optimistic concurrency control. But it intentionally leaves maintenance to the operator. The format's spec defines what snapshots, manifests, and data files are. It does not define when they should be cleaned up, how files should be reorganized, or which tables need attention first.

A managed Iceberg data lake is one where these operations are handled continuously, automatically, and with awareness of the broader system — not just individual tables.

There are roughly six categories of maintenance work:

Compaction — merging small files into optimally-sized ones
Snapshot lifecycle management — expiring old snapshots and reclaiming storage
Manifest maintenance — rewriting manifests, consolidating position deletes, refreshing statistics
Orphan file cleanup — removing data files that are no longer referenced by any snapshot
Observability — knowing which tables are healthy, which are degrading, and why
Policy and governance — enforcing consistent maintenance rules across catalogs, namespaces, and teams

Each of these is straightforward in isolation. The difficulty is doing all of them, for every table, at the right frequency, in the right order, without breaking anything. Most teams start with custom Spark scripts scheduled in Airflow. That works at 10 tables. At 200 tables across multiple catalogs and engines, the scripts become the problem — brittle, uncoordinated, and blind to the interactions between operations.

LakeOps treats these six categories as a single coordinated system. You connect your catalog and storage — typically in under 10 minutes, with no agents installed, no data movement, and no pipeline changes — and the platform continuously manages all six across your entire fleet.

The small file problem and why compaction matters

Every Iceberg write produces new data files. Streaming pipelines that commit every few seconds can generate thousands of files per hour. Batch jobs with high partition cardinality scatter data across many small files. Even well-designed pipelines accumulate file fragmentation over time as tables evolve.

The consequences compound:

Query planning overhead: the query engine must open and parse every manifest entry to build a scan plan. More files means more manifest entries means slower planning. A table with 100,000+ files can push planning time from milliseconds to tens of seconds before a single byte of data is read.
I/O amplification: each file requires a separate object storage GET request. Object stores are optimized for throughput on large sequential reads, not for opening thousands of small files. A query that should scan 10 GB across 20 files instead scans the same 10 GB across 2,000 files — same data volume, dramatically worse latency.
Metadata bloat: more files means larger manifests. Larger manifests means more data for the coordinator to load into memory. At extreme scale, this can cause OOM failures during query planning.

The standard solution is compaction: periodically rewriting groups of small files into fewer, larger files targeting 256–512 MB each. Iceberg supports this natively through the RewriteDataFiles action in Spark, or through engine-specific SQL commands. But how you compact — the engine, the strategy, the awareness of actual query patterns — makes a dramatic difference.

How LakeOps handles compaction

LakeOps replaces Spark-based compaction with a purpose-built Rust engine built on Apache Arrow and DataFusion. The difference is architectural: no JVM, no garbage collection pauses, bounded memory, vectorized execution. In production benchmarks on a 5.5 TB dataset across 10 tables, the engine achieved up to 99.8% file reduction and peak throughput of 2,522 MB/s — completing jobs that caused Spark OOM on identical hardware.

But raw speed is only part of it. LakeOps compaction is query-aware: it analyzes actual query patterns against each table to determine which file groups to prioritize and which sort orders will produce the best scan pruning. This means compaction cycles directly reduce I/O for the queries your team actually runs, rather than blindly reorganizing data no one reads.

Two strategies are available:

Binpack consolidates small files into optimally-sized files (~512 MB) without changing sort order. It is the default, and it resolves the small file problem with minimal compute.
Sort compaction rewrites data in a column order derived from query filter and join patterns — enabling Iceberg's min/max metadata pruning to skip entire files. On TPC-H benchmarks, sorted layouts reduced bytes scanned by 51% with an additional ~9% compression improvement.

LakeOps also supports branch-based simulations: you can test a layout change on an Iceberg branch, see the projected impact on scan volume and query latency, and promote only if the results justify it — without touching the production snapshot.

Snapshot lifecycle management

Every Iceberg commit creates a new snapshot. Snapshots enable time travel, but they are not free:

Each snapshot references a manifest list, which references manifests, which reference data files. Old snapshots keep references alive to data files that may have been logically deleted or replaced by compaction.
A table with 120 days of hourly commits accumulates ~2,880 snapshots. Each snapshot's metadata must be tracked, and the data files it references cannot be garbage collected until the snapshot is expired.
In production, unexpired snapshots are one of the largest sources of storage waste. Teams routinely discover that 30–50% of their object storage bill is data referenced only by old snapshots that should have been expired weeks ago.

Snapshot expiration removes old snapshots and makes their exclusively-referenced data files eligible for deletion. The parameters are straightforward — retention window, minimum snapshot count, schedule — but the execution at fleet scale is where things break down. Different tables need different retention windows. Streaming tables commit far more frequently than batch tables. And expiration must happen before orphan cleanup to avoid deleting files still referenced by unexpired snapshots.

LakeOps automates snapshot lifecycle per table, respecting configurable retention policies at every scope level. You set the rules once — the platform handles scheduling, sequencing, and cleanup continuously. It also provides a snapshot explorer in the UI: browse snapshots, compare states, tag versions, and roll back — with full visibility into how much storage each retention window is costing you.

Manifest maintenance

Manifests are Iceberg's indexing layer. Each manifest file tracks a subset of data files along with their partition values, column-level min/max statistics, and file sizes. Over time, they accumulate problems that silently degrade query performance.

Manifest fragmentation: as tables evolve through many small commits, manifests proliferate. A table with 500+ manifests forces the query planner to open and parse each one during scan planning. Rewriting manifests consolidates them — reducing the number of files the planner must read and cutting planning latency.

Position delete files: Iceberg v2 supports row-level deletes via position delete files (merge-on-read). At query time, the engine reads both data files and their associated delete files, filtering out deleted rows. As delete files accumulate, read amplification grows. Rewriting position deletes physically merges them into the data files, converting merge-on-read to copy-on-write and eliminating per-query overhead.

Puffin statistics: Puffin files store column-level statistics (NDV, histograms, null counts) that enable cost-based query optimization. Stale Puffin stats lead to suboptimal join ordering and filter selectivity estimates. They need to be refreshed after significant data changes.

These operations interact with each other and with compaction. Manifest rewrites should happen after compaction (since compaction changes the file layout). Position delete merges should be coordinated with compaction to avoid redundant rewrites. Puffin refresh should follow any significant layout change.

LakeOps treats all of these as coordinated operations within a single autonomous maintenance loop. The platform understands the dependency graph — expire snapshots first, then clean orphans, then compact, then rewrite manifests and refresh statistics — and executes the full sequence at the right cadence for each table based on its ingestion rate and query patterns. No manual scheduling, no sequencing bugs, no forgotten tables.

Orphan file cleanup

Orphan files are data files that exist in storage but are not referenced by any current snapshot or metadata file. They accumulate from failed writes, completed compaction (old files remain until explicitly deleted), snapshot expiration (references removed, files left behind), and table drops or partition deletes.

Left unchecked, orphan files consume enormous storage. Production lakes regularly accumulate hundreds of terabytes of dead data — files that cost money every month but serve no purpose. One team discovered ~200 TB of orphan files across their lake, costing roughly $4,000/month in pure storage waste.

Cleanup involves listing all files in the table's storage location, comparing against all file references in current metadata, and deleting files that are unreferenced and older than a configurable age threshold. The age threshold is critical — set it shorter than your longest-running write job and you risk deleting files that are part of an in-progress transaction.

LakeOps runs orphan cleanup as part of its coordinated maintenance sequence, after snapshot expiration has properly released references. The age threshold is configurable per policy (default 7 days), and the platform ensures cleanup never runs ahead of expiration. For teams that have never cleaned orphans, the first run often reclaims a surprising amount of storage — one documented case saw a 350 TB lake shrink to 230 TB in 10 minutes.

Observability: knowing what is broken before users notice

The most dangerous state for a data lake is one where tables are degrading but no one knows. Queries get slower by 5% per week. Storage costs creep up. A table that used to plan in 200ms now takes 4 seconds. No single commit is the cause — it is the accumulation of hundreds of small writes without maintenance.

Effective Iceberg observability requires monitoring at multiple levels:

Table-level health metrics: file count, average file size, manifest count, snapshot count and age, position delete file count, orphan file volume. A table with 50,000 files averaging 2 MB is in trouble. A table with 500+ manifests has a planning bottleneck. These signals are available via Iceberg metadata tables ($files, $snapshots, $manifests), but querying them across hundreds of tables and correlating the results into actionable classifications is a significant engineering effort on its own.

Query-level signals: planning time vs. execution time (if planning exceeds 30% of total query time, metadata or file count is the bottleneck), data scanned vs. data returned (high ratios mean poor file layout), and engine-specific metrics like Trino split counts or Spark task counts.

Fleet-level views: you need to see across all tables in all catalogs to identify systemic patterns — which namespaces have the most critical tables, whether streaming tables degrade faster than batch tables, and where storage cost is concentrating.

LakeOps provides this observability out of the box. Every table gets a health classification (critical, warning, healthy) based on file count, file size distribution, manifest fragmentation, snapshot depth, and orphan accumulation. The platform dashboard shows fleet-wide health, engine metrics, cost signals, and query performance — correlated across engines so you see the full picture rather than stitching together engine-specific monitoring. Insights surface specific issues (e.g., "12 tables have small-file hotspots," "3 tables have snapshot bloat") with severity and recommended action, so you know exactly where your lake needs attention and what to do about it.

Policy and governance at fleet scale

When you have 10 tables, you configure maintenance manually. When you have 800 tables across 5 catalogs, you need policies.

A policy defines what maintenance operations should run, on which scope, at what schedule, with what parameters. Policies should cascade — an organization-wide default applies everywhere, but a namespace-level policy can override it for streaming tables that need more aggressive compaction, and a table-level policy can override that for a specific high-priority table.

Common policy dimensions:

Compaction: target file size, strategy (binpack/sort), schedule, concurrency limits
Snapshot expiration: retention period, minimum snapshots to retain, schedule
Manifest rewrite: schedule, trigger threshold (e.g., rewrite when manifest count exceeds 50)
Orphan cleanup: age threshold, schedule, scope
Table configuration standards: enforce Iceberg v2, default Parquet compression, write distribution modes, commit retry settings

LakeOps implements a hierarchical policy engine with four scopes: table → namespace → catalog → organization. More specific policies override broader ones. You define the rules once, and the platform enforces them continuously across your entire fleet. Every action is logged, auditable, and reversible — which matters when you are explaining to compliance why a table's data was reorganized at 2 AM.

For reference, a reasonable manual scheduling baseline looks like this:

Operation	Schedule	Rationale
Snapshot expiration	Hourly	High-frequency commits need frequent cleanup
Compaction	Daily, 2 AM	Off-peak, after expiration has freed references
Orphan cleanup	Daily, 3 AM	After expiration, before compaction
Manifest rewrite	Daily, 4 AM	After compaction produces new file layout

With LakeOps, you do not need to manage these schedules yourself — the platform's autonomous loop handles sequencing and adapts frequency to each table's ingestion rate — but the table above is useful for understanding the dependency order.

Multi-engine considerations

One of Iceberg's core value propositions is engine independence. A single table can be read and written by Spark, queried by Trino, accessed by Snowflake's external tables, and analyzed by DuckDB — all through the same catalog.

This creates two challenges: maintenance operations must be compatible with all engines accessing the table, and routing decisions affect both cost and performance.

Catalog as the coordination point

The Iceberg REST Catalog API has emerged as the standard for multi-engine access. LakeOps connects to your existing catalog — AWS Glue, Apache Polaris, Gravitino, Nessie, Lakekeeper, or REST-compatible catalogs — and operates through it. All maintenance commits go through the same catalog path your query engines use, ensuring atomic commits and proper concurrency handling. No separate catalog, no sidecar processes, no data movement.

Query routing

With multiple engines available, the question becomes: which engine should run which query? A small lookup query costs $0.01 on DuckDB and $0.08 on Snowflake. A complex analytical join runs in 2 seconds on Trino and 45 seconds on DuckDB. Sending every query to the same engine wastes money or wastes time.

LakeOps includes a multi-engine query routing layer that classifies queries and routes them to the optimal engine based on cost, latency, or throughput targets. Routing groups present a single SQL endpoint to downstream consumers (via PostgreSQL, MySQL, or Arrow Flight wire protocols) while the platform handles backend engine selection. Applications, BI tools, and agents connect to one endpoint — the routing layer optimizes which engine actually executes each query, and adapts as table health and engine load change.

Agentic AI and the data lake

A newer challenge: AI agents that issue SQL queries autonomously. LLM-powered agents, feature store pipelines, and autonomous data workflows increasingly query Iceberg tables directly — often at unpredictable volumes, with unpredictable query shapes, and without the human judgment that would avoid scanning a 500 TB table with SELECT *.

This changes the maintenance equation:

Latency sensitivity increases: agents expect sub-second responses. A table with a 4-second planning time because of small file accumulation is unusable for agent workflows, even if it is "fine" for batch analytics. Benchmarks show uncompacted tables impose a 5–10x latency penalty on agent queries.
Query volume increases: agents can issue 50 queries per interaction, multiplied across thousands of concurrent agents. Poorly laid-out tables amplify costs multiplicatively.
Safety becomes critical: an agent with write access to production tables can issue DDL, trigger full table scans on petabyte tables, or expose PII. Without guardrails, autonomous SQL access is a liability.

LakeOps addresses this through a dedicated agentic AI enablement layer. Agents connect via Model Context Protocol (MCP) with structured tools for schema discovery, query execution, and statistics retrieval — no raw JDBC connections. A layered guardrail chain enforces read-only access, row limits, cost estimation (via EXPLAIN), PII masking, and optional human approval for high-risk operations. The routing layer classifies agent queries and sends them to the right engine automatically. And critically, the maintenance system monitors agent query patterns and adjusts sort orders, compaction priority, and file sizes accordingly — so the lake self-optimizes for the workloads actually hitting it.

Common anti-patterns

Running compaction without expiring snapshots first. Compaction creates new files and commits a new snapshot, but the old files remain referenced by old snapshots. If you compact aggressively without expiring, you increase storage consumption — you now have both the old files and the new compacted files.

Setting orphan cleanup age threshold too low. If your longest-running write job takes 3 hours, and you clean orphans older than 1 hour, you will delete files that are part of an in-progress transaction. Set the threshold to at least 2x your longest write job.

Compacting everything on the same schedule. Not all tables need the same compaction frequency. A streaming table committing every 10 seconds needs hourly compaction. A daily batch table needs weekly compaction. One-size-fits-all scheduling either wastes compute on cold tables or under-maintains hot tables. LakeOps adapts compaction frequency per table based on ingestion rate and health signals, eliminating this tradeoff entirely.

Ignoring manifest count. Teams focus on file count and miss manifest fragmentation. A table with 500 manifests and well-sized data files will still have slow query planning because the planner must parse every manifest. Manifest rewriting is cheap — do it regularly.

Sort compaction without query analysis. Sorting by arbitrary columns wastes compute and can actually make queries slower if the sort order does not match dominant filter patterns. Always analyze query patterns before choosing sort columns — or let a system that observes actual query traffic make the decision for you.

No observability before automation. Automating maintenance without visibility into table health is flying blind. You will not know if your automation is effective, if it is missing tables, or if it is making things worse. This is why LakeOps starts with observability — classifying every table's health before taking any action.

Getting started

If you are standing up or improving a managed Iceberg data lake, here is a practical path:

Audit your current state. Query the $files, $snapshots, and $manifests metadata tables for your most important tables. How many files? What is the average file size? How many snapshots? This gives you a baseline — and usually a few surprises.
Connect LakeOps to your catalog. The platform connects to your existing catalog and storage in under 10 minutes — no agents, no data movement, no pipeline changes. Once connected, it automatically classifies every table as healthy, warning, or critical, giving you immediate visibility into your fleet's health.
Let autonomous maintenance run. LakeOps handles snapshot expiration, orphan cleanup, compaction, and manifest maintenance in the correct sequence, at the right frequency for each table. Start in observation mode if you want to review actions before they execute, then switch to full autopilot once you are confident.
Define policies for your fleet. Set organization-wide defaults for retention windows, compaction targets, and cleanup thresholds. Override at the namespace or table level where specific workloads need different treatment.
Enable query routing. If you run multiple engines, configure routing groups to optimize cost and latency across your engine mix. Present a single endpoint to consumers and let the routing layer handle engine selection.
Evaluate sort compaction selectively. For your most-queried tables, use LakeOps simulations to test sort strategies against actual query patterns. Promote layouts that show significant scan reduction.
Plan for agents. If AI agents will query your lake, enable the MCP interface and guardrail chain. Monitor agent workload patterns and let the platform optimize table layouts accordingly.

Closing

The work of managing an Iceberg data lake is not glamorous. It is compaction schedules, orphan cleanup thresholds, manifest fragmentation, and snapshot retention policies. It is the kind of work that, when done well, is invisible — queries are fast, storage is efficient, and engineers are not firefighting.

When done poorly — or not done at all — it is the reason your dashboards are slow, your cloud bill is unexplainable, and your platform team spends their time writing one-off maintenance scripts instead of building the things that actually matter.

Iceberg gives you a powerful foundation. LakeOps is what turns that foundation into a production-grade data platform — autonomously managed, fully observable, and ready for whatever workloads come next.

Introducing QueryFlux: Open-Source Universal Multi-Engine Query Router and SQL Proxy

joni sar — Mon, 06 Apr 2026 09:07:13 +0000

Efficiently routing multiple query engines is a critical challenge.

QueryFlux is a universal SQL proxy and multi-engine query router written in Rust. It sits between clients and query engines. Clients connect to QueryFlux using a protocol they already know. QueryFlux routes each query to the right backend, translates SQL dialects when needed, enforces concurrency limits, and gives you a unified observability surface.

Open table formats unified the data. QueryFlux unifies the access.

If you already run more than one query engine, you know the problem is not only where data lives. The harder part is how query access works in practice.

Which engine should this run on?
Which client should connect where?
How do you protect low-latency traffic from batch workloads?
What happens when one cluster is saturated?
How much routing logic ends up hardcoded across the stack?

That is the problem QueryFlux is built to solve.

Why QueryFlux exists

Modern data platforms are multi-engine by design.

A team may use Trino for federated queries, DuckDB for embedded analytics, StarRocks for low-latency serving, and Athena for pay-per-scan workloads on cold data. That mix is not a sign of architectural drift. In many cases, it is the right shape of the system.

Open table formats made this possible. With Apache Iceberg, Delta Lake, or Hudi, multiple engines can read the same data in object storage without duplicating it. That solved storage interoperability.

What it did not solve is compute access.

Each engine still comes with its own protocol, its own SQL dialect, its own connection handling, and its own operational behavior. Clients still need to know where to connect. Routing logic still leaks into notebooks, applications, dashboards, and team conventions. Capacity management is still fragmented across backends.

QueryFlux adds the missing layer above the table format: one access layer in front of the engine fleet.

What QueryFlux does

At a high level, QueryFlux handles three things:

protocol ingestion
routing
dispatch and dialect translation

Clients connect using protocols they already speak. QueryFlux supports:

Trino HTTP
PostgreSQL wire
MySQL wire
Arrow Flight SQL
Admin REST API

On the backend side, it already supports:

Trino
DuckDB
StarRocks
Athena

That gives it a very specific place in the stack. It is not trying to replace engines, and it is not introducing a custom client model. It is making a heterogeneous engine fleet look coherent from the access layer.

How a query flows through the system

A client connects to QueryFlux using a native protocol.

The query is evaluated against an ordered routing chain.

The first matching rule selects the cluster group that should handle the query.

From there, QueryFlux selects a healthy cluster in that group, optionally rewrites the SQL into the target dialect using sqlglot, and dispatches the query.

If the group is already at its concurrency limit, the query can queue at the proxy instead of failing immediately.

That is the important design move. QueryFlux is not just a forwarder. It is the runtime layer where access, routing, translation, and capacity handling meet.

Architecture

Client (psql / Trino CLI / mysql / BI tool)
    │
    │ native protocol
    ▼
┌─────────────────────────────────────────────┐
│                 QueryFlux                   │
│                                             │
│  Frontend ──► Router ──► Dialect translation│
│                    │                        │
│              Cluster group                  │
│         (concurrency limit + queue)         │
└──────────────────┬──────────────────────────┘
                   │
      ┌────────────┼────────────┐
      ▼            ▼            ▼
   Trino       StarRocks      Athena
      └────────────┴────────────┘
          Apache Iceberg / Delta / Hudi

The architecture is simple enough to understand quickly, but deep enough to be useful in real environments.

The simplicity is at the edge. Clients keep using the protocols they already know.

The depth is inside the routing and dispatch path, where QueryFlux can apply routing policy, translation, concurrency limits, queueing, health-aware selection, and load balancing without pushing that complexity back into every client.

Routing is where the value becomes obvious

QueryFlux evaluates each query against an ordered router chain.

Routing can be based on:

protocol
HTTP headers
SQL text using regex
client tags
Python script logic
compound rules
fallback routing

That matters because real routing logic is rarely a single condition. In practice, you may want to steer PostgreSQL wire traffic to a low-latency group, send ETL-tagged traffic to a batch-oriented cluster, and use query patterns to catch common fast-path cases.

A simple example looks like this:

routes:
  - name: fast_queries
    match:
      query_regex: "SELECT .* LIMIT \\d+"
    target: duckdb_group

  - name: dashboard_queries
    match:
      protocol: mysql
    target: starrocks_group

  - name: heavy_analytics
    match:
      query_regex: "JOIN|GROUP BY|WINDOW"
    target: trino_group

  - name: fallback
    fallback: true
    target: athena_group

The point is not that every deployment should use this exact policy. The point is that the policy becomes explicit, traceable, and shared.

That alone removes a surprising amount of hidden operational drag.

Cluster groups make routing operational

Once a route resolves to a cluster group, QueryFlux handles execution there.

It supports these load-balancing strategies:

roundRobin
leastLoaded
failover
engineAffinity
weighted

It also supports:

per-group concurrency limits
proxy-side queueing when groups are full
health-aware cluster selection
background health checks

This is where QueryFlux starts to feel deeper than a typical proxy.

It is not only deciding where a query should go. It is also giving operators a place to control how traffic behaves when systems are under load, how overflow is absorbed, and how healthy capacity is chosen.

That is the part that makes the system practical.

SQL translation is built into the path

Multi-engine routing is much more useful when SQL dialect differences do not immediately get in the way.

QueryFlux integrates dialect-only translation through sqlglot. When needed, it can rewrite SQL into the target engine’s dialect during dispatch.

That means:

clients can keep speaking the SQL they naturally emit
QueryFlux can normalize for the backend that will actually execute the query
teams do not need to maintain multiple versions of the same query only because engines differ

The current design is disciplined here. What is implemented today is dialect-only translation. Schema-aware translation is explicitly on the roadmap.

That is a good balance: the system is already useful now, and the path to deeper translation is clear.

Observability is part of the product, not an add-on

A routing layer only works if operators can see what it is doing.

QueryFlux includes:

Prometheus metrics
Grafana dashboard
Admin REST API
QueryFlux Studio

The current observability surface covers:

query counts
query duration
translation metrics
running queries
queued queries

It also supports routing traces, which matters in practice. When you introduce a routing layer, one of the first questions engineers ask is: why did this query land there? QueryFlux has a real answer to that.

Where this becomes useful quickly

The value of QueryFlux is easier to see in real scenarios than in abstract feature lists.

A multi-engine platform

BI tools connect through one access layer
different workloads are routed to the engines they fit best
backend topology becomes configuration instead of client code

Dashboard SLA protection

low-latency groups can be protected with concurrency limits
overflow can queue or spill instead of degrading the serving path

Incremental engine migration

weighted routing makes gradual traffic shifts possible
clients do not need to change while the migration happens

Mixed workloads on shared data

batch, interactive, and exploratory traffic can be separated by policy
routing intent lives in one place instead of being spread across the stack

These are practical benefits. They show up immediately once a platform becomes multi-engine.

Getting started is intentionally simple

One of the nice things about the project is that the first run experience is straightforward.

A minimal setup looks like this:

git clone https://github.com/lakeops-org/queryflux.git
cd queryflux/examples/minimal-trino
docker compose up -d --wait

That gives you:

QueryFlux on http://localhost:8080
Trino direct on http://localhost:8081
Admin API on http://localhost:9000
Studio on http://localhost:3000
Postgres on localhost:5433

You can then send a simple query through the Trino HTTP frontend:

curl -X POST http://localhost:8080/v1/statement \
  -H "X-Trino-User: dev" \
  -d "SELECT 42"

There are also examples for:

a minimal in-memory setup
a Prometheus + Grafana stack
a full stack with Trino, StarRocks, and Iceberg-related services

That combination is important. The system is conceptually ambitious, but the on-ramp is short.

It feels like deep infrastructure without feeling heavy to try.

What is already shipped

QueryFlux already includes a substantial set of capabilities on the main branch:

Trino HTTP frontend
PostgreSQL wire frontend
MySQL wire frontend
Arrow Flight SQL frontend
Admin REST API
Trino backend
DuckDB backend
StarRocks backend
Athena backend
ordered router chains and routing fallback
route tracing support
per-group concurrency limits
proxy-side queueing
multiple load-balancing strategies
health-aware cluster selection
dialect-only translation through sqlglot
in-memory persistence
PostgreSQL persistence
authentication providers including none, static, OIDC, and LDAP
authorization modes including allow-all, simple policy, and OpenFGA
Prometheus metrics
Grafana dashboard
QueryFlux Studio
dynamic config reload from Postgres

This matters because the project already feels like infrastructure, not just an idea.

What comes next

The roadmap extends the same core design.

Near-term work includes:

schema-aware SQL translation
ClickHouse backend and HTTP frontend
richer routing telemetry in Studio

Medium-term work includes:

cost- and performance-aware routing
Snowflake backend
BigQuery backend
Redis persistence

That roadmap makes sense. It deepens the same access layer instead of changing the project’s center of gravity.

Solving the data ccess side

The most interesting thing about QueryFlux is not that it is a proxy.

It is that it is a carefully placed layer in a part of the modern data stack that is still surprisingly underbuilt.

Open table formats solved the data side.

QueryFlux is solving the access side.

That creates an appealing combination:

conceptually clean architecture
obvious operational benefits
room for sophisticated policy and routing logic
low-friction adoption path

It feels like the kind of infrastructure that becomes more valuable as the rest of the stack becomes more heterogeneous.

Getting started

Once a data platform becomes multi-engine, the missing piece is usually not another engine.

It is the access layer.

Clients still need to know where to connect. Routing still leaks into tools and applications. SQL dialect differences still show up at the edges. Capacity handling is still fragmented.

QueryFlux gives that layer a shape.

It makes multi-engine access easier to reason about, easier to operate, and easier to evolve.

That is why it is a compelling project: the idea is deep, the benefits are immediate, and the first experience is simple.

To try it out visit: https://queryflux.dev/

11 Compaction Optimizations for Iceberg Data Lakes

joni sar — Mon, 16 Feb 2026 12:55:54 +0000

Compaction should provide an easy solution to a very difficult problem: controlling file count, minimizing the cost of delete operations over read costs, and keeping metadata growth from turning every query plan into a deep, time-consuming walk through snapshots and manifests.

The data layer has compaction as its mechanism for solving these issues, but only if compaction is run using some defined set of rules governing the scope of compactions, the thresholds above which compactions are run, and the synchronization of compaction runs with the timing of snapshot expirations and the maintenance of manifests.

Compaction can be run manually via scripts and schedules, or automatically by a control plane.

When manual scripts are used to run compaction, they can effectively manage compaction for a small number of tables and one engine. However, as soon as there are many tables, and/or multiple engines, the manual process becomes guesswork; scripts may rewrite too much, may run too infrequently, may interfere with ongoing ingestions, and will cause churn in both the snapshots and manifests.

A control plane flips this model completely around.

Instead of rewriting everything all the time, a control plane continuously monitors the health and workload characteristics of tables. Then, only when necessary, a control plane spends rewrite budget on the parts of the tables that actually change performance or cost, while also managing the entire lifecycle of maintaining the table.

This article will teach you how to run compaction like the production lakes do it: how to choose your base line strategy (bin-packing vs sorting) for compaction, how to prevent rewrites of healthy partitioning, how to limit the scope of each compaction so maintenance remains invisible, how to focus on hot and delete-heavy areas first, how to prevent the continuous commit cadence of streaming data from creating a factory of snapshot partitions, and how to synchronize compaction with the metadata cleanup that is needed to maintain stable query planning.

For further reading on other aspects of optimizing and maintaining Iceberg, please also refer to:

Let's move on to the compaction strategies that actually work in real production lakes.

1. Add a Control Plane for 20x Faster Compaction and Optimized Table Maintenance

If you add a control plane to your lake, LakeOps comes with the most powerful and intelligent compaction engine that exists today, and will also manage and optimize table maintenance and lake operations for you.

Snowflake-like experience for Iceberg with 10x performance (source: lakeops.dev)

Instead of being confined to fixed schedules, LakeOps operates as a control plane for Iceberg tables and can know when and how to compact what. It treats compaction as a continuous operational problem rather than a periodic batch job, and optimizes it in real time.

It analyzes telemetry data from query engines and Iceberg catalogs and uses that data to decide when compaction is actually needed, what to compact, and how. It takes actual usage patterns into account as well.

Under the hood, LakeOps uses a dedicated Rust-based compaction engine that is designed specifically for Iceberg layouts and metadata behavior. Compaction is coordinated with snapshot expiration, manifest rewrites, orphan cleanup, and statistics maintenance so these operations reinforce each other instead of fighting.

The results are ~20x faster compaction, ~15x faster queries, and ~80% CPU/Storage cost saving.

🚢 Apache Iceberg compaction is not “background maintenance.” It’s a time-critical optimization problem that directly impacts query latency, metadata growth, and infrastructure cost.

Learn more about it here:

Apache Iceberg Compaction: Time-Critical Optimization | Amit Gilad

In addition to compaction LakeOps gives you control with manual and autopilot modes for all maintenance operations in youe tables and coordinates them with compaction. That includes expiring snapshots, manifest rewrites, orphan file cleanups and more.

Real-Time comoaction and maintenance optimization with a control plane (source: lakeops.dev/)

You can choose between manual mode and auto-pilot per table or for groups of tables to control compaction and maintenance proccesses.

LakeOps also lets you define policies across the lake to enforce your standards, and provides you with dashboards to see and manage all compaction and maintenance processes per table and for the entire lake.

Learn more: https://lakeops.dev

2. Use bin-pack as the baseline correction

Most iceberg tables do not require complex layout schemes. However, they all suffer from file fragmentation.

Before attempting to fix the problem using sort-based layouts, clustering, or partitioning, take a closer look at the most obvious source of file fragmentation: the writing process itself. In almost all cases, the initial performance decline caused by file fragmentation is due to the streaming ingestion of very small batches; micro-batch commits occur very frequently; and backfill data will always come in very unevenly-sized chunks.

As a result of the write process, many small Parquet files exist within each partition. None of these files are "broken," and queries will still continue to provide accurate answers. However, as more and more small Parquet files exist within each partition, planning time will increase, the overhead associated with task scheduling will grow, and the number of object store calls will grow.

This is not a layout issue; it is a file count issue.

The easiest and most reliable method to solve the file count issue is to use bin-pack compaction. Bin-pack compaction combines small data files into a smaller number of larger, properly sized files. Bin-pack compaction does not alter the existing file sort order, nor does it re-cluster data; it merely normalizes the file size and decreases the metadata overhead associated with having a high number of files.

In practice, this is usually sufficient.

Why This Approach to Compaction Really Does Improve Performance

Iceberg engines operate at the file level. The more files you add to an engine, the more the engine must plan:

More manifest entries must be read

More file footers must be inspected

More scan tasks must be scheduled

More file references to delete must be tracked

As the number of files grows exponentially, so too does the planning time. Bin-pack compaction eliminates the number of files physically on disk, while maintaining the existing logical layout. Therefore, there are fewer planning reads, and fewer tasks to schedule without requiring additional shuffling.

A good rule of thumb for most production tables is to target file sizes ranging from 128 MB to 512 MB. The specific size range will depend on the engine, and the workload. What is important is consistency.

Start with the Default Rewriting Method

Unless you specify otherwise, Iceberg uses the bin-pack rewriting method:

CALL prod.system.rewrite_data_files( 
  table => 'db.events'
);

Verify the effectiveness of the rewrite:

SELECT
  count(*) AS file_count, 
  sum(file_size_in_bytes)/1024/1024/1024 AS total_size_gb
FROM prod.db.events.files;

You want to see fewer files and the same total size. If the total size is substantially changed, something other than file fragmentation is occurring.

Specify Your Target File Size

If file fragmentation continues, specify a target file size at the table level:

ALTER TABLE prod.db.events SET TBLPROPERTIES ( 
  'write.target-file-size-bytes'='536870912' -- 512MB
);

And then perform the rewrite with the same target file size:

CALL prod.system.rewrite_data_files( 
  table => 'db.events', 
  options => map( 
    'target-file-size-bytes','536870912' 
  ) 
);

Without specifying a target file size, the engine and writer will create files of varying sizes, which will require subsequent compactions to restore the target file size.

Define Thresholds to Prevent Unnecessary Rewrites

At scale, performing rewrites on healthy data wastes compute resources. Define the following thresholds to prevent unnecessary rewrites:

CALL prod.system.rewrite_data_files( 
  table => 'db.events', 
  options => map( 
    'min-input-files','5', 
    'min-file-size-bytes','134217728' -- 128MB 
  ) 
);

This will ensure that only those partitions with a minimum of five input files are rewritten. Partitions with one or two files of reasonable sizes are ignored.

These thresholds can be made tighter by defining multiple conditions:

CALL prod.system.rewrite_data_files( 
  table => 'db.events', 
  options => map( 
    'min-input-files','5', 
    'min-file-size-bytes','134217728', -- 128MB 
    'target-file-size-bytes','536870912' -- 512MB 
  ) 
);

Therefore, compaction will now only run under the following conditions:

There are enough small files to warrant consolidation

Files are currently below a reasonable size threshold

There is a clear target to normalize to

This transforms compaction from an automated rewrite operation to a targeted repair operation.

Prior to executing a compaction operation, review the distribution of files:

SELECT 
  partition.event_date, 
  count(*) AS file_count, 
  avg(file_size_in_bytes)/1024/1024 AS avg_mb 
FROM prod.db.events.files 
GROUP BY partition.event_date 
ORDER BY file_count DESC 
LIMIT 20;

If a partition contains four files averaging 480 MB and your target file size is 512 MB, rewriting the partition will not significantly affect either planning time or scan time.

However, if another partition contains 180 files averaging 25 MB, that partition clearly requires compaction.

Decisions regarding compaction operations should be based on this type of signal. Not a schedule.

3. Conditional Compaction

Another way to waste compute cycles in an Iceberg lake is to blindly compact data on a regular basis.

It usually begins innocently. A periodic rewrite job is created to "keep things tidy." For a period of time, it appears to help. Eventually, however, it begins to rewrite partitions that were previously healthy. Each rewrite generates new files, new snapshots, and updates the manifest list. Although none of the files are "broken," the system is spending compute cycles on processing data that does not improve performance.

Compaction should be used as a corrective measure, not as a routine activity.

Why Unconditional Rewrites Cause Churn

Each time you rewrite data files, Iceberg:

Creates new data files

Generates a new snapshot

Updates manifest lists

Increases metadata history

If the files being rewritten are already close to their target size, you are essentially cycling the data through the system. The added churn causes increased metadata depth and longer planning times over time.

At scale, this overhead becomes noticeable.

Your objective is not to compact frequently. It is to compact when the layout is measurably unhealthy.

Implement Gateways to Control Rewrites

Iceberg provides a rewrite_data_files procedure that allows you to implement gateway conditions. The most effective condition is min_input_files.

CALL prod.system.rewrite_data_files( 
  table => 'db.events', 
  options => map( 
    'min-input-files','3' 
  ) 
);

Using this condition, Iceberg will only rewrite file groups that contain at least three files. Partitions that already have one or two files that are of reasonable size are excluded.

This is a relatively small change to make, but in large lakes, it will significantly reduce unnecessary compaction.

You can implement additional conditions to make this gateway even tighter:

CALL prod.system.rewrite_data_files( 
  table => 'db.events', 
  options => map( 
    'min-input-files','5', 
    'min-file-size-bytes','134217728', -- 128MB 
    'target-file-size-bytes','536870912' -- 512MB 
  ) 
);

Compaction will now only run when the following conditions are met:

There are sufficient small files to warrant consolidation

Files are currently below a reasonable size threshold

There is a valid target to normalize to

This converts compaction from an automatic rewrite into a targeted repair operation.

Let the Table State Drive the Decision

Prior to initiating a compaction operation, review the distribution of files:

SELECT 
  partition.event_date, 
  count(*) AS file_count, 
  avg(file_size_in_bytes)/1024/1024 AS avg_mb 
FROM prod.db.events.files 
GROUP BY partition.event_date 
ORDER BY file_count DESC 
LIMIT 20;

If a partition has four files averaging 480 MB and your target file size is 512 MB, rewriting the partition will not materialy affect either planning or scanning time.

However, if another partition has 180 files averaging 25 MB, that partition is a prime candidate for compaction.

Compaction decisions should be based on signals such as this. Not a schedule.

4. Limit Rewrite Scope Per Run

Big compaction jobs appear great on paper. In reality, they are among the easiest ways to create instability in a production lake.

Backfills, partition evolutions, or long stretches of time without maintenance can quickly turn terabytes of data into rewrite candidates. If you don't specify any boundaries, Iceberg will rewrite everything that meets its criteria. The outcome is well understood: long-running jobs, significant shuffles, large amounts of object store I/O, significant increases in snapshot sizes, and sometimes even cluster contention with users' queries.

Compaction operates at its best when it is incremental.

Why "Rewrite Everything" Is A Risk

When you rewrite a large section of a table in a single pass, you are executing a number of costly operations simultaneously:

Reading numerous data files

Shuffle and rewrite them

Create a lot of new files

Create a new large snapshot

Possibly rewrite manifests

Regardless of whether it is successful, you have produced a major maintenance event. Even if it fails in the middle of its execution, you will lose some resources and extend the length of your maintenance period.

Operationally, smaller, and more frequent corrections are safer than infrequent large-scale rewrites.

Restrict rewrite size explicitly

Rewrite_data_files is a method that allows Iceberg to provide parameters that can help limit the amount of effort that is put into a single run of rewriting data.

To illustrate, you may desire to restrict the maximum number of file group rewrites:

CALL prod.system.rewrite_data_files(
  table => 'db.events',
  options => map(
    'max-file-group-rewrites','20'
  )
);

This restricts the number of rewrite groups executed in a single pass of rewrite_data_files. Instead of rewriting hundreds of partitions in one pass, you will execute a controlled sequence of batch passes.

You can also utilize these with eligibility thresholds:

CALL prod.system.rewrite_data_files(
  table => 'db.events',
  options => map(
    'min-input-files','5',
    'max-file-group-rewrites','20',
    'target-file-size-bytes','536870912'
  )
);

Compaction now becomes predictable. Each pass of compaction corrects a finite amount of drift.

Disperse correction over cycles

There is rarely a good reason to try to "Fix everything tonight" if a table contains months of small files due to a large number of small writes.

Instead, a more sustainable and steady state approach would be:

Compact data with limited rewrite scope.

Permit normal operation of user workloads.

Repeat on the subsequent maintenance cycle.

Within a couple of cycles, fragmentation will decrease significantly, without generating a maintenance peak.

This also produces less "snapshot shock." Instead of a large, single-pass rewrite snapshot replacing nearly half of the table, you generate a series of smaller, incremental snapshots.

Prioritize rather than rewriting randomly

Limiting rewrite scope, prioritizing becomes essential.

Practically speaking, you want to rewrite:

Partition groups with the greatest number of files.

Partition groups with the least average file size.

Partition groups with the largest numbers of deletions.

Partition groups with the most user queries.

You can find the worst offending partition groups using the following query:

SELECT
  partition.event_date,
  count(*) AS file_count,
  avg(file_size_in_bytes)/1024/1024 AS avg_mb
FROM prod.db.events.files
GROUP BY partition.event_date
ORDER BY file_count DESC
LIMIT 50;

Then either target the offending partition groups directly using a WHERE statement or permit your orchestration layer to determine the highest impact groups to rewrite first.

For example:

CALL prod.system.rewrite_data_files(
  table => 'db.events',
  WHERE => 'event_date >= DATE ''2026-01-01''',
  options => map(
    'max-file-group-rewrites','10'
  )
);

This limits both the logical scope (only recent partitions) and the physical rewrite volume.

Maintain Maintenance Invisible To Users

The ultimate objective of limiting rewrite scope is not merely cluster stability. It is predictability.

When compaction is done in a manner such that each run is relatively small and bounded:

Maintenance windows are brief.

Resource spikes are under control.

Snapshots grow gradually.

Query performance improves incrementally, rather than suddenly.

In production lakes, stability is usually more important than the rate of correction. Incremental correction is generally preferred to dramatic restructuring.

5. Focus On Hot Partition Groups

In the majority of Iceberg tables, compaction impact is not uniformly distributed.

A small segment of partitions is responsible for most of the pain: They receive the most writes (thus they tend to fragment the quickest) and they receive the most reads (thus every additional file appears as both planning + scan overhead). If you rewrite the partitions that are "hot", you will typically gain 80% of the benefits with only a fraction of the rewrite volume.

The simplest approach to achieve this is to treat compaction as a rolling window problem.

"Hot" Generally Means Two Things

Hot partitions generally represent partitions that are still active:

They are still receiving new files from streaming or micro-batch systems

They are the partitions that your analysts / dashboards / downstream jobs are accessing constantly.

This results in two operational principles:

Compact relatively recent partitions regularly, since they will accumulate the most small files.

Do not compact the actively written partitions unless you know you can tolerate collisions with writers.

AWS describes this for Iceberg compaction: Use a where predicate to exclude actively written partitions, so that you do not encounter data conflicts with writers and leave only metadata conflicts that Iceberg can normally resolve.

Identify Your Rolling Window With Where

Iceberg's Spark procedure provides a where predicate for filtering which files (and hence which partitions) qualify for rewriting.

An extremely common use case is "Compact everything older than the current ingest window":

CALL prod.system.rewrite_data_files(
  table => 'db.events',
  WHERE => 'event_date < DATE ''2026-02-10''',
  options => map(
    'target-file-size-bytes', '536870912',
    'min-input-files', '5'
  )
);

This maintains compaction away from the partition(s) that are currently being mutated, while continually cleaning up yesterday's and previous data.

If your table is partitioned hourly, perform the same concept at the hour granularity:

CALL prod.system.rewrite_data_files(
  table => 'db.events',
  WHERE => 'event_hour < TIMESTAMP ''2026-02-11 12:00:00''',
  options => map(
    'target-file-size-bytes', '536870912',
    'min-input-files', '5'
  )
);

The main principle here is not the specific cut-off. It is maintaining a buffer so that compaction does not conflict with ingestion.

Locate the Worst Partition Groups First

Even within the "hot-ish" window, not all partition groups are equally bad. You can typically find the worst offender simply by examining the file count and average size:

SELECT
  partition.event_date,
  count(*) AS file_count,
  avg(file_size_in_bytes)/1024/1024 AS avg_mb
FROM prod.db.events.files
GROUP BY partition.event_date
ORDER BY file_count DESC
LIMIT 30;

If you plan to compact only a limited portion of the data per pass (as you probably should), this query will indicate where you will obtain the largest initial benefit.

If You Must Compact Partition Groups That Are Still Receiving Late Data

There are certain types of workloads that receive late-arriving events, updates, or merges that keep older partition groups "active." If you compact them regardless, you may periodically collide with writers.

Iceberg includes a partial progress mode that commits compaction in smaller portions, rather than committing a large block, which reduces the collision risk associated with retries when conflicts occur.

CALL prod.system.rewrite_data_files(
  table => 'db.events',
  WHERE => 'event_date >= DATE ''2026-02-01'' AND event_date < DATE ''2026-02-10''',
  options => map(
    'partial-progress.enabled', 'true',
    'partial-progress.max-commits', '10',
    'max-concurrent-file-group-rewrites', '10'
  )
);

You are making a tradeoff of "one clean commit" versus "multiple smaller commits that fail at lower expense." In actual production lakes with continuous write activity, that trade is typically worthwhile.

Where a Control Plane Helps

Once you have numerous tables, "hot partition groups" ceases to be something you determine through experience. You require a loop that continuously determines the "hot partition groups" based on the actual read/write usage of your system and then applies the rolling window concept to those partition groups.

That is the point at which a control plane such as LakeOps becomes useful: It is not adding a new compaction algorithm as much as it is determining where to expend your rewrite budget based on real workload telemetry and applying that determination in a consistent manner across hundreds of tables.

6. Sort or Z-Order When Scan Efficiency Is The Bottleneck

Bin-packing compaction decreases the number of files. However, it does not affect how the data is organized internally within those files.

If partitions are appropriately sized and file counts are reasonable, but selectivity-based queries are scanning a disproportionately larger number of data files than anticipated, the cause is likely clustering. You will commonly observe the following pattern: Planning times are consistent, partition pruning is functioning, but filtered queries are reading a large percent of files within a given partition. This is the point where sorting-based compaction becomes relevant.

Data engines depend on file-based statistical information (such as min and max values) to determine whether a file can be excluded from processing. When data is written in a random fashion, the value ranges between files overlap greatly. Therefore, even selective predicates typically cannot exclude a large number of files. Sorting changes this. When data is ordered by a frequently used filter column, each file will generally contain a narrower value range than previously existed, thus allowing more files to be skipped based upon the predicate and reducing the number of bytes to be scanned.

If there is one column that dominates your predicates (e.g., event_time within a date-partitioned table), a simple sort-based rewrite is typically sufficient:

CALL prod.system.rewrite_data_files(
  table => 'db.events',
  strategy => 'sort',
  sort_order => 'event_time ASC NULLS LAST',
  options => map(
    'target-file-size-bytes','536870912',
    'min-input-files','5'
  )
);

This rearranges the rows of data within files so that time-based predicates can remove more files early in the process. The effect is evident not only in terms of runtime, but also in the reduction in scanned bytes and the number of splits generated.

If your workload filters on multiple columns (e.g., user_id, event_type, and occasionally device_type), Z-order is typically a superior choice:

CALL prod.system.rewrite_data_files(
  table => 'db.events',
  strategy => 'sort',
  sort_order => 'zorder(user_id, event_type)',
  options => map(
    'target-file-size-bytes','536870912',
    'min-input-files','5'
  )
);

Z-ordering enhances locality across multiple dimensions. While Z-ordering will never perfectly optimize any individual column, it typically minimizes overall scan expansion when filter patterns vary.

Important Note: Defining a sort order at the table level does not rewrite historical data. It only affects newly-written data. All existing files will remain unmodified until a rewrite occurs.

It is typical to define a sort order and then expect improvements, only to discover that no changes occurred to the physical layout of the data.

After performing a sort-based compaction, verify it correctly. Examine the number of files that are scanned for common predicates. Compare the total bytes scanned before and after. Runtime can be difficult to quantify in shared environments; however, file count and total bytes scanned are more reliable metrics.

Sorting is more resource-intensive than bin-packing. Sorting involves additional shuffle and CPU overhead during compaction. If you were to blindly apply sorting to all partitions, the costs of maintenance could potentially exceed the query performance improvements. In general, sorting works best when applied selectively: Target high-traffic partitions; Align the sort with real filter patterns; Apply rewrites incrementally.

When scan efficiency is the primary bottleneck rather than file count, sorting or Z-order is one of the few techniques that will reliably enhance pruning. The key is to apply sorting or Z-order in a manner that aligns with the characteristics of your workload.

7. Compact delete-heavy partitions deliberately

You can view a table, note that file sizes are healthy, and believe that compaction is being handled - yet the queries are slowing down.

A common cause is delete files.

Iceberg does not immediately write out data files when rows are modified or deleted. Rather, it will store the row position deletes or equality deletes with the data. At read time, the engine will combine the data files with their associated delete files. Although this provides an efficient mechanism for writing, as delete files become numerous, every query will incur additional cost.

The affect is subtle. File sizes appear fine. Bin-pack has already normalized fragmentation. However, scan CPU increases, and partitions that are update-heavy begin to perform poorer than append-only partitions.

You may usually verify this by examining the delete file distribution:

SELECT
  partition.event_date,
  count(*) AS delete_file_count
FROM prod.db.events.delete_files
GROUP BY partition.event_date
ORDER BY delete_file_count DESC
LIMIT 20;

If certain partitions contain a high concentration of delete files, that is an indication that reads are performing more work than necessary.

Iceberg supports delete aware compaction. Instead of rewriting files solely based upon their size, you may specify a threshold for the ratio of deleted rows and have Iceberg rewrite data files that meet this criteria. For example:

CALL prod.system.rewrite_data_files(
  table => 'db.events',
  options => map(
    'delete-ratio-threshold','0.3',
    'remove-dangling-deletes','true',
    'target-file-size-bytes','536870912'
  )
);

In this case, Iceberg will rewrite data files that are severely impacted by deletes and will remove the physical representation of deleted rows as well as the associated delete files.

The practical result is that queries will no longer require the merging of as many delete files at runtime; CPU decreases; scan cost stabilizes; and planning becomes easier since there are fewer auxiliary files to track.

This has a significant impact primarily on tables that have been subjected to upserts, CDC pipelines, and/or frequent merges. Event tables that are used exclusively for appending data do not typically exhibit this behavior. Similarly, dimension tables and slowly changing datasets do.

Just like with any other compaction strategy, maintain focus. Use the combination of delete thresholds, partition filters, and rewrite limits to optimize the compaction strategy. It is unnecessary to rewrite the entire table simply because a handful of partitions contain a high amount of delete activity.

Healthy file size does not ensure healthy performance. If delete files comprise the majority of a partition, the compaction strategy must specifically target these delete files - otherwise, read cost will continue to increase regardless of whether the layout appears to be "correct".

8. Rewrite position delete files individually when needed

Rewriting data files does not necessarily resolve all delete related issues.

In many update-intensive workloads, position delete files accumulate more quickly than data files are rewritten. Even if you execute a delete aware compaction strategy, you can still find yourself with a large number of position delete files attached to otherwise healthy data files.

Even after executing a compaction strategy that reduces the number of data files, the engine still has to open and apply the delete files during a read operation. Therefore, as delete files accumulate, the scan overhead will remain greater than it should be.

This is particularly prevalent in tables that receive regular upserts or merges. Tables that are subject to append-only inserts do not typically exhibit this type of behavior. However, CDC pipelines and dimension tables do.

Iceberg permits the explicit rewriting of position delete files as follows:

CALL prod.system.rewrite_position_delete_files('db.events');

Rewrite smaller delete files into fewer, larger ones and attempt to eliminate obsolete entries wherever possible. The objective of this approach is not merely to reduce the total number of files, but rather to minimize the number of files that the engine has to open during a read operation and thus improve performance.

You may also limit the scope of this rewrite operation, similar to how you limit the scope of data file rewrites, by specifying the partition(s) to be rewrote:

CALL prod.system.rewrite_position_delete_files(
  table => 'db.events',
  where => 'event_date >= DATE ''2026-02-01'''
);

If you have already rewritten the data files and the performance of your application has not improved, inspect the distribution of delete files. In some cases, rewriting data files will reduce fragmentation, but will leave a heavy delete layer behind. In such cases, the separate rewriting of delete files will be required.

Similar to all of the strategies described throughout this guide, keep the rewriting of delete files focused. There is little to be gained by rewriting delete files across the entire table if only a limited number of partitions are subject to upserts.

In addition, treating the maintenance of delete files as a separate entity maintains the predictability of read cost. Otherwise, even though the data files themselves are sized appropriately, the accumulated overhead that is generated by the engine processing the delete files can slow over time.

9. Lower the commit frequency for streaming workloads

When writing to Iceberg from streaming or micro-batch applications, the commit frequency is one of the largest factors contributing to the overall cost multiplier in the system.

Each commit generates a new snapshot and produces new metadata work. This includes updating the manifest and creating new, small data files. As you commit every few seconds, you don't simply create small files; you create a long chain of snapshots and a continuous flow of metadata churn. While nothing "breaks," the planning is slowed and maintenance must continually struggle to keep pace with the increasing overhead.

The frustrating aspect is that teams typically attempt to resolve this issue by applying more compaction, while the true solution lies upstream: stop committing as often.

The benefits of modifying the commit frequency

When you increase the interval between commits, you generally gain three tangible benefits at once.

First, you generate fewer snapshots, resulting in fewer pieces of metadata that the engine has to evaluate during planning.

Second, you generate fewer manifests / manifest updates overall.

Third, each commit contains more data, resulting in larger files (or fewer small files) and therefore reduced compaction pressure.

You're essentially lowering entropy at the source.

Structured Streaming using Spark: Set a valid trigger interval

A common antipattern is to configure structured streaming to run "as fast as possible" or with a very short trigger. If you are writing to Iceberg tables, avoid this practice unless you have a true requirement for sub-minute freshness.

The following shows the configuration for setting a reasonable commit interval in PySpark:

(
  df.writeStream
  .format("iceberg")
  .outputMode("append")
  .option("checkpointLocation", "s3://prod-checkpoints/events/")
  .trigger(processingTime="1 minute")
  .toTable("prod.db.events")
)

If your service level agreement (SLA) allows it, increase the commit interval to 2–5 minutes. In most analytics lakes, this tradeoff is worthwhile: slightly increased data freshness lag in exchange for significantly decreased metadata churn and less maintenance overhead.

Flink: Commit frequency follows checkpointing

For Flink, Iceberg commits typically follow the checkpoint intervals. If you checkpoint every 30 seconds, you are essentially committing every 30 seconds. That's a lot.

A more reasonable interval would be minutes, not seconds, unless you are operating a low-latency serving pipeline.

Example:

// 5 minutes
env.enableCheckpointing(300_000);

Ultimately, the best value will depend on recovery requirements and end-to-end latency needs. However, the underlying premise is the same: do not checkpoint so frequently that you turn your Iceberg table into a snapshot factory.

A simple method to select the interval

Do not overcomplicate things. Ask yourself: what is the longest delay that your downstream consumers can tolerate for data freshness?

If the response is "near real-time", you may still be fine at 1 minute. If the response is "a few minutes", take advantage of the situation and commit every few minutes.

If the response is "we run dashboards hourly", then committing every 10 seconds is just self-imposed suffering.

Sanity check

If you observe thousands of snapshots being created daily for a single table, this is typically an indication that your commit cadence is too aggressive for an analytics lake. You can certainly use Iceberg as a means of generating data in this manner - it is designed to be correct - but you will pay for it in terms of planning overhead and ongoing maintenance.

Lowering the commit frequency is one of the few optimization techniques that will decrease costs and improve stability, independent of whether you adjust the compaction strategy. Fixing this earlier is beneficial because once you have multiple dozen or hundred of streaming-written tables, this behavior will dictate your operational overhead.

10. Stop the repair loop; fix the write path

Write paths that produce too many small files or heavily skewed partitions will make compaction a never-ending battle. As long as you continue to re-write the same issues, the lake will always drift back into an unhealthy condition.

The majority of "we need more compaction" situations are actually "our write path is poorly configured."

Begin with Distribution Mode

Small file generation is a common result of poor data distribution during the write process. A common scenario is when one writer (task) has the majority of the data for a given partition, it emits a couple of large files, while the remaining writers (tasks) emit a large number of smaller files. Worse, if the data distribution is unstable between batches, you will experience fragmentation regardless of how often you compact.

Iceberg allows you to configure the way data is written across multiple writers. A good baseline configuration for many workloads is hash, as it generally spreads rows out more evenly:

ALTER TABLE prod.db.events SET TBLPROPERTIES (
  'write.distribution-mode' = 'hash'
);

This does not completely remove the necessity for compaction, but it helps slow down how quickly fragmentation occurs again.

Establish a Target File Size for Writers at the Table Level

When writers do not have a target file size, you will see variability in the file sizes produced by writers across the engine and job. Some will produce 16MB files, some will produce 1GB files, etc., and compaction will continually attempt to normalize the mess.

Create a target file size for writers at the table level and maintain it consistent:

ALTER TABLE prod.db.events SET TBLPROPERTIES (
  'write.target-file-size-bytes' = '536870912' -- 512MB
);

Once you have established a target file size for writers at the table level, compaction will transition from "fix everything" to "fix the outliers."

Optimize the Writer and Not Just the Table

Another common reason writers produce small files is due to the number of tasks that are utilized when writing versus the amount of data in each micro-batch or partition. The easiest method to optimize this is to adjust the degree of parallelism at the point of write.

If you are experiencing hundreds of files per partition per batch, consider reducing the number of output partitions prior to writing:

(
  df
  .repartition(200) // pick a number that corresponds to your cluster and batch size
  .writeTo("prod.db.events")
  .append()
)

You don't have to find the optimal number. All you need to do is stop creating 1000 small files because your job happened to run with 1000 tasks.

Do Not Create "Hot Partitions" by Design

Some datasets inherently skew towards certain items, such as one customer producing 70% of the events, or a specific date receiving a massive backfill. When a partitioning scheme directs a large amount of data into a single partition, you will continually be fighting it with compaction.

This is one of the few instances where adjusting the partitioning scheme to provide less skewness can greatly reduce compaction load. A common strategy is to add another dimension to the partitioning scheme (or create a derived shard key) to ensure that a single logical partition does not become a physical hotspot.

You do not need to re-design the entire table. One additional dimension may be sufficient to prevent the worst skewness.

11. Maintain Metadata With Compaction

You can obtain the desired file sizes, reduce the number of files, and yet still end up with a table whose performance and cost characteristics degrade over time. This is typically a metadata issue and not a physical layout issue.

Every time you run compaction, you create a new snapshot. Every snapshot adds to the table's history. Each manifestation accumulates. The old metadata remains until something removes the history. If nothing removes the history, the table becomes deeper and more expensive to reason through, even if the physical data files appear to be clean.

This is the most common trap: Teams focus on rewrite_data_files and neglect what happens to the snapshots and manifestations after the fact.

In general, compaction should be run immediately followed by snapshot expiration. If you keep thousands of historical snapshots around "just in case," the engine still has to traverse the lineage when performing planning. Over time, this shows up as slower metadata reads and longer planning times.

Typically, a snapshot expiration would look something like this:

CALL prod.system.expire_snapshots(
  table => 'db.events',
  retain_last => 10
);

The actual number of retained snapshots will vary based on your rollback and time-travel policies, however, it is critical that you establish a clear retention policy. Retention policies greater than infinite are rarely what you truly need.

After expiring snapshots, it is also beneficial to perform manifest consolidation:

CALL prod.system.rewrite_manifests('db.events');

Even if your file sizes are acceptable, fragmented manifests will require the planner to open and evaluate numerous small metadata files. Manifest consolidation will reduce the fan-out and stabilize the planner costs.

Then there is the removal of orphans. Failed jobs, speculative tasks, and partial re-writes will leave files in object storage that are no longer referenced by the table. Over months, this is a lot of money. Removing these orphans will help to predictably manage the lake.

CALL prod.system.remove_orphan_files(
  table => 'db.events',
  older_than => TIMESTAMP '2026-02-10 00:00:00'
);

The older_than guardrail is essential. You cannot afford to be racing with active writers in a production environment. Safety is more important than aggression.

What complicates this is that the above actions are not independent. How you retain snapshots impacts what you can remove. How frequently the table is updated determines how frequently you need to rewrite manifests. Removing orphans is related to when commits occur and streaming jobs.

Therefore, compaction is not simply a single maintenance action. It is part of a life cycle. Physical data files, snapshots, manifests, and physical storage move in tandem.

At small scales, you can run these actions manually and get away with it. At larger scales, you need to be able to enforce consistency. Tables drift in various ways at varying rates. Without coordinated metadata maintenance, you will continue to repair file layout, while the metadata layer quietly continues to grow.

Your goal is not simply to minimize the number of small files. Your goal is to maintain a table whose performance and cost characteristics remain stable over time. Compaction addresses the data layer. The three above actions address the metadata layer to prevent it from becoming the next bottleneck.

Recap and Conclusion

Compaction in Iceberg is not about scheduling rewrite_data_files. It is about maintaining the alignment between the layout, deletes, and metadata of a table with its actual usage.

Compaction plus table maintenance is now a coordination problem. Therefore, having a control plane to continuously assess the health of tables, prioritize the top partitions requiring compaction, and coordinate compaction with metadata maintenance rather than treat these as separate jobs is a home run on the first step:

Manual work and scripting are typically the alternatives.

We previously reviewed the practical aspects of this:

Use bin-pack to control the file count

Escalate to sorting only when the scan efficiency is the limiting factor

Use gates to restrict the re-writing of healthy data

Limit the scope to make the maintenance predictable

Proactively resolve delete-heavy partitions

Reduce the commit entropy in streaming jobs

Correct the write path to prevent constant repair of the same issues

Connect compaction to snapshot expiration, manifest rewrites, and orphan removal

Most importantly, we connected compaction to snapshot expiration, manifest rewrites, and orphan removal - because the physical data layout and the metadata health are interdependent.

Your goal is not to simply minimize the number of small files. Your goal is to maintain a lake that remains predictable - in terms of performance, cost, and operational overhead - as it grows.

If you are operating Iceberg in production, I would appreciate your feedback regarding what has worked (and failed) for you. Real world patterns are always more interesting than theoretical ones.

Thank you for reading 🍺

Iceberg Rewrite Manifest Files: A Guide

joni sar — Sun, 08 Feb 2026 15:31:17 +0000

Iceberg Rewrite Manifest Files: A Guide

A data engineer happily runing into the quicksand

Manifest rewrites are a critical ongoing operation in Iceberg table maintenance.

Data keeps landing, queries stay correct, and nothing looks obviously wrong. But over time, query planning takes longer, metadata reads increase, and latency creeps up even though the amount of data scanned hasn’t really changed. In most production systems, the root cause is not data layout — it’s metadata, and specifically how manifest files accumulate and degrade over time.

Manifest files are central to how Iceberg works. They’re what allow engines to plan efficiently without listing object storage. But with frequent commits, streaming writes, deletes, and long snapshot histories, manifests naturally fragment. Iceberg doesn’t reorganize them automatically, so planning cost quietly grows until it starts to matter.

This guide focuses on rewrite manifests: what they actually do, when they help, and how to run them correctly in production. You’ll learn how to detect when manifest rewrites are needed, how they interact with snapshot expiration and compaction, and why running them in isolation often delivers disappointing results.

We’ll also contrast two operational models: managing all of this manually with scripts and schedules, and handling it continuously through a Control Plane like LakeOps, which optimizes table maintenance based on real workload behavior instead of fixed timers.

The rest of the article guides you through mannual optimization with practical examples. No spec theory, no generic advice — just what actually works when Iceberg tables grow, change, and age in real systems.

Let’s begin then 🙂

Smart Automation vs Manual Scripts

Before getting into mechanics, it’s important to understand the two main ways teams approach manifest management: Automated maintenance optimization with a Control Plane like LakeOps, and doing this operation manually and ongoingly by hand, and using generic scripts.

Continuous Optimization with a Control Plane

A control plane is a layer that sits above your data lake, catalogs, and query engines and takes responsibility for operating and optimizing tables over time. Iceberg defines table structure and guarantees correctness, but it intentionally does not decide when, where, or how aggressively maintenance should run. That operational and optimization gap is exactly what a control plane fills.

Instead of running maintenance because a schedule says it’s time, a control plane continuously optimizes tables based on what is actually happening in the system. Operations run only when and where they are needed, or according to explicit policies you define, rather than blindly across all tables.

LakeOps acts as a control plane for Iceberg by continuously analyzing telemetry from Iceberg catalogs and query engines. Using this data, LakeOps builds a live understanding of how each table behaves in practice.

Cotinious Optimization and Maintenance with an Iceberg Control Plane (source: lakeops.dev)

From that telemetry, LakeOps continuously optimizes table maintenance. Manifest rewrites are triggered only when metadata fragmentation begins to impact planning or cost. Snapshot expiration runs only when retained history no longer provides real value. Compaction is optimized continuously to reduce small files before they create downstream metadata pressure. Orphan cleanup runs when metadata and data files are no longer referenced and can safely be removed.

Coordination is central to optimization. Rewrite manifests, snapshot expiration, compaction, and cleanup are not independent jobs. They are executed as part of a single, continuous optimization loop that ensures only the required operations run, only on the tables that need them, and only at the point where they actually improve performance or cost.

Automating Smart Rewrite manfiest operations (source: lakeops.dev)

Engineers don’t tune per-table schedules or chase drifting thresholds. They decide what should be optimized and within what constraints, and the control plane decides when and where to run each operation. The result is stable metadata, predictable performance, and far less work.

Learn more:

9 Apache Iceberg Table Maintenance Tools You Should Know

Start at the beginning: What Manifest Files Are

Manifest files are the core metadata units Iceberg uses to describe which data files exist and what is inside them. They sit between snapshots and actual data files and are the reason Iceberg can plan queries efficiently without scanning directories or listing objects in storage.

Each manifest file is essentially a list of data file entries. For every data file, the manifest records:

the partition values for that file
record count
per-column statistics such as min and max values
file size and other low-level attributes

When a query runs, the engine does not discover data files by walking object storage. Instead, it reads manifests referenced by the current snapshot and uses the stored statistics to decide which data files can be skipped entirely. This is how Iceberg enables predicate and partition pruning at planning time.

Manifests are immutable. Every commit creates new metadata. When a write happens, Iceberg typically creates one or more new manifest files describing the files added or removed by that commit. Over time, a snapshot references many manifests, some created recently and some carried forward from older snapshots.

This design is powerful, but it has predictable operational consequences.

Frequent small commits, especially from streaming or micro-batch ingestion, tend to produce many small manifest files. For example, a streaming job that commits every minute may generate hundreds or thousands of manifests per day, each describing only a handful of data files. From Iceberg’s point of view this is correct, but for the query engine it means more metadata to read and evaluate during planning.

Another issue is manifest clustering. Manifests are not automatically reorganized around how tables are queried. If files are appended over time with mixed partitions or evolving data distributions, manifests may contain entries that are poorly aligned with common filters. The engine still prunes correctly, but it has to examine more metadata to do so.

Snapshots make this worse if they are not expired. Each snapshot retains references to the manifests that describe its table state. Even if newer snapshots supersede old ones, the metadata remains live as long as those snapshots are kept. This means manifests that are no longer useful for active queries still participate in metadata reads and storage costs.

The net effect is subtle but significant. Query planning time increases even though data size stays flat. Metadata I/O grows quietly. Storage costs creep up due to retained metadata. None of this breaks correctness, which is why it often goes unnoticed until performance degrades.

Manifest rewrites exist specifically to address these issues. They allow Iceberg to reorganize and consolidate manifests so that the metadata layer reflects the current table state and access patterns, rather than the historical accident of how data arrived over time.

What Rewrite Manifests Does

A rewrite manifests operation restructures the metadata layer of an Iceberg table without touching the data itself.

At a high level, Iceberg takes the manifest files referenced by the current snapshot, reads the data-file entries inside them, and writes a new set of manifest files that describe the exact same live data files, just in a layout that’s cheaper for engines to plan against. The commit updates table metadata to point the current snapshot at the new manifests. The old manifests become obsolete once nothing references them anymore (usually after snapshot expiration and cleanup).

This is a metadata rewrite, not a data rewrite. No Parquet/ORC/Avro files are rewritten.

What actually improves

Rewrite manifests helps in three very concrete ways.

It reduces manifest fan-out. When you have many small commits (streaming, micro-batch), you often end up with lots of tiny manifest files. Each query has to open and evaluate those manifests during planning. Rewriting consolidates many small manifests into fewer, larger ones, which reduces metadata I/O and planning latency.

It aligns manifest layout with partitioning. Iceberg sorts data-file entries in manifests by fields in the partition spec. In practice, this tends to make partition pruning cheaper because related entries are adjacent and engines do less work to decide what to skip.

It removes “historical write shape” from the current snapshot. Without rewrites, manifests reflect how data arrived over time, not how it’s queried. Rewriting reorganizes metadata around the current state, which is usually what you actually care about for planning.

What rewrite manifests does not do

It does not compact data files. Tiny data files stay tiny. It does not change partitioning or rewrite records.

It does not delete old manifests by itself. If old snapshots still reference them, they’ll remain. Cleanup is a separate step.

Practical code examples

Below are a few examples that are actually useful in day-to-day operations, not just “hello world”.

1) Measure the problem before you touch anything

Start by inspecting the metadata table that lists manifests. Don’t assume column names — Iceberg versions and engines can differ — so first look at the schema:

\-- Spark: inspect the manifests metadata table schema  
DESCRIBE TABLE EXTENDED prod.db.my\_table.manifests;

Then get a baseline count:

\-- How many manifests does the current snapshot reference?  
SELECT COUNT(\*) AS manifest\_count  
FROM prod.db.my\_table.manifests;

If this number grows steadily week over week while the table isn’t exploding in size, planning overhead is usually creeping up.

2) Rewrite manifests via Spark SQL procedure (the most common operational path)

This runs the rewrite in parallel using Spark:

CALL prod.system.rewrite_manifests('db.my_table');

In Spark, this returns a small result set with counters (how many manifests were rewritten, how many were added). In practice, you run the call, note the counters, and then re-check my_table.manifests to see the manifest count drop.

3) Rewrite manifests for a specific partition spec (when you’ve done partition evolution)

If your table has evolved partition specs over time, you may want to rewrite manifests for a particular spec id:

CALL prod.system.rewrite\_manifests(  
  table   => 'db.my\_table',  
  spec\_id => 1  
);

This is useful when an older spec still contributes a lot of manifest fragmentation and you want to target it instead of doing everything blindly.

4) Disable Spark caching if executors get memory pressure during rewrites

Some environments prefer to avoid caching during maintenance to reduce executor memory footprint:

CALL prod.system.rewrite_manifests('db.my_table', false);

If you’ve ever seen maintenance jobs destabilize executor memory, this is one of the first knobs to reach for.

5) Validate the effect (simple but important)

After the rewrite, validate that you actually improved the metadata shape:

SELECT COUNT(\*) AS manifest\_count\_after  
FROM prod.db.my\_table.manifests;

If the count doesn’t drop (or drops only slightly), the usual causes are that snapshots weren’t expired (old manifests still referenced), or the table’s write pattern keeps producing fragmentation faster than your maintenance cadence.

That’s the point where you either tighten the full maintenance loop (expire snapshots, rewrite manifests, remove orphans, and revisit compaction) or stop doing this manually and let a control plane keep it stable continuously.

When You Should Rewrite Manifests

Manifest rewrites are not something you run on a fixed schedule “just in case”. They are most effective when there is a clear signal that metadata, not data, is becoming the bottleneck.

The most common trigger is planning getting slower while data size stays flat. If query runtimes increase but the amount of data scanned is roughly the same, the extra time is often spent in planning and metadata evaluation. This is especially visible in engines that log planning or analysis time separately.

Another strong signal is manifest growth that outpaces data growth. If storage size grows slowly but the number of manifests keeps climbing, you are accumulating metadata fragmentation. This usually happens in tables with frequent commits, even if each commit is small.

Tables that receive streaming or micro-batch writes are prime candidates. Frequent commits tend to generate many small manifests. Even if data files are reasonably sized, the metadata layer becomes increasingly expensive to process.

A very common real-world pattern is a table that “looks healthy” in storage metrics but becomes steadily slower to query over weeks. Nothing is broken, nothing obvious changed, but planning time creeps up. That is almost always a manifest problem.

Managing Manifest Rewrites Manually

If you don’t use a control plane, the following sequence reflects what works well in production if done right and in context.

Step 1: Inspect Metadata Health

Before you decide to rewrite manifests, you need visibility into the live metadata — not guesswork, not periodic dashboards, but concrete numbers that reflect how fragmented the metadata has become.

Iceberg exposes metadata tables that you can query just like regular tables. These include tables like …$manifests, …$files, …$snapshots, etc. You can use these directly in SQL to inspect current state and spot trouble early.

Iceberg stores metadata in layers:

-   A **manifest list** per snapshot points to all manifests for that snapshot.
-   Each **manifest file** lists a subset of data files, partition values, and column statistics (min/max/null counts).
-   Manifests may be reused across snapshots.

As a result:

Lots of small commits → many small manifests.
Old snapshots hold onto old manifests.
Query engines read manifests during plan time to prune partitions/files.

If manifests are fragmented or numerous, query planning becomes slow because engines read and evaluate many metadata files before they touch actual data.

This is why metadata health matters early, not late.

What to Look At

Here are the core checks you should be doing regularly — ideally automated — to monitor manifest health.

🔍 1) Count the Current Manifests

Run a live count of manifests referenced by the current snapshot:

SELECT COUNT(\*) AS active\_manifest\_count  
FROM prod.db.my\_table$manifests;

A sudden jump in this number relative to data size usually correlates with planning overhead.

A steady climb over time, without data volume growth, is a strong indicator your metadata is fragmenting.

2) Look at Files per Manifest

Iceberg metadata stores statistics such as file counts per manifest. Pull a distribution:

SELECT  
  CASE  
    WHEN record\_count < 10 THEN '<10 rows'  
    WHEN record\_count BETWEEN 10 AND 100 THEN '10–100 rows'  
    ELSE '100+ rows'  
  END AS manifest\_size\_bucket,  
  COUNT(\*) AS manifests  
FROM prod.db.my\_table$manifests  
GROUP BY 1  
ORDER BY 1;

If you see lots of manifests with very few rows/files, that means fragmentation. It means many small manifests (from tiny commits) that blow up planning work.

You can also look at larger manifests: if lots of manifests hold small amounts of data, it’s a sign that maintenance will be valuable.

3) Compare Manifests to Data Growth

If you track how data size and manifest count change together, you can spot divergence:

\-- number of data files  
SELECT COUNT(\*) AS data\_file\_count  
FROM prod.db.my\_table$files;

\-- number of manifests  
SELECT COUNT(\*) AS manifest\_count  
FROM prod.db.my\_table$manifests;

If manifest_count grows faster than data_file_count, that’s another sign of metadata inefficiency.

4) Look at Snapshots (Optional but Useful)

Snapshots tell you how many historical versions you’re retaining, which impacts how many manifests persist:

SELECT  
  committed\_at,  
  snapshot\_id  
FROM prod.db.my\_table$snapshots  
ORDER BY committed\_at DESC  
LIMIT 10;

Long snapshot histories mean old manifests may still be referenced and not cleaned up until expiration happens.

Interpreting the Results

Here are practical heuristics data engineers use:

High manifest count with small average manifest size → metadata fragmentation (good candidate for rewrite).
Stable manifest count but slow query planning → the problem might be clustering, not count; manifest rewrites can help.
Lots of snapshots older than retention needs → metadata is being kept too long; expire them first so rewrites can be effective.
Manifest growth outpacing data file growth → metadata is drifting away from the current shape of data.

Example Scenario

Imagine a streaming table ingesting updates every minute. You might see:

5,000 data files
2,000 manifests
70% of manifests contain <10 files

That’s a classic candidate for consolidated manifests: smaller number of larger manifests will cut planning time dramatically, especially if queries filter on partitions that aren’t well clustered yet.

Step 2: Expire Snapshots First

Always expire snapshots before rewriting manifests. This is not a best-practice nicety — it directly determines whether a manifest rewrite will actually do anything useful.

The easiest way to achieve this is using a Control Plane for automated and optimized maintenance operations that include snapshot expirations in addition to manifest rewrites. Learn more:

Automated and optimized snapshot expiration with a Control Plane (source: lakeops.dev)

Her’es a deep dive into the topic and practical solutions:

11 Expire Snapshots Optimizations for Apache Iceberg

Snapshots are what keep manifests alive. Every snapshot references a specific set of manifest files that describe the table state at that point in time. As long as a snapshot exists, all of its manifests must remain reachable, even if they describe data that is no longer relevant for current queries.

If you run a manifest rewrite while old snapshots are still retained, Iceberg can only optimize the manifests referenced by the current snapshot. Older snapshots will continue to reference older manifests, which means:

old manifests stay in storage,
metadata fan-out remains higher than expected,
storage costs don’t drop,
and in some engines, planning still touches more metadata than necessary.

This is the most common reason teams say “we ran rewrite manifests and it didn’t really help”.

Why snapshot expiration comes first

Think of snapshot expiration as pruning the metadata graph.

Until you expire snapshots, Iceberg is obligated to preserve historical metadata for correctness and time travel. A rewrite cannot remove or consolidate manifests that are still referenced by retained snapshots. Expiring snapshots reduces the metadata surface area first, so the rewrite can actually consolidate what remains.

In practice, snapshot expiration is what turns a rewrite from “cosmetic” into “effective”.

Inspect snapshot history before expiring

Before expiring anything, look at what you’re retaining:

SELECT  
  snapshot\_id,  
  committed\_at,  
  operation  
FROM prod.db.my\_table$snapshots  
ORDER BY committed\_at DESC;

In many production systems, you’ll find snapshots going back weeks or months, even though nobody ever queries historical versions beyond a few days.

That’s usually accidental, not intentional.

Expire snapshots based on real needs

Snapshot retention should reflect actual recovery and audit requirements, not defaults or copy-pasted examples.

If you only need:

a few days of rollback for operational safety, or
short-term auditability,

then retaining dozens or hundreds of snapshots actively hurts metadata efficiency with no upside.

A common pattern is:

retain snapshots newer than a time threshold, and
always keep the last N snapshots as a safety net.

Example: expire old snapshots in Spark

Here’s a practical Spark SQL example:

CALL prod.system.expire\_snapshots(  
  table => 'db.my\_table',  
  older\_than => TIMESTAMP '2024-01-01',  
  retain\_last => 2  
);

This removes snapshots older than the specified timestamp while keeping the most recent snapshots for safety.

After this runs, many old manifests will become unreferenced — which is exactly what you want before rewriting manifests.

Validate the effect

After expiring snapshots, re-check your metadata:

SELECT COUNT(\*) AS remaining\_snapshots  
FROM prod.db.my\_table$snapshots;

You should see a much smaller snapshot set. At this point:

old manifests are no longer protected,
rewrite manifests can actually consolidate metadata,
and orphan cleanup will be able to reclaim storage.

Step 3: Run Rewrite Manifests

At this point, the current snapshot references only the metadata that still matters. That gives Iceberg room to consolidate and reorganize manifests instead of carrying forward historical baggage.

What this step actually does now

After snapshot expiration, rewrite manifests can:

merge many small manifests into fewer, larger ones,
reorganize data-file entries so they’re better clustered by partition and statistics,
reduce the amount of metadata the engine has to read during planning.

If you skip snapshot expiration, most of these benefits are muted. After expiration, they show up immediately in planning time and metadata size.

Running rewrite manifests (Spark example)

In Spark-based environments, this is usually done via a system procedure:

CALL prod.system.rewrite\_manifests('db.my\_table');

This executes the rewrite in parallel across the cluster. Spark will read the existing manifests, generate a new optimized set, and commit a new snapshot that references them.

The command itself is simple. The impact depends entirely on whether you prepared the table correctly in the earlier steps.

Validate that it actually worked

After the rewrite, always check the result:

SELECT COUNT(\*) AS manifest\_count\_after  
FROM prod.db.my\_table$manifests;

You should see a noticeable drop in manifest count or, at the very least, fewer very small manifests. If nothing changes, the usual reasons are:

snapshots were not expired, so old manifests are still referenced,
the table’s write pattern is fragmenting metadata faster than maintenance runs,
or the table is already in a reasonably healthy state.

Why this step is cheap — but not free

Rewrite manifests does not rewrite data files, so it’s much cheaper than compaction. However, it still:

reads all manifests referenced by the current snapshot,
writes new manifest files,
and commits new metadata.

On large tables with many manifests, this can still consume noticeable CPU, memory, and I/O. That’s why you should not run it blindly across hundreds of tables at once.

Practical scheduling guidance

If you’re doing this manually:

avoid peak query hours,
stagger rewrites across tables,
and gate execution on actual metadata health signals rather than time alone.

Running rewrite manifests selectively, when metadata drift is real, is what keeps it a high-ROI operation instead of background noise.

Step 4: Remove Orphan Files

Once snapshots are expired and manifests are rewritten, you need to clean up what is no longer referenced.

Orphan files are data or metadata files that exist in storage but are no longer referenced by any snapshot. They typically appear after snapshot expiration, manifest rewrites, failed jobs, or aborted commits. Iceberg does not delete these files automatically, because doing so without coordination would risk correctness.

If you stop after rewriting manifests, those unreferenced files will remain in object storage indefinitely.

From Iceberg’s point of view, everything is correct after a rewrite. From your cloud bill’s point of view, nothing changed.

Skipping orphan cleanup is one of the most common reasons teams see storage costs grow even though they “ran all the maintenance jobs.” The metadata graph is clean, but the physical files are still sitting in S3, GCS, or ADLS.

This step is what turns logical cleanup into actual cost reduction.

What orphan cleanup actually removes

Orphan cleanup removes:

old manifest files no longer referenced by any snapshot,
metadata files left behind by rewrites and expired snapshots,
data files created by failed or rolled-back writes.

It does not remove any file that is reachable from a live snapshot. If a file is still referenced, it stays.

Running orphan cleanup (Spark example)

In Spark environments, orphan cleanup is typically done with a system procedure:

CALL prod.system.remove\_orphan\_files(  
  table \=\> 'db.my\_table',  
  older\_than \=\> TIMESTAMP '2024-01-01'  
);

The older_than guard is critical. It ensures Iceberg only deletes files older than a safe cutoff, protecting against races with in-flight or recently committed jobs.

Never run orphan cleanup without a time threshold.

Validate the effect

After cleanup, you should see:

a reduction in storage usage over time,
fewer unreferenced metadata files,
and no change in query results or correctness.

Storage metrics won’t always drop instantly due to object store reporting delays, but the trend should flatten instead of creeping upward.

The key takeaway

Snapshot expiration and manifest rewrites clean up logical metadata. Orphan cleanup is what turns that into physical cleanup.

If you skip this step, maintenance looks successful on paper but storage costs keep rising. If you include it consistently, metadata maintenance finally translates into real, measurable savings.

Step 5: Coordinate with Compaction

Manifest rewrites optimize how files are described. Compaction optimizes how many files exist. If you ignore compaction, manifest rewrites will help briefly — then fragmentation will return.

Small data files are the main upstream cause of manifest churn. Every time a write job produces many small files, Iceberg must record them in metadata. Even if you rewrite manifests perfectly, frequent small-file writes will recreate fragmentation within days.

The optimal solution is to use an Iceberg Control Plane. LakeOps for example, compacts data 95% faster and cheaper than alternatives thanks to a rust based engine and analyzed cross-system data. Compaction is also smarter, so query times go up ans costs go down dramatically.

![](https://cdn-images-1.medium.com/max/1600/1*7w1IT-CzuDQRQsVuCionDA.png)

Compaction optimizagtion with a dcontrol plane (source:lakeops/dev)

In LakeOps, compaction processes are also synchronized with maintenance processes like manifest rewrites, so everything runs smoothly and you don’t have to connect and coordinate it yourself. Results are optimized and are usually far better than a home-made solution.

Example: why rewrites alone don’t hold

Let’s go back to the core problem for a second, and then see how to manually address it if you don’t use a control plane.

Consider a table with streaming ingestion committing every few minutes:

Each commit writes 20–50 small Parquet files.
Each commit creates one or more new manifests.
After a week, the table has thousands of data files and hundreds of manifests.

You run snapshot expiration and rewrite manifests. Planning time improves.

Two days later, the table is slow again.

Nothing is broken. The write pattern simply recreated the same metadata pressure. This is what happens when compaction is missing or misaligned.

Use metadata to confirm compaction pressure

Before scheduling more manifest rewrites, check whether small files are the real problem:

SELECT  
  COUNT(\*) AS file\_count,  
  AVG(file\_size\_in\_bytes) / 1024 / 1024 AS avg\_file\_mb  
FROM prod.db.my\_table$files;

If the average file size is far below your engine’s sweet spot, manifest rewrites are treating symptoms, not the cause.

Another useful signal:

SELECT  
  COUNT(\*) AS manifests,  
  SUM(added\_data\_files\_count) AS total\_files\_tracked  
FROM prod.db.my\_table$manifests;

If file counts are high and keep growing quickly, metadata pressure will return unless compaction slows it down.

How compaction stabilizes manifest rewrites

When compaction is running correctly:

fewer data files are created per write cycle,
manifests grow more slowly and stay denser,
rewrite manifests becomes an occasional cleanup, not a recurring firefight.

In stable tables, teams often find that:

compaction runs frequently (or continuously),
manifest rewrites run infrequently,
snapshot expiration runs regularly.

That balance is what keeps planning predictable.

Practical coordination rule

In production, a simple rule holds up well: If manifest rewrites are needed often, compaction is not doing enough.

If you find yourself rewriting manifests weekly or daily on the same tables, it’s usually a sign that upstream file layout is unstable.

Should you add compaction code here?

At this point in the guide, full compaction code examples are usually not helpful. Compaction is engine-specific, workload-specific, and already well-covered elsewhere. What matters here is understanding the dependency:

compaction reduces metadata churn,
reduced churn makes manifest rewrites effective,
without compaction, rewrites are temporary relief.

That mental model is more valuable than a generic compaction snippet.

Step 6: Add policies and guardrails

If you manage maintenance with scripts or schedulers, automation needs guardrails or it will drift out of alignment with reality.

The simpest way is to add a Control Plane.

Then you can define policies per table or for your entire lake.

Define maintenance policies with a control plane (source:lakeops.dev/)

Start by skipping inactive tables. Tables that are rarely queried or written to don’t need aggressive maintenance. Running rewrites on them just burns cluster resources without benefit.

Avoid peak query hours. Even though manifest rewrites are cheaper than data compaction, they still consume CPU, memory, and I/O. Running them during high query load increases contention and hurts user-facing performance.

Trigger maintenance based on observed metadata health, not time alone. Manifest count, average files per manifest, snapshot growth, and planning time trends are far better signals than “once a day” or “once a week”.

Finally, expect thresholds to change. Write patterns evolve, query behavior shifts, and what worked six months ago may be wrong today. Scripts that never get revisited slowly turn into background noise or, worse, a source of instability.

This is the point where many teams decide that maintaining guardrails manually is more work than it’s worth and move metadata maintenance into a control plane.

Practical steps to take

Rewrite manifests is one of those Iceberg operations that looks optional until it isn’t. When metadata is healthy, planning is fast and predictable. When it drifts, everything still works — just slower and more expensively. That’s why manifest issues often go unnoticed for a long time.

In this guide, we walked through what manifest files actually are, why they fragment in real systems, and what rewrite manifests really does under the hood. We covered when rewrites are worth running, why snapshot expiration has to come first, how orphan cleanup turns logical cleanup into real cost savings, and why compaction is the long-term stabilizer that keeps metadata from degrading again.

If you’re managing this manually, the step-by-step approach will get you there. If you want this handled continuously and optimzied, a Control Plane exists to do exactly that — operating and optimizing Iceberg tables based on real workload behavior instead of fixed schedules.

The big takeaway is that rewrite manifests only works well as part of a coordinated maintenance loop. Run it in isolation and the benefits are usually temporary. Pair it with snapshot expiration, compaction, and cleanup, and it becomes one of the highest-ROI metadata optimizations Iceberg offers.

If you’ve run into edge cases, different patterns, or lessons learned the hard way, feel free to share them in the comments.

Thanks for reading, and hope this helps keep your Iceberg tables healthy, fast, predictable, and boring in all the right ways.

Cheers 🍺

Learn more

11 Iceberg Performance Optimizations You Should Know.

13 Apache Iceberg Optimizations You Should Know.

11 Apache Iceberg Cost Reduction Strategies You Should Know.

9 Data Lake Cost Optimization Tools You Should Know.

11 Must-Know FrontEnd Trends for 2020

joni sar — Sun, 29 Dec 2019 12:30:59 +0000

Or- how to sound smart in frontEnd lunch conversations!

Sounding smart at your team's lunch talks is obviously a great reason to stay updated with the latest frontend trends. It might even help you become a better developer, build better technology and better products. Maybe.

So, please allow me to make this honorable quest easier by pointing you in a few interesting directions. I will not explain every concept A-Z, but will introduce the concept, how it’s useful and direct to further resources.

For example, we’ll shortly cover an introduction to Micro Fontends, Atmoic Design, Web components TS take-over, ESM CDN and even Design tokens. Feel free to scroll through and mark the topics you’d like to learn more about. For any questions or more suggestions, just drop a comment below.

Short disclaimer: I'm on the team building Bit. This doesn't make any of the following less true though. Enjoy!

1. Micro frontends

Micro Frontends are the buzziest frontend topic for lunch conversations.
Ironically, while frontend development enjoys the modular advantages of components, it is still largely more monolithic than backend microservices.

Micro frontends bring the promise of splitting your frontend architecture into different frontends for different teams working on different parts of your app. Each team can gain autonomy over the end-to-end lifecycle of their micro frontend, which can be developed, versioned, tested, built, rendered, updated and deployed independently (using tools like Bit for example).
Instead of explaining the whole concept here, **read this great post** by @thecamjackson published at the @martinfowler blog. It’s really good and should cover everything you need to start digging into this concept.

However, there are still certain shortages in today’s ecosystem. Mostly, people are worried by issues like the deployments of separate frontends, bundling, environment differences etc. Bit already lets you isolate, version, build, test and update individual frontends/components. For now, this is mainly useful when working with multiple applications (though It’s already commonly used for gradually refactoring parts of existing apps via components).
When Bit will introduce deployments in 2020, independent teams will get the power to develop, compose, version, deploy and update standalone frontends. It will let you compose UI apps together and let teams create simple decoupled codebases with independent continuous deployments and incremental upgrades. The composition of these frontends will end up creating your application. Here's how a UI app composed with Bit looks like:

Learn more:
Micro Frontends - Martin Fowler

2. Atomic Design

Atomic Design is yet another super interesting topic for lunch talks, which I like to think about more of as a philosophy than a pure methodology.
Simply put, the theory introduced by Brad Frost compares the composition of web applications to the natural composition of Atoms, Molecules, Organisms and so on- ending with concrete web pages. Atoms compose molecules (e.g. text-input + button + label atoms = search molecule). Molecules compose an organism. Organisms live in a layout template, which can be concretized into a page delivered to your users.
Here’s a *detailed 30-seconds explanation with visual examples. *It includes very impressive drawings I made with great artistic talent, which you can copy-paste to your office board 😆
The advantages of Atomic components go beyond building modular UI applications through modular and reusable components. This paradigm forces you to think in composition so you better understand the role and API of every component, their hierarchy, and how to abstract the building process of your application in an effective and efficient way. Take a look.

3. Encapsulated Styling and Shadow Dom

Source: developer.mozzila.org
An important aspect of components is encapsulation — being able to keep the markup structure, style, and behavior hidden and separate from other code on the page so that different parts do not clash, and the code can be kept nice and clean. The Shadow DOM API is a key part of this, providing a way to attach a hidden separated DOM to an element.
Shadow DOM is actually used by browsers for a long time now. You can think of the shadow DOM as a “DOM within a DOM”. It is its own isolated DOM tree with its own elements and styles, completely isolated from the original DOM.
It allows hidden DOM trees to be attached to elements in the regular DOM tree — this shadow DOM tree starts with a shadow root, underneath which can be attached to any elements you want, in the same way as the normal DOM. The main implication of this is that we have no need for a namespace for our classes, as there’s no risk of name clashing or style spilling. There also additional advantages. It is often referred to as the long-promised solution to a true encapsulation of styles for web components. Learn more:
Using shadow DOM

4. The TypeScript take over

So lately every conversation makes it sound like TS is taking over frontend development. It is reported that **80% of developers admit they would like to use or learn TypeScript in their next project**.
Although it has it’s shortcomings, TS code is easier to understand, faster to implement, it produces less bugs and requires less boilerplate. Want to refactor your React app to work with TS? Go for it. Want to start gradually? Use tools like Bit to gradually refactor components in your app to TS and use the React-Typescript compiler to build them independently from your app. This way to can gradually upgrade your code one component at a time.
learn more:
Why TypeScript is the best way to write Front-end in 2019And why you should convince everybody to use it.
TypeScript at Lyft
TypeScript at Slack

5. Web components- Stencil, Svelte, Lit & friends!

So basically, this is the future. Why? because these pure web components are framework agnostic and can work without a framework or with any framework- spelling standardization. Because they are free from JS fatigue and are supported by modern browsers. Because their bundle size and consumption will be optimal, and VDOM rendering is mind-blowing.
These components provide Custom Element, a Javascript API that allows you to define a new kind of html tag, HTML templates to specify layouts, and of course the Shadow DOM which is component-specific by nature.
Prominent tools to know in this space are **Lit-html (and Lit-element), **StencilJS, **SvelteJS and of course Bit**, for reusable modular components which can be directly shared, consumed and developed anywhere.
When thinking of the future of our UI development, and of how principles of modularity, reusability, encapsulation, and standardization should look like in the era of components, web components are the answer. Learn more:

6. From component libraries to dynamic collections

Organize components in dynamic collections; reuse, compose, stay independent
The emergence of component-driven development gave birth to a verity of tools. One prominent tool is Bit, alongside it’s hosting platform Bit.dev.
Instead of working hard to build a cumbersome and highly-coupled component-library, use Bit to continuously isolate and export existing components into a dynamically reusable shared-collection.
Using Bit (GitHub) you can independently isolate, version, build, test and update UI components. It streamlines the process of isolating a component in an existing app, harvesting it to a remote collection, and using it anywhere. Every component can build, test, and render outside of any project. You can update a single component (and it’s dependants) and not the whole app.

In the bit.dev platform (or on your own server) your components can be remotely hosted and organized for different teams, so that every team can control the development of their own components. Every team can share and reuse components but keep their independence and control.
The platform also provides the all-in-one ecosystem for a shared components out-of-the-box: It auto-documents UI components, renders components in an interactive playground, and even provides a built-in registry to install components using npm/yarn. In addition, you can bit import components for modifications in any repository.

In the short run, this revolutionizes the process of sharing and composing components in a similar way to how Spotify/iTunes changed the process of previously sharing Music through static CD Music Albums. It’s a dynamic and modular solution that lets everyone share and use components together.
In the long run, Bit helps pave the way to micro-frontends. Why? Because it already lets you independently version, test, build and update parts of your UI application. In 2020 it will introduce independent deployments, which will finally allow different teams to own parts of your apps end-to-end: keep decoupled and simple codebases, let teams cautiously and continuously build and deploy incremental UI upgrades, and compose frontends together.

7. State management: Bye Bye Redux? (Not….)

Redux is a hard beast to kill. While the pains of globally managing states in your app are becoming more clear as frontend becomes more modular, the sheer usefulness of Redux makes it a go-to solution for many teams.
So will we say bye-bye to Redux in 2020? Probably not entirely 😄
However, the uprising of new features within frameworks that handle states (React hooks, Context-API etc) are painting the way to a future without a global store. Tools like Mobx, which only a year ago were rather scarcely adopted, are becoming more popular every day thanks to their component-oriented and scalable nature. You can explore more alternatives here.
Read: *Making Sense of React Hooks* — by Dan Abramov

8. ESM CDN

ES Modules is the standard for working with modules in the browser, standardized by ECMAScript. Using ES modules you can easily encapsulate functionalities into modules which can be consumed via CDN etc. With the release of Firefox 60, all major browsers will support ES modules, and the Node mteam is working on adding ES module support to Node.js. Also, ES module integration for WebAssembly is coming in the next few years. Just imagine modular Bit UI components composed in your app via CDN…

9. Progressive web apps. Still growing.

Progressive web applications take advantage of the latest technologies to combine the best of web and mobile apps. Think of it as a website built using web technologies but that acts and feels like an app. Recent advancements in the browser and in the availability of service workers and in the Cache and Push APIs have enabled web developers to allow users to install web apps to their home screen, receive push notifications and even work offline.
Since PWAs provide an intimate user experience and because all network requests can be intercepted through service workers, it is imperative that the app be hosted over HTTPS to prevent man-in-the-middle attacks, which also spells better security. Here’s a great talk by Facebook developer Omer Goldberg outlining best practices for PWAs.

10. Designer-developer integrations

With the uprise of component-driven design systems to enable a consistent UI across products and teams, new tools have emerged to bridge the gap between designers and developers. This is no simple task however; While code itself is really the only source of truth (this is what your user really gets), most tools try to bridge the gap from the designer’s end. In this category you can find Framer, Figma, Invision DSM and more.
From the developer’s end you can see how platforms like Bit.dev, which host your next-gen component library and helps create adoption for shared components. The platform provides rendered visualization for your actual source-code so that designers can collaborate wit developers and create discussions over the source-code itself, in a visual way.
Another promising idea to take note of is design-tokens. Placing tokens in your code through which designers can really control simple styling aspects (e.g. colors) directly through external collaboration tools. Integrated with platforms like Bit.dev, this can create a tighter workflow than ever before.

UI Component Design System: A Developer’s Guide

JoniSar ・ Oct 23 '19 ・ 10 min read

#design #ui #javascript #frontend

UI Design System and Component Library: Where Things Break | by Jonathan Saring | codeburst

Jonathan Saring ・ Aug 22, 2019 ・ 8 min read
Medium

7 Tools for Building Your Design System in 2020 | by Jonathan Saring | Bits and Pieces

Jonathan Saring ・ Dec 4, 2019 ・ 11 min read
Medium

11. Web assembly — into the future?

Web assembly brings language diversity into web development to cover gaps created by JavaScript. It is defined as a “a binary instruction format for a stack-based virtual machine. Wasm is designed as a portable target for compilation of high-level languages like C/C++/Rust, enabling deployment on the web for client and server applications”.
In his post, Eric Elliott elegantly outlines the concept’s benefits:

An improvement to JavaScript: Implement your performance critical stuff in wasm and import it like a standard JavaScript module.
A new language: WebAssembly code defines an AST (Abstract Syntax Tree) represented in a binary format. You can author and debug in a text format so it’s readable.
A browser improvement: Browsers will understand the binary format, which means we’ll be able to compile binary bundles that compress smaller than the text JavaScript we use today. Smaller payloads mean faster delivery. Depending on compile-time optimization opportunities, WebAssembly bundles may run faster than JavaScript, too!
A Compile Target: A way for other languages to get first-class binary support across the entire web platform stack To learn more about this concept, why it’s useful, where it will be used and why it’s not here yet, I suggest this great post and this great video. Why We Need WebAssembly: An Interview with Brendan Eich *Brendan Eich & Eric Elliott Discuss WebAssembly Details*medium.com

Learn more

13 Top React Component Libraries for 2020 | by Fernando Doglio | Bits and Pieces

Fernando Doglio ・ Jun 8, 2020 ・ 13 min read
Medium

11 Top Angular Developer Tools for 2020 | by Giancarlo Buomprisco | Bits and Pieces

Giancarlo Buomprisco ・ Dec 24, 2019 ・ 8 min read
Medium

11 Top VueJS Developer Tools for 2020 | by Shanika Wickramasinghe | Bits and Pieces

Shanika Wickramasinghe ・ Dec 24, 2019 ・ 8 min read
blog.bitsrc.io

Reuse React Components Between Apps Like a Pro

joni sar — Wed, 20 Nov 2019 13:58:52 +0000

One of the reasons we love React is the truly reusable nature of its components, even compared to other frameworks. Reusing components means you can save time writing the same code, prevent bugs and mistakes, and keep your UI consistent for users across your different applications.

But, reusing React between apps components can be harder than it sounds. In the past, this process involved splitting repositories, boiler-plating packages, configuring builds, refactoring our apps and more.

In this post, I'll show how to use Bit (GitHub) in order make this process much easier, saving around 90% of the work. Also, it will allow you to gradually collect existing components from your apps into a reusable collection for your team to share - like these ones.

In this short tutorial, we'll learn how to:

Quickly setup a Bit workspace
Track and isolate components in your app
Define a zero-config React compiler
Version and export components from your app
Use the components in a new app

Bonus: Leveraging Bit to modify the component from the consuming app (yes), and syncing the changes between the two apps.

Quick Setup

So for this tutorial, we've prepared an example React App on GitHub you can clone.

$ git clone https://github.com/teambit/bit-react-tutorial
$ cd bit-react-tutorial
$ yarn

Now, go ahead and install Bit.

$ npm install bit-bin -g

Next, we'll need a remote collection to host the shared components. You can set up on your own server, but let's use Bit's free component hub instead. This way our collection can be visualized and shared with our team, which is very useful.

quickly head over to bit.dev and create a free collection. It should take less than a minute.

Now return to your terminal and run bit login to connect your local workspace with the remote collection, where we'll export our components.

$ bit login

Cool. Now return to the project you've cloned and init a Bit workspace:

$ bit init --package-manager yarn
successfully initialized a bit workspace.

That's it. Next, let's track and isolate a reusable component from the app.

Track and isolate reusable components

Bit lets you track components in your app, and isolates them for reuse, including automatically defining all dependencies. You can track multiple components using a glob pattern (src/components/*) or specify a path for a specific component. In this example, we'll use the later.

Let's use the bit add command to track the "product list" component in the app. We'll track it with the ID 'product-list'. Here's an example of how it will look like as a shared component in bit.dev.

$ bit add src/components/product-list
tracking component product-list:
added src/components/product-list/index.js
added src/components/product-list/product-list.css
added src/components/product-list/products.js

Let's run a quick bit status to learn that Bit successfully tracked all the files of the component. You can use this command at any stage to learn more, it's quite useful!

$ bit status
new components
(use "bit tag --all [version]" to lock a version with all your changes)

     > product-list ... ok

Define a zero-config reusable React compiler

To make sure the component can run outside of the project, we'll tell Bit to define a reusable React compiler for it. This is part of how Bit isolates components for reuse, while saving you the work of having to define a build step for every component.

Let's import the React compiler into your project's workspace. You can find more compiler here in this collection, including react-typescript.

$ bit import bit.envs/compilers/react --compiler
the following component environments were installed
- bit.envs/react@0.1.3

Right now the component might consume dependencies from your project. Bit's build is taking place in an isolated environment to make sure the process will also succeed on the cloud or in any other project. To build your component, run this command inside your react project:

$ bit build

Version and export reusable components

Now let's export the component to your collection. As you see, you don't need to split your repos or refactor your app.

First, let's tag a version for the component. Bit lets you version and export individual components, and as it nows about each component's dependants, you can later bump versions for single component and all its dependants at once.

$ bit tag --all 0.0.1
1 component(s) tagged
(use "bit export [collection]" to push these components to a remote")
(use "bit untag" to unstage versions)

new components
(first version for components)
     > product-list@0.0.1

You can run a quick 'bit status' to verify if you like, and then export it to your collection:

$ bit export <username>.<collection-name>
exported 1 components to <username>.<collection-name>

Now head over to your bit.dev collection and see how it looks!
You can save a visual example for your component, so you and your team can easily discover, try and use this component later on.

Install components in a new app

Create a new React app using create-create-app (or your own).

$ npx create-react-app my-new-app

Move over to the new app you created.
Install the component from bit.dev:

$ yarn add @bit/<username>.<collection-name>.product-list --save

That's it! you can now use the component in your new app!

If you want to use npm, run npm install once after the project is created so a package-lock.json will be created and npm will organize dependencies correctly.

Modify components from the consuming app

Now let's use Bit to import the component's source-code from bit.dev and make some changes, right from the new app.

First, init a Bit workspace for the new project:

$ bit init

And import the component

$ bit import <username>.<collection-name>/product-list
successfully imported one component

Here is what happened:

A new top-level components folder is created that includes the code of the component, with its compiled code and node_modules (in this case the node_modules are empty, as all of your node_modules are peer dependencies and are taken from the root project.

The .bitmap file was modified to include the reference to the component
The package.json file is modified to point to the files rather than the remote package. Your package.json now displays:

"@bit/<username>.<collection-name>.product-list": "file:./components/product-list"

Start your application to make sure it still works. As you'll see, no changes are required: Bit takes care of everything.

Then, just go ahead and make changes to the code anyway you like!
Here's an example.

Now run a quick bit status to see that the code is changed. Since Bit tracks the source-code itself (via a Git extension), it "knows" that the component is modified.

$ bit status
modified components
(use "bit tag --all [version]" to lock a version with all your changes)
(use "bit diff" to compare changes)

     > product-list ... ok

Now tag a version and export the component back to bit.dev:

$ bit tag product-list
1 component(s) tagged
(use "bit export [collection]" to push these components to a remote")
(use "bit untag" to unstage versions)

changed components
(components that got a version bump)
     > <username>.<collection-name>/product-list@0.0.2

and...

$ bit export <username>.<collection-name>
exported 1 components to <username>.<collection-name>

You can now see the updated version with the changes in bit.dev!

Update changes in the first app (checkout)

Switch back to the react-tutorial app you cloned and exported the component from, and check for updates:

$ bit import
successfully imported one component
- updated <username>.<collection-name>/product-list new versions: 0.0.2

Run bit status to see that an update is availabe for product-list:

$ bit status
pending updates
(use "bit checkout [version] [component_id]" to merge changes)
(use "bit diff [component_id] [new_version]" to compare changes)
(use "bit log [component_id]" to list all available versions)

    > <username>.react-tutorial/product-list current: 0.0.1 latest: 0.0.2

Merge the changes done to the component to your project. The structure of the command is bit checkout <version> <component>. So you run:

$ bit checkout 0.0.2 product-list
successfully switched <username>.react-tutorial/product-list to version 0.0.2
updated src/app/product-list/product-list.component.css
updated src/app/product-list/product-list.component.html
updated src/app/product-list/product-list.component.ts
updated src/app/product-list/product-list.module.ts
updated src/app/product-list/products.ts

Bit performs a git merge. The code from the updated component is now merged into your code.

Run the application again to see it is working properly with the updated component:

$ yarn start

That's it. A change was moved between the two projects. Your application is running with an updated component.

Happy coding!

Conclusion

By being able to more easily reuse React components between applications you can speed your development velocity with React, keep a consistent UI, prevent bugs and mistakes and better collaborate as a team over a collection of shared components. It's also a useful way to create a reusable UI component library for your team in a gradual way without having to stop everything or lose focus.

Feel free to try it out yourself, explore the project in GitHub. Happy coding!

UI Component Design System: A Developer’s Guide

joni sar — Wed, 23 Oct 2019 12:08:08 +0000

Component design systems let teams collaborate to introduce a consistent user visual and functional experience across different products and applications.

On the designer's side, a predefined style guide and set of reusable master components enable consistent design and brand presented to users across all different instances (products etc) built by the organization. This is why great teams like Uber, Airbnb, Shopify and many others work so hard to build it.

On the developer's side, a reusable set of components helps to standardize front-end development across different projects, save time building new apps, reduce maintenance overhead and provide easier onboarding for new team members.

Most importantly, on the user's side, a successful component design system means less confusion, better navigation of your products, warm and fuzzy brand-familiarity feeling and better overall satisfaction and happiness. For your business, this means better results.

But, building a successful design system can be trickier than you might think. Bridging the gap between designers and developers is no simple task, both in the process of building your system as well as over time. In this post, we’ll walk-through the fundamentals of successfully building a component design system, using it across projects and products, and growing a thriving and collaborative component ecosystem within the organization, that brings everyone together. We’ll also introduce some shiny modern tools that can help you build it. Please feel free to comment below, ask anything, or share from your own experience!

Bridging the gap between design and development through components

When building your system you will face several challenges. The first, is achieving true collaboration between designers, developers and everyone else (product, marketing etc). This is hard. Designers use tools like Photoshop, Sketch etc which are built for generating “flat” visual assets that don’t translate into real code developers will use. Tools like Framer aim to bridge this gap on the designer’s side.

Developers work with Git (and GitHub) and use different languages and technologies (such as component-based frameworks: React, Vue etc) and have to translate the design into code as the source of truth of the design’s implementation. Tools like Bit turn real components written in your codebase into a visual and collaborative design system (examples), making it easy to reuse and update components across apps, and visualizing them for designers.

Modern components are the key to bridging this gap. They function as both visual UI design elements as well as encapsulated and reusable functional units that implement UX functionality that can be used and standardized across different projects in your organization’s codebase.

To bridge the gap, you’d have to let designers and other non-coding stakeholders collaborate over the source of truth, which is code. You can use Bit or similar tools to bridge this gap and build a collaborative component economy where developers can easily build, distribute and adopt components while designers and everyone else can collaborate to build and align the design implementation of components across applications.

Choosing your stack and tools

The choice of technologies and tools is a major key in the success of your design system. We’ll try to narrow it down to a few key choices you’d have to make along the way:

Framework or no framework?

Modern frameworks like React, Vue and Angular provide an environment where you can build components and build applications with components. Whether you choose a view library or a full-blown MVC, you can start building your components with a mature and extensive toolchain and community behind you. However, such frameworks might not be future proof, and can limit the reuse and standardization of components on different platforms, stacks and use-cases.

Another way to go is framework-agnostic web components. Custom components and widgets that build on the Web Component standards, will work across modern browsers, and can be used with any JavaScript library or framework that works with HTML.

This means more reuse, better stability, abstraction and standardization, less work and pretty much everything else that comes with better modularity. While many people are sitting around waiting on projects like web-assembly, in the past year we see new tools and techs rise to bring the future today.

The core concept of a standardized component system that work everywhere goes naturally well with the core concept of web components, so don’t be quick to overlook it despite the less mature ecosystem existing around it today.

Component library or no library?

Building a component library is basically a way to reduce the overhead that comes with maintaining multiple repositories for multiple components. Instead, you group multiple components into one repository and distribute it like a multi-song CD music album.

The tradeoff? App developers (component consumers) can’t use, update or modify individual components they need. They are struggling with the idea of coupling the development of their products to that of the library. Component collaboration platforms like Bit can greatly mitigate this pain, by sharing your library as a “playlist” like system of components that people can easily discover, use, update and collaborate-over across projects and teams. Every developer can share, find, use and update components right from their projects.

Most larger organization implement a library (examples) to consolidate the development of their components, consolidate all development workflows around the project and control changes. In today's ecosystem, it’s hard to scale component-based design systems without libraries mostly due to development workflows (PRs, issues, deployment etc). In the future, we might see more democratized component economies where everyone can freely share and collaborate.

When building your library you effectively build a multi-component monorepo. Open-source tools like bit-cli can help you isolate each component, automatically define all its dependencies and environments, test and build it in isolation, and share it as a standalone reusable unit. It also lets app-developers import and suggest updates to components right from their own projects, to increase the adoption of shared components.

Component discoverability and visualization

When building and distributing components you must create a way for other developers, and for non-developers collaborating with you, to discover and learn exactly which components you have, what they look like, how they behave in different states and how to use them.

If working with tools like Bit you get this out of the box, as all your components are visualized in a design system made from your actual components. Developers can use and develop components from the same place designers, marketers and product managers can view and monitor the components.

If not, you can create your own documentation portal or leverage tools like Storybook to organize the visual documentation of the components you develop in a visual way. Either way, without making components visually discoverable it will be hard to achieve true reusability and collaboration over components.

Building your design system: top-down vs. bottom-up

There are two ways to build a component design system. Choosing the right one is mostly based on who your are and what you need to achieve.

Design first, then implement reusable components

The first, mostly used by larger organizations that need to standardize UX/UI and development across multiple teams and products, is to design components first and then make sure this design is implemented as components (often building a library) and used everywhere.

A super over-simplified structure of this workflow looks like this:

Build a visual language and design components
Implement components in a git-based project in GitHub/Gitlab etc
Distribute using component-platforms like Bit and/or to package managers
Standardize instances of components across projects and apps
Collaboratively monitor, update and evolve components (using Bit or other tools)

Code first, then collect components into a design system

The second, often used by smaller and younger teams or startups, is to build-first and then collect existing components from your apps into one system, align the design, and keep going from there. This approach saves the time consumed by the design-system project, time which startups often can’t afford to spend. bit-cli introduces the ability to virtually isolate components from existing repositories, building and exporting each of them individually as a standalone reusable unit, and collect them into one visual system made of your real code. So, you can probably use it to collect your components into a system in a few hours without having to refactor, split of configure anything, which is a quick way to do it today.

This workflow looks like this:

Isolate and collect components already existing in your apps into one collection (Bit is useful)
Bring in designers and other stakeholders to learn what you have and introduce your visual language into this collection
Update components across projects to align to your new collection
Use these components to build more products and apps
Collaboratively monitor, update and evolve components (using Bit or other tools)

Design systems and atomic design

Through the comparison of components and their composition to atoms, molecules, and organisms, we can think of the design of our UI as a composition of self-containing modules put together.

Atomic Design helps you create and maintain robust design systems, allowing you to roll out higher quality, more consistent UIs faster than ever before.

Learn more in this post: Atomic Design and UI Components: Theory to Practice.

Collaboratively manage and update components

Over time your design system is a living creature that changes as the environment does. Design might changes, and so should the components. Components might change to fit new products, and so should the design. So, you must think of this process as a 2-way collaborative workflow.

Controlling components changes across projects

When a component is used in 2 or more projects, sooner or later you will have to change it. So, you should be able to update a component from one project to another, consolidate code-changes and update all dependent components impacted by the change.

If you are using Bit this is fairly easy. You can import a component into any project, make changes, and update them as a new version. Since Bit “knows” exactly which other components depend on this component in different projects, you can update all of them at once and learn that nothing breaks before updating. Since Bit extends Git, you can merge the changes across projects just like you do in a single repository. All the changes will be visually availbe to view and monitor in your shared bit.dev component collection.

If not, things become trickier, and your component infrastructure team will have to enforce updates to their libraries for all projects using these libraries, which impairs flexibility, creates friction and makes it hard to achieve true standardization through adoption. Yet, this is harder but not impossible, here is how Walmart Labs do it. You will also have to make sure that both changes to code and design are aligned in both your design tools and library docs wikis, to avoid misunderstandings and mistakes.

Grow a component ecosystem in your organization

Building a design system is really about building a growing component ecosystem in your organization. This means that managing components isn’t a one-way street; you have to include the app-builders (component consumers) in this new economy, so that the components you build will actually use them in their applications and products.

Share components that people can easily find and use. Let them collaborate and make it easy and fun to do so. Don’t force developers to install heavy libraries or dive-in too deep into your library just to make a small pull-request. Don’t make it hard for designers to learn exactly which components changes over time and make it easy for them to collaborate in the process.

Your component design system is a living and breathing organism that grows and evolves over time. If you try to enforce it on your organization, it might die. Instead, prefer legalization and democratization of components, their development and their design. Regulate this process to achieve standardization, but don’t block or impair adoption- at all costs. Bit is probably the most prominent power-tool here too, but please do share more if you know them.

Conclusion

Design systems help to create consistency in the visual and functional experience you give you users, while forming your brand across different products and applications. Modern components, with or without a framework, let you implement this system as a living set of building blocks that can and should be shared across projects to standardize and speed development.

As designers and developers use different tools, it’s critical to bring them together over a single source of truth, which is really your code since this is what your users really experience. A democratized and collaborative process between developers, designers, products, marketers and everyone else is the only way to grow a thriving and sustainable component ecosystem that breathes life into your design system.

Modern tools built for this purpose, such as Bit and others (FramerX and BuilderX are also interesting on the designer’s end) can be used to build, distribute and collaborate over components to turn your design system into a consistent and positive user experience everywhere, and to manage and collaborate over components across teams within the organization.

Thanks for reading!

DEV Community: joni sar

Apache Iceberg Query Performance: A Practical Guide

How Iceberg scan planning works

Why tables degrade

Two paths to query performance

Path 1: Intelligent continuous optimization

Query-aware data layout

Compaction at streaming speed

Coordinated maintenance pipeline

Event-driven triggers

Multi-engine query routing and workload optimization

Observability and self-improvement

What setup looks like

Path 2: The manual approach

Diagnosing the bottleneck

Sort order configuration

File consolidation (compaction)

Preventing small files at write time

Bloom filters for point lookups

Manifest consolidation

Snapshot expiration and orphan cleanup

Delete file cleanup (MoR tables)

Partition strategy

Puffin statistics for join planning

Parquet-level tuning

Copy-on-Write vs. Merge-on-Read

The correct maintenance sequence

Performance troubleshooting checklist

Choosing your path

Summary

AI Data Lake and Lakehouse

Table of Contents

Automation Is Not Intelligence

The AI Control Plane

How AI Transforms Each Layer of the Lakehouse

AI-Driven Compaction: Learning How Data Is Read

Query Pattern Analysis

From Analysis to Action

Adaptive Sort Evolution

Speed as an AI Enabler

AI-Driven Maintenance: Predicting What Breaks

Health Prediction, Not Health Reaction

Intelligent Sequencing

Operation Outcome Learning

AI-Driven Query Routing: Matching Queries to Engines

The Three-Router AI Stack

Data-Quality-Aware Routing

AI-Driven Observability: From Dashboards to Diagnosis

Cross-Signal Correlation

Severity-Ranked Insights

Predictive Degradation

AI-Driven Governance: Policies That Adapt

Intelligent Policy Recommendations

Policy Inheritance with Workload Awareness

The Closed Loop: How Every Query Makes the Lake Smarter

AI Agents as Lake Citizens

The Infrastructure Problem

The LakeOps MCP: Agent-Native Connectivity

Intelligent Guardrails

AI Solving the AI Problem

AI-Native vs. AI-Washed: How to Tell the Difference

The Compounding Effect

Where This Is Heading

Summary

Managed Data Lake: A Guide for 2026

In this article

What "managed" actually means in 2026

Why data lakes degrade — the mechanics of silent failure

Strategy 1: Smart automation with a control plane

Observability and health classification

Intelligent compaction

Snapshot, orphan, and manifest lifecycle

Policies, routing, and AI readiness

The compound effect

Strategy 2: Manual management with open-source tooling

The manual stack

Compaction by hand

Snapshot expiration, orphan cleanup, and manifests by hand

Health monitoring and the safe sequence

Where manual management breaks down