Is Your Data Lake Actually A Landfill?

Why I chose this topic: I spent three weeks debugging a "simple" read query that took four minutes to return ten rows because the metadata layer had ballooned to 200,000 snapshots. I’m writing this so you don’t have to explain to your CTO why the data platform is hemorrhaging AWS credits.

You ship the job. It passes CI. Your dbt tests are green, and the data lands in S3 right on time. Then, three weeks later, your Spark jobs start failing with java.lang.OutOfMemoryError during metadata initialization, and your query latency looks like a jagged mountain range.

You aren’t experiencing a "data growth" problem. You are experiencing a "neglect" problem. Apache Iceberg is elegant, but it isn’t magic. If you treat it like a traditional Hive table—set and forget—you are effectively building a digital landfill.

Every time you commit a transaction, Iceberg creates a new snapshot. Every time you overwrite a partition, you leave behind orphaned files in your object storage. Left unchecked, the metadata layer becomes so heavy that your catalog service will choke, and your cloud provider will send you a bill that makes you wish you’d stayed in the finance sector.

The real problem

The problem isn't the data volume itself; it’s the fragmentation and the metadata overhead. When you perform frequent streaming writes or micro-batching, you create hundreds of tiny files. These files are the death of performance. Your query engine’s planner has to do more work just to resolve the table state than it does to actually scan the data.

Furthermore, Iceberg doesn't automatically delete the data you tell it to "remove." It marks it as expired in the metadata. If you don't prune those snapshots and physically delete the files, you are paying for storage that no longer exists in your current view of reality. It’s a classic "zombie file" scenario.

Photo by Anton Acosta on Unsplash

Step: Compacting the small file disaster

Small files are the primary reason your S3 LIST calls take forever. Use Spark’s rewriteDataFiles procedure. Don't run this on every single commit unless you want to crash your cluster. Schedule it.

I prefer the bin-pack strategy for general purpose, but use sort if you have high-cardinality filters in your WHERE clauses.

CALL catalog.system.rewrite_data_files(
  table => 'db.my_table',
  strategy => 'sort',
  sort_order => 'event_timestamp DESC',
  options => map(
    'min-input-files', '5',
    'max-concurrent-file-group-rewrites', '10'
  )
)

If you’re running this on EMR or Databricks, monitor the rewrite-data-files job memory. If it fails with OOM, increase your executor memory; don't just reduce the number of files. You want fewer, larger files (aim for 128MB to 512MB), not more, smaller ones.

Step: Expiring snapshots with a death clock

Metadata accumulates linearly. If you are doing 50 writes a day, you have 18,000+ snapshots a year. Your catalog (Glue, Nessie, or Postgres) will eventually hit a latency wall trying to load the table state.

You need a strict retention policy. I typically default to a 7-day retention period. If you need to "time travel" further back than a week, you’re doing audit logs wrong—keep those in a separate, immutable table.

CALL catalog.system.expire_snapshots(
  table => 'db.my_table',
  older_than => timestamp '2023-10-20 00:00:00.000',
  retain_last => 5
)

Note the retain_last => 5. This ensures that even if you accidentally drop the table or run an aggressive expiry, you always have a safety net of the most recent commits. Never set this to zero unless you are decommissioning the table.

Step: Cleaning up the orphans

expire_snapshots removes the reference, but the physical files might still exist in S3 if you don't run the cleanup process. If you have "orphaned" files—files that exist in the storage layer but aren't in the metadata—you’re burning money.

Use remove_orphan_files. Be careful with the older_than parameter. If you run this while a long-running read query is still scanning the table, you might delete files that the reader is currently trying to access, leading to a FileNotFoundException.

CALL catalog.system.remove_orphan_files(
  table => 'db.my_table',
  older_than => current_timestamp - interval '3' days
)

I always leave a 72-hour buffer here. It’s safer to pay for three days of ghost storage than to have a production job fail because a partition was mid-read when the janitor job kicked in.

Photo by Annie Spratt on Unsplash

Lessons learned from production

• Glue Catalog Throttling: If you are using AWS Glue as your Iceberg catalog, frequent maintenance triggers will hit the Glue API limits. Increase your Glue throughput quotas before you scale your maintenance jobs, or expect constant 400/500 errors.
• The write.format.default trap: If you are migrating from Parquet to Iceberg, ensure your maintenance jobs aren't silently converting back to Parquet via session defaults. Always verify the output format in your DESCRIBE DETAIL output.
• Maintenance shouldn't be "on-write": Everyone wants to set write.metadata.delete-after-commit.enabled = true. Don't. It makes your ingestion jobs brittle. Decouple maintenance into a separate, scheduled Airflow DAG.
• Monitoring is non-negotiable: If your maintenance job fails, it doesn't alert you by default unless you hook it into your logging stack. Pipe the Spark logs from these maintenance procedures into Datadog/CloudWatch. A silent failure here is a ticking time bomb for your storage costs.
• Know your partition evolution: If you change your partitioning scheme, your compaction strategy must change. Compacting across old and new partition specs can be computationally expensive. Read the docs on rewrite_position_delete_files if you’re doing heavy deletions or updates.

Conclusion

Iceberg is a powerful tool, but it is not a "set-and-forget" database. It is a complex distributed file system management task disguised as a SQL table. If you don't treat your maintenance jobs with the same rigor as your ingestion pipelines, you’re not an engineer—you’re just a temporary occupant waiting for the system to collapse under its own weight.

Try it: Run SELECT count(*) FROM "db.my_table.snapshots" in your query engine today. If the number is in the thousands, stop what you are doing, write a cleanup script, and schedule it immediately. Your cloud bill will thank you.

Cover photo by Bodega on Unsplash.