Most data leaders sell "Open Table Formats" like they’re a magic bullet for vendor independence. They aren't. They’re a way to ensure your data stays usable when you inevitably decide that BigQuery’s STORAGE_BILLING_MODEL is eating your entire cloud budget. I’ve spent the last 18 months moving petabyte-scale healthcare datasets onto BigLake and Iceberg. It works, but Google didn't build it to make leaving easy; they built it to keep the friction of moving data in GCP lower than the friction of moving it out.
If you’re expecting a plug-and-play experience, stop. You are about to deal with metadata locking, Hive-metastore nightmares, and the specific joy of debugging 403 Forbidden errors that actually mean "your service account doesn't have the right Storage Object Admin role on the underlying GCS bucket." This guide is for the engineer who wants the architecture to be actually maintainable, not just "production-ready" on a slide deck.
1. Stop treating BigLake like a standard BigQuery table
BigQuery tables are black boxes. BigLake tables are external pointers. The biggest mistake I see? Treating them as if they have the same ingestion SLAs. When you run a MERGE statement on a native BQ table, Google manages the optimization. When you do it on Iceberg, you are responsible for the maintenance of the underlying Parquet files.
If you don't run CALL sys.rewrite_data_files and CALL sys.rewrite_manifests regularly, your query performance will degrade into a slow-motion car crash. You aren't just a data engineer here; you’re an amateur database administrator managing compaction intervals. Don't automate this via a simple cron job; hook it into your Airflow DAGs as a post-load quality gate.
-- Don't let your manifest files grow to the moon
CALL my_dataset.system.rewrite_data_files(
table => 'my_project.my_dataset.my_iceberg_table',
where => 'date_partition = "2023-10-01"'
);
Photo by Damien Schnorhk on Unsplash
2. Partitioning is your only defense against egress costs
In the traditional BigQuery world, we got lazy with clustering. In Iceberg, if you aren't partition-pruning correctly, you are scanning your entire GCS bucket. And guess what? Google charges for data processed. If you scan 10TB to find one patient record, that’s on you.
Always, and I mean always, use hidden partitioning. Don't create a date_string column in your source data just to satisfy the partition requirement. Use Iceberg’s ability to derive partitions from timestamps. If you’re partitioning by day, use days(timestamp_col). If you don't, you'll be rewriting your entire metadata layer when you realize you need to change your partition strategy.
3. The IAM dance is a multi-act tragedy
BigLake requires the BigQuery Connection resource to act as an intermediary between the engine and the storage. You’ll need a Google-managed service account for that connection. The failure mode here is subtle: users will have BigQuery Data Viewer permissions, but they won't have Storage Object Viewer on the GCS bucket.
The error message in the console will tell you "Access Denied," and it will lie to you about which permission is missing. Always check the service account tied to the connection resource first.
# The CLI command that saves you 4 hours of debugging
gcloud projects add-iam-policy-binding [PROJECT_ID] \
--member="serviceAccount:[CONNECTION_SERVICE_ACCOUNT]" \
--role="roles/storage.objectViewer"
4. Manifest snapshots are not backups
A common misconception is that Iceberg's metadata history serves as a disaster recovery strategy. It doesn't. If someone accidentally deletes the underlying Parquet files from GCS, your Iceberg metadata is just a list of pointers to ghosts.
You need to enable GCS Object Versioning on your buckets. Period. If you don't, and a pipeline goes rogue and deletes a partition, you have no recourse. Iceberg's expire_snapshots procedure is useful for storage cleanup, but keep at least 7 days of snapshots. If you set this too aggressively, you lose the ability to perform time-travel queries, which is the only reason you’re using Iceberg in the first place.
5. Schema evolution is a trap
Iceberg supports schema evolution, and BigLake respects it. That’s the theory. The reality is that if you rename a column in your Parquet files via a tool that doesn't respect the Iceberg manifest, you break the contract.
Never manually modify the Parquet files outside of the Iceberg engine. If you need to fix a data type or rename a column, use ALTER TABLE. If you go behind the engine's back, you'll encounter the "orphaned data file" problem where the metadata points to a column that no longer exists in the Parquet schema, leading to NullPointerException or generic engine failures that don't point to the root cause.
-- Keep the engine as the single source of truth for metadata
ALTER TABLE my_iceberg_table
RENAME COLUMN old_name TO new_name;
6. The "Hidden" cost of metadata storage
Every time you commit a transaction in Iceberg, you create a new manifest file. If you have a high-frequency ingestion pipeline (e.g., streaming small batches every minute), your metadata layer will explode.
I’ve seen metadata directories hit 50,000 files in three weeks. This slows down query planning significantly. You aren't just paying for data storage; you're paying for GCS read operations on every single metadata file during the EXPLAIN phase of your query. Micro-batching is the enemy of Iceberg performance. Buffer your data in memory or use a staging area before committing to the table.
7. Avoid the "BigQuery-only" mindset
If you chose Iceberg, it’s likely because you want to use Trino, Spark, or DuckDB on the same data. If you write your data using only BigQuery’s INSERT or MERGE statements, you might be creating files that are optimized for BigQuery but are absolute garbage for Trino.
Check your Parquet writer settings. Ensure you are using Snappy compression and reasonable row group sizes. If you write 1GB row groups, Spark/Trino will choke on memory when trying to read them. Stick to the 128MB to 256MB range. It’s boring, it’s standard, and it keeps your compute engines from crashing.
Conclusion
BigLake and Iceberg are powerful tools, but they shift the burden of performance tuning from Google’s black box to your own infrastructure. You get "openness," but you pay for it in complexity. You have to be the one to manage the metadata, the IAM policies, the file compaction, and the storage lifecycles.
It’s worth it if you’re tired of being held hostage by proprietary formats. It’s a disaster if you treat it like "just another table." Before you migrate that production workload, ask yourself: are you actually prepared to own the metadata layer, or are you just looking for a new way to break your pipelines?
Tags: #gcp #bigquery #iceberg #dataengineering
Cover photo by Mick Haupt on Unsplash.
Top comments (0)