If you work with data — or you're just getting started in the cloud — you've probably heard terms like Data Warehouse, Data Lake, and Data Lakehouse. This article explains those concepts from scratch, details what Lakehouse for Apache Iceberg on Google Cloud (formerly BigLake) and Apache Iceberg actually are, and shows how to make three different tools read and modify the same data at the same time — no mess, no duplicated data.
Everything below was validated for real on GCP: Spark committed through the REST catalog, Trino read and wrote the same table, BigQuery ran DML on it, and we captured a genuine OCC rejection (HTTP 409 ConflictException). Full logs are in the repo's evidence/ folder.
The Problem with Data Copies
In most companies, different teams use different tools:
- BI analysts use SQL inside BigQuery to build sales reports.
- Data scientists run Apache Spark jobs to train forecasting models.
- Infrastructure engineers use tools like Trino for fast ad-hoc queries.
In the traditional model, the BI team copied data into BigQuery. The data science team copied the same data to a Cloud Storage directory as Parquet.
That separation created three immediate problems:
- Double or triple storage bills: the same terabytes were stored and billed multiple times.
- Stale data: when an order was cancelled in the main system, one copy updated while the others kept the old value.
- No unified security: you had to configure who could see what inside each tool separately.
The Data Lakehouse concept was born to end this duplication. The idea is simple: keep a single copy of the files in cheap cloud storage (like Google Cloud Storage) and let any query engine (SQL, Spark, Python, Trino) work directly on it, under clear transaction rules.
Core Concepts
Before the code, let's pin down what each piece does:
1. Google Cloud Storage (GCS)
The cloud hard drive. It stores raw files (CSV, JSON, Parquet) durably and cheaply. In our project, it's where the physical data and metadata files live.
2. The Parquet Format
A way to save tables to files. Instead of storing row by row like CSV, Parquet organizes data by column and applies heavy compression — queries read less disk and answer faster.
3. Apache Iceberg
Parquet alone is just a loose file. Try updating a single row inside a Parquet file and the whole file must be rewritten. Let two processes write to the same folder at once and one can silently overwrite the other.
Apache Iceberg is an open metadata layer. It organizes Parquet files into a real table, providing:
- ACID transactions: either the whole write succeeds, or nothing changes. No half-written data.
- Time travel: query exactly how the table looked 3 hours ago or last week — Iceberg keeps a history of versions called snapshots.
- Schema evolution: adding or renaming columns doesn't break old queries.
4. What is Lakehouse for Apache Iceberg (formerly BigLake)?
Lakehouse for Apache Iceberg is the Google Cloud service — previously called BigLake — that breaks the barrier between file storage (Cloud Storage) and SQL analytics (BigQuery). If you find "BigLake" in older articles, APIs, or commands, it's the same product: Google renamed it to reflect that it is the platform's official open lakehouse foundation.
It offers two huge benefits:
- Native Iceberg REST Catalog: Iceberg defines a web communication standard (the REST Catalog) to register tables and control which version of the data is current. Google Cloud implemented that official catalog inside Lakehouse — the Lakehouse Iceberg REST catalog, a central hub providing read/write interoperability between BigQuery, Google Cloud Managed Service for Apache Spark, and Iceberg-compatible OSS engines (Spark, Trino, Flink). No data is trapped in a proprietary format.
-
Unified security via Workload Identity: instead of downloading risky password files (like Service Account
.jsonkeys), Lakehouse uses the cloud's own execution identity to authorize queries, keeping access permissions centralized.
Renaming (BigLake → Lakehouse): although the product name is now Lakehouse, technical identifiers still use
biglake— the API endpoint (biglake.googleapis.com), the CLI (gcloud biglake), table properties (gcp.biglake.*), the managed service account (blirc-*@gcp-sa-biglakerestcatalog...), the catalog type (CATALOG_TYPE_BIGLAKE), and the warehouse scheme (bl://). In this article we say "Lakehouse" in prose and keepbiglakeonly inside literal commands and names.
How Three Engines Talk to the Same Table
In our project, we configured a single retail orders table (orders_iceberg) in Cloud Storage. Three applications access it in a coordinated way:
Figure 1: Lakehouse architecture with Apache Iceberg and the Lakehouse Iceberg REST Catalog — processing engines (BigQuery, Spark, Trino) accessing data through a centralized catalog on Google Cloud Storage.
What each tool does:
-
Google BigQuery: runs analytical queries and mutation commands like
MERGE INTO,UPDATE, andDELETEdirectly on the table in GCS. - Serverless Spark (Dataproc): processes large data batches in the background without anyone provisioning servers.
- Trino: a distributed SQL engine used by engineering teams for fast queries without locking production tables.
How Concurrency Control (OCC) Works
When Spark and BigQuery try to change the same table at the same time, how do we stop one from erasing the other's work?
Iceberg doesn't lock the table with pessimistic locks (which would make queries wait in line). It uses Optimistic Concurrency Control (OCC).
The process works like this:
-
Initial read: BigQuery and Spark both read the current table version from the Lakehouse catalog — say, snapshot number
10. - Isolated work: BigQuery computes its changes and writes new files to Cloud Storage. Spark does the same with its batch. Neither knows what the other is doing.
-
First to arrive wins: BigQuery finishes first and asks Lakehouse: "Commit my changes on top of snapshot 10." Since the current version is still 10, the write is accepted. The table's official snapshot becomes
11. - Conflict detection: seconds later, Spark finishes and asks Lakehouse: "Commit my changes on top of snapshot 10." Lakehouse refuses: "The table already moved to version 11 — your starting point is stale."
-
Automatic reconciliation: Spark doesn't break the system. It downloads the new version 11, checks whether the files BigQuery touched collide with its own, and — since they don't — applies its changes, creating snapshot
12.
The sequence diagram below shows this race step by step:
Figure 2: The optimistic commit cycle — BigQuery wins the race (snapshot 11), Spark's commit on the stale base (snapshot 10) is rejected by the catalog, and Spark retries on top of snapshot 11, producing snapshot 12.
All of this happens with no human intervention and without corrupting a single record.
What does this look like under the hood? The REST commit
Before the code, a concept that confuses many people: what is Iceberg's "main"?
Think of an Iceberg table as a Git repository:
- Each snapshot is like a Git commit: an immutable version of the table (the list of Parquet files making up that state).
-
mainis the table's default branch: a simple pointer saying "the current official snapshot is this one." When you runSELECT, your engine asks the catalog "where doesmainpoint?" and reads that snapshot. - A commit is the operation that moves that pointer — exactly like
git pushmoves a repo'smain. And, like Git, Iceberg lets you create other branches and tags pointing to specific snapshots (great for auditing and time travel), butmainis always the "official" version everyone reads.
With that in mind, every engine talks to the catalog through the Iceberg REST API. Here's the full timeline of a write — notice the POST happens only once, at the end:
Figure 3: Timeline of a write — the engine fetches metadata with GET, moves Parquet files directly to/from GCS, and calls POST (the commit) only once at the end. If main moved in between, the catalog answers HTTP 409 and the engine retries on the new base.
Important details of that sequence:
-
Data never flows through the REST API. The
biglake.googleapis.comendpoint only carries metadata (lightweight JSON). The heavy Parquet flows straight between the engine and GCS — that's why the architecture scales. -
The
POSTis the commit: it carries two parts —requirements(safety conditions, like "only accept ifmainis still at snapshot X that I read in step 1") andupdates(the changes: new snapshot, new pointer). -
The catalog — not the engine — writes the final
metadata.json. In the REST protocol, the client never moves the pointer by itself — it requests the change and the server validates it, writes the metadata, and updatesmain. That's why arbitration is central and identical for every engine.
That's exactly how we provoked a real conflict on purpose — we sent a commit claiming the base was an already-stale snapshot:
// POST https://biglake.googleapis.com/iceberg/v1/restcatalog
// /v1/projects/{project}/catalogs/{catalog}/namespaces/{ns}/tables/{table}
{
"requirements": [
{
// RACE CONDITION GUARD: "only commit if the main branch pointer
// is still exactly at this snapshot" — the snapshot I read when
// I started working. If another engine committed first, main
// has moved and this number is stale → rejection.
"type": "assert-ref-snapshot-id",
"ref": "main", // the table's official branch
"snapshot-id": 4808015153575422177 // the snapshot I read at the start
}
],
"updates": [
{
// What it would request if the requirement passed: move main's
// pointer to the NEW snapshot described in this commit (with the
// Parquet files I already wrote to GCS in step 3).
// The catalog then materializes the corresponding metadata.json.
"action": "set-current-snapshot",
"snapshot-id": 9999999999
}
]
}
Lakehouse's answer was exactly the protection we expected — HTTP 409 ConflictException:
Requirement failed: branch main has changed:
expected id 4808015153575422177 != 4196013214950993553
In plain English: "you claimed main was at snapshot 4808..., but it's already at 4196... — another engine committed before you. Commit rejected." That's step 4 of the flow happening for real, on the live API (full evidence in evidence/49_occ_conflict_409.txt).
Notice that nothing was locked: the losing engine simply re-reads main (now at the new snapshot), redoes its work on that base, and tries to commit again. It's the equivalent of git pull --rebase and pushing again — but automatic, at table level.
Step by Step: Running the Project
Prerequisites
- Python 3 installed on your machine.
- Google Cloud SDK (
gcloud) installed and authenticated. - Docker installed (needed to run local Trino in Step 7).
Clone the repository: all the code, scripts, SQL, Mermaid diagrams, and evidence from this article are on GitHub:
git clone git@github.com:carlosrgomes/lakehouse-iceberg.git
cd lakehouse-iceberg
Step 1: Run the Local Unit Tests
You don't need to spend cloud money to understand the logic. We built a complete Iceberg REST catalog simulator that runs locally using only Python's standard library:
python3 -m unittest discover -s tests -p "test_*.py" -v
This runs 7 automated tests validating table creation, data insertion, MERGE INTO updates, and the correct rejection of conflicting writes.
Step 2: Run the Multithreaded Concurrency Simulation
To watch multiple engines racing for the same table in real time on your machine:
python3 -m src.concurrency_simulation
The script fires parallel processes simulating BigQuery, Spark, and Trino writing to the same simulated catalog, showing the safe progression of snapshots.
Step 3: Generate Synthetic Data
We built a data generator simulating a real e-commerce store with 5,000 orders (amounts, customers, categories, delivery statuses):
python3 scripts/generate_synthetic_data.py
It produces the local files scripts/synthetic_orders.csv and scripts/synthetic_orders.jsonl.
Step 4: Provision the Infrastructure on Google Cloud
When you're ready to run everything in the cloud, execute the provisioning script:
export PROJECT_ID="your-gcp-project"
export REGION="us-central1"
./scripts/deploy_gcp_resources.sh
The script safely performs the following steps:
- Enables the required APIs (
biglake.googleapis.com— the Lakehouse API — plusBigQuery,Storage, andDataproc). - Creates the Cloud Storage bucket (
lakehouse-iceberg-PROJECT_ID). - Creates the native Iceberg REST catalog in Lakehouse (
lakehouse-rest-catalog). - Creates a catalog connection in BigQuery using Workload Identity, generating no private
.jsonkey files. - Grants the connection permission to read and write directly on the GCS bucket.
- Configures environment variables for the Spark integration.
Creating the Iceberg table: after deployment, create the table manually via CLI (the table name comes from the JSON file):
gcloud biglake iceberg tables create \
--catalog="lakehouse-rest-catalog" \ # Lakehouse catalog created by the deploy
--namespace="retail_lakehouse" \ # logical "database" inside the catalog
--project="${PROJECT_ID}" \
--create-from-file="scripts/orders_iceberg_table.json" # schema + properties
The scripts/orders_iceberg_table.json file holds the table's schema definition and settings. The essential parts, commented:
{
"name": "orders_iceberg",
// WHERE THE PHYSICAL FILES LIVE: the table's root inside the bucket.
// This is where data/*.parquet and metadata/*.json|*.avro appear.
"location": "gs://lakehouse-iceberg-PROJECT_ID/iceberg-warehouse/retail_lakehouse/orders_iceberg",
"schema": {
"type": "struct",
"fields": [
// Stable column IDs let you rename fields without breaking reads
{ "id": 1, "name": "order_id", "type": "long", "required": true },
{ "id": 2, "name": "customer_id", "type": "string", "required": false },
{ "id": 3, "name": "amount", "type": "double", "required": false },
{ "id": 4, "name": "status", "type": "string", "required": false }
]
},
"partition-spec": { "fields": [] }, // no partitioning (small demo table)
"properties": {
"write.format.default": "parquet" // physical format of files in data/
}
}
Enable BigQuery DML (needed once, for INSERT/MERGE/UPDATE/DELETE):
gcloud biglake iceberg tables update orders_iceberg \
--catalog="lakehouse-rest-catalog" \
--namespace="retail_lakehouse" \
--project="${PROJECT_ID}" \
# Enables BigQuery writes on the catalog table (reads work without it)
--update-properties="gcp.biglake.bigquery-dml.enabled=true"
Step 5: Run a Batch Load on Serverless Spark (Dataproc)
To test Apache Spark writing to the same table managed by the Lakehouse Iceberg REST Catalog — with no manual servers — submit the job to Dataproc Serverless.
Required Iceberg dependencies: the job needs two jars on the classpath — iceberg-spark-runtime-3.5_2.12-1.10.2.jar (~45 MB) and iceberg-gcp-bundle-1.10.2.jar (~59 MB). They're kept in this repo's jars/ folder and staged to the lakehouse bucket before submitting:
# (optional) download from Maven Central if jars/ is empty
curl -L -o jars/iceberg-spark-runtime-3.5_2.12-1.10.2.jar \
"https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-spark-runtime-3.5_2.12/1.10.2/iceberg-spark-runtime-3.5_2.12-1.10.2.jar"
curl -L -o jars/iceberg-gcp-bundle-1.10.2.jar \
"https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-gcp-bundle/1.10.2/iceberg-gcp-bundle-1.10.2.jar"
# stage them into the lakehouse bucket
gcloud storage cp jars/iceberg-spark-runtime-3.5_2.12-1.10.2.jar jars/iceberg-gcp-bundle-1.10.2.jar \
"gs://lakehouse-iceberg-${PROJECT_ID}/jars/"
gcloud dataproc batches submit pyspark scripts/spark_iceberg_batch_job.py \
--region="us-central1" \
--project="${PROJECT_ID}" \
--version=2.2 \
--deps-bucket="gs://lakehouse-iceberg-${PROJECT_ID}" \
--jars="gs://lakehouse-iceberg-${PROJECT_ID}/jars/iceberg-spark-runtime-3.5_2.12-1.10.2.jar,gs://lakehouse-iceberg-${PROJECT_ID}/jars/iceberg-gcp-bundle-1.10.2.jar" \
-- "${PROJECT_ID}" "lakehouse-iceberg-${PROJECT_ID}" "lakehouse-rest-catalog"
Note: the job uses
iceberg-spark-runtime-3.5_2.12andiceberg-gcp-bundleversion 1.10.2 or newer —rest.auth.typeauthentication (GoogleAuthManager) only exists in Apache Iceberg 1.10+. Older versions (1.5.x, 1.9.x) fail withClassNotFoundExceptionor401. The Spark commit really happens in the catalog: it's an atomic snapshot arbitrated by Lakehouse's OCC, not a loose Parquet write.
The config that connects Spark to the catalog (commented)
This is the most important snippet in the project — every property answers a real error we hit during testing. File: scripts/spark_iceberg_batch_job.py.
warehouse_path = f"bl://projects/{project_id}/catalogs/{catalog_id}"
# ^-- NOT gs:// ! For CATALOG_TYPE_BIGLAKE catalogs, the "warehouse"
# is the catalog's logical address (bl://...), which supports
# tables spread across multiple buckets.
spark = SparkSession.builder \
.appName("IcebergManagedSparkBatchWriter") \
.config("spark.sql.defaultCatalog", "iceberg") \
.config("spark.sql.catalog.iceberg", "org.apache.iceberg.spark.SparkCatalog") \
.config("spark.sql.catalog.iceberg.type", "rest") \
# The catalog is REST — any engine speaking this protocol sees the table
.config("spark.sql.catalog.iceberg.uri",
"https://biglake.googleapis.com/iceberg/v1/restcatalog") \
# ^-- Lakehouse's Iceberg REST endpoint (NOT the control-plane API
# v1/projects/.../catalogs — that one is only for gcloud/admin)
.config("spark.sql.catalog.iceberg.warehouse", warehouse_path) \
.config("spark.sql.catalog.iceberg.header.x-goog-user-project", project_id) \
# ^-- required header: which project Google bills/quota-counts
# catalog calls against
.config("spark.sql.catalog.iceberg.rest.auth.type",
"org.apache.iceberg.gcp.auth.GoogleAuthManager") \
# ^-- native Google auth: uses Dataproc's ambient credentials
# (Workload Identity), no JSON key. Requires Iceberg 1.10+
.config("spark.sql.catalog.iceberg.oauth2-server-uri",
"https://biglake.googleapis.com/iceberg/v1/restcatalog/v1/oauth/tokens") \
# ^-- endpoint for the catalog's OAuth2 token exchange/refresh
.config("spark.sql.catalog.iceberg.io-impl",
"org.apache.iceberg.gcp.gcs.GCSFileIO") \
# ^-- who reads/writes the physical Parquet files in GCS
.config("spark.sql.catalog.iceberg.header.X-Iceberg-Access-Delegation",
"vended-credentials") \
# ^-- CREDENTIAL VENDING: the catalog issues short-lived,
# scoped tokens so Spark can only reach the table's files —
# the engine doesn't need broad bucket access
.config("spark.sql.catalog.iceberg.gcs.oauth2.refresh-credentials-endpoint",
"https://oauth2.googleapis.com/token") \
.config("spark.sql.extensions",
"org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") \
.getOrCreate()
And the write itself is trivial — all the complexity lives in the configuration:
new_orders = spark.createDataFrame(
[(901, "spark_batch_1", 450.0, "PROCESSED_BY_SPARK"),
(902, "spark_batch_2", 275.5, "PROCESSED_BY_SPARK")],
["order_id", "customer_id", "amount", "status"])
# .writeTo().append() = writes new Parquet to GCS + POSTs the commit to
# the catalog with assert-ref-snapshot-id. Stale base → 409 → retry.
new_orders.writeTo("iceberg.retail_lakehouse.orders_iceberg").append()
The job runs the PySpark load, processes the data, and writes results back through the catalog.
Step 6: Query and Update Data via BigQuery SQL
With the table created, BigQuery reaches the Lakehouse catalog directly using the project.catalog.namespace.table syntax — no external table needed:
SELECT
customer_id,
status,
COUNT(*) as total_orders,
ROUND(SUM(amount), 2) as total_revenue
FROM `your-project.lakehouse-rest-catalog.retail_lakehouse.orders_iceberg`
GROUP BY customer_id, status
ORDER BY total_revenue DESC;
And to change an order without rewriting files by hand:
Note: to run DML (INSERT/MERGE/UPDATE/DELETE) on the catalog table, enable the
gcp.biglake.bigquery-dml.enabled=truetable property (viagcloud biglake iceberg tables updateor a REST commit).
-- Atomic UPSERT: BigQuery writes the new Parquet files to GCS and the
-- catalog arbitrates the commit via OCC (rejects if another engine won).
MERGE INTO `your-project.lakehouse-rest-catalog.retail_lakehouse.orders_iceberg` T
-- ^-- project.catalog.namespace.table syntax: NOT an external table,
-- it's the Lakehouse catalog's Iceberg table directly
USING (
SELECT 101 AS order_id, 'cust_1' AS customer_id, 200.0 AS amount, 'SHIPPED' AS status
) S
ON T.order_id = S.order_id
WHEN MATCHED THEN
UPDATE SET amount = S.amount, status = S.status -- order exists → update
WHEN NOT MATCHED THEN
INSERT (order_id, customer_id, amount, status) -- missing → insert
VALUES (S.order_id, S.customer_id, S.amount, S.status);
Step 7: Run Trino (Local Docker and GCP Architecture)
Trino is a distributed (MPP) SQL engine that requires continuous, persistent communication between nodes for state discovery (/v1/info/state). For that reason, on production Google Cloud environments it runs on Google Kubernetes Engine (GKE) with StatefulSets, or locally via Docker:
cd docker
# Copy the environment variables example
cp .env.example .env
# Edit .env with your GCP values
# Start Trino
./start-trino.sh
The Trino connector pointing at the same catalog (docker/iceberg.properties), commented:
connector.name=iceberg # the "iceberg" catalog inside Trino
iceberg.catalog.type=rest # speaks the Iceberg REST protocol
iceberg.rest-catalog.uri=https://biglake.googleapis.com/iceberg/v1/restcatalog
iceberg.rest-catalog.warehouse=bl://projects/YOUR_PROJECT/catalogs/lakehouse-rest-catalog
iceberg.rest-catalog.security=GOOGLE # native Google auth (needs Trino ~460+; we use 483)
iceberg.rest-catalog.google-project-id=YOUR_PROJECT
iceberg.file-format=PARQUET
fs.gcs.enabled=true # native gs:// support
gcs.auth-type=APPLICATION_DEFAULT # uses ambient ADC — no JSON key
And in docker-compose.yml, the whole auth secret is just mounting your gcloud default credentials into the container:
services:
trino-coordinator:
image: trinodb/trino:483 # version with security=GOOGLE support
volumes:
- ./iceberg.properties:/etc/trino/catalog/iceberg.properties:ro
- ${HOME}/.config/gcloud/application_default_credentials.json:/gcp/adc.json:ro
environment:
- GOOGLE_APPLICATION_CREDENTIALS=/gcp/adc.json # ADC inside the container
After gcloud auth application-default login on the host, Trino reads and writes the same table:
-- inside the Trino CLI (docker exec -it trino-iceberg trino)
INSERT INTO iceberg.retail_lakehouse.orders_iceberg
VALUES (903, 'trino_adhoc_1', 99.99, 'ADDED_BY_TRINO');
-- atomic commit through the same REST catalog — shows up as
-- engine-name=trino in the snapshot history
Open the console in your browser at http://localhost:8088.
Tip: if the initial query list looks empty after logging in, clear the User filter field at the top of the screen to see all queries run on the cluster.
Step 8: Tearing Everything Down (Destroy Script)
To avoid unwanted charges after testing the lab, we built a script that safely and completely deletes every resource created:
./scripts/destroy_gcp_resources.sh
It cancels pending Spark executions, removes leftovers from previous Trino deployments (if any), deletes the Lakehouse namespaces and catalogs, removes the BigQuery datasets, and deletes the Cloud Storage bucket with all test files.
What We Actually Proved on GCP
This isn't just a local simulation. On the real barbero-gde project we ran the full cycle:
-
Spark committed 2 rows via the REST catalog (batch
spark-iceberg-catalog-06, SUCCEEDED) -
Trino read Spark's rows and appended its own (
903, trino_adhoc_1) -
BigQuery ran
INSERT,MERGE, andUPDATEon the same table — including updating Trino's row toamount=500.0 -
Snapshot history shows
engine-name=spark,trino,bigquery— three engines, one table, one copy of the data - A deliberately stale commit returned the real HTTP 409
ConflictExceptionfrom Lakehouse
Final table state, visible identically from all three engines:
101 cust_1 200.0 SHIPPED
102 cust_2 85.5 COMPLETED
901 spark_batch_1 450.0 PROCESSED_BY_SPARK
902 spark_batch_2 275.5 PROCESSED_BY_SPARK
903 trino_adhoc_1 500.0 ADDED_BY_TRINO
Best Practices and Security
- Zero passwords on disk: never download service-account JSON keys. Lakehouse uses native connections that automatically renew short-lived auth tokens.
- Small-file control: continuous writes can generate hundreds of small Parquet files. In corporate environments, run periodic Spark compaction routines to merge small files into 128–512 MB blocks.
- History and retention: configure policies to expire old snapshots after 7–14 days, keeping the metadata tree lean and queries fast.
This article was created for the Lakehouse/Iceberg Content Sprint 2026. Full source, scripts, and execution evidence: github.com/carlosrgomes/lakehouse-iceberg — if this helped you, a star is appreciated! #LakehouseIceberg
Top comments (0)