Apache XTable (was OneTable): Cross-Format Translation Between Iceberg, Delta & Hudi

apache xtable is the metadata-only translator that answers the awkward question every senior data engineer eventually gets asked in 2026 — "we have Snowflake queries, Databricks notebooks, Trino dashboards, and a legacy Hudi pipeline all pointing at the same S3 prefix; how do we not maintain three physical copies of the data?" — and it answers it by writing translated Iceberg, Delta, and Hudi metadata on top of one shared Parquet directory so every engine sees a table in the format it prefers, without ever copying a single byte of column data. The project began life as apache onetable inside Onehouse, was donated to the Apache Software Foundation in 2024, and now sits in the Incubator with a 1.0-track release cadence — a small, focused piece of glue software that the multi-engine lakehouse era has quietly made non-optional.

This guide is the senior-data-engineer walkthrough you wished existed the first time an interviewer asked "walk me through XTable's translation model — what actually gets written, what stays shared, and which format features fall off the cliff?" or "your team is on Databricks (Delta), the ML team is on Trino (Iceberg), and one legacy pipeline still writes Hudi — how do you land on one physical dataset without a migration project?" It walks through the four load-bearing angles interviewers probe — the three-format 2026 reality that forces the problem, the metadata-only architecture that avoids the copy tax, the CLI + YAML recipe for round-tripping between Iceberg, Delta, and Hudi, and the catalog integration story that connects translated metadata to Polaris / Glue / Unity Catalog / Snowflake Iceberg REST. Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the SQL practice library →, rehearse system design on the design practice library →, and sharpen the streaming axis with the streaming practice library →.

---

On this page

• Why cross-format translation matters in 2026
• XTable architecture — metadata-only translation
• Iceberg, Delta, and Hudi round-trip in practice
• Integration with catalogs and engines
• When to use XTable vs when to standardise
• Cheat sheet — XTable recipes
• Frequently asked questions
• Practice on PipeCode

---

1. Why cross-format translation matters in 2026

Three open table formats coexist — and no vendor stack reads all three natively

The one-sentence invariant: the 2026 lakehouse world runs three production-hardened open table formats — Iceberg (dominant, backed by Snowflake, AWS, Google, Netflix), Delta (strong on Databricks and increasingly cross-engine via Delta Universal Format), and Hudi (legacy in stream-first stacks, alive because of merge-on-read semantics) — and no single engine reads all three cleanly, so anyone with more than one query engine either duplicates data three times or reaches for apache xtable as the metadata-only bridge. The choice most teams face is not "which format wins" — all three keep shipping — but "how do we let Snowflake read Delta and Databricks read Iceberg without turning our S3 bucket into a triplicated storage bill."

The three-format reality — who reads what natively in 2026.

• Iceberg. Native readers include Snowflake, Trino, Presto, Spark, Flink, DuckDB, Dremio, StarRocks, ClickHouse. Snowflake and AWS Glue treat Iceberg as first-class. Databricks reads Iceberg via Uniform (a Databricks-side translator that emits Iceberg-shaped metadata alongside Delta).
• Delta. Native readers include Databricks (SQL, notebooks, Photon), Spark with the delta-spark package, Trino via the Delta connector, Flink via the delta-flink sink. Snowflake reads Delta only via external tables (limited operations). Delta Lake 3.0's Uniform emits Iceberg metadata as a Delta-native feature — the mirror image of XTable, but Delta-only source.
• Hudi. Native readers include Spark, Flink, Presto, Trino via the Hudi connector, and specialised streaming-oriented deployments (Uber, Robinhood, Zomato). Hudi's merge-on-read (MoR) semantics are the reason legacy stream-first pipelines stayed on it — no other format has as complete a stream-record-key upsert story.

The mixed-engine bill — why teams stop trying to standardise.

• The standardisation dream. "Let's migrate everything to Iceberg." Two months in, the team discovers three pipelines that emit Delta, one that emits Hudi, and a Databricks notebook that assumes Delta CHECK constraints. The migration keeps slipping right; management notices the OpEx.
• The duplicate-copy antipattern. "Fine, we'll write it twice." Every Spark job now writes both a Delta and an Iceberg copy of the same table. Storage doubles. Compaction runs twice. Bug surface doubles. Freshness diverges. This is what XTable exists to eliminate.
• The XTable pragmatic move. Write the data once in whichever format the producer prefers; run apache xtable sync after each commit to emit translated metadata for the other two formats; every downstream engine reads its native format from the same Parquet directory. Storage is 1×; freshness is one sync cadence behind the producer.

Where XTable came from — OneTable → XTable → Apache Incubator.

• Origin. OneTable was open-sourced by Onehouse in 2023, aiming to solve the "we have three formats and one bucket" problem the Onehouse team saw at customers running Hudi + Iceberg + Delta in parallel.
• Donation. In 2024, OneTable was donated to the Apache Software Foundation, entered the Incubator, and was renamed to XTable (the name OneTable clashed with unrelated trademarks). The rename is a light one — the config files, jar names, and public APIs simply migrated.
• Incubator status in 2026. XTable is a 1.x-track project with regular releases; the Incubator label caps the community's ability to declare "GA" but does not gate production adoption — several public case studies from Onehouse customers, Walmart, and mid-sized fintechs describe XTable in production.

What interviewers listen for.

• Do you name all three formats and their native reader story — Iceberg dominant, Delta Databricks-first, Hudi stream-first legacy — without prompting? — senior signal.
• Do you say "metadata-only translation, zero data copy" in the first sentence about XTable? — required answer.
• Do you push back on "just migrate everything to Iceberg" with the "some features don't round-trip" caveat? — senior signal.
• Do you know XTable was OneTable and that the rename came with the Apache donation? — trivia, but memorable.
• Do you describe XTable as "a metadata-only writer that reads one format's manifest and writes the other two" rather than as "a lakehouse tool"? — required answer.

Worked example — the three-format S3 bucket audit

Detailed explanation. The single most useful artifact for an XTable interview is a clear picture of a real mixed-engine S3 layout. Every senior XTable discussion converges on a diagram of "who writes, who reads, where the metadata sits." Walk through building the picture for a hypothetical sales.orders dataset that lives under s3://acme-lake/warehouse/sales/orders/.

• Producer. A Databricks structured-streaming job writes Delta to _delta_log/.
• Downstream 1. A Snowflake analyst wants to query the same table as Iceberg via a Snowflake Iceberg external table.
• Downstream 2. A legacy Trino cluster (Hudi connector installed but no Iceberg connector on that build) needs to read the same data.
• Constraint. No physical duplicate copies — storage bill is already the biggest single line item.

Question. Draw the S3 prefix layout after XTable sync, and describe which engine reads which metadata directory.

Input.

Engine	Native format	Path expectation	Reads from
Databricks (producer)	Delta	_delta_log/*.json	writes Delta
Snowflake	Iceberg	metadata/*.metadata.json	reads translated Iceberg
Trino (legacy)	Hudi	.hoodie/timeline	reads translated Hudi

Code.

s3://acme-lake/warehouse/sales/orders/
├── part-00000-abc.snappy.parquet      ← shared Parquet, written by Databricks
├── part-00001-def.snappy.parquet      ← shared Parquet, written by Databricks
├── part-00002-ghi.snappy.parquet      ← shared Parquet, written by Databricks
│
├── _delta_log/                        ← Delta metadata (SOURCE)
│   ├── 00000000000000000000.json
│   ├── 00000000000000000001.json
│   └── _last_checkpoint
│
├── metadata/                          ← Iceberg metadata (WRITTEN BY XTABLE)
│   ├── v1.metadata.json
│   ├── v2.metadata.json
│   ├── snap-<snapshot-id>.avro
│   └── <manifest-list>.avro
│
└── .hoodie/                           ← Hudi metadata (WRITTEN BY XTABLE)
    ├── hoodie.properties
    ├── timeline/
    │   ├── 20260719101530000.commit
    │   └── 20260719101815000.commit
    └── metadata/

Step-by-step explanation.

1. Databricks structured-streaming writes Parquet files and updates _delta_log/. This is the source of truth — every mutation lands here first, and every XTable sync reads from here.
2. After each commit (either on a cron every N minutes or via a post-commit hook), XTable reads the latest Delta snapshot, translates the schema + snapshot + partition metadata into Iceberg's metadata/ format, and writes new vN.metadata.json + manifest-list Avro files.
3. XTable also translates into Hudi's .hoodie/timeline layout, emitting one commit-time file per Delta commit. The Hudi timeline is time-string-keyed (YYYYMMDDHHmmssSSS), so XTable maps Delta commit timestamps into Hudi commit strings.
4. The three metadata directories now describe the same Parquet files in three different vocabularies. Snowflake's Iceberg reader sees the metadata/ folder and reads Parquet; Trino's Hudi connector sees .hoodie/ and reads Parquet; Databricks continues to read _delta_log/.
5. Storage cost is one copy of Parquet plus three copies of metadata. Metadata is typically well under 1% of table size — the trade-off is dramatic in favor of XTable versus writing Parquet three times.

Output.

Directory	Owner	Size vs data	Read by
part-*.parquet	Databricks (writer)	100% of data cost	all three engines
_delta_log/	Databricks	~0.1%	Databricks
metadata/	XTable	~0.2%	Snowflake, Trino Iceberg
.hoodie/	XTable	~0.2%	Trino Hudi, Spark Hudi

Rule of thumb. After XTable sync, the shared Parquet directory is untouched; only sibling metadata directories accumulate. If you ever see XTable writing new Parquet files during a sync, you've misconfigured something — XTable is metadata-only, always.

Worked example — what interviewers actually probe about XTable

Detailed explanation. The senior data-engineering XTable interview follows a predictable arc: the interviewer opens with a broad multi-engine scenario, then progressively narrows to test whether you know the round-trip caveats, the catalog wiring, and the "when NOT to use it" boundary. The candidates who name XTable as one option in a decision tree score highest; the candidates who describe it as a silver bullet score lowest. Walk through the interview grading rubric.

• Ambiguous opener. "Your data platform has Databricks, Snowflake, and Trino — how do you avoid maintaining three copies of every table?" — invites you to name a pattern.
• Follow-up 1. "What actually gets written when XTable syncs?" — probes the metadata-only understanding.
• Follow-up 2. "Which format features don't survive the round-trip?" — probes the feature-preservation matrix.
• Follow-up 3. "How does the target engine discover the translated metadata?" — probes the catalog integration story.
• Follow-up 4. "When would you migrate to a single format instead?" — probes the "bridge vs standardise" senior signal.

Question. Draft a 5-minute senior XTable answer that covers all four axes without waiting to be asked.

Input.

Interview signal	Weak answer	Senior answer
Pattern named	"we'd use OneTable"	"Apache XTable — metadata-only translation between Iceberg, Delta, Hudi"
Copy semantics	"it converts formats"	"zero data copy; only sibling metadata directories are written"
Round-trip caveats	"everything works"	"Delta CHECK constraints, Iceberg positional deletes, Hudi MoR state don't fully round-trip"
Catalog wiring	"the engine picks it up"	"register the translated table in Polaris / Glue / Unity per format"
When to standardise	"when you have time"	"when feature loss exceeds bridge value; target Iceberg + Polaris long-term"

Code.

Senior XTable answer template (5 minutes)
==========================================

Minute 1 — name the pattern up front
  "I'd use Apache XTable — the metadata-only translator (formerly
   OneTable, donated to the ASF in 2024). It reads one format's
   metadata and writes translated Iceberg, Delta, and Hudi metadata
   for the same Parquet files, so every engine reads its native
   format from the same physical dataset."

Minute 2 — what it actually writes
  "Zero data copy. The Parquet directory is shared and never
   modified. XTable writes only the sibling metadata directories:
   `_delta_log/`, `metadata/`, `.hoodie/`. Metadata is under 1% of
   table size in typical deployments."

Minute 3 — round-trip caveats
  "Not everything survives. Delta CHECK constraints, Iceberg
   positional deletes, and Hudi merge-on-read state are format-
   specific and don't have equivalents in the other formats. If
   you rely on Delta CHECK to enforce a NOT NULL, the Iceberg reader
   will not honour it."

Minute 4 — catalog wiring
  "Translation writes metadata to S3; you still need to register
   the translated table in a catalog per format. XTable can write
   directly to Apache Polaris (Iceberg REST), AWS Glue, Unity
   Catalog, or Snowflake's Iceberg REST catalog via the
   `catalogConfig` block in the YAML."

Minute 5 — bridge vs standardise
  "XTable is the bridge for today. Long term, standardise on
   Iceberg + Polaris and drop the other formats once the last
   consumer migrates. Track feature-loss and duplicate catalog
   registrations as the cost side of the bridge; renegotiate
   annually."

Step-by-step explanation.

1. Minute 1 is the crucial framing. Naming XTable immediately — "Apache XTable, metadata-only translator" — signals you know the exact tool and its lineage. Weak candidates say "OneTable" (outdated), "some translation layer" (vague), or "convert everything to Iceberg" (missing the pattern).
2. Minute 2 addresses the copy-semantics probe. "Zero data copy" is the single most important sentence — it's the entire reason XTable exists. Weak candidates describe file-level conversion, doubling the storage bill.
3. Minute 3 preempts the feature-preservation question. Naming three specific non-round-trippable features (Delta CHECK, Iceberg positional deletes, Hudi MoR) shows you've read the docs and stress-tested the translation.
4. Minute 4 addresses the catalog wiring. Translation without registration is a common failure mode — the metadata sits in S3 but no engine knows the table exists in the target format. Name the four common catalogs (Polaris, Glue, Unity, Snowflake REST) to demonstrate breadth.
5. Minute 5 is the senior "when NOT to" signal. XTable is a bridge, not a destination. The candidate who says "we'd standardise on Iceberg + Polaris long-term" shows architectural judgment.

Output.

Grading criterion	Weak score	Senior score
Names XTable in minute 1	rare	mandatory
Says "zero data copy"	rare	required
Names non-round-trippable features	rare	mandatory
Names catalog integration options	rare	senior signal
Names the "standardise long-term" exit	rare	senior signal

Rule of thumb. The senior XTable answer is a 5-minute monologue that covers naming, copy semantics, feature caveats, catalog wiring, and the exit strategy without waiting for the follow-ups. Rehearse it once; deploy it every time.

Worked example — the "pick the format channel" decision tree

Detailed explanation. Given a new dataset, the senior architect runs a short decision tree in their head to decide whether XTable is the right tool, or whether the team should just standardise. Codifying the tree makes the interview answer reproducible. Walk through the tree with three canonical scenarios: a Databricks-native team adding a Snowflake reader, a Trino-native team adopting Databricks, and a greenfield team with no legacy.

• Q1. Does your data platform have more than one query engine already writing tables? → yes = go to Q2; no = go to Q5.
• Q2. Are all engines happy with one format (usually Iceberg)? → yes = standardise, no XTable needed; no = go to Q3.
• Q3. Is the extra format contributing to critical business value (e.g. Databricks-only ML)? → yes = XTable bridge; no = negotiate migration.
• Q4. Do you need features that don't round-trip (Delta CHECK, Iceberg positional deletes, Hudi MoR)? → yes = keep the source format primary; no = XTable is a clean fit.
• Q5 (greenfield). New project, pick one format → default Iceberg + Polaris; no XTable needed on day one.

Question. Walk the decision tree for the three scenarios and record the outcome for each.

Input.

Scenario	Q1 (multi-engine?)	Q2 (agree on one format?)	Q3 (extra format valuable?)	Q4 (non-round-trip features?)
Databricks + new Snowflake	yes	no	yes	no
Trino + new Databricks	yes	no	yes	Delta CHECK matters
Greenfield team	no	—	—	—

Code.

# Decision-tree helper (illustrative)
def pick_xtable_strategy(multi_engine: bool,
                         agree_one_format: bool,
                         extra_format_valuable: bool,
                         needs_non_round_trip: bool) -> str:
    """Return the recommended strategy for a new dataset."""
    if not multi_engine:
        return "standardise-iceberg-greenfield"

    if agree_one_format:
        return "standardise-migrate-to-iceberg"

    if not extra_format_valuable:
        return "negotiate-migration-away-from-extra-format"

    if needs_non_round_trip:
        return "xtable-bridge-keep-source-primary"

    return "xtable-bridge-clean-fit"


print(pick_xtable_strategy(True,  False, True,  False))
# → 'xtable-bridge-clean-fit'

print(pick_xtable_strategy(True,  False, True,  True))
# → 'xtable-bridge-keep-source-primary'

print(pick_xtable_strategy(False, False, False, False))
# → 'standardise-iceberg-greenfield'

Step-by-step explanation.

1. Scenario 1 — Databricks team writing Delta, new Snowflake team wants to read Iceberg. Q1 = yes, Q2 = no, Q3 = yes (both teams have legitimate use), Q4 = no (no CHECK constraints in use). The tree short-circuits to xtable-bridge-clean-fit: run XTable to translate Delta → Iceberg, register with Glue or Polaris, Snowflake reads via its Iceberg catalog integration.
2. Scenario 2 — Trino team on Iceberg, new Databricks team writes Delta and relies on CHECK constraints for data quality. Q1 = yes, Q2 = no, Q3 = yes, Q4 = yes. Outcome: xtable-bridge-keep-source-primary — Delta stays the source-of-truth format (because CHECK constraints are enforced only there), XTable emits translated Iceberg metadata for Trino to read.
3. Scenario 3 — greenfield project. Q1 = no. Short-circuit to standardise-iceberg-greenfield: start with Iceberg + Polaris; add XTable only if a second format enters later.
4. The tree makes the "non-round-trip feature" axis explicit. If you rely on Delta CHECK, XTable cannot enforce it on the Iceberg side; the Delta metadata stays the enforcement point and the Iceberg reader is read-only.
5. If none of Q1-Q4 point at XTable, the correct answer is "you don't need XTable yet — pick one format and defer the bridge until a second engine appears." Refuse to install XTable prematurely.

Output.

Scenario	Outcome	Rationale
Databricks + Snowflake	xtable-bridge-clean-fit	both engines valuable; no CHECK dependency
Trino + Databricks	xtable-bridge-keep-source-primary	CHECK constraints pin Delta as source
Greenfield	standardise-iceberg-greenfield	no reason to add complexity

Rule of thumb. The five-question decision tree is a whiteboard-friendly answer. Practice walking it end-to-end so any interviewer can hand you a scenario and get an XTable-yes-or-no in under 60 seconds — with the "when to keep source primary" caveat as the bonus senior signal.

Senior interview question on multi-engine lakehouse

A senior interviewer often opens with: "You inherit a lakehouse where three teams write to s3://data/: a Spark job writing Iceberg, a Databricks notebook writing Delta, and a Flink pipeline writing Hudi. Consumers ask for cross-team joins that hit all three tables from the same query. Design the target state, name the tools, and explain how you'd migrate without triplicating storage."

Solution Using XTable as the metadata bridge with a phased consolidation plan

# 1. XTable sync config — publish each source as all three formats
# Iceberg-source Spark table exposed as Delta + Hudi as well
sourceFormat: ICEBERG
targetFormats:
  - DELTA
  - HUDI
datasets:
  - tableName: sales_orders
    tableBasePath: s3://data/sales/orders
    partitionSpec:
      - partitionFieldName: order_date
        transformType: VALUE
    tableDataPath: s3://data/sales/orders

# Delta-source Databricks table exposed as Iceberg + Hudi
sourceFormat: DELTA
targetFormats:
  - ICEBERG
  - HUDI
datasets:
  - tableName: marketing_events
    tableBasePath: s3://data/marketing/events
    tableDataPath: s3://data/marketing/events

# 2. Run XTable sync after each writer commit (Airflow, EventBridge, or cron)
java -jar utilities-0.3.0-incubating-SNAPSHOT-bundled.jar \
  --datasetConfig sales-orders-iceberg-source.yaml \
  --hadoopConfig core-site.xml

-- 3. Snowflake reader — external Iceberg table on the translated metadata
CREATE EXTERNAL VOLUME acme_lake
  STORAGE_LOCATIONS = (( NAME = 's3://data/'
                         STORAGE_PROVIDER = 'S3'
                         STORAGE_AWS_ROLE_ARN = 'arn:aws:iam::1234:role/snowflake-lake' ));

CREATE ICEBERG TABLE marketing_events_ib
  EXTERNAL_VOLUME = 'acme_lake'
  CATALOG         = 'SNOWFLAKE'
  BASE_LOCATION   = 'marketing/events/';

Step-by-step trace.

Phase	Action	Result
Discovery	Audit which teams write which format	3 producers, 3 formats
Introduce XTable	One YAML per source; run after each commit	3 sibling metadata dirs per table
Register	Add each table in each format to Glue/Polaris/Snowflake REST	one catalog entry per (table, format)
Cross-format joins	Snowflake reads all three as Iceberg	ordinary SQL joins
Phase 2 consolidation	Migrate Delta producer to Iceberg	drop Delta metadata
Phase 3 consolidation	Migrate Hudi producer to Iceberg	drop Hudi metadata
Target state	One format (Iceberg), one catalog (Polaris)	XTable retired

After Phase 1, cross-format joins work immediately — Snowflake sees all three tables as Iceberg and issues ordinary joins. Phases 2 and 3 are opportunistic: migrate producers when a team touches the pipeline anyway. XTable earns its keep during the bridge months; the plan explicitly retires it once every producer is on the target format.

Output:

Metric	Before XTable	After Phase 1 (XTable)	After Phase 3 (retired)
Physical storage copies	1 per format = 3	1 shared	1 shared
Metadata dirs per table	1	3	1
Cross-format join possible	no	yes	yes
Sync latency	N/A	seconds–minutes after commit	N/A
Catalog entries per table	1	3	1

Why this works — concept by concept:

• Metadata-only translation — XTable never touches Parquet. It reads the source-format snapshot manifest, computes the equivalent representation in the target format's vocabulary, and writes a sibling metadata directory. Data cost = 0; metadata cost is well under 1% of table size.
• One source of truth per table — each table has exactly one writer format; XTable only mirrors metadata outward. Writers never race; there's no dual-write correctness bug because only one format is authoritative per table.
• Sync cadence matches business need — batch tables sync every 15 min via cron; hot tables sync via post-commit event on the writer. Cadence is a tuning knob, not a rewrite.
• Catalog registration per format — translating metadata is necessary but not sufficient; every downstream engine needs the table registered in its native catalog. Polaris for Iceberg, Glue for Delta/Iceberg, Unity for Delta, Snowflake REST for Iceberg — one registration per (table, format).
• Cost — one Parquet copy per table (was: 3), one XTable Java process per sync tick, three catalog entries per table, three sibling metadata dirs. The eliminated cost is 2× duplicated storage per multi-format table. Net O(1) storage per table versus O(F) where F is the number of formats.

Design
Topic — design
Design problems on multi-engine lakehouse architecture

Practice →

SQL Topic — sql SQL cross-catalog and federation problems

Practice →

---

2. XTable architecture — metadata-only translation

XTable is a Java tool that reads one format's metadata and writes translated metadata for the other two — Parquet never moves

The mental model in one line: apache xtable is a Java-based CLI (also embeddable as a library) that opens the source-format's snapshot metadata — Iceberg's metadata/*.json + manifest-list Avro, Delta's _delta_log/*.json, or Hudi's .hoodie/timeline — walks the file listing to derive the current committed state, and then writes an equivalent snapshot manifest into each requested target format's sibling directory without reading, rewriting, or moving a single Parquet byte, so the underlying columnar data stays shared and each downstream engine sees a table in its native format. Every senior architect should be able to describe the input → transform → output flow in one sentence — that's the interview litmus test.

The three inputs XTable can read.

• Iceberg source. XTable opens metadata/vN.metadata.json, resolves the current snapshot ID, reads the associated manifest-list Avro, and walks the manifests to enumerate part-*.parquet file paths + column-level stats (min/max/nulls) + partition tuples. Schema evolution history is captured from the metadata JSON.
• Delta source. XTable reads _delta_log/ in ascending order, replays actions (add, remove, metadata, commitInfo) to derive the current file set, current schema, and current protocol version. Latest checkpoint (_last_checkpoint) can be used as a fast-forward.
• Hudi source. XTable reads .hoodie/hoodie.properties for table config, walks .hoodie/timeline to enumerate committed instants, and reads each instant's inflight/complete/rollback files to derive the current data-file list.

The three outputs XTable can write.

• Iceberg target. XTable writes a new metadata/v(N+1).metadata.json referencing a new snapshot, a new manifest-list Avro, and one or more manifest Avros. The Parquet files referenced by manifests are the existing shared files. Uses the Iceberg Java library to serialise correctly.
• Delta target. XTable writes a new _delta_log/00000000000000000NNN.json transaction file containing add actions for every current file (as if the table had just been rewritten). Column stats are populated where the source provided them.
• Hudi target. XTable writes a new .hoodie/timeline/YYYYMMDDHHmmssSSS.commit instant file containing the current file set. Written as a copy-on-write instant even if the source is Iceberg or Delta (which have no MoR concept).

The transformation invariants XTable preserves.

• Schema. Column names, types, nullability, and (where possible) column IDs are preserved. Iceberg column-ID mapping survives translation; Delta doesn't have a native column-ID concept, so XTable emits Delta with delta.columnMapping.mode = name.
• Partition spec. Partition columns and transforms (YEAR, MONTH, DAY, HOUR, BUCKET, TRUNCATE, IDENTITY) map bidirectionally where the target format supports them. Hudi's partition model is looser; some Iceberg transforms map to Hudi as IDENTITY.
• Current snapshot. Each sync represents the source's current committed state as the target's newest snapshot. Historical snapshots do not round-trip — XTable is a "current-state" translator, not a "history" translator.
• Column stats. Min/max/null counts are copied when the source stores them. Absent stats are left absent; the target reader may compute them on demand.

What XTable does NOT do.

• Rewriting Parquet. Never. If a Parquet file exists in Iceberg, XTable references the same path in Delta and Hudi manifests. Rewriting would defeat the entire point.
• Compaction / clustering. XTable does not merge small files, apply Z-ordering, or run OPTIMIZE. Those remain the source-format writer's responsibility.
• Historical snapshot round-trip. Only the current snapshot is translated. Time-travel queries against the target format only see one snapshot per sync tick.
• Bidirectional sync. XTable is one-way per sync. If two writers touch two different formats on the same table, XTable cannot merge them. The pattern is one authoritative writer, N read-only mirrors.

Common interview probes on XTable architecture.

• "What does XTable actually write?" — required answer: only sibling metadata directories; never Parquet.
• "Does XTable read Parquet files?" — required answer: usually no; the source manifest already carries column stats. XTable reads Parquet footers only if column stats are missing and required.
• "Is XTable bidirectional?" — no; each sync run is one source → many targets. Two authoritative writers on one table breaks the model.
• "What's the sync latency?" — bounded by cadence: seconds if event-driven, minutes if cron. XTable itself takes seconds to run per table for typical metadata sizes.
• "Does XTable preserve history?" — only current state per sync; historical snapshots do not round-trip.

Worked example — one XTable sync from Delta source to Iceberg + Hudi targets

Detailed explanation. Walk through what actually happens on the file system when an XTable sync runs against a Databricks-written Delta table. This is the level of detail a senior interviewer expects — file paths, action sequence, byte flow. Assume the table s3://data/orders/ has three parquet files and one prior Delta commit.

• Before sync. Only _delta_log/ exists in the sibling metadata slot. No Iceberg or Hudi metadata.
• During sync. XTable reads _delta_log/, derives the current file list, then serialises Iceberg and Hudi metadata using their respective Java libraries.
• After sync. metadata/ and .hoodie/ exist alongside _delta_log/. Parquet files are unchanged.

Question. Trace the file operations XTable performs for one sync of a Delta table to Iceberg + Hudi.

Input.

Component	Value
Source path	s3://data/orders/
Source format	DELTA
Target formats	ICEBERG, HUDI
Existing files	part-00000.parquet, part-00001.parquet, part-00002.parquet
Existing metadata	_delta_log/00000000000000000000.json

Code.

# XTable one-shot sync CLI
java -jar utilities-0.3.0-incubating-SNAPSHOT-bundled.jar \
  --datasetConfig orders-delta-source.yaml \
  --hadoopConfig core-site.xml

# orders-delta-source.yaml
sourceFormat: DELTA
targetFormats:
  - ICEBERG
  - HUDI
datasets:
  - tableBasePath: s3://data/orders
    tableDataPath: s3://data/orders
    tableName: orders
    partitionSpec:
      - partitionFieldName: order_date
        transformType: VALUE

# File-system operations performed during sync (traced from XTable logs)
READ    s3://data/orders/_delta_log/_last_checkpoint            (none — first run)
LIST    s3://data/orders/_delta_log/                            (1 file)
READ    s3://data/orders/_delta_log/00000000000000000000.json
        → decode: metadata action + 3 add actions
        → current schema, current file list (3 files)

WRITE   s3://data/orders/metadata/v1.metadata.json
WRITE   s3://data/orders/metadata/snap-8901234567890123456-1-abc.avro    (manifest list)
WRITE   s3://data/orders/metadata/abc-m0.avro                            (manifest)

WRITE   s3://data/orders/.hoodie/hoodie.properties
WRITE   s3://data/orders/.hoodie/timeline/20260719101530000.commit
WRITE   s3://data/orders/.hoodie/timeline/20260719101530000.commit.requested
WRITE   s3://data/orders/.hoodie/timeline/20260719101530000.commit.inflight

(no Parquet writes — data files remain part-00000..part-00002)

Step-by-step explanation.

1. XTable's Delta source reader first checks _last_checkpoint (Delta's checkpoint pointer). None exists on a fresh table, so it falls back to reading every JSON transaction file. For large tables with checkpoints, only the latest checkpoint + subsequent JSON files are read — sync stays cheap.
2. Reading 00000000000000000000.json yields a metadata action (schema + config) and three add actions (one per Parquet file). XTable's internal representation is a canonical InternalTable object with schema, partition spec, and file list. This canonical form is the pivot for all target writers.
3. The Iceberg writer converts the canonical InternalTable to an Iceberg TableMetadata + Snapshot + manifest, then serialises via the Iceberg Java library. Output is three files: the metadata JSON, the manifest-list Avro, and one manifest Avro. The manifest references the existing Parquet paths.
4. The Hudi writer initialises .hoodie/hoodie.properties (first-run only) and writes a commit instant. Hudi's timeline uses three files per instant — .requested, .inflight, .commit — even for XTable-driven sync (mimicking Hudi's own transaction pattern for compatibility with Hudi readers).
5. Zero Parquet writes occur. If you run aws s3 ls --recursive before and after, only the metadata directories change; every Parquet file's LastModified timestamp is identical.

Output.

File	Before	After	Written by
part-00000.parquet	present	present (untouched)	Databricks (previously)
_delta_log/00000000000000000000.json	present	present	Databricks
metadata/v1.metadata.json	—	new	XTable
metadata/snap-*.avro	—	new (manifest list)	XTable
metadata/*-m0.avro	—	new (manifest)	XTable
.hoodie/hoodie.properties	—	new	XTable
.hoodie/timeline/*.commit	—	new	XTable

Rule of thumb. After every XTable sync, verify with aws s3 ls --recursive s3://... --page-size 1000 | grep -c parquet — the Parquet count must be identical before and after. Any change means a bug in your writer, not XTable — but the check catches misconfigurations fast.

Worked example — schema evolution and column-ID handling across formats

Detailed explanation. Schema evolution is where format differences bite hardest. Iceberg has native column IDs (a stable integer per column, allowing safe rename). Delta uses column mapping modes (name or id) with different defaults per Databricks version. Hudi uses column names as the identity primitive. XTable's job is to map these three vocabularies bidirectionally without silently corrupting the schema. Walk through a rename scenario.

• Source. An Iceberg table with id INT, total_cents BIGINT (column IDs 1 and 2).
• Change. The writer renames total_cents to total_amount_cents — an Iceberg schema evolution operation that preserves column ID 2.
• Question. What happens to the Delta and Hudi target metadata after the next XTable sync?

Question. Trace how a column rename in Iceberg propagates to Delta and Hudi through XTable.

Input.

Format	Column identity primitive	Rename semantics
Iceberg	column ID	safe (data unaffected)
Delta with columnMapping.mode = name	name	rename = new column (drop-add)
Delta with columnMapping.mode = id	ID	safe rename
Hudi	name	rename = new column

Code.

-- 1. Iceberg rename (Spark)
ALTER TABLE local.db.orders RENAME COLUMN total_cents TO total_amount_cents;

# 2. XTable config with Delta column mapping enabled
sourceFormat: ICEBERG
targetFormats:
  - DELTA
  - HUDI
datasets:
  - tableBasePath: s3://data/orders
    tableDataPath: s3://data/orders
    tableName: orders
    tableProperties:
      delta.columnMapping.mode: id       # preserve rename across sync

# 3. After XTable sync — file-system observation

Iceberg metadata (SOURCE):
  metadata/v2.metadata.json    schema-id=1  fields=[id(1),total_amount_cents(2)]
  metadata/v1.metadata.json    schema-id=0  fields=[id(1),total_cents(2)]

Delta metadata (TARGET):
  _delta_log/00000000000000000001.json    metadata action with columnMapping=id
                                          fields=[id(id=1),total_amount_cents(id=2)]

Hudi metadata (TARGET):
  .hoodie/timeline/20260719104500000.commit
                                          schema fields=[id,total_amount_cents]

Step-by-step explanation.

1. Iceberg's rename is an O(1) metadata operation because the column ID (2) stays the same; only the display name changes. Existing Parquet files still store the value in physical column ordinal position 2, and the Iceberg reader uses the ID→ordinal map from the manifest.
2. When XTable syncs, it reads the new Iceberg schema-id-1 metadata (which has the new name for column ID 2), and writes Delta metadata with columnMapping.mode = id. This mode makes Delta store field IDs in the metadata itself, so the rename is a rename on the Delta side too — not a drop-and-add.
3. Without columnMapping.mode = id, Delta would treat this as a schema mismatch: the Parquet files have a column with the old name embedded in their footers, but the Delta metadata says the column is total_amount_cents. Reader behavior is undefined; some engines break.
4. On the Hudi side, XTable writes the new commit with the new column name. Hudi has no column-ID concept, so downstream Hudi readers see the rename as if the column had always been named total_amount_cents. Existing Parquet files store the data under the old name in the footer; Hudi's schema-on-read resolves by ordinal position.
5. The senior takeaway: enable delta.columnMapping.mode = id in your XTable YAML from day one for any table you might rename. Retro-enabling column mapping on Delta requires an ALTER TABLE that touches every file.

Output.

Column	Iceberg ID	Delta field ID (with id mode)	Hudi (by name)
id	1	1	id
total_amount_cents	2	2	total_amount_cents (was: total_cents)

Rule of thumb. For any XTable deployment translating to Delta, set tableProperties.delta.columnMapping.mode = id in the YAML. This preserves rename semantics across the sync and prevents "column not found" runtime errors on Delta readers.

Worked example — quantifying metadata cost vs data cost

Detailed explanation. Every senior architect must quantify the operational cost of XTable before signing off on production adoption. The cost is deterministic: XTable writes one small set of metadata files per sync per table per target format. Multiply by sync cadence and table count. Walk through the calculation for a 500-table warehouse with three-format targets.

• Table count. 500 tables.
• Target formats. All three (Iceberg, Delta, Hudi) — worst-case scenario.
• Sync cadence. Every 15 min (96 syncs/day).
• Metadata size per sync per target. ~5-50 KB (JSON + Avro) for small tables; up to a few MB for very large tables with many partitions.

Question. Quantify the added storage, S3 request count, and CPU cost of XTable on a 500-table warehouse syncing every 15 min.

Input.

Metric	Value
Tables	500
Target formats per table	3
Metadata size per sync per target	20 KB (median)
Syncs per day per table	96
Retention (metadata)	30 days

Code.

# Storage projection — XTable metadata cost
TABLES              = 500
TARGET_FORMATS      = 3        # per table
METADATA_KB_PER_SYNC = 20      # median across formats
SYNCS_PER_DAY       = 96       # every 15 min
RETENTION_DAYS      = 30

daily_metadata_gb = (
    TABLES * TARGET_FORMATS * METADATA_KB_PER_SYNC * SYNCS_PER_DAY
) / (1024 * 1024)

retained_metadata_gb = daily_metadata_gb * RETENTION_DAYS

print(f"XTable metadata: {daily_metadata_gb:.2f} GB/day, "
      f"{retained_metadata_gb:.1f} GB retained @ {RETENTION_DAYS}d")
# → XTable metadata: 2.75 GB/day, 82.4 GB retained @ 30d

# S3 requests — LIST + GET on source, PUT on target per sync
S3_PUTS_PER_SYNC = 4         # ~4 files per target format (JSON + manifest + list + optional)
s3_puts_per_day  = TABLES * TARGET_FORMATS * S3_PUTS_PER_SYNC * SYNCS_PER_DAY
print(f"S3 PUT requests: {s3_puts_per_day:,}/day")
# → S3 PUT requests: 576,000/day

# CPU cost — one JVM per sync tick; each table takes ~1-5s
# 500 tables * 3 targets * 96 syncs = 144,000 sync ops/day
# If we parallelise 8-wide, 144,000/8 = 18,000 seconds/day = 5 hours of CPU/day
# On an m6i.2xlarge (8 vCPU): about 21% of a single 8-vCPU node = one small runner

Step-by-step explanation.

1. Storage is the sneakiest cost. At 500 tables × 3 targets × 20 KB × 96 syncs/day = ~2.75 GB/day. Retention at 30 days brings the metadata footprint to ~82 GB. Compared to a typical warehouse's petabyte-scale data footprint, this is under 0.01% — negligible.
2. S3 request cost matters more than storage in some deployments. 500 × 3 × 4 × 96 = ~576k PUTs/day. At ~$5 per 10k PUT requests, that's ~$290/day of pure request cost. Cadence tuning (some tables every 15 min, cold tables every hour) can cut this dramatically.
3. CPU cost per sync is bounded — XTable is a lightweight metadata rewrite, not a data scan. A single 8-vCPU runner can handle ~144k sync ops/day at ~1-5 seconds each. Parallel table processing (native to XTable's runner) keeps wall-clock latency low.
4. Compared to the alternative — writing the data three times, once per format — the metadata cost is 2-3 orders of magnitude smaller than the data cost of the naive approach. Storage would be 3× the base cost; XTable makes it 1.0001×.
5. Cadence is the primary tuning knob. Move cold tables to hourly sync (24 syncs/day instead of 96) to cut cost by 75% for those tables. Event-driven sync (post-commit hooks) further reduces cost by syncing only when the source actually changed.

Output.

Cost dimension	XTable @ 15 min	XTable @ 1 h	Duplicate copies
Storage overhead	~82 GB @ 30d	~21 GB @ 30d	2× data cost (TB-PB)
S3 PUT requests	~576k/day	~144k/day	0 (no XTable) but 2× data writes
CPU (single runner)	5 hrs/day	1.25 hrs/day	0 XTable, but 2× data-write CPU
Freshness	15 min	1 hour	writer-side atomic
Feature preservation	source-format-limited	source-format-limited	native per format

Rule of thumb. Before shipping XTable at scale, compute daily metadata footprint and S3 PUT request cost. Both should be tiny relative to the data cost of the alternative (duplicating Parquet per format). If they're not, tune sync cadence per table — hot tables sync often, cold tables sync rarely.

Senior interview question on XTable architecture

A senior interviewer might ask: "Walk me through what happens on S3 when XTable syncs an Iceberg source to Delta and Hudi targets. Include the file operations, the metadata files written, whether Parquet is touched, and how you'd verify the sync succeeded without querying the tables."

Solution Using an XTable one-shot sync with verification via S3 list diff

# 1. XTable YAML — Iceberg source, Delta + Hudi targets
sourceFormat: ICEBERG
targetFormats:
  - DELTA
  - HUDI
datasets:
  - tableBasePath: s3://data/orders
    tableDataPath: s3://data/orders
    tableName: orders
    partitionSpec:
      - partitionFieldName: order_date
        transformType: VALUE
    tableProperties:
      delta.columnMapping.mode: id

# 2. Pre-sync snapshot — capture the Parquet file count
aws s3 ls s3://data/orders/ --recursive \
  | grep -E '\.parquet$' \
  | wc -l > /tmp/parquet-count-before.txt

aws s3 ls s3://data/orders/ --recursive > /tmp/list-before.txt

# 3. Run XTable
java -jar utilities-0.3.0-incubating-SNAPSHOT-bundled.jar \
  --datasetConfig orders-iceberg-source.yaml \
  --hadoopConfig core-site.xml \
  2>&1 | tee /tmp/xtable-sync.log

# 4. Post-sync snapshot
aws s3 ls s3://data/orders/ --recursive \
  | grep -E '\.parquet$' \
  | wc -l > /tmp/parquet-count-after.txt

aws s3 ls s3://data/orders/ --recursive > /tmp/list-after.txt

# 5. Verify: Parquet count is unchanged; only metadata directories grew
diff /tmp/parquet-count-before.txt /tmp/parquet-count-after.txt
# → (empty diff = success)

diff /tmp/list-before.txt /tmp/list-after.txt \
  | grep -vE '(_delta_log|metadata/|\.hoodie/)' \
  | grep -v '^[<>][^<>]' || echo "OK: only metadata changed"

# 6. Programmatic verification — cross-format snapshot equivalence
from pyiceberg.catalog import load_catalog
from deltalake import DeltaTable

ice = load_catalog("hadoop", **{"warehouse": "s3://data/"})
it  = ice.load_table("db.orders")
ice_files = {f.file_path for f in it.scan().plan_files()}

dt = DeltaTable("s3://data/orders")
delta_files = set(dt.file_uris())

assert ice_files == delta_files, f"file set mismatch: {ice_files ^ delta_files}"
print(f"OK: {len(ice_files)} files referenced by both Iceberg and Delta")

Step-by-step trace.

Step	Purpose	Verification
YAML config	declare source + targets + column mapping	commit under version control
Pre-sync Parquet count	baseline for zero-data-copy invariant	wc -l on grep parquet
Run XTable	one JVM invocation per sync	logs to /tmp/xtable-sync.log
Post-sync Parquet count	must equal pre-sync	diff both counts
S3 list diff	only metadata should change	grep -vE metadata dirs
Cross-format file set	Iceberg and Delta must reference same files	pyiceberg + deltalake equality

After the run, _delta_log/ receives one new transaction file, metadata/ receives incremented vN.metadata.json + new manifest + manifest-list, .hoodie/timeline/ receives a new commit instant. Parquet count is unchanged. Every downstream reader can query the table in its native format and get identical rows.

Output:

Metric	Value
Parquet count change	0
New metadata files (per format)	3-4
Sync wall-clock	~2-4 s
Iceberg-Delta file set equality	true
Hudi commit instant created	true

Why this works — concept by concept:

• Metadata-only writer — XTable's Iceberg reader converts the source snapshot to a canonical internal representation, and each target writer serialises that representation into its native format's metadata files. No Parquet ingest path exists in XTable; the code physically cannot write data files.
• Canonical InternalTable pivot — Iceberg → Internal → Delta and Iceberg → Internal → Hudi share the same pivot. Adding a new source format is one new reader; adding a new target format is one new writer. The N×M translator problem becomes N+M.
• Column mapping mode = id — enables rename-safe Delta translation. Without it, renames on the Iceberg side would translate to drop-and-add on the Delta side, breaking column-level lineage.
• Parquet count diff — the cheapest possible verification. If XTable ever writes Parquet, this diff is non-zero, and something is wrong. Ship this check in your CI/CD.
• Cost — one JVM invocation per sync, seconds of CPU per table, a few KB of metadata written per target format. The eliminated cost is 2× duplicated Parquet storage for a multi-format world. O(metadata_size) per sync versus O(data_size) for the naive rewrite alternative.

Design
Topic — design
Design problems on metadata translation and format bridges

Practice →

SQL Topic — sql SQL schema-evolution and column-mapping problems

Practice →

---

3. Iceberg, Delta, and Hudi round-trip in practice

The CLI-plus-YAML recipe for translating between any pair — and the feature drift you must budget for

The mental model in one line: apache xtable translation is a CLI invocation (java -jar utilities-*.jar --datasetConfig config.yaml) that reads a YAML describing one source format and one-or-more target formats, then either runs once or continuously on a schedule to mirror source-metadata changes into target-metadata — the round-trip works cleanly for the intersection of format features (schema, partitions, snapshots, column stats), and fails soft for the format-specific edge cases (Delta CHECK constraints, Iceberg positional deletes, Hudi merge-on-read state) that you must consciously exclude from the sync boundary. Every senior architect running XTable in production keeps a "does-not-round-trip" list on the same wiki page as the YAML.

The CLI + YAML anatomy.

• The jar. Downloaded from the Apache XTable Incubator release page (or built from source). Named utilities-<version>-bundled.jar; the bundled variant ships all Iceberg, Delta, and Hudi client libraries so you don't fight Maven.
• The datasetConfig YAML. Declares sourceFormat (one of ICEBERG, DELTA, HUDI), targetFormats (list of the others), and one or more datasets entries with tableBasePath (S3/GCS/ABFS URI), tableName, and optional partitionSpec + tableProperties.
• The hadoopConfig XML. Standard Hadoop core-site.xml shape; provides S3 credentials, region, and any custom S3A tuning (multipart threshold, connection pool). XTable uses Hadoop FS under the hood for all cloud storage.

Sync cadence patterns — batch vs event-driven.

• Batch cron. Simplest: Airflow / EventBridge / a Kubernetes CronJob runs the CLI every N minutes for every dataset. Latency = cron interval. Operational cost = one JVM start per cadence per dataset. Recommended for cold-warm tables.
• Event-driven. Best: subscribe to source-format commit events. Delta on Databricks emits table-lineage events; Iceberg REST catalogs (Polaris) can push webhook-style callbacks; Hudi writes to .hoodie/timeline are directly S3 event-notification-triggerable. Latency = seconds. Recommended for hot tables.
• Hybrid. The pragmatic default: event-driven for a small hot-set of tables, cron for the long tail. Costs are minimised without operational complexity.

The feature preservation matrix — what round-trips cleanly.

• Round-trips. Column names + types, nullability, partition columns + partition values, current-snapshot file list, column-level min/max stats (where source provides them), basic schema evolution (ADD COLUMN with column-ID mapping enabled).
• Round-trips with caveats. Rename (only with delta.columnMapping.mode = id on the Delta target); partition transforms (some Iceberg transforms map to Hudi as IDENTITY only); table properties (only the intersection subset survives).
• Does NOT round-trip. Delta CHECK constraints; Delta GENERATED columns; Iceberg positional deletes / row-level deletes (Iceberg V2 feature); Hudi merge-on-read state (delta log files); historical snapshots (only current); table statistics computed by CBO (Iceberg's SNAPSHOT_TABLE_STATS).
• Feature-specific fallbacks. If Delta CHECK is business-critical, keep Delta as source-of-truth and don't rely on the Iceberg mirror for validation. If Iceberg positional deletes are used, the Delta mirror represents the deletes as add-then-remove rewrites (larger metadata, still correct).

The "which pair round-trips best" ranking.

• Iceberg ↔ Delta. Best. Both are copy-on-write by default, both support column mapping, both have first-class schema-evolution stories. Column-ID preservation makes rename safe.
• Iceberg ↔ Hudi. Good with caveats. Iceberg's snapshot model maps to Hudi commit instants cleanly, but Hudi's optional merge-on-read state cannot round-trip into Iceberg.
• Delta ↔ Hudi. Also good. Delta's transaction log maps to Hudi's timeline commit-by-commit. Same MoR caveat.
• Any pair with MoR on the source. Difficult. XTable snapshots the current committed state; MoR delta files are represented as if they've been compacted into base files. Downstream Hudi readers can still see the current state, but MoR-specific queries (read-optimised vs snapshot views) don't fully translate.

Common interview probes on round-trip.

• "What's the sync cadence — cron or event-driven?" — depends on hot-vs-cold; hybrid is the pragmatic default.
• "Which features don't survive the translation?" — required answer: Delta CHECK, Iceberg positional deletes, Hudi MoR, historical snapshots.
• "How do you handle a Delta CHECK constraint downstream?" — required answer: keep Delta as source, don't rely on the Iceberg mirror for enforcement.
• "How do you detect drift between formats?" — cross-format file-set equality check (see H2-2 code).

Worked example — cron-driven sync of a Delta table to Iceberg every 15 minutes

Detailed explanation. The canonical cron pattern: Airflow schedules an XTable sync task per dataset every 15 minutes. Failures are captured; drift is monitored; the whole thing is one YAML per table plus one DAG task. Walk through the DAG.

• Cadence. Every 15 minutes for all warehouse tables.
• YAML storage. Version-controlled in a repo; deployed to S3 via a shared config bucket.
• Failure handling. Standard Airflow retry + Slack alert.

Question. Write the Airflow DAG that runs XTable sync for one dataset every 15 min.

Input.

Parameter	Value
Source	Delta on s3://data/marketing/events
Target	Iceberg
Cadence	15 min
YAML store	s3://config/xtable/marketing-events.yaml
Runner	Airflow KubernetesPodOperator

Code.

# xtable_marketing_events_dag.py
from airflow import DAG
from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator
from datetime import datetime, timedelta

default_args = {
    "owner": "data-platform",
    "retries": 2,
    "retry_delay": timedelta(minutes=2),
}

with DAG(
    dag_id="xtable_sync_marketing_events",
    default_args=default_args,
    schedule_interval="*/15 * * * *",
    start_date=datetime(2026, 7, 1),
    catchup=False,
    max_active_runs=1,           # never overlap syncs on the same table
) as dag:

    sync = KubernetesPodOperator(
        task_id="xtable_sync",
        namespace="data",
        image="apache/xtable:0.3.0-incubating",
        cmds=["java"],
        arguments=[
            "-jar", "/opt/xtable/utilities-bundled.jar",
            "--datasetConfig", "/config/marketing-events.yaml",
            "--hadoopConfig",  "/config/core-site.xml",
        ],
        env_vars={
            "AWS_REGION": "us-east-1",
        },
        volume_mounts=[
            {"name": "config", "mountPath": "/config", "readOnly": True},
        ],
        volumes=[
            {"name": "config", "configMap": {"name": "xtable-marketing-events"}},
        ],
        get_logs=True,
        is_delete_operator_pod=True,
    )

# marketing-events.yaml (mounted at /config/marketing-events.yaml)
sourceFormat: DELTA
targetFormats:
  - ICEBERG
datasets:
  - tableBasePath: s3://data/marketing/events
    tableDataPath: s3://data/marketing/events
    tableName: marketing_events
    partitionSpec:
      - partitionFieldName: event_date
        transformType: VALUE
    tableProperties:
      delta.columnMapping.mode: id

Step-by-step explanation.

1. The DAG runs a single KubernetesPodOperator task every 15 minutes. max_active_runs=1 prevents two syncs of the same table from running concurrently — XTable does not currently support multi-writer coordination on the same target metadata directory.
2. The XTable image bundles the jar and its dependencies (Iceberg, Delta, Hudi client libraries). Version pin in the image tag (0.3.0-incubating) — do not use latest in production; XTable's Incubator release cadence means breaking changes can arrive quickly.
3. The datasetConfig YAML lives in a ConfigMap so it can be updated independently of the DAG. Changes to the YAML — new partitions, new column mapping — take effect on the next sync tick with no DAG deployment.
4. Airflow retries twice with a 2-minute backoff on transient failures (S3 throttling, JVM startup issues). Persistent failures fire a Slack alert via the standard Airflow SLA/failure callback machinery.
5. Latency is bounded by the 15-minute schedule + the ~2-4 second XTable sync time. For a table receiving one Delta commit per hour, the average freshness lag is ~7.5 minutes (half the schedule); for a table with continuous writes, freshness is ~15 minutes.

Output.

Cadence tick	Delta commits since last sync	XTable writes	Downstream visible
10:00	0	none (idempotent no-op)	last snapshot from prior tick
10:15	3	new Iceberg v2.metadata.json + manifest	new snapshot with 3 commits merged
10:30	1	new Iceberg v3.metadata.json + manifest	one more snapshot
10:45	0	none	unchanged

Rule of thumb. For cron-driven XTable sync, use Airflow (or any DAG scheduler) with max_active_runs=1, retries on transient failures, and version-pinned XTable images. Never deploy XTable with latest; Incubator releases can break.

Worked example — event-driven sync via S3 event notification on Delta commit

Detailed explanation. For hot tables where 15 minutes is too slow, wire XTable to run on-demand as soon as a new Delta commit lands. The trigger: S3 event notification fires on PutObject in _delta_log/, an SQS queue buffers events, a Lambda (or a long-running consumer) invokes XTable. Latency drops from minutes to seconds.

• Trigger. S3 event notification on _delta_log/*.json.
• Queue. SQS to absorb bursts and deduplicate.
• Runner. ECS Fargate task or Lambda (Lambda if the jar fits in 10 GB).
• Backoff. Coalesce multiple rapid commits into one sync — no need to sync every single commit if they arrive within seconds of each other.

Question. Design the event-driven XTable sync for a hot Delta table.

Input.

Component	Value
S3 event	ObjectCreated on _delta_log/*.json
Queue	SQS with 30s visibility timeout
Runner	Fargate task
Coalesce window	30 s
Target	Iceberg

Code.

# lambda_dispatcher.py — receives S3 events, coalesces, invokes XTable
import boto3
import json
import os
import time

sqs   = boto3.client("sqs")
ecs   = boto3.client("ecs")
QUEUE = os.environ["QUEUE_URL"]
COALESCE_SEC = 30

def handler(event, context):
    """Triggered by S3 PutObject on _delta_log/*.json — enqueue for sync."""
    for record in event["Records"]:
        key = record["s3"]["object"]["key"]
        if not key.endswith(".json"):
            continue
        # Extract table base path from the S3 key
        table_prefix = key.rsplit("/_delta_log/", 1)[0]
        sqs.send_message(
            QueueUrl=QUEUE,
            MessageBody=json.dumps({"table_prefix": table_prefix, "ts": time.time()}),
            MessageDeduplicationId=table_prefix,   # coalesce per table
            MessageGroupId=table_prefix,
        )

# consumer.py — long-running loop that drains SQS and launches Fargate tasks
import boto3
import json
import os
import time

sqs   = boto3.client("sqs")
ecs   = boto3.client("ecs")
QUEUE = os.environ["QUEUE_URL"]
CLUSTER = os.environ["ECS_CLUSTER"]
COALESCE_SEC = 30

def drain_and_launch():
    last_launch: dict[str, float] = {}   # table_prefix -> ts

    while True:
        resp = sqs.receive_message(
            QueueUrl=QUEUE,
            MaxNumberOfMessages=10,
            WaitTimeSeconds=5,
        )
        for msg in resp.get("Messages", []):
            body = json.loads(msg["Body"])
            table_prefix = body["table_prefix"]

            now = time.time()
            if now - last_launch.get(table_prefix, 0) < COALESCE_SEC:
                # Recent sync launched for this table; delete this event
                sqs.delete_message(QueueUrl=QUEUE, ReceiptHandle=msg["ReceiptHandle"])
                continue

            # Launch XTable Fargate task
            ecs.run_task(
                cluster=CLUSTER,
                taskDefinition="xtable-sync:1",
                overrides={
                    "containerOverrides": [{
                        "name": "xtable",
                        "environment": [
                            {"name": "TABLE_PREFIX", "value": table_prefix},
                        ],
                    }],
                },
                launchType="FARGATE",
            )
            last_launch[table_prefix] = now
            sqs.delete_message(QueueUrl=QUEUE, ReceiptHandle=msg["ReceiptHandle"])

# S3 event notification config (bucket-level)
QueueConfigurations:
  - Id: delta-commit-notify
    QueueArn: arn:aws:sqs:us-east-1:1234:xtable-sync-queue.fifo
    Events:
      - s3:ObjectCreated:*
    Filter:
      Key:
        FilterRules:
          - Name: suffix
            Value: .json
          - Name: prefix
            Value: warehouse/

Step-by-step explanation.

1. S3 event notification on _delta_log/*.json fires within seconds of any Delta commit. The event carries the object key; the Lambda dispatcher extracts the table base path (everything before /_delta_log/).
2. The message is enqueued into an SQS FIFO queue with MessageDeduplicationId = table_prefix — SQS FIFO auto-deduplicates messages with the same ID within 5 minutes, so a burst of 100 commits on one table becomes at most one queued message.
3. The consumer maintains an in-memory last_launch timestamp per table_prefix. If a message arrives within 30 seconds of the last launch for that table, it's dropped — the coalesce window ensures a burst of commits produces one XTable run, not one per commit.
4. Each XTable run is a fresh Fargate task with the datasetConfig YAML fetched from the config bucket. Task lifetime is bounded by the sync itself (~2-4 seconds) plus Fargate cold-start (~30 seconds). At-scale, warm consumer pools can eliminate the cold-start hit.
5. The result: end-to-end latency from Delta commit → Iceberg metadata is ~5-10 seconds in the warm-path, ~30-60 seconds cold. This is dramatically better than 15-minute cron and pays for itself for tables serving latency-sensitive readers (Snowflake dashboards, Trino ad-hoc queries).

Output.

Path	Latency	Cost per commit	When to use
Cron every 15 min	0-15 min	~$0.001 (JVM start)	cold-warm tables
Event-driven Fargate	5-30 s	~$0.003 (Fargate task)	hot tables
Event-driven Lambda	2-5 s	~$0.0002	very hot small tables
Hybrid	as needed	mixed	production default

Rule of thumb. For event-driven XTable sync, always coalesce within a ~30-second window — a Delta writer that flushes every 5 seconds does not need XTable to run every 5 seconds. Coalescing is the difference between "operationally cheap" and "expensive."

Worked example — the feature-preservation matrix in practice

Detailed explanation. Some features round-trip; some don't. A senior architect never claims otherwise. Walk through six real feature scenarios: schema ADD COLUMN, RENAME COLUMN, Delta CHECK constraint, Iceberg positional delete, Hudi merge-on-read commit, and historical snapshot access. For each, state whether the round-trip preserves the feature or drops it.

• Scenario 1. Delta writer adds a new column. Iceberg mirror sees the new column.
• Scenario 2. Delta writer renames a column with columnMapping.mode = id. Iceberg mirror sees the rename (not drop-add).
• Scenario 3. Delta writer adds a CHECK constraint. Iceberg mirror silently drops it.
• Scenario 4. Iceberg writer applies a positional delete. Delta mirror represents it as add-then-remove.
• Scenario 5. Hudi writer emits a MoR delta log. Iceberg mirror sees the current logical state (as if compacted).
• Scenario 6. Iceberg time-travel to snapshot N-5. Delta mirror only knows about the most-recent XTable sync.

Question. Build the feature-preservation matrix and mark each scenario as safe / caveat / not-preserved.

Input.

Feature	Source format	Target format
ADD COLUMN	any	any
RENAME COLUMN	Iceberg, Delta w/ id-mapping	any
CHECK constraint	Delta	Iceberg / Hudi
Positional delete	Iceberg V2	Delta / Hudi
MoR delta log	Hudi	Iceberg / Delta
Time-travel to old snapshot	any	any

Code.

Feature-preservation matrix (rows = feature, cols = round-trip outcome)

FEATURE                       | ICE→DELTA | ICE→HUDI | DELTA→ICE | DELTA→HUDI | HUDI→ICE | HUDI→DELTA
------------------------------|-----------|----------|-----------|------------|----------|------------
Schema ADD COLUMN             | ✅        | ✅       | ✅        | ✅         | ✅       | ✅
Schema RENAME COLUMN          | ✅ (id)   | ⚠️ (name)| ✅ (id)   | ⚠️ (name)  | ⚠️       | ⚠️
Delta CHECK constraint        | n/a       | n/a      | ❌ drop   | ❌ drop    | n/a      | n/a
Delta GENERATED column        | n/a       | n/a      | ❌ drop   | ❌ drop    | n/a      | n/a
Iceberg positional delete     | ⚠️ rewrite| ⚠️ rewrite| n/a      | n/a        | n/a      | n/a
Hudi MoR delta files          | n/a       | n/a      | n/a       | n/a        | ⚠️ view  | ⚠️ view
Historical snapshots          | ❌        | ❌       | ❌        | ❌         | ❌       | ❌
Partition transforms (Y/M/D)  | ✅        | ⚠️ id    | ✅        | ⚠️ id      | ⚠️       | ⚠️
Column stats                  | ✅        | ✅       | ✅        | ✅         | ✅       | ✅

# Programmatic drift-detector — compares source + target row counts
from pyiceberg.catalog import load_catalog
from deltalake import DeltaTable

def check_row_count_parity(iceberg_table: str, delta_path: str) -> tuple[int, int, bool]:
    ice = load_catalog("hadoop", warehouse="s3://data/").load_table(iceberg_table)
    ice_rows = sum(f.record_count for f in ice.scan().plan_files())

    dt = DeltaTable(delta_path)
    delta_rows = sum(int(f["size"] or 0) for f in dt.get_add_actions().to_pylist())
    # simpler: use dt.files_by_partitions() + row_group stats
    delta_rows = dt.to_pyarrow_dataset().count_rows()

    return ice_rows, delta_rows, ice_rows == delta_rows

Step-by-step explanation.

1. Schema ADD COLUMN round-trips cleanly across all three formats. New columns are always additive; every format supports schema evolution with a new column. Existing Parquet files without the column read as NULL in that column.
2. RENAME COLUMN round-trips only when column-ID mapping is enabled on the target. For Delta, set delta.columnMapping.mode = id. For Hudi, the concept is looser and the rename is represented as a schema swap; readers may see the old name in older Parquet footers.
3. Delta CHECK constraints and GENERATED columns are Delta-only features. XTable silently drops them from the Iceberg/Hudi mirror. If you rely on CHECK for data quality, Delta must remain the source-of-truth and the mirror must be read-only.
4. Iceberg positional deletes (V2 spec) are the trickiest case. XTable represents them in Delta/Hudi as remove-then-add rewrites — larger metadata footprint but semantically equivalent for readers. Rewrites are logical, not physical (no Parquet is touched).
5. Historical snapshots are the single feature no XTable version preserves. Time-travel is a per-format concept. If Iceberg holds 50 snapshots, the Delta mirror only knows about the most recent XTable sync as version 0 and every subsequent sync as version N.

Output.

Feature	Round-trip outcome	Recommended posture
ADD COLUMN	✅ safe	trust the mirror
RENAME COLUMN (with id mapping)	✅ safe	enable columnMapping.mode=id
CHECK constraint	❌ dropped	keep source authoritative
GENERATED column	❌ dropped	keep source authoritative
Positional delete	⚠️ rewrite	acceptable; metadata slightly larger
MoR delta log	⚠️ view-of-current	acceptable for snapshot readers
Historical snapshot	❌ dropped	do not rely on cross-format time-travel

Rule of thumb. Every XTable production deployment maintains a per-table "does-not-round-trip" note. If a feature on the "no" list is business-critical, that table's source format is the authoritative writer, and the mirrors are read-only. Never rely on a mirror to enforce something the mirror format cannot express.

Senior interview question on XTable round-trip

A senior interviewer might ask: "Your team writes Iceberg from Spark, but the data-science group needs a Delta view for MLflow. Design the XTable sync pipeline including cadence, config, feature-preservation caveats, and how you'd detect if the two format views ever drift out of sync."

Solution Using event-driven sync with a drift-detection cronjob

# 1. XTable datasetConfig — Iceberg source, Delta target with id-mapping
sourceFormat: ICEBERG
targetFormats:
  - DELTA
datasets:
  - tableBasePath: s3://data/ml/features
    tableDataPath: s3://data/ml/features
    tableName: ml_features
    partitionSpec:
      - partitionFieldName: feature_date
        transformType: DAY
    tableProperties:
      delta.columnMapping.mode: id
      delta.minReaderVersion: '2'
      delta.minWriterVersion: '5'

# 2. Event-driven sync — SQS-triggered Fargate task
import subprocess

def handle_sqs_message(msg: dict) -> None:
    table_prefix = msg["table_prefix"]
    subprocess.run([
        "java", "-jar", "/opt/xtable/utilities-bundled.jar",
        "--datasetConfig", "/config/ml-features.yaml",
        "--hadoopConfig",  "/config/core-site.xml",
    ], check=True)

# 3. Drift-detection cronjob — runs hourly; compares row counts and file sets
from pyiceberg.catalog import load_catalog
from deltalake import DeltaTable
import boto3
import json
import sys

def check_drift(iceberg_id: str, delta_path: str) -> dict:
    ice = load_catalog("hadoop", warehouse="s3://data/").load_table(iceberg_id)
    dt  = DeltaTable(delta_path)

    ice_files = {f.file_path for f in ice.scan().plan_files()}
    delta_files = set(dt.file_uris())

    only_in_iceberg = ice_files - delta_files
    only_in_delta   = delta_files - ice_files

    ice_rows   = ice.scan().to_arrow().num_rows
    delta_rows = dt.to_pyarrow_dataset().count_rows()

    return {
        "iceberg_files":   len(ice_files),
        "delta_files":     len(delta_files),
        "only_in_iceberg": len(only_in_iceberg),
        "only_in_delta":   len(only_in_delta),
        "iceberg_rows":    ice_rows,
        "delta_rows":      delta_rows,
        "healthy":         ice_files == delta_files and ice_rows == delta_rows,
    }

report = check_drift("ml.features", "s3://data/ml/features")
print(json.dumps(report, indent=2))

if not report["healthy"]:
    boto3.client("sns").publish(
        TopicArn="arn:aws:sns:us-east-1:1234:xtable-drift",
        Subject="XTable drift detected: ml.features",
        Message=json.dumps(report),
    )
    sys.exit(1)

-- 4. Downstream Databricks reads the translated Delta mirror
SELECT feature_date, feature_id, feature_value
FROM   delta.`s3://data/ml/features`
WHERE  feature_date = current_date()
LIMIT  100;

Step-by-step trace.

Layer	Component	Purpose
Producer	Spark Iceberg writer	authoritative source-of-truth
Trigger	S3 event on metadata/vN.metadata.json	fires within seconds of commit
Queue	SQS FIFO with dedupe	coalesces bursts to one sync
Sync	Fargate + XTable jar	writes _delta_log/
Consumer	Databricks / MLflow	reads Delta natively
Drift check	hourly Python job	compares file set + row count
Alert	SNS topic	pages if drift detected

After deployment, every Iceberg commit becomes a Delta commit within 5-30 seconds. The drift check catches any XTable failure or misconfiguration within an hour. Databricks consumers see Delta metadata; they never know Iceberg is upstream.

Output:

Metric	Value
End-to-end sync latency (steady state)	5-30 s
Feature drift (with id mapping)	zero
Delta metadata size overhead	~0.2% of table size
Fargate cost per sync	~$0.003
Drift-check false-positive rate	< 0.01%
CHECK constraint enforcement	Iceberg only (Delta mirror can't enforce)

Why this works — concept by concept:

• Iceberg-as-source authoritative model — one writer format is the source of truth; all mirrors are derived. This eliminates the multi-writer conflict scenario that XTable cannot arbitrate. Spark writes Iceberg; XTable mirrors to Delta; Databricks reads-only.
• Column mapping mode = id — makes RENAME COLUMN safe across the sync. Without it, a rename on the Iceberg side becomes a drop-add on the Delta side, breaking column lineage in every downstream consumer.
• Event-driven + coalesce — the ~5-30s latency is achieved by triggering on S3 events, coalescing bursts within 30s, and running a lightweight Fargate task. Cron every 15 min would cost 15× more latency.
• Drift-detection cronjob — the operational safety net. It catches misconfiguration (wrong bucket, wrong region), XTable bugs, or Iceberg-Delta version mismatches. False positives are rare because both formats reference the same Parquet.
• Cost — one Iceberg writer (authoritative), one XTable Fargate task per commit (event-driven), one drift-check cron per hour, one SNS topic. The eliminated cost is a duplicate Delta pipeline (writers, compaction jobs, cost). Net O(1) storage + O(commits) sync CPU versus O(2) writers.

Streaming
Topic — streaming
Streaming event-driven sync problems

Practice →

Design Topic — design Design problems on drift detection and consistency checks

Practice →

---

4. Integration with catalogs and engines

Translated metadata is inert until you register it — Polaris for Iceberg, Glue for Delta and Iceberg, Unity for Delta, Snowflake for Iceberg REST

The mental model in one line: apache xtable writes translated metadata files to S3, but every downstream engine discovers tables through a catalog — Apache Polaris for Iceberg REST clients, AWS Glue for Delta and Iceberg on AWS engines, Databricks Unity Catalog for Delta consumers, Snowflake's Iceberg REST catalog for the Snowflake-native path — and until you register the translated table in each catalog, the metadata sits on S3 unread; the operational reality is that XTable solves the format-translation problem but leaves you with one catalog registration per (table × target format), which is the second-order cost that senior architects budget for. Every senior data engineer running XTable at scale writes a small deployment script that registers each table in every catalog it targets.

The four catalogs that matter in 2026.

• Apache Polaris. The Snowflake-donated, ASF-incubating Iceberg REST catalog. Cloud-agnostic, multi-engine, first-class Iceberg support. XTable can auto-register translated Iceberg tables in Polaris via the catalogConfig block. Polaris is emerging as the neutral-ground catalog for teams that want Iceberg across Snowflake + Databricks + Trino + Spark.
• AWS Glue Data Catalog. The AWS-native metastore. Supports Iceberg and Delta table types. XTable can register tables via the Glue AWS SDK. Used by Athena, EMR, Redshift Spectrum, and (via foreign-catalog integration) Snowflake.
• Databricks Unity Catalog. Databricks' first-party governance-plus-catalog product. Supports Delta natively, Iceberg via Uniform. XTable can register Delta tables in Unity; Iceberg registration requires Uniform + Delta on the Databricks side, which slightly changes the story.
• Snowflake Iceberg REST Catalog. Snowflake exposes an Iceberg REST endpoint so external Iceberg clients can consume Snowflake-managed Iceberg tables. In the XTable direction, Snowflake reads translated Iceberg metadata via its CREATE ICEBERG TABLE ... CATALOG='OBJECT_STORE' mode, which points at a metadata JSON directly.

The XTable catalogConfig block.

• Purpose. Tells XTable to register the translated table in a catalog immediately after writing the metadata files.
• Shape. catalogConfig: catalogImpl: <fully-qualified-class>, catalogName: <name>, catalogOptions: {...} — the plugin model lets you target Polaris, Glue, Nessie, or a custom catalog.
• Order. Write metadata to S3 first; register second. If registration fails, the metadata is still on S3 and can be re-registered manually.
• Idempotency. Registration is idempotent — CREATE-OR-REPLACE semantics on the catalog side. Re-running XTable never breaks catalog state.

The engine-per-format reader story.

• Snowflake reads Iceberg. Two flavours: (a) CREATE ICEBERG TABLE ... CATALOG='OBJECT_STORE' METADATA_FILE_PATH='s3://.../metadata/v1.metadata.json' for direct metadata-pointer reads; (b) CATALOG='POLARIS' for a live catalog integration that follows XTable sync updates automatically.
• Databricks reads Delta. Native. Zero setup once the table is registered in Unity (or via a direct file-system path). XTable's Delta mirror shows up as DELTA in DESCRIBE HISTORY.
• Trino reads all three. Configure per-catalog connectors — iceberg-catalog, delta-catalog, hudi-catalog — pointing at Glue, Polaris, Nessie, or metadata paths. One physical Parquet dataset, three logical tables in Trino, all identical rows.
• Spark reads all three. Standard Iceberg / Delta / Hudi client libraries. Configuration is per-session catalog config.

Hidden costs of the catalog layer.

• Duplicate catalog entries. One table in three formats = three catalog entries. Data governance tools (Purview, Alation) may see this as three different assets; label them clearly.
• Metadata file proliferation. Every XTable sync writes new metadata files. Iceberg is worst-offender (one JSON + one manifest-list + one manifest per sync). Compaction / retention on the metadata directory is not automatic — enable Iceberg's remove_orphan_files or Delta's VACUUM on the metadata dirs.
• Catalog auth complexity. Four catalogs = four auth models. Polaris uses OAuth2; Glue uses IAM; Unity uses OAuth or SP; Snowflake uses key-pair or PAT. Every XTable runner needs credentials for every target catalog.
• Cross-catalog governance drift. A grant applied in Unity does not propagate to Glue. Access-control policies must be replicated (or centralised via a policy-as-code layer like Open Policy Agent).

Common interview probes on catalog integration.

• "How does the downstream engine discover the XTable-translated table?" — required answer: catalog registration in the target format's native catalog.
• "Which catalog do you pick for XTable's Iceberg output?" — depends: Polaris for multi-engine neutrality; Glue for AWS-native; Snowflake REST for Snowflake-native.
• "What's the operational cost of running three catalogs?" — cred management, governance replication, metadata retention on each.
• "Does XTable coordinate with the catalog on schema evolution?" — yes; on each sync, XTable updates the catalog entry with the new schema.

Worked example — registering an XTable-Iceberg mirror in Apache Polaris

Detailed explanation. The canonical Polaris integration: XTable writes Iceberg metadata to S3, then calls Polaris's REST API to register the table. Every subsequent sync updates the Polaris entry with the new snapshot. Downstream Snowflake, Trino, and Spark clients configured against Polaris see the fresh metadata immediately.

• Polaris deployment. Assume a Polaris service at https://polaris.internal:8181.
• XTable YAML. Includes catalogConfig with Polaris details.
• Downstream. Snowflake configured with CATALOG_INTEGRATION pointing at Polaris; Trino Iceberg connector pointing at Polaris.

Question. Write the XTable YAML with Polaris registration, plus the Snowflake and Trino downstream configs.

Input.

Component	Value
Source	Delta on s3://data/sales/orders
Target	Iceberg (with Polaris registration)
Polaris URL	https://polaris.internal:8181
Polaris namespace	sales
Table name	orders

Code.

# 1. XTable YAML — Delta source, Iceberg target with Polaris catalog registration
sourceFormat: DELTA
targetFormats:
  - ICEBERG
datasets:
  - tableBasePath: s3://data/sales/orders
    tableDataPath: s3://data/sales/orders
    tableName: sales.orders
    partitionSpec:
      - partitionFieldName: order_date
        transformType: VALUE

catalogConfig:
  catalogImpl: org.apache.iceberg.rest.RESTCatalog
  catalogName: polaris
  catalogOptions:
    uri: https://polaris.internal:8181/api/catalog
    warehouse: acme-lakehouse
    credential: ${env:POLARIS_CLIENT_ID}:${env:POLARIS_CLIENT_SECRET}
    scope: PRINCIPAL_ROLE:ALL

-- 2. Snowflake — one-time catalog integration for Polaris
CREATE CATALOG INTEGRATION polaris_catalog
  CATALOG_SOURCE = ICEBERG_REST
  TABLE_FORMAT   = ICEBERG
  REST_CONFIG    = (
    CATALOG_URI = 'https://polaris.internal:8181/api/catalog'
    WAREHOUSE   = 'acme-lakehouse'
  )
  REST_AUTHENTICATION = (
    TYPE            = OAUTH
    OAUTH_TOKEN_URI = 'https://polaris.internal:8181/api/catalog/v1/oauth/tokens'
    OAUTH_CLIENT_ID = '<snowflake-polaris-client>'
    OAUTH_CLIENT_SECRET = '<...>'
    OAUTH_ALLOWED_SCOPES = ('PRINCIPAL_ROLE:ALL')
  )
  ENABLED = TRUE;

-- Snowflake reads the XTable-translated table through Polaris
CREATE ICEBERG TABLE orders_ib
  EXTERNAL_VOLUME     = 'acme_lake'
  CATALOG             = 'polaris_catalog'
  CATALOG_NAMESPACE   = 'sales'
  CATALOG_TABLE_NAME  = 'orders';

SELECT order_date, COUNT(*) AS cnt
FROM   orders_ib
GROUP  BY order_date
ORDER  BY order_date DESC
LIMIT  10;

# 3. Trino — iceberg-polaris.properties
connector.name=iceberg
iceberg.catalog.type=rest
iceberg.rest-catalog.uri=https://polaris.internal:8181/api/catalog
iceberg.rest-catalog.warehouse=acme-lakehouse
iceberg.rest-catalog.security=oauth2
iceberg.rest-catalog.oauth2.credential=${ENV:POLARIS_CLIENT_ID}:${ENV:POLARIS_CLIENT_SECRET}
iceberg.rest-catalog.oauth2.scope=PRINCIPAL_ROLE:ALL

Step-by-step explanation.

1. The catalogConfig block tells XTable to instantiate a RESTCatalog client after writing the translated Iceberg metadata files. XTable then calls Polaris's registerTable (or updateTable on subsequent syncs) via the Iceberg REST protocol.
2. On the first sync, Polaris creates the table entry under namespace sales with name orders, pointing at the metadata JSON on S3. On subsequent syncs, XTable updates the entry to point at the new metadata file.
3. Snowflake's CATALOG INTEGRATION is a one-time setup per catalog instance, not per table. Once created, every CREATE ICEBERG TABLE ... CATALOG = 'polaris_catalog' uses the same OAuth token flow.
4. Trino's Iceberg connector, configured with iceberg.catalog.type=rest, queries Polaris for the table entry, resolves the metadata JSON pointer, reads the Iceberg metadata, and plans the scan against the shared Parquet files. Same data, three engines, one catalog.
5. Grants are managed centrally in Polaris — a role like sales-read granted on the sales namespace propagates to every engine reading through the same Polaris instance. This is a governance win over per-engine catalogs.

Output.

Downstream engine	Catalog	Read path	Grant propagation
Snowflake	polaris_catalog integration	CREATE ICEBERG TABLE ... CATALOG='polaris_catalog'	via Polaris role
Trino	iceberg connector	SELECT * FROM iceberg.sales.orders	via Polaris role
Spark	Iceberg REST catalog	spark.catalog("polaris").sales.orders	via Polaris role
Databricks	Unity Catalog federation	(federated Iceberg via Uniform)	separate Unity grants

Rule of thumb. For any XTable Iceberg deployment serving more than two engines, register with Polaris (or Nessie) as the neutral REST catalog. It centralises grants and gives every engine one URL to talk to; per-engine catalogs (Glue, Unity, Snowflake internal) are fine for single-engine scenarios but become governance nightmares at 3+ engines.

Worked example — Delta mirror registered in AWS Glue for Athena and Redshift Spectrum

Detailed explanation. For AWS-centric deployments, Glue is often the pragmatic default. XTable writes Delta metadata; Glue is registered with the Delta table type; Athena and Redshift Spectrum discover the table via Glue and read the shared Parquet directly. Walk through the registration.

• Source. Iceberg on s3://data/customers/.
• Target. Delta.
• Catalog. AWS Glue Data Catalog.
• Downstream. Athena for ad-hoc; Redshift Spectrum for scheduled reports.

Question. Write the XTable YAML for Glue registration and the Athena MSCK REPAIR / Redshift Spectrum external table setup.

Input.

Component	Value
Source	Iceberg on s3://data/customers
Target	Delta
Glue database	analytics
Glue table	customers_delta

Code.

# 1. XTable YAML with Glue catalog registration
sourceFormat: ICEBERG
targetFormats:
  - DELTA
datasets:
  - tableBasePath: s3://data/customers
    tableDataPath: s3://data/customers
    tableName: analytics.customers_delta
    partitionSpec:
      - partitionFieldName: signup_date
        transformType: DAY
    tableProperties:
      delta.columnMapping.mode: id

catalogConfig:
  catalogImpl: org.apache.iceberg.aws.glue.GlueCatalog
  catalogName: glue
  catalogOptions:
    warehouse: s3://data/
    glue.region: us-east-1

# 2. Manual Glue registration for Delta (if XTable's Glue integration
#    doesn't yet handle the delta table type — verify against your XTable
#    version's release notes)
import boto3

glue = boto3.client("glue", region_name="us-east-1")

glue.create_table(
    DatabaseName="analytics",
    TableInput={
        "Name": "customers_delta",
        "TableType": "EXTERNAL_TABLE",
        "Parameters": {
            "table_type":       "DELTA",
            "spark.sql.sources.provider": "delta",
            "delta.location":   "s3://data/customers",
            "classification":   "delta",
        },
        "StorageDescriptor": {
            "Location":     "s3://data/customers",
            "InputFormat":  "org.apache.hadoop.hive.ql.io.SymlinkTextInputFormat",
            "OutputFormat": "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat",
            "SerdeInfo": {
                "SerializationLibrary":
                    "org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe",
            },
        },
    },
)

-- 3. Athena — reads the Delta table registered in Glue
SELECT signup_date, COUNT(*) AS new_customers
FROM   analytics.customers_delta
WHERE  signup_date >= DATE '2026-07-01'
GROUP  BY signup_date
ORDER  BY signup_date;

-- 4. Redshift Spectrum — external schema pointing at Glue
CREATE EXTERNAL SCHEMA IF NOT EXISTS analytics
  FROM DATA CATALOG
  DATABASE 'analytics'
  IAM_ROLE 'arn:aws:iam::1234:role/redshift-spectrum-glue';

SELECT signup_date, COUNT(*)
FROM   analytics.customers_delta
GROUP  BY signup_date;

Step-by-step explanation.

1. The XTable YAML's catalogConfig uses GlueCatalog; XTable calls the Glue AWS SDK to register the table. For Delta specifically, older XTable versions may not yet auto-register the Delta table type — the boto3 fallback in step 2 covers that case.
2. Glue registration includes the table_type: DELTA parameter — this tells Athena to use the Delta reader path instead of the default Hive path. Without this parameter, Athena tries to read Parquet directly and misses Delta's delete/update semantics.
3. Athena queries analytics.customers_delta and Delta's transaction-log-aware reader handles the file selection correctly (respecting remove actions from later commits). Every ordinary SELECT "just works."
4. Redshift Spectrum uses an external schema mapped to the Glue database. Once the schema is created, Spectrum queries the same table via the same Glue entry. AWS's Delta support in Spectrum is somewhat newer than Iceberg — pin your cluster version if using this path.
5. Grants are managed in Lake Formation (if enabled) or via IAM policies on the Glue table. A grant to analytics.customers_delta propagates to both Athena and Redshift Spectrum consumers automatically.

Output.

Engine	Reads via	Grant surface	Delta feature coverage
Athena	Glue → Delta reader	Lake Formation / IAM	full Delta reader
Redshift Spectrum	Glue → Delta reader	Lake Formation / IAM	most reads (some feature lag)
EMR (Spark on EMR)	Glue → delta-spark	IAM	full Delta reader/writer
Snowflake (via Iceberg export)	not via Glue directly	—	use Iceberg mirror instead

Rule of thumb. For AWS-only deployments, Glue is the pragmatic default catalog for both Iceberg and Delta mirrors. Add Polaris on top when you need to serve Snowflake, GCP, or Azure consumers — Polaris centralises grants across clouds; Glue is AWS-only.

Worked example — Snowflake reading Delta directly via translated Iceberg REST

Detailed explanation. Snowflake's Iceberg REST support is the cleanest path for Snowflake reading an XTable-translated Iceberg mirror. Instead of Snowflake trying to read Delta (which requires external tables with limited feature coverage), XTable emits Iceberg metadata and Snowflake reads that. The Iceberg REST catalog is the connective tissue.

• Source. Delta on Databricks writing s3://data/products/.
• Target. Iceberg with Polaris registration.
• Snowflake reads. Via Polaris catalog integration.

Question. Write the end-to-end config: XTable YAML + Polaris registration + Snowflake catalog integration + SELECT.

Input.

Component	Value
Databricks producer	Delta writer on s3://data/products
XTable sync cadence	event-driven (Databricks table listener)
Polaris table id	catalog.products.products
Snowflake table	products_ib

Code.

# 1. XTable YAML — Delta source, Iceberg target, Polaris registration
sourceFormat: DELTA
targetFormats:
  - ICEBERG
datasets:
  - tableBasePath: s3://data/products
    tableDataPath: s3://data/products
    tableName: products.products
    partitionSpec:
      - partitionFieldName: category
        transformType: VALUE

catalogConfig:
  catalogImpl: org.apache.iceberg.rest.RESTCatalog
  catalogName: polaris
  catalogOptions:
    uri: https://polaris.internal:8181/api/catalog
    warehouse: acme-lakehouse
    credential: ${env:POLARIS_CLIENT_ID}:${env:POLARIS_CLIENT_SECRET}

-- 2. Snowflake — one-time catalog integration
CREATE CATALOG INTEGRATION polaris_catalog
  CATALOG_SOURCE = ICEBERG_REST
  TABLE_FORMAT   = ICEBERG
  REST_CONFIG    = (
    CATALOG_URI = 'https://polaris.internal:8181/api/catalog'
    WAREHOUSE   = 'acme-lakehouse'
  )
  REST_AUTHENTICATION = (
    TYPE                = OAUTH
    OAUTH_TOKEN_URI     = 'https://polaris.internal:8181/api/catalog/v1/oauth/tokens'
    OAUTH_CLIENT_ID     = '<snowflake-client>'
    OAUTH_CLIENT_SECRET = '<...>'
    OAUTH_ALLOWED_SCOPES = ('PRINCIPAL_ROLE:ALL')
  )
  ENABLED = TRUE;

-- 3. Snowflake — external volume for S3 access
CREATE EXTERNAL VOLUME acme_lake
  STORAGE_LOCATIONS = ((
    NAME = 'acme-s3'
    STORAGE_PROVIDER = 'S3'
    STORAGE_BASE_URL = 's3://data/'
    STORAGE_AWS_ROLE_ARN = 'arn:aws:iam::1234:role/snowflake-lake'
  ));

-- 4. Snowflake — Iceberg table pointing at Polaris
CREATE ICEBERG TABLE products_ib
  EXTERNAL_VOLUME     = 'acme_lake'
  CATALOG             = 'polaris_catalog'
  CATALOG_NAMESPACE   = 'products'
  CATALOG_TABLE_NAME  = 'products';

-- 5. Query — Snowflake reads Delta-source data as if it were Iceberg
SELECT category, COUNT(*) AS n_products, AVG(price_cents) AS avg_price
FROM   products_ib
GROUP  BY category
ORDER  BY n_products DESC;

# 6. Databricks-side XTable trigger — post-commit listener via Delta lineage
# (illustrative; requires Databricks Unity Catalog lineage events)
def on_delta_commit(event):
    if event["table_path"] == "s3://data/products":
        # Trigger the XTable sync
        subprocess.run(["kubectl", "create", "-f", "xtable-products-job.yaml"])

Step-by-step explanation.

1. Databricks writes Delta to s3://data/products/. The Databricks table event stream (or a Unity Catalog lineage subscriber) fires on every commit; a small dispatcher launches the XTable job.
2. XTable reads the new Delta transaction, writes an incremented Iceberg metadata/vN.metadata.json + manifest, and calls Polaris to update the products.products table entry to point at the new metadata JSON.
3. Snowflake's products_ib table is defined once with CATALOG='polaris_catalog'. On every SELECT, Snowflake calls Polaris via the REST API, gets the current metadata pointer, fetches the metadata + manifest, and plans the scan.
4. The freshness lag from a Databricks Delta commit to a Snowflake SELECT sees the new data is bounded by: Delta commit → XTable trigger (~seconds) + XTable sync (~2-4s) + Polaris update (~ms) + Snowflake's metadata cache (~30s default). Total: ~30-60s in the warm path.
5. Alternative Snowflake-side setup: CATALOG='OBJECT_STORE' with METADATA_FILE_PATH pointing at a specific metadata JSON — this avoids the Polaris hop but requires manual re-pointing on every sync (or a periodic ALTER ICEBERG TABLE ... REFRESH).

Output.

Layer	Latency	Ownership
Delta commit → XTable trigger	seconds (event-driven)	XTable-Lambda
XTable sync	2-4 s	XTable jar
Polaris update	ms	Iceberg REST
Snowflake metadata cache	30 s (default)	Snowflake
End-to-end freshness	30-60 s	full path

Rule of thumb. For Snowflake reading a Delta source via XTable, always go through Polaris (not direct METADATA_FILE_PATH). The Polaris path auto-follows XTable's updates; the direct-metadata path requires a manual ALTER TABLE ... REFRESH on every sync — an operational tax you don't want.

Senior interview question on catalog integration

A senior interviewer might ask: "You're running XTable to translate an Iceberg source into Delta and Hudi mirrors. Snowflake will read the Iceberg source, Databricks will read the Delta mirror, and a legacy Trino cluster with only the Hudi connector will read the Hudi mirror. Design the catalog layer: which catalog per engine, which grants live where, and how you'd unify governance across the three."

Solution Using Polaris as the neutral catalog for Iceberg, Unity for Delta, and Trino Hudi connector with per-catalog RBAC

# 1. XTable YAML — Iceberg source, Delta + Hudi targets, Polaris + Unity registration
sourceFormat: ICEBERG
targetFormats:
  - DELTA
  - HUDI
datasets:
  - tableBasePath: s3://data/orders
    tableDataPath: s3://data/orders
    tableName: sales.orders
    partitionSpec:
      - partitionFieldName: order_date
        transformType: DAY
    tableProperties:
      delta.columnMapping.mode: id

# Iceberg source is registered in Polaris (its own catalog)
catalogConfig:
  catalogImpl: org.apache.iceberg.rest.RESTCatalog
  catalogName: polaris
  catalogOptions:
    uri: https://polaris.internal:8181/api/catalog
    warehouse: acme-lakehouse
    credential: ${env:POLARIS_CLIENT_ID}:${env:POLARIS_CLIENT_SECRET}

# 2. Post-XTable step — register Delta mirror in Unity Catalog via SDK
from databricks.sdk import WorkspaceClient
from databricks.sdk.service.catalog import TableInfo, TableType, DataSourceFormat

w = WorkspaceClient()
w.tables.create(
    catalog_name="production",
    schema_name="sales",
    name="orders_delta",
    table_type=TableType.EXTERNAL,
    data_source_format=DataSourceFormat.DELTA,
    storage_location="s3://data/orders",
)

# 3. Trino Hudi connector — hudi.properties
connector.name=hudi
hive.metastore=glue
hive.metastore.glue.region=us-east-1
hive.s3.aws-access-key=${ENV:AWS_ACCESS_KEY_ID}
hive.s3.aws-secret-key=${ENV:AWS_SECRET_ACCESS_KEY}

-- 4. Trino side — register the Hudi mirror in Glue
-- (Trino's Hudi connector reads through the Hive metastore / Glue)
CREATE TABLE hudi.sales.orders_hudi (
    id BIGINT,
    customer_id BIGINT,
    total_cents BIGINT,
    order_date DATE
)
WITH (
    format = 'ORC',
    external_location = 's3://data/orders',
    partitioned_by = ARRAY['order_date']
);

# 5. Governance layer — OPA policy replicated across all three catalogs
# open-policy-agent/policies/sales-orders.rego
package sales.orders

default allow = false

allow {
    input.principal.role == "sales-analyst"
    input.action in {"select"}
}

allow {
    input.principal.role == "sales-admin"
    input.action in {"select", "insert", "update", "delete"}
}

Step-by-step trace.

Consumer	Catalog	Table id	Grants
Snowflake / Trino	Polaris	polaris:sales.orders (Iceberg)	Polaris principal roles
Databricks	Unity	production.sales.orders_delta	Unity account-level grants
Legacy Trino	Glue (via Hudi connector)	glue:sales.orders_hudi	Glue Lake Formation
OPA policy layer	central	one policy per table	replicated to each catalog

After deployment, one Iceberg source table produces two mirrors (Delta + Hudi), each registered in a different catalog. Every engine reads through its native catalog path. A central OPA policy per table is replicated (via CI) into Polaris principals, Unity grants, and Lake Formation grants so a single logical role (sales-analyst) resolves consistently across all three engines.

Output:

Metric	Value
Physical Parquet copies	1
Catalog registrations per table	3 (Polaris + Unity + Glue)
End-to-end sync latency	5-30 s
Grant-drift risk (without OPA)	high
Grant-drift risk (with OPA replication CI)	low
Ops overhead per table	one YAML + one register call per catalog

Why this works — concept by concept:

• Polaris as the Iceberg neutral catalog — Snowflake and Trino both talk to Polaris via Iceberg REST. One grant surface for the Iceberg source; two engines read through it.
• Unity as the Delta-native catalog — Databricks needs a Unity entry for full governance features (data lineage, Delta Sharing). Registering the Delta mirror in Unity gives Databricks its native experience.
• Glue as the Hudi legacy path — the legacy Trino cluster uses Glue via the Hudi connector because that's what's already deployed. Migrating this cluster to Polaris is a Phase 2 goal, not a Phase 1 blocker.
• OPA policy replication — the governance layer that keeps three catalogs in sync. One Rego policy per table; CI replicates it to Polaris principals, Unity grants, and Lake Formation grants. Grants never drift because they're generated from one source.
• Cost — one Iceberg writer, one XTable sync per commit, three catalog registrations per table, one OPA CI pipeline. The eliminated cost is per-catalog manual grant maintenance and the "who accidentally granted access to X" audit incidents. Net O(1) grant maintenance versus O(3) manual grants per table per role.

Design
Topic — design
Design problems on catalog federation and governance

Practice →

SQL Topic — sql SQL cross-catalog query and join problems

Practice →

---

5. When to use XTable vs when to standardise — and interview signals

XTable is a bridge, not a destination — pick it when the standardisation cost exceeds the bridge cost this quarter

The mental model in one line: apache xtable is a pragmatic bridge for the transition period when your data platform genuinely runs multiple open table formats — but the long-term architectural play is almost always to standardise on one format (usually Iceberg + Polaris in 2026) and retire XTable — so every deployment should include an explicit retirement path, and every senior interview should probe whether you know when the bridge stops earning its keep. XTable earns its place when the cost of migrating every producer to one format this quarter exceeds the cost of maintaining a translator; when that ratio flips, XTable becomes technical debt.

The four scenarios where XTable earns its keep in 2026.

• Multi-vendor engine mix. Databricks (Delta) + Snowflake (Iceberg) + Trino (Iceberg or Hudi) on shared S3. Migrating any single producer to a common format takes months; XTable buys immediate compatibility.
• Legacy Hudi pipelines. A team invested in Hudi merge-on-read semantics for a specific streaming workload. Rewriting to Iceberg is a multi-quarter project; XTable emits an Iceberg mirror for the warehouse in the meantime.
• M&A integration. Two companies merge, one on Delta, one on Iceberg. XTable is the "acquire the data first, migrate later" bridge.
• Vendor-locked producers. A SaaS tool writes Delta because Delta is what it supports; the rest of your stack is Iceberg. XTable translates the SaaS output.

The three scenarios where you should NOT use XTable.

• Greenfield deployment. No existing format constraints. Pick Iceberg + Polaris and skip XTable entirely; add it later if a second format arrives.
• Single-engine, single-format shop. Everyone on Databricks + Delta and no reader outside that stack. XTable adds cost with no benefit; keep it out.
• Feature-dependent workloads. Your business logic relies on Delta CHECK constraints, Iceberg positional deletes, or Hudi MoR. XTable's mirror cannot enforce these; the source format must be the sole reader, and XTable delivers no value.

The migration path from XTable-bridge to standardised format.

• Phase 1 (Bridge). XTable translates every source to every needed target. Every engine reads its native format. Duration: 1-3 quarters.
• Phase 2 (Consolidate producers). Migrate producers one at a time to the target format (usually Iceberg). Each migrated producer drops from XTable's YAML. Duration: 2-6 quarters, dependent on producer complexity.
• Phase 3 (Retire mirrors). Once no producer writes a non-target format, decommission the XTable jobs, delete the sibling metadata directories, retire the extra catalog entries.
• Phase 4 (Target state). One format, one catalog (Polaris), one physical dataset per table.

The risk register — what breaks XTable deployments in production.

• Feature drift. A team enables Delta CHECK constraints and forgets that the Iceberg mirror doesn't enforce them. Downstream Iceberg readers see rows the CHECK would have rejected. Fix: policy that the mirror is read-only for CHECK-guarded tables.
• Sync-cadence blind spot. A hot table syncs every 15 min via cron; a Snowflake dashboard querying the Iceberg mirror sees 15-minute-old data even though the source is fresh. Fix: event-driven for tables with SLA < 15 min.
• Metadata bloat. Every sync writes new Iceberg metadata + manifest + manifest-list. Over months, metadata/ directory has thousands of files. Reads slow down as the reader downloads more manifests. Fix: schedule remove_orphan_files or Iceberg metadata compaction weekly.
• Catalog cred drift. Polaris client secret rotates; XTable YAML in ConfigMap doesn't. Sync fails silently until someone notices Snowflake data is stale. Fix: use IAM roles or short-lived tokens where possible; alert on sync failures loudly.
• Version mismatch. XTable Incubator releases can break YAML compatibility. Pinning latest bites eventually. Fix: version-pin XTable images; test upgrades in staging.

Senior interview signals — what listeners score.

• Do you name XTable as "a metadata-only translator between Iceberg, Delta, and Hudi" without prompting? — required.
• Do you recognise it as the renamed OneTable, donated to Apache in 2024? — trivia bonus.
• Do you push back on "we should use XTable forever" with the "bridge, not destination" argument? — senior signal.
• Do you name at least three non-round-trippable features without prompting? — senior signal.
• Do you describe the migration path from bridge to standardised format as an explicit multi-quarter plan? — senior signal.

Worked example — the "bridge cost vs migration cost" ROI calculation

Detailed explanation. Every senior architect must be able to defend the XTable-bridge decision with numbers. The comparison is: (a) cost of running XTable indefinitely versus (b) cost of migrating every non-target producer to the target format. The break-even is usually 6-18 months out; XTable wins the short-term ROI, migration wins the long-term. Walk through the calculation for a 50-producer platform.

• Producers on Delta. 30 (Databricks native).
• Producers on Iceberg. 15 (Spark, Flink).
• Producers on Hudi. 5 (legacy Uber-style streaming).
• Target state. Iceberg + Polaris.
• Bridge cost. XTable sync ops + catalog dupes.
• Migration cost. Engineer-hours to rewrite each producer.

Question. Compare the annual bridge cost vs the one-time migration cost, and identify the break-even month.

Input.

Cost dimension	XTable bridge	Full migration
Compute (annual)	~$5k (XTable runners)	$0 (once done)
S3 requests (annual)	~$10k	$0
Catalog dupes (annual)	~$20k (Glue + Polaris + Unity overhead)	$0
Engineer maintenance (annual)	~$50k (0.5 FTE)	~$10k (single-format ops)
Feature drift risk incidents (annual)	~$30k (avg)	$0
Migration engineering (one-time)	$0	~$500k (50 producers × 40h × $250/h)
Downstream consumer changes (one-time)	$0	~$100k
Total year-1 cost	~$115k	~$610k
Total year-2 cost	~$115k	~$10k

Code.

# ROI break-even calculator
def bridge_vs_migrate_breakeven(
    annual_bridge_cost:    float,
    annual_migrated_cost:  float,
    one_time_migration:    float,
) -> tuple[int, float]:
    """Return (break-even month, cumulative-cost at break-even)."""
    m_annual = annual_bridge_cost - annual_migrated_cost
    if m_annual <= 0:
        return (0, 0.0)   # migration is never cheaper on ops alone
    months = one_time_migration / (m_annual / 12.0)
    return (int(months + 0.5), one_time_migration + (annual_migrated_cost * months / 12.0))


bp = 115_000
mp = 10_000
mig = 600_000

months, breakeven_cost = bridge_vs_migrate_breakeven(bp, mp, mig)
print(f"Break-even at month {months}; cumulative cost ~${breakeven_cost:,.0f}")
# → Break-even at month 69; cumulative cost ~$657,500

# Sensitivity — what if XTable maintenance is 1.0 FTE (not 0.5)?
bp_high = 165_000
months_h, _ = bridge_vs_migrate_breakeven(bp_high, mp, mig)
print(f"Break-even (high maintenance): month {months_h}")
# → Break-even (high maintenance): month 46

Step-by-step explanation.

1. The bridge cost is dominated by two categories: engineer maintenance (~$50k/year for 0.5 FTE keeping XTable YAMLs, cron jobs, and catalog registrations aligned) and feature-drift-risk incidents (~$30k/year for one or two data-quality incidents per year caused by mirror limitations).
2. The migration cost is dominated by the engineering to rewrite 50 producers (~40 hours each) plus updating downstream consumers where the format-change breaks assumptions. At a $250/h all-in engineering rate, that's ~$500k of engineering.
3. The break-even is roughly 69 months for this profile — 5.75 years. That's a long time. The naive interpretation "just run XTable forever" is not wrong when the migration cost is high and the bridge cost is low.
4. Sensitivity matters. If bridge maintenance grows to 1.0 FTE (~$100k/year), break-even shrinks to 46 months. If a feature-drift incident costs $200k instead of $30k, break-even collapses to under 24 months. Every architect should compute this per their org's cost profile.
5. The correct senior answer is nuanced: "XTable earns its keep for a well-defined multi-year bridge period; the retire-XTable plan is opportunistic — migrate producers as they change anyway. Do not commit to a big-bang migration project purely to retire XTable; the ROI does not usually justify it."

Output.

Scenario	Break-even month	Recommendation
0.5 FTE maintenance, low incident cost	69 months	run XTable indefinitely; opportunistic migration
1.0 FTE maintenance, low incident cost	46 months	migrate within 2-3 years
0.5 FTE maintenance, high incident cost	22 months	migrate ASAP if feature drift is happening
Producers already on target format	N/A	no bridge needed

Rule of thumb. Compute the XTable-vs-migrate break-even for your org before defending either path. The naive "we should migrate to one format" answer often loses to the "we should keep the bridge for 3+ years and migrate opportunistically" answer once you plug real numbers in.

Worked example — retiring XTable one producer at a time

Detailed explanation. Once you decide to migrate, the pattern is producer-by-producer. Never migrate consumers before their producer; consumers can keep reading through XTable-produced mirrors as long as the mirror exists. Walk through the retirement of one Delta producer.

• Producer. A Databricks structured-streaming job writing Delta to s3://data/inventory/.
• Target format. Iceberg via Spark.
• Consumers. Snowflake (reads Iceberg mirror), Trino (reads Iceberg mirror), MLflow (reads Delta mirror).
• Retirement. Rewrite producer to Iceberg; MLflow now reads Iceberg via a compatibility layer; drop XTable for this table.

Question. Sequence the retirement of one XTable-managed producer with zero-downtime for consumers.

Input.

Phase	Action
Before	Producer writes Delta; XTable mirrors to Iceberg + Hudi
Phase 1	Set up parallel Iceberg producer; XTable mirrors from both
Phase 2	Cut over consumers to read the new authoritative Iceberg
Phase 3	Stop the Delta producer
Phase 4	Drop XTable job; retire Delta + Hudi metadata dirs

Code.

# 1. Original state — Delta source, Iceberg + Hudi targets
sourceFormat: DELTA
targetFormats:
  - ICEBERG
  - HUDI
datasets:
  - tableBasePath: s3://data/inventory
    tableDataPath: s3://data/inventory
    tableName: inventory.inventory

# 2. Phase 1 — write a shadow Iceberg producer in parallel
#    (both write to the same shared Parquet, but only one owns the metadata)
from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .config("spark.sql.catalog.iceberg", "org.apache.iceberg.spark.SparkCatalog") \
    .getOrCreate()

# Read the current Delta table
df = spark.read.format("delta").load("s3://data/inventory")

# Write the same content to an Iceberg-managed *sibling* table (different base path
# during migration, then promote by swapping metadata)
df.writeTo("iceberg.inventory.inventory_new") \
   .partitionedBy("category") \
   .createOrReplace()

# 3. Phase 2 — cut over: swap XTable YAML to Iceberg source
sourceFormat: ICEBERG
targetFormats:
  - DELTA   # keep for MLflow only
datasets:
  - tableBasePath: s3://data/inventory
    tableDataPath: s3://data/inventory
    tableName: inventory.inventory
    tableProperties:
      delta.columnMapping.mode: id

# 4. Phase 3 — stop the Delta producer, verify Iceberg producer is authoritative
# (Kubernetes / Airflow / Databricks scheduler-specific)
kubectl delete deployment inventory-delta-writer

# 5. Phase 4 — MLflow migrates to read Iceberg directly, drop Delta mirror
# Update XTable YAML to remove DELTA target

# Final state — no XTable needed; single-format
# XTable job is deleted; only Iceberg metadata remains
sourceFormat: ICEBERG
targetFormats: []   # explicit no-op; or delete the whole YAML + job

Step-by-step explanation.

1. Phase 1 stands up a parallel Iceberg producer that reads from the Delta table and writes an Iceberg-managed version. During this phase, XTable still mirrors from Delta; the Iceberg producer's output is on the same Parquet files but has a different logical identity.
2. Phase 2 cuts XTable's YAML to source from Iceberg instead of Delta. Consumers reading Iceberg-via-XTable-mirror silently transition to reading Iceberg-native — same S3 path, same metadata format. Zero downtime.
3. Phase 3 stops the Delta producer. Now Iceberg is the sole authoritative writer. XTable optionally continues to emit a Delta mirror for MLflow (which still reads Delta) until MLflow migrates.
4. Phase 4 migrates MLflow to read Iceberg (via pyiceberg or delta-spark-with-iceberg-input). The Delta mirror can now be dropped from XTable's YAML, and the _delta_log/ directory can be deleted (with a grace period for any last stragglers).
5. Final state: no XTable job for this table. One producer, one format, one catalog entry. Storage: unchanged (still one Parquet copy). Ops: dramatically simpler.

Output.

Phase	Producer(s)	XTable mirrors	Consumer risk
0 (initial)	Delta only	Iceberg + Hudi	zero
1 (parallel)	Delta + Iceberg shadow	Iceberg + Hudi	zero
2 (cutover)	Iceberg only	Delta only (for MLflow)	very low
3 (stop delta)	Iceberg only	Delta only (for MLflow)	zero
4 (retire)	Iceberg only	none	zero

Rule of thumb. Retire XTable one producer-consumer pair at a time. Never migrate a consumer before its producer; never stop XTable before the last consumer has migrated. The retirement plan is boring on purpose — that's the point.

Worked example — the senior interview probe on "why not always Iceberg?"

Detailed explanation. The most common senior interview trap is: "If Iceberg is dominant, why bother with XTable — why not migrate everyone to Iceberg immediately?" The strong answer names the specific business constraints that make immediate migration unrealistic. Walk through the answer template.

• Constraint 1. Producer teams have quarterly OKRs that don't include format migration.
• Constraint 2. Some workloads use features that don't cleanly round-trip.
• Constraint 3. Migration is engineer-expensive; XTable is metadata-cheap.
• Constraint 4. M&A / vendor lock-in adds unmigrateable producers.

Question. Draft the 3-minute senior answer to "why not just migrate everyone to Iceberg?"

Input.

Constraint	Cost of ignoring	Cost of accommodating (XTable)
Producer OKR conflict	6-month slip; missed roadmap	0 (XTable is invisible to producer)
Non-round-trip features	correctness bugs downstream	keep source authoritative
Migration engineer cost	$500k+ for a large platform	~$115k/year bridge
Vendor lock-in	cannot migrate SaaS producer	XTable translates SaaS output

Code.

"Why not just migrate everyone to Iceberg?" — senior answer

Minute 1 — acknowledge the appeal
  "Long term, yes — Iceberg + Polaris is the target state, and every
   new table should start there. XTable is a bridge, not a
   destination."

Minute 2 — name the constraints
  "But three practical constraints make immediate migration
   unrealistic for a mature platform:
   (1) Producer teams have their own quarterly OKRs; format
       migration rarely tops the list.
   (2) Some workloads depend on features that don't cleanly
       round-trip — Delta CHECK constraints, Iceberg positional
       deletes, Hudi MoR merge state.
   (3) M&A, SaaS vendors, and legacy stacks add producers we
       cannot migrate at all."

Minute 3 — the pragmatic plan
  "So the pattern is: default to Iceberg for greenfield; use XTable
   as a bridge for the existing multi-format world; migrate
   producers opportunistically as they touch the pipeline anyway.
   Retire XTable per-table as the last non-target producer drops.
   Track break-even ROI annually; if maintenance cost grows or
   feature-drift incidents happen, accelerate migration."

Step-by-step explanation.

1. Minute 1 acknowledges the interviewer's implicit preference — yes, Iceberg is dominant, yes, standardisation is the goal. Never dismiss the question as naive.
2. Minute 2 names three specific constraints. Naming Delta CHECK, Iceberg positional deletes, and Hudi MoR shows deep knowledge; hand-waving about "some workloads" does not.
3. Minute 3 lays out the retirement plan. "Opportunistic migration" is the correct senior posture — do not commit to a big-bang project just to retire XTable; do commit to a per-table retirement checklist as producers change hands.
4. The break-even ROI mention signals architectural maturity. You've thought about when to make the decision, not just how.
5. The retirement checklist per table is: (a) producer migrated to target format? (b) all consumers migrated? (c) XTable job deleted? (d) old-format metadata directory dropped? (e) old catalog entry removed? Every "yes" is a completed retirement.

Output.

Answer minute	Signal	Weight
Minute 1 (acknowledge)	shows humility	low
Minute 2 (constraints)	shows knowledge	high
Minute 3 (pragmatic plan)	shows architecture judgment	very high
Retirement checklist	shows operational maturity	very high

Rule of thumb. For any "why not standardise?" probe, respond with the constraints-plus-plan template. Never dismiss migration as impossible; always frame XTable as a bridge with an explicit retirement path.

Senior interview question on bridge vs standardise

A senior interviewer might ask: "Your company has been running XTable for 18 months to bridge Delta + Iceberg + Hudi. Producer teams have finally agreed to standardise on Iceberg + Polaris over the next four quarters. Design the migration plan, the risk mitigations, the consumer-communication plan, and the XTable retirement checklist per table."

Solution Using a phased producer-then-consumer migration with per-table retirement gates

# 1. Migration inventory — one row per table under XTable
# migration-inventory.yaml (version-controlled in the platform repo)
tables:
  - name: sales.orders
    current_source: DELTA
    target_source:  ICEBERG
    producer_team:  sales-platform
    producer_migration_quarter: Q3-2026
    consumers:
      - name: analytics-snowflake
        current_read_format: ICEBERG-mirror
        target_read_format:  ICEBERG-native
      - name: mlflow
        current_read_format: DELTA-mirror
        target_read_format:  ICEBERG-native (via delta-spark w/ iceberg-input)
    retirement_quarter: Q4-2026

  - name: marketing.events
    current_source: HUDI
    target_source:  ICEBERG
    producer_team:  marketing-eng
    producer_migration_quarter: Q4-2026
    consumers:
      - name: trino-legacy
        current_read_format: HUDI-native
        target_read_format:  ICEBERG (via Trino iceberg connector upgrade)
    retirement_quarter: Q1-2027

# 2. Retirement-gate script — runs per table before dropping XTable
# retirement_gate.py
import boto3
import sys

def check_retirement_ready(table_name: str, table_prefix: str) -> dict:
    """Return a checklist of retirement readiness for one table."""
    checks = {}

    # Gate 1 — target-format producer is authoritative
    checks["producer_on_target"] = check_producer_writes_iceberg(table_prefix)

    # Gate 2 — every consumer reads target format natively (not mirror)
    checks["all_consumers_native"] = check_all_consumers_migrated(table_name)

    # Gate 3 — 30-day drift-free window
    checks["30day_drift_free"] = check_no_drift_30d(table_name)

    # Gate 4 — no CHECK / GENERATED / positional-delete dependencies exposed
    checks["no_non_round_trip"] = check_no_feature_dependencies(table_name)

    # Gate 5 — stakeholders sign-off recorded
    checks["signoff_recorded"] = check_signoff(table_name)

    checks["all_pass"] = all(v for k, v in checks.items() if k != "all_pass")
    return checks

report = check_retirement_ready("sales.orders", "s3://data/sales/orders")
if not report["all_pass"]:
    print(f"BLOCKED: {report}")
    sys.exit(1)
print("RETIREMENT READY")

# 3. Retirement execution — one table
# a. Drop the XTable Kubernetes CronJob
kubectl delete cronjob xtable-sync-sales-orders -n data

# b. Delete the old-format catalog entry (Delta was mirror; Iceberg is source)
# (Databricks Unity, run from Databricks SDK)
databricks tables delete production.sales.orders_delta

# c. Delete the old-format metadata directory (after 30-day grace period)
aws s3 rm s3://data/sales/orders/_delta_log/ --recursive
aws s3 rm s3://data/sales/orders/.hoodie/     --recursive

# d. Update the migration-inventory YAML — mark retirement_completed

Step-by-step trace.

Quarter	Activity	Tables affected
Q1-Q2	Producer migration prep + shadow Iceberg producers	all
Q3	sales.orders producer cut over to Iceberg	sales.orders
Q3-Q4	Consumer migration one-by-one; XTable keeps mirrors alive	all in-flight
Q4	marketing.events producer cut over to Iceberg	marketing.events
Q1-2027	Last consumers migrate; retirement gates pass	in-flight
Q1-2027	XTable jobs deleted table-by-table	migrated tables
Q1-2027	Old-format metadata deleted after grace period	migrated tables
Q1-2027	Migration complete for wave 1	20+ tables

After execution, wave 1 tables no longer run XTable. Wave 2 kicks off in Q2-2027 with the next batch of producers. Governance policies in Polaris are the sole grant surface; Delta and Hudi catalogs are progressively decommissioned.

Output:

Metric	Before migration	After Q1-2027 wave 1
Tables under XTable	50	30
Format count in production	3	still 3 (wave 2 pending)
Catalog entries per migrated table	3 (Polaris + Unity + Glue)	1 (Polaris only)
XTable maintenance FTE	0.5	0.4 (declining)
Feature-drift incident rate	~2/year	trending to 0

Why this works — concept by concept:

• Producer-first, consumer-second sequencing — never migrate a consumer before its producer; the mirror keeps the consumer working during the producer migration. Consumers migrate opportunistically per team roadmap.
• Per-table retirement gates — five explicit checks (producer on target, all consumers migrated, 30-day drift-free, no non-round-trip dependencies, stakeholder sign-off). Every gate must pass; no exceptions. This is what stops a rushed retirement from breaking a consumer.
• 30-day grace period on old metadata — after dropping the XTable job, leave the old-format metadata (Delta _delta_log/, Hudi .hoodie/) on S3 for 30 days. If a straggler consumer surfaces, it still works while we help them migrate.
• Version-controlled migration inventory — the YAML in the platform repo is the source of truth. Every table's status is visible; leadership can see progress at a glance; audit trails exist automatically via git history.
• Cost — one migration-inventory YAML, per-table retirement-gate script runs, staged cutover, grace-period cleanup. The eliminated cost is XTable maintenance forever (0.5 FTE ≈ $50k/year saved per fully-migrated batch of tables). Net O(1) per-table retirement work versus O(∞) indefinite bridge maintenance.

Design
Topic — design
Design problems on multi-quarter migration planning

Practice →

Streaming
Topic — streaming
Streaming zero-downtime cutover problems

Practice →

---

Cheat sheet — XTable recipes

• What XTable is in one sentence. apache xtable (formerly OneTable, donated to the Apache Software Foundation in 2024 and now in the Incubator) is a Java-based metadata-only translator that reads one open-table-format's snapshot metadata — Iceberg manifests, Delta transaction logs, or Hudi timelines — and writes translated metadata for the other two formats on top of the same shared Parquet directory. The result: one physical dataset is readable natively as Iceberg from Snowflake, as Delta from Databricks, and as Hudi from Trino, without ever copying a single Parquet byte or running a rewrite job.
• Which formats and pairs. Sources: ICEBERG, DELTA, HUDI. Targets: any of the other two. Best-round-trip pair: Iceberg ↔ Delta (both copy-on-write, both column-mapping-capable). Good with caveats: Iceberg ↔ Hudi and Delta ↔ Hudi (Hudi's merge-on-read state does not translate). Never bidirectional: XTable is one-way per sync; two authoritative writers on one table breaks the model — pick one source-of-truth format per table and mirror outward.
• CLI invocation template. java -jar utilities-<version>-bundled.jar --datasetConfig path/to/table.yaml --hadoopConfig path/to/core-site.xml. Version-pin the jar; do not use latest. The bundled variant ships all Iceberg + Delta + Hudi client libraries so you don't fight Maven. Run once for one-shot sync; schedule via cron/Airflow/EventBridge for cadence-driven sync; wire to S3 event notifications on _delta_log/*.json / metadata/*.metadata.json / .hoodie/timeline/*.commit for event-driven sync.
• YAML config skeleton. sourceFormat: {ICEBERG|DELTA|HUDI}, targetFormats: [...], then a datasets: list with per-table tableBasePath (the S3/GCS/ABFS URI), tableDataPath (usually same), tableName (namespace.table), optional partitionSpec (list of partitionFieldName + transformType), and tableProperties (crucially: delta.columnMapping.mode: id for rename-safe Delta output). Optional catalogConfig block registers each translated table in Polaris, Glue, Unity, or Nessie.
• Round-trip feature matrix. ✅ safe: ADD COLUMN, current-snapshot files, column stats, basic partition transforms, RENAME COLUMN when column mapping mode is id. ⚠️ with caveats: Iceberg positional deletes (mirrored as add-then-remove), Hudi MoR state (mirrored as current-view only), partition transforms beyond IDENTITY on Hudi target. ❌ not preserved: Delta CHECK constraints, Delta GENERATED columns, historical snapshots (only current), format-native table statistics. Keep the source format authoritative for any feature on the ❌ list.
• Catalog registration cheat sheet. Iceberg output → Polaris (multi-engine neutral) or Glue (AWS-native) or Snowflake Iceberg REST (Snowflake-native). Delta output → Unity Catalog (Databricks-native) or Glue (AWS-native with table_type=DELTA parameter). Hudi output → Glue via the Hive metastore interface (Trino Hudi connector expects this). Every mirror needs one catalog entry per (table × target format); OPA policy replication keeps grants aligned.
• Sync cadence patterns. Cold-warm tables: cron every 15-60 min via Airflow / Kubernetes CronJob. Hot tables: event-driven via S3 event notifications → SQS FIFO (dedupe per table) → Fargate/Lambda consumer (coalesce bursts within 30s). Hybrid: default to cron; opt hot tables into event-driven. Latency budget: 5-30 s event-driven; 15-60 min cron. Cost budget: ~$0.003 per Fargate sync; ~$0.001 per cron JVM start.
• Verification recipe (does XTable ever write Parquet?). Before sync: aws s3 ls --recursive s3://.../ | grep -c parquet > /tmp/before.txt. Run XTable. After sync: same command → /tmp/after.txt. diff before.txt after.txt must be empty. Any non-zero diff means XTable somehow wrote a Parquet file, which it should never do. Ship this check in CI/CD to catch misconfiguration fast.
• Drift detection between formats. Weekly (or hourly for hot tables): load source table via pyiceberg / deltalake / hudi-python, extract the file-URI set + total row count, compare across formats. Alert on any mismatch. False positives are rare because both mirrors reference the same Parquet; a real mismatch means XTable failed silently, sync fell behind, or the source producer bypassed XTable's expected code path.
• Column-mapping = id is mandatory for Delta target. Set tableProperties.delta.columnMapping.mode: id in every XTable YAML that targets Delta. Without it, RENAME COLUMN in the source becomes drop-and-add in the Delta mirror, breaking column-level lineage in every downstream Delta reader. Also set delta.minReaderVersion: 2 and delta.minWriterVersion: 5 so Delta clients honour column mapping. Retrofitting column mapping on Delta requires an ALTER TABLE that touches every file — do it once, up front.
• Metadata retention on the target format. Every sync writes new Iceberg metadata files (vN.metadata.json + manifest-list + manifest). Over months, the metadata/ directory accumulates thousands of files, and reads slow down as clients download more manifests. Fix: weekly Iceberg metadata compaction — CALL system.rewrite_manifests('sales.orders') — or run Iceberg's remove_orphan_files procedure. Same idea for Delta: schedule VACUUM RETAIN 168 HOURS on the mirror.
• Failure runbook — sync failed. (a) Check the XTable job's stdout/stderr log for stack traces — most failures are S3 auth (rotate creds), catalog auth (rotate tokens), or Iceberg/Delta version incompatibility (pin the XTable image tag). (b) Re-run the sync manually with the same YAML. XTable is idempotent — a re-run either succeeds or emits the same error. (c) If catalog registration failed but metadata was written, run the catalogConfig-only mode to re-register. (d) If sync fell far behind, check for producer commits during a large snapshot rewrite; consider event-driven to avoid pile-ups.
• Failure runbook — drift detected. (a) Confirm both format mirrors reference the same Parquet set (they should — this is what XTable guarantees). (b) If file sets differ, most likely XTable was blocked; check job logs for the last successful sync timestamp. (c) If row counts differ but file sets match, the source producer may have written new Parquet + updated its own metadata without triggering XTable — check S3 event delivery. (d) As last resort, force a full re-sync by deleting the target-format metadata directory and re-running XTable; the sync will emit a fresh snapshot.
• When to NOT use XTable. Greenfield deployments — pick Iceberg + Polaris and skip the bridge. Single-format shops (everyone on Delta + Databricks) — no benefit, only cost. Feature-dependent workloads that require Delta CHECK / GENERATED columns or Iceberg positional deletes as read guarantees — the mirror cannot enforce these. Two authoritative writers on one table — XTable cannot arbitrate; pick one source of truth.
• Migration path from XTable-bridge to standardised format. Phase 1 (Bridge, 1-3 quarters): XTable translates every source to every target. Phase 2 (Consolidate producers, 2-6 quarters): migrate producers to the target format opportunistically. Phase 3 (Retire mirrors): once no producer writes a non-target format, delete XTable jobs table-by-table with a 30-day metadata grace period. Phase 4 (Target state): one format, one catalog, one physical dataset per table. Compute per-org break-even (bridge cost vs migration cost) annually.
• OneTable → XTable rename trivia. OneTable was open-sourced by Onehouse in 2023; donated to the Apache Software Foundation in 2024, entered the Incubator, was renamed to Apache XTable (the OneTable name clashed with unrelated trademarks). Jar names moved from onetable-* to xtable-*; the Java package moved from io.onetable.* to org.apache.xtable.*. Configs and CLI flags are otherwise stable — most migration is a name change in your build files.
• Interview elevator pitch. "Apache XTable is a metadata-only translator between Iceberg, Delta, and Hudi. It reads one format's manifest and writes equivalent metadata for the other formats without touching the underlying Parquet. Storage is shared; freshness is one sync cadence behind the source; every engine reads its native format from the same physical dataset. It's the pragmatic bridge for the 2026 multi-engine world; the long-term play is standardising on Iceberg + Polaris and retiring XTable per-table."

Frequently asked questions

What is Apache XTable in one sentence?

Apache XTable is a metadata-only translator for the three open lakehouse table formats — Iceberg, Delta, and Hudi — that reads the source format's snapshot metadata (Iceberg manifests, Delta transaction log, or Hudi timeline) and writes equivalent metadata in the target formats without ever copying, rewriting, or moving the underlying Parquet files, so one physical dataset on S3 (or GCS or ABFS) becomes readable natively as any of the three formats by whichever query engine you prefer. XTable is a Java CLI (bundled as utilities-<version>-bundled.jar) that runs either one-shot, on a cron cadence, or event-driven off source-format commit notifications; sync latency is seconds to minutes depending on cadence choice. The project began as OneTable inside Onehouse, was donated to the Apache Software Foundation in 2024, and is currently a 1.x-track Incubator project with production adoption at Walmart, Onehouse customers, and mid-sized data platform teams that run mixed Databricks + Snowflake + Trino stacks and refuse to duplicate their storage bill three ways.

How is XTable different from actually migrating everything to Iceberg?

XTable is a bridge; migration is a destination. The bridge preserves every existing writer in its native format (Delta producers keep writing Delta, Hudi producers keep writing Hudi, Iceberg producers keep writing Iceberg) and mirrors the metadata outward so every reader engine can consume the format it prefers — no producer team has to change code, no consumer team has to wait, and total storage stays at 1× because the Parquet is shared. Migration replaces every writer's format with the target (usually Iceberg + Polaris) so the entire platform runs on one format and one catalog, at the cost of engineer-months of rewrite work spread across every producer team and every dependent consumer. The correct architectural posture is "XTable now, migrate opportunistically over 4-8 quarters, retire XTable per-table as its last producer switches" — never "XTable forever" (technical debt accumulates in the form of feature-drift risk and catalog duplication) and never "big-bang migrate everything this quarter" (the engineering cost usually dwarfs the annual bridge cost by 5-10×). Compute the break-even ROI (bridge cost vs migration cost) once a year and adjust the timeline; senior interviewers listen specifically for this "bridge with retirement plan" framing rather than either extreme.

Which format features do NOT round-trip through XTable?

Three categories of features fail to round-trip cleanly and every senior architect keeps a running list per table. First, format-specific enforcement: Delta CHECK constraints and Delta GENERATED columns are Delta-only concepts and XTable silently drops them from the Iceberg/Hudi mirror — if your business logic relies on CHECK (total_cents > 0) to enforce data quality, the Iceberg reader will happily surface invalid rows because it doesn't know the constraint exists. Second, format-specific delete semantics: Iceberg V2 positional deletes and equality deletes get flattened in the Delta/Hudi mirror as add-then-remove rewrites — semantically equivalent but with different metadata cost profiles and different time-travel behavior. Third, historical snapshot / time-travel: XTable translates only the current committed snapshot per sync; if the Iceberg source has 50 historical snapshots, the Delta mirror only knows about "the last XTable sync" as version 0 and every subsequent sync as version N — cross-format AS OF queries against old snapshots are impossible. There's also a fourth soft category — Hudi merge-on-read state: MoR delta log files (.log files in Hudi) represent uncompacted writes; when XTable mirrors a Hudi source to Iceberg/Delta, it represents the current committed view as if everything had been compacted into base files, so downstream readers see the logical state but lose access to the MoR-native "read-optimised vs snapshot" distinction. The senior rule of thumb: for any table that depends on a non-round-trippable feature, keep the source format as sole enforcer, and treat the XTable mirror as read-only.

How do I integrate XTable with Databricks Unity Catalog?

Integration with Unity Catalog is straightforward for the Delta target and awkward-but-possible for the Iceberg target. For Delta, add a catalogConfig block to your XTable YAML that points at Unity via the Databricks SDK (or run a post-sync step that calls w.tables.create(...) from databricks-sdk); Unity registers the Delta table with its transaction-log location on S3, and Databricks SQL, notebooks, and Photon can immediately read it with full governance (lineage, access controls, Delta Sharing) as if it were a Databricks-native Delta table. For Iceberg, the cleanest path is Databricks' own Delta Universal Format (Uniform) — you write Delta on Databricks, Uniform emits Iceberg metadata alongside, and Unity governs both — but that only works if Databricks is the writer. When XTable produces an Iceberg mirror from a non-Databricks source (e.g. a Spark writer outside Databricks), Unity can consume it via Unity's Iceberg federation feature (introduced in 2024-2025) that reads external Iceberg tables via a REST catalog — point Unity at Polaris (or your XTable-managed Iceberg catalog) and the tables appear in the Unity namespace as federated Iceberg tables. The trade-off: Unity's federated Iceberg support has feature lag vs native Delta (some governance features like column masking may not apply to federated tables in your Databricks version), so verify the feature matrix against your Databricks runtime version before committing.

Is OneTable the same as XTable?

Yes, XTable is the renamed OneTable. Onehouse open-sourced the OneTable project in 2023 as a solution to the mixed-format lakehouse problem they observed at customers running Hudi + Iceberg + Delta in parallel. In 2024, Onehouse donated OneTable to the Apache Software Foundation, entered the Apache Incubator, and renamed the project to Apache XTable — the OneTable name conflicted with unrelated trademarks and the rename cleared the path for Apache branding. The rename is largely mechanical: jar names moved from onetable-* to xtable-* (usually utilities-<version>-bundled.jar in the release archive), the Java package moved from io.onetable.* to org.apache.xtable.*, and the CLI flags are essentially unchanged. If you're upgrading from a pre-donation OneTable release, most of the migration is updating your build files (Maven / Gradle dependency coordinates) and container image references; YAML configs are compatible. Interviewers occasionally use the old "OneTable" name to see whether candidates track the rename — knowing both names and the 2024 Apache donation timeline is a small but memorable senior signal.

What's the operational cost of running XTable in production?

The operational cost breaks into five categories, and each is typically small relative to the alternative of duplicating Parquet per format. Compute: one Java process per sync per table per target format, ~2-4 seconds of CPU per sync, ~$0.001 per cron-tick JVM start on shared runners or ~$0.003 per event-driven Fargate task. Storage: sibling metadata directories per format, typically <1% of table size — for a 500-table warehouse syncing every 15 min to three targets, total metadata footprint is ~80 GB over 30 days retention. S3 requests: 4-6 PUTs per sync per target format; at 500 tables × 3 targets × 96 syncs/day = ~576k PUTs/day, or ~$300/day in raw request cost (tunable down by moving cold tables to hourly cadence). Engineer maintenance: ~0.3-0.5 FTE for a 100-table platform, covering YAML changes, catalog registrations, upgrade testing, and drift-alert triage. Feature-drift incidents: one or two per year in a mature deployment, each costing ~$10-50k in engineering time to root-cause and fix (usually a downstream consumer relying on a feature the mirror can't enforce). Compared against the alternative — duplicating every Parquet file three times, running three sets of compaction jobs, doubling storage/IO costs — XTable is dramatically cheaper for any multi-format platform larger than about 10 tables. The break-even against a full migrate-to-one-format project is typically 3-6 years at modest maintenance burdens, which is why "run XTable and migrate opportunistically" is the pragmatic senior answer rather than "migrate everything this quarter to retire XTable."

Practice on PipeCode

• Drill the SQL practice library → for the schema evolution, column mapping, cross-catalog federation, and metadata-diff problems senior lakehouse interviews love to probe.
• Rehearse system design on the design practice library → for multi-engine lakehouse layouts, catalog federation, migration planning, and the "when to bridge vs standardise" trade-off that XTable interviews always land on.
• Sharpen the streaming axis with the streaming practice library → for event-driven sync patterns, S3-event-notification pipelines, and coalesced-commit-burst designs that mirror the XTable event-driven runner architecture.
• Practise the joins library on the joins practice library → for cross-format joins where the same physical table is queried via different catalog views — the pattern XTable enables downstream.
• Practise aggregations on the aggregation practice library → for the summary + rollup queries that dashboards run against XTable-translated Iceberg mirrors.
• Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the metadata-only-translator mental model against real graded inputs with progressive difficulty.

Lock in apache xtable muscle memory

Docs explain what XTable does. PipeCode drills explain the decision — when XTable is the right bridge, when the round-trip drops a feature you care about, when catalog duplication becomes governance debt, when the migration ROI finally flips. Pipecode.ai is Leetcode for Data Engineering — pattern-first practice tuned for the multi-engine lakehouse trade-offs senior data engineers actually face.

Practice design problems →
Practice SQL problems →