Apache Gravitino: A Federated Metadata Lake Across Catalogs, Clouds & Engines

Apache Gravitino is the layer that finally lets a data platform stop asking "which metastore knows about this table?" — a single, governed, read/write metadata service that sits above every catalog you already run and turns them into one addressable namespace — instead of a scatter of Hive metastores, Iceberg catalogs, JDBC databases, Kafka schema registries, and object-store filesets, each wired separately into every engine that needs it. The hard problem was never storing metadata; every system does that. The problem was that each system stores its own metadata in its own place, so a Spark job, a Trino cluster, and a Flink pipeline each carry a different, partial, hand-maintained picture of the same estate, and no single place can answer "what tables exist, who may read them, and where did this column come from."

This guide is the senior-data-engineering walkthrough for closing that gap — for building a metadata lake that federates the catalogs you have rather than migrating them into one more silo — framed the way interviewers actually probe it: why a federated catalog is a different animal from a metastore or a discovery tool, how Gravitino's metalake → catalog → schema → table model gives every source one unified catalog namespace, how a single Iceberg REST endpoint can front a Hive metastore and a JDBC catalog at once so any engine points at one URI, how Spark, Trino, and Flink resolve the same fully-qualified name across a multi-cloud estate, and how governance — unified tags, role-based access, credential vending, and lineage — lives in the metadata layer instead of being re-implemented per engine. Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the ETL practice library →, rehearse metadata-platform patterns on the data processing practice library →, and sharpen the architecture axis with the system design practice library →.

---

On this page

• Why a federated metadata lake over catalog sprawl
• Architecture — metalake, catalog, schema, table
• Federation — one Iceberg REST endpoint over many backends
• Connecting engines — Spark, Trino & Flink
• Governance & lineage — tags, access, credential vending
• Cheat sheet — Apache Gravitino
• Frequently asked questions
• Practice on PipeCode

---

1. Why a federated metadata lake over catalog sprawl

The sprawl problem — every engine and cloud grows its own metastore, and nothing spans them

The one-sentence invariant: a metadata lake is a single read/write metadata service that federates the catalogs an organisation already runs — Hive metastores, Iceberg catalogs, JDBC databases, Kafka schema registries, object-store filesets — into one governed namespace, so the reason Apache Gravitino exists is that without it every engine (Spark, Trino, Flink) and every cloud carries its own partial copy of "what tables exist and who may read them," which is an N-engines × M-catalogs coupling explosion that no team can keep consistent, govern uniformly, or trace lineage across. Wire five engines to four catalogs the old way and you maintain twenty separate configurations; put a metadata lake in the middle and each engine points at one place, and each catalog is registered once.

The four axes interviewers actually probe.

• Unification. Is there one namespace that addresses every table, fileset, and topic the same way? A federated catalog gives you metalake.catalog.schema.table as a single fully-qualified name regardless of whether the metadata physically lives in a Hive metastore, a Postgres database, or an S3 fileset. The senior answer names the unified identifier first, because it is what makes everything else — governance, lineage, portability — tractable.
• Federation. Do you copy metadata into a new store, or front the stores you have? The senior answer is emphatic: federation exposes existing metastores in place, it does not migrate them. A metadata lake that forced a rewrite of every catalog would be just one more silo.
• Governance. Where do tags, access control, and credentials live? The senior answer puts classification (tags), authorization (roles/privileges on securable objects), and secret handling (credential vending) in the metadata layer, enforced once, instead of re-implemented in each engine.
• Portability. Can the same table be read from Spark on one cloud and Trino on another without re-registering it? The senior answer frames multi-cloud, multi-engine portability as the payoff of a unified namespace: register once, reach it everywhere.

The 2026 reality — the metadata layer became a first-class system.

• Gravitino is a server, not a library. It runs as a service with a REST API and an entity store, and it speaks several access protocols — its own REST/CLI/SDK, plus the Iceberg REST catalog spec — so engines integrate through whichever door fits.
• Catalogs are connectors, not copies. A Gravitino catalog wraps a provider (hive, lakehouse-iceberg, jdbc-postgresql, hadoop fileset, kafka) and delegates to the real backend; the metadata still lives where it lived.
• Governance is metadata. Tags, a role-based access-control model over securable objects, optional Apache Ranger pushdown, and credential vending are all properties of the metadata lake, applied uniformly across every registered catalog.
• Lineage rides the namespace. Because every dataset has one fully-qualified name across engines, lineage events (OpenLineage) from different engines refer to the same nodes — which is what makes cross-engine, cross-catalog lineage possible at all.

What interviewers listen for.

• Do you distinguish a metadata lake (federated, read/write, engines operate through it) from a metastore (one system's own store) and a data catalog (a read-only discovery/search index)? — senior signal.
• Do you say federation, not migration — expose existing catalogs in place? — required answer.
• Do you put governance in the metadata layer (tags, RBAC, credential vending) rather than per engine? — senior signal.
• Do you name the unified fully-qualified name as the thing that makes lineage and portability possible? — required answer.

Worked example — inventory the sprawl and the N×M coupling

Detailed explanation. The most useful artifact for a metadata-platform interview is an honest inventory of the silos and the coupling they create. Every serious discussion starts here: list the metadata stores, the engines, and the connections between them, then count.

• The silos. A Hive metastore for legacy tables, an Iceberg catalog for the lakehouse, a Postgres database, a Kafka schema registry, and a pile of filesets on S3 and GCS.
• The engines. Spark for ETL, Trino for interactive SQL, Flink for streaming.
• The tension. Each engine needs its own configuration block for each catalog, so wiring is engines × catalogs, and every governance or credential change must be repeated everywhere.

Question. For an estate of 3 engines and 5 metadata stores, quantify the wiring before and after a metadata lake, and name what breaks without one.

Input.

Metadata store	Owner	Reached today by	Governed by
Hive metastore	platform	Spark, Trino (separate configs)	HMS + ad hoc
Iceberg catalog	lakehouse team	Spark, Flink, Trino	per-engine
Postgres (JDBC)	app team	Trino, Spark	DB grants
Kafka registry	streaming	Flink	registry ACLs
S3 / GCS filesets	many	Spark, Flink	IAM per cloud

Code.

Before a metadata lake — every engine configures every catalog itself.
================================================================
spark-defaults.conf   : hive uri, iceberg uri, jdbc url, s3 keys, gcs keys ...
trino/catalog/*.props : hive.props, iceberg.props, postgres.props ...
flink-conf + catalogs : iceberg catalog, kafka registry, s3 creds ...

Wiring   = engines x catalogs = 3 x 5 = 15 configuration surfaces
Governance = repeated in each = 15 places to change a grant or rotate a key
Lineage  = none across stores (each engine sees only its own configs)

After a metadata lake (Gravitino) — engines point at ONE server.
================================================================
Each engine    : gravitino.uri + metalake      (1 config each)  -> 3
Each catalog   : registered ONCE in Gravitino   (1 registration) -> 5

Wiring   = engines + catalogs = 3 + 5 = 8   (N x M  ->  N + M)
Governance = one place (tags / RBAC / credentials in the metadata lake)
Lineage  = possible: every dataset has ONE fully-qualified name

Step-by-step explanation.

1. In the "before" world, the cost is multiplicative: three engines each carrying five catalog configurations is fifteen surfaces, and every one of them independently holds connection strings, credentials, and (implicitly) an authorization posture. That is fifteen places a mistake can leak data or drift out of sync.
2. A change amplifies through the whole matrix. Rotate an S3 key or revoke a grant and you must touch every engine that reached that store — miss one and you have either an outage or an open door.
3. Registering each catalog once in Gravitino turns the multiplicative N × M into an additive N + M: each engine has one config (the Gravitino URI plus a metalake), and each catalog is registered a single time behind the server.
4. Governance collapses from fifteen surfaces to one: a tag, a role, or a vended credential is defined in the metadata lake and applies to every engine that reaches the catalog through it.
5. Lineage becomes possible — not automatic, but possible — because the precondition for tracing a column across Spark and Trino is that both call it by the same name. A unified namespace is that precondition; sprawl structurally prevents it.

Output.

Concern	Sprawl (N × M)	Metadata lake (N + M)
Wiring surfaces (3×5)	15	8
Change a grant / rotate a key	15 edits	1 edit
One namespace for a table	no	yes (metalake.catalog.schema.table)
Cross-engine lineage	impossible	possible

Rule of thumb. Count the wiring: sprawl is engines × catalogs, a metadata lake is engines + catalogs. If a single grant change or key rotation touches more than one place per catalog, you are paying the coupling tax that a federated metadata layer exists to eliminate.

Worked example — metastore vs data catalog vs metadata lake

Detailed explanation. The trap question is "isn't Gravitino just a Hive metastore, or a DataHub?" The weak answer conflates all three. The senior answer separates them by what they do: store metadata for one system, index metadata for humans to search, or federate metadata that engines operate through.

• The metastore. A Hive metastore (or a Glue catalog) stores metadata for the systems that speak its protocol — one store, one protocol, one system's worldview.
• The data catalog. A discovery tool (DataHub, Amundsen, OpenMetadata) ingests metadata into a search index so humans can find and document datasets — read-mostly, engines do not run queries through it.
• The metadata lake. Gravitino federates many metastores/catalogs into one read/write namespace that engines actually operate through — create a table in Spark and it is immediately visible to Trino under the same name.

Question. Place a Hive metastore, DataHub, and Gravitino on the axes of read/write, federation, and whether engines operate through it.

Input.

Capability	Hive metastore	Data catalog (DataHub)	Metadata lake (Gravitino)
Scope	one system's metadata	ingested index of many	federates many, live
Read/write	read/write (its own)	read-mostly (discovery)	read/write across sources
Engines operate through it	Hive-family only	no (out-of-band)	yes (Spark/Trino/Flink)
Governance	table/db grants	documentation/tags	tags + RBAC + credentials

Code.

Metastore  (e.g. Hive Metastore)
  role : store metadata for ONE system family; engines that speak its protocol use it
  gap  : each engine/cloud runs its own; no single cross-system namespace

Data catalog (e.g. DataHub / OpenMetadata / Amundsen)
  role : INGEST metadata from many systems into a SEARCH index for humans
  gap  : discovery only — you don't CREATE or QUERY a table THROUGH it

Metadata lake (Apache Gravitino)
  role : FEDERATE many catalogs into ONE read/write namespace engines operate THROUGH
         create in Spark -> visible in Trino under the SAME name, governed the SAME way
  note : it can also FEED a data catalog and SIT OVER metastores — complementary, not a clone

Step-by-step explanation.

1. A metastore is the source of truth for one system: powerful, but its worldview stops at its protocol boundary, which is exactly why an estate ends up with several of them.
2. A data catalog solves a human problem — search, documentation, ownership — by ingesting metadata after the fact. It is deliberately out of the query path; you do not CREATE TABLE through DataHub, and an engine does not resolve a table by asking it.
3. A metadata lake solves the engine problem: it is in the operational path. When Spark creates a table through Gravitino, Trino sees it immediately under the identical fully-qualified name, and the same tag and grant apply — because there is one namespace, not three copies.
4. The three are complementary, not competitors: Gravitino can sit over your metastores (federating them) and feed your data catalog (as a metadata source), so the honest answer is "different layer," not "replacement."
5. The interview tell is whether you say engines operate through the metadata lake. Someone who calls Gravitino "a fancier DataHub" has missed that it is read/write and in the query path; someone who calls it "just a metastore" has missed that it federates many.

Output.

Question	Metastore	Data catalog	Metadata lake
"Create a table through it?"	yes (its system)	no	yes (any provider)
"Search/document datasets?"	no	yes	yes (tags)
"One namespace across engines?"	no	index only	yes
"In the engine query path?"	its family	no	yes

Rule of thumb. A metastore stores one system's metadata; a data catalog indexes many for humans to search; a metadata lake federates many into one read/write namespace that engines operate through. Gravitino is the third — and it can sit over the first and feed the second.

Worked example — the senior "why federate" answer

Detailed explanation. The metadata-platform interview escalates predictably: an ambiguous opener ("we have too many catalogs"), then narrowing to test whether you reach for federation, a unified namespace, and layer governance — or whether you propose yet another migration.

• Ambiguous opener. "Every team runs its own metastore and nothing lines up. Fix it."
• Follow-up 1. "Do we migrate everything into one catalog?" — probes federation vs migration.
• Follow-up 2. "How does a Trino user find a table a Spark job just made?" — probes the unified namespace.
• Follow-up 3. "Where do access rules and cloud keys live now?" — probes governance placement.

Question. Draft a 4-minute senior answer that pre-empts the follow-ups without proposing a big-bang migration.

Input.

Interview signal	Weak answer	Senior answer
Consolidation	"migrate all into one metastore"	"federate in place with a metadata lake"
Discovery	"everyone learns each catalog"	"one FQN: metalake.catalog.schema.table"
Governance	"each engine enforces its own"	"tags + RBAC + credentials in the metadata layer"
Portability	"re-register per cloud"	"register once; reach from any engine/cloud"

Code.

Senior "why a metadata lake" answer (4 minutes)
===============================================

Minute 1 — name the layer, not a migration
  "I would NOT migrate five metastores into a sixth. I'd put a federated
   metadata lake (Gravitino) over them — each catalog registered once,
   metadata staying where it lives. Federation, not a rewrite."

Minute 2 — the unified namespace
  "Every dataset gets ONE fully-qualified name, metalake.catalog.schema.table,
   whether it's Hive, Iceberg, JDBC, a fileset, or a Kafka topic. A Trino user
   finds a table a Spark job made because it's the SAME name in one server."

Minute 3 — governance in the layer
  "Classification (tags), authorization (roles + privileges on securable
   objects, optionally pushed to Ranger), and secrets (credential vending of
   short-lived cloud tokens) live in the metadata lake — defined once, enforced
   across every engine that reaches the catalog through it."

Minute 4 — portability + lineage
  "Because it's one namespace and one governed layer, the same table is
   reachable from Spark, Trino, and Flink across clouds without re-registering,
   and lineage events from different engines refer to the SAME nodes — so
   cross-engine lineage finally becomes tractable."

Step-by-step explanation.

1. Minute 1 refuses the migration trap up front. Proposing "consolidate into one metastore" is the junior move; naming federation-in-place signals you understand that the value is exposing what exists, not rebuilding it.
2. Minute 2 makes the unified namespace concrete with the four-part FQN, and answers the discovery follow-up before it is asked — the same name in one server is why a Trino user sees a Spark-made table.
3. Minute 3 places governance in the metadata layer and enumerates the three governance surfaces (tags, RBAC, credentials) so the interviewer hears that you know each is defined once, not per engine.
4. Minute 4 closes on the two payoffs that only a unified, governed layer can deliver — multi-cloud/multi-engine portability and cross-engine lineage — tying them back to the single namespace introduced in minute 2.
5. The through-line is "one layer over many sources." Every follow-up is answered by the same idea, which is what a senior answer sounds like: a single organising principle, not a list of tools.

Output.

Grading criterion	Weak score	Senior score
Federation over migration	rare	mandatory
Unified FQN named	occasional	mandatory
Governance in the layer	rare	senior signal
Portability + lineage payoff	rare	senior signal

Rule of thumb. The senior "why federate" answer is one idea — a governed layer over the catalogs you already have — expressed as a unified namespace, layered governance, and the portability/lineage it unlocks. Rehearse the four-part FQN and you have pre-empted the discovery, governance, and migration follow-ups at once.

Senior interview question on federated metadata strategy

A senior interviewer often opens with: "Your company has a Hive metastore, an Iceberg catalog, several JDBC databases, a Kafka registry, and filesets across two clouds — and Spark, Trino, and Flink each wired to them separately. Metadata drifts, governance is copied everywhere, and no one can trace lineage. Design the fix without a big-bang migration: how you unify the namespace, where governance and credentials live, and how each engine reaches every source."

Solution Using one metalake, federated catalogs, and layered governance

-- Step 1 — put ONE metadata lake (Gravitino) over the estate; migrate nothing.
--          Each existing store becomes a registered catalog (federated in place).
metalake: company
  catalogs (registered once each, metadata stays put):
    cat_hive      -> provider hive              (thrift://hms:9083)
    cat_iceberg   -> provider lakehouse-iceberg (backend=jdbc, warehouse=s3://.../wh)
    cat_pg        -> provider jdbc-postgresql   (jdbc:postgresql://pg/app)
    cat_events    -> provider kafka             (bootstrap=kafka:9092)
    cat_files_s3  -> provider hadoop (fileset)  (s3a://bucket/)
    cat_files_gcs -> provider hadoop (fileset)  (gs://bucket/)

-- Step 2 — one namespace: every dataset is metalake.catalog.schema.table.
company.cat_hive.legacy.orders
company.cat_iceberg.sales.orders
company.cat_pg.app.customers
company.cat_events.streams.clickstream        (topic)
company.cat_files_s3.landing.raw_events       (fileset)

-- Step 3 — governance in the metadata layer, defined once, enforced across engines.
tags        : pii, gold, deprecated ... attached to catalogs/schemas/tables/columns
rbac        : role analyst_ro = [USE_CATALOG cat_iceberg, SELECT_TABLE cat_iceberg.sales.*]
              (optionally pushed down to Apache Ranger for engine-side enforcement)
credentials : credential vending -> short-lived S3/GCS tokens per request (no static keys)
lineage     : engines emit OpenLineage events keyed by the SAME FQN -> cross-engine graph

-- Step 4 — every engine points at ONE server + metalake (N + M, not N x M).
Spark : spark.sql.gravitino.uri=http://gravitino:8090 ; spark.sql.gravitino.metalake=company
Trino : connector.name=gravitino ; gravitino.uri=... ; gravitino.metalake=company
Flink : catalog store = gravitino (uri + metalake=company)

Step-by-step trace.

Decision	Before (sprawl)	After (metadata lake)
Namespace	5 stores, 5 worldviews	one FQN across all
Wiring (3 engines × 5 stores)	15 configs	8 (3 engine + 5 catalog)
Grant / key change	15 places	1 place (the layer)
Cross-engine lineage	impossible	possible (same nodes)
Migration required	—	none (federated in place)
Credentials	static keys per engine	vended, short-lived

After the rollout, the metalake company federates all six catalogs in place; every dataset is addressable as company.<catalog>.<schema>.<table>; tags, roles, and vended credentials are defined once in the metadata layer and honoured by every engine; and Spark, Trino, and Flink each carry a single Gravitino URI plus the metalake name. No metastore was migrated — each was registered once and left where it lives — and because all three engines name a dataset identically, their OpenLineage events land on the same graph nodes.

Output:

Metric	Before (sprawl)	After (Gravitino)
Namespaces to learn	one per store	one FQN total
Config surfaces (3×5)	15	8
Governance edit blast radius	every engine	the metadata layer
Lineage across engines	none	unified graph
Static cloud keys in engines	many	zero (vended)
Migration effort	—	none

Why this works — concept by concept:

• Federation, not migration — each existing metastore/catalog is registered once and left in place, so the metadata lake exposes the estate you have instead of forcing a rewrite into one more silo. The metadata never moves; only the addressing unifies.
• One unified namespace — metalake.catalog.schema.table addresses a Hive table, an Iceberg table, a JDBC table, a fileset, and a Kafka topic identically, which is the precondition for cross-engine discovery, governance, and lineage.
• Governance in the metadata layer — tags, RBAC over securable objects (optionally pushed to Ranger), and credential vending are defined once and enforced across every engine, collapsing an N × M governance matrix into a single surface.
• Portability across engines and clouds — because a dataset has one governed name, Spark, Trino, and Flink reach it across clouds without re-registration, and lineage events from different engines refer to the same nodes.
• Cost — one server plus one config per engine and one registration per catalog, versus engines × catalogs hand-maintained wiring with duplicated credentials and no lineage. The eliminated cost is the coupling tax itself — O(N + M) to run a governed estate instead of O(N × M) to keep a sprawling one barely consistent.

Design
Topic — design
Design problems on federated metadata and platform architecture

Practice →

Data processing Topic — data-processing Data processing problems on multi-catalog and multi-engine estates

Practice →

---

2. Architecture — metalake, catalog, schema, table

One four-level namespace over every source, with each catalog wrapping a provider

The mental model in one line: Apache Gravitino organises all metadata under a strict four-level namespace — metalake (the top-level tenant/container) → catalog (a connector to one backend, chosen by provider) → schema (a database/namespace inside it) → table/fileset/topic/model (the entity) — so a unified catalog is not a pile of imported tables but a tree where each catalog wraps a provider (hive, lakehouse-iceberg, jdbc-postgresql, hadoop, kafka) and delegates to the real metastore behind it, while the Gravitino server keeps its own light entity store of the registrations and serves the whole tree over a REST API. Get the four levels right and every dataset in the company has exactly one fully-qualified name; get them wrong and you are back to per-source addressing.

The four levels of the namespace.

• Metalake — the top container. A metalake is the highest-level grouping, usually one per tenant, environment, or business unit; everything below it shares governance and identity. Engines connect to a metalake, and all names are resolved within it.
• Catalog — a connector to one backend. A catalog is created with a type (relational, fileset, messaging, model) and a provider that names the backend integration. The catalog holds the connection properties (URI, warehouse, backend) and delegates all operations to that backend.
• Schema — a namespace inside a catalog. A schema (a.k.a. database) groups entities within a catalog, mapping to the backend's own database/namespace concept.
• Table / fileset / topic / model — the entity. The leaf is a relational table, a fileset (a managed or external set of files/paths), a Kafka topic, or a registered ML model — each addressed as metalake.catalog.schema.<entity>.

Providers — the connector catalogue.

• Relational providers. hive (Hive metastore), lakehouse-iceberg (Iceberg with a Hive/JDBC/REST backend), lakehouse-paimon, and jdbc-mysql / jdbc-postgresql / jdbc-doris for operational databases exposed as read/write tables.
• Fileset provider. hadoop exposes filesets over any Hadoop-compatible filesystem — hdfs://, s3a://, gs://, abfss://, oss:// — so raw paths become governed, named entities.
• Messaging provider. kafka exposes topics as first-class entities so streaming data sits in the same namespace as tables and filesets.
• Model provider. A model catalog registers and versions ML models, giving them the same naming, tagging, and access model as data.

The server and entity store.

• A stateful service. The Gravitino server exposes the REST API, resolves names, applies governance, and persists its own metadata — the registrations, tags, roles, and properties — in an entity store.
• The entity store. Backed by a relational database (or an embedded key-value store for small deployments), it holds Gravitino's view: which catalogs exist, their properties, tags, and access rules — not a copy of every backend table's schema, which is fetched from the backend.
• The access surfaces. The same tree is reachable through the REST API, the gcli command-line client, language SDKs (Java/Python), and — for Iceberg tables — the Iceberg REST catalog service covered in the next section.

The failure modes senior engineers pre-empt.

• Flat naming. Treating catalogs as cosmetic and dumping everything into one blurs ownership and governance. Mitigation: model catalogs by backend/ownership boundary so the FQN reflects reality.
• Provider mismatch. Choosing hive for what is really an Iceberg table (or vice versa) breaks capabilities. Mitigation: pick the provider that matches the backend's true format and catalog.
• Treating Gravitino as the source of truth for table data. The entity store holds registrations, not the authoritative table schemas of federated backends. Mitigation: remember schemas are read through the provider; Gravitino governs and addresses, the backend still owns.

Common interview probes on the model.

• "What are the four levels?" — metalake → catalog → schema → table/fileset/topic/model.
• "What's a provider?" — the connector a catalog uses (hive, lakehouse-iceberg, jdbc-*, hadoop, kafka).
• "Where does Gravitino store metadata?" — its own entity store (registrations, tags, roles); backend schemas are read live.
• "How is a fileset different from a table?" — a fileset names files/paths as a governed entity; a table is relational — both share the namespace.

Worked example — create a metalake and catalogs of different providers

Detailed explanation. The bootstrap of any Gravitino deployment: create a metalake, then create catalogs that each wrap a different provider. Do it once through the REST API and once through the CLI to show they are the same operations.

• The metalake. company — the container for everything.
• The catalogs. An Iceberg catalog (JDBC-backed, S3 warehouse) and a Postgres JDBC catalog.
• The point. Two providers, one namespace, created with the same shape of call.

Question. Create a metalake company, an Iceberg catalog, and a Postgres catalog via REST and via the CLI.

Input.

Object	Type	Provider	Key properties
company	metalake	—	—
cat_iceberg	relational	lakehouse-iceberg	catalog-backend, uri, warehouse
cat_pg	relational	jdbc-postgresql	jdbc-url, jdbc-user

Code.

# --- REST: create the metalake ---
curl -s -X POST http://gravitino:8090/api/metalakes \
  -H 'Content-Type: application/json' \
  -d '{ "name": "company", "comment": "federated metadata lake" }'

# --- REST: create an Iceberg catalog (JDBC backend, S3 warehouse) ---
curl -s -X POST http://gravitino:8090/api/metalakes/company/catalogs \
  -H 'Content-Type: application/json' \
  -d '{
        "name": "cat_iceberg",
        "type": "RELATIONAL",
        "provider": "lakehouse-iceberg",
        "properties": {
          "catalog-backend": "jdbc",
          "uri": "jdbc:postgresql://pg:5432/iceberg_catalog",
          "warehouse": "s3a://lake/warehouse"
        }
      }'

# --- CLI (gcli): the SAME operations, terser ---
gcli metalake create --metalake company --comment "federated metadata lake"

gcli catalog create --metalake company --name cat_iceberg \
  --provider lakehouse-iceberg \
  --properties catalog-backend=jdbc,uri=jdbc:postgresql://pg:5432/iceberg_catalog,warehouse=s3a://lake/warehouse

gcli catalog create --metalake company --name cat_pg \
  --provider jdbc-postgresql \
  --properties jdbc-url=jdbc:postgresql://pg:5432/app,jdbc-user=gravitino,jdbc-password=***

# --- CLI: list what now exists under the metalake ---
gcli catalog list --metalake company
# -> cat_iceberg (lakehouse-iceberg), cat_pg (jdbc-postgresql)

Step-by-step explanation.

1. Creating the metalake company establishes the top-level container; every catalog, name, tag, and role from here on is scoped inside it, so two metalakes (say company and sandbox) are fully isolated worldviews.
2. The Iceberg catalog is created with provider: lakehouse-iceberg and the properties that tell Gravitino how to reach the real Iceberg catalog — here a JDBC backend and an S3 warehouse. Gravitino does not store the tables; it stores this registration and delegates.
3. The Postgres catalog uses provider: jdbc-postgresql, so an operational database's tables become read/write entities in the same namespace as the lakehouse tables — the federation in action.
4. The CLI calls are the same operations as the REST calls, just terser; the REST API, gcli, and the SDKs are three doors onto one server, which matters because CI/CD pipelines script the CLI while services use the SDK.
5. After catalog list, the metalake holds two catalogs of two providers, and every table under them is now addressable as company.cat_iceberg.<schema>.<table> or company.cat_pg.<schema>.<table> — one namespace, two very different backends.

Output.

Step	Result
create metalake company	container ready
create cat_iceberg (lakehouse-iceberg)	Iceberg tables federated
create cat_pg (jdbc-postgresql)	Postgres tables federated
catalog list	both visible under company

Rule of thumb. Bootstrap a metadata lake by creating the metalake first, then one catalog per backend with the provider that matches it — and script it through the CLI/SDK so registrations are version-controlled infrastructure, not clicks. The REST API, CLI, and SDK are interchangeable doors onto the same operations.

Worked example — one catalog per backend type: relational, fileset, messaging

Detailed explanation. The unifying power of the model is that a relational table, a set of files, and a Kafka topic all live under the same tree. Register one catalog of each of the three types and address an entity in each.

• Relational. cat_iceberg — tables.
• Fileset. cat_files (provider hadoop) — managed/external file paths.
• Messaging. cat_events (provider kafka) — topics.

Question. Register a fileset catalog and a Kafka catalog alongside the relational one, and show the fully-qualified name of an entity in each.

Input.

Catalog	Type	Provider	Entity example
cat_iceberg	relational	lakehouse-iceberg	sales.orders (table)
cat_files	fileset	hadoop	landing.raw_events (fileset)
cat_events	messaging	kafka	streams.clickstream (topic)

Code.

# Fileset catalog over S3 — file paths become governed, named entities.
gcli catalog create --metalake company --name cat_files \
  --provider hadoop \
  --properties location=s3a://lake/landing

# A fileset: a named handle to a path (external here — Gravitino governs, doesn't own the files).
gcli fileset create --metalake company \
  --name cat_files.landing.raw_events \
  --properties location=s3a://lake/landing/raw_events,kind=external

# Messaging catalog over Kafka — topics join the same namespace.
gcli catalog create --metalake company --name cat_events \
  --provider kafka \
  --properties bootstrap.servers=kafka:9092

# Three entity kinds, ONE addressing scheme — metalake.catalog.schema.entity:
company.cat_iceberg.sales.orders          # relational table
company.cat_files.landing.raw_events      # fileset (files at s3a://lake/landing/raw_events)
company.cat_events.streams.clickstream    # Kafka topic

# A single governance vocabulary now spans all three:
#   tag 'pii'        can sit on a column of orders OR a fileset OR a topic
#   role 'analyst'   can be granted SELECT on the table and USE on the others

Step-by-step explanation.

1. The fileset catalog (provider hadoop) turns raw object-store paths into governed entities: raw_events is now a name with tags, ownership, and access — not just a prefix someone remembers.
2. kind=external marks the fileset as one Gravitino governs but does not own (it will not delete the files on drop), versus a managed fileset whose lifecycle Gravitino controls — the same managed/external distinction relational catalogs use for tables.
3. The Kafka catalog (provider kafka) brings topics into the namespace, so streaming data is addressable and governable exactly like tables and filesets — one vocabulary for batch and streaming.
4. The payoff is the shared addressing scheme: company.<catalog>.<schema>.<entity> names a table, a fileset, and a topic identically, so tooling that understands the FQN understands all three.
5. Governance follows the addressing: a pii tag or an analyst role is expressed against the FQN, so the same classification and access model covers relational, file, and streaming data without three separate systems.

Output.

Entity	Fully-qualified name	Governed like
Iceberg table	company.cat_iceberg.sales.orders	table
Fileset	company.cat_files.landing.raw_events	files/paths
Kafka topic	company.cat_events.streams.clickstream	topic
all three	one FQN scheme	one tag/RBAC vocabulary

Rule of thumb. Model relational tables, filesets, and Kafka topics as catalogs of their respective types so all three share one FQN and one governance vocabulary. The value of a unified catalog is precisely that a pii tag or an access grant means the same thing whether the entity is a table, a file path, or a topic.

Worked example — the fully-qualified name as the single addressing scheme

Detailed explanation. Everything downstream — engine SQL, tags, grants, lineage — keys on the fully-qualified name. Make the FQN explicit and show how the same name resolves through the tree to a backend.

• The name. company.cat_iceberg.sales.orders.
• The resolution. metalake → catalog (provider + properties) → schema → table.
• The guarantee. The same name means the same entity from every access surface.

Question. Trace how company.cat_iceberg.sales.orders resolves, and show why the identical name works from CLI, SQL, and a tag.

Input.

FQN part	Level	Resolves to
company	metalake	the container/tenant
cat_iceberg	catalog	provider lakehouse-iceberg + backend props
sales	schema	Iceberg namespace sales
orders	table	the Iceberg table

Code.

Resolving  company.cat_iceberg.sales.orders
============================================
company        -> metalake        : scope + governance boundary
  cat_iceberg  -> catalog         : provider=lakehouse-iceberg, backend=jdbc, warehouse=s3a://lake/warehouse
    sales      -> schema          : Iceberg namespace "sales"
      orders   -> table           : Iceberg table -> metadata via backend, data on s3a://lake/warehouse/sales/orders

-- SQL engines address the entity by the SAME name (catalog.schema.table within the metalake):
SELECT region, sum(revenue_cents) AS rev
FROM cat_iceberg.sales.orders           -- Trino/Spark resolve this via Gravitino
GROUP BY region;

# CLI and governance key on the SAME FQN — no second naming scheme to learn:
gcli table details --metalake company --name cat_iceberg.sales.orders
gcli tag  set     --metalake company --name cat_iceberg.sales.orders --tag gold
# lineage events (OpenLineage) will also identify this dataset by the same name

Step-by-step explanation.

1. Resolution walks the tree top-down: the metalake fixes the scope, the catalog supplies the provider and backend connection, the schema maps to the backend namespace, and the table is the leaf — so the name itself encodes how to reach the entity, not just what to call it.
2. Crucially, the catalog level is where federation lives: cat_iceberg carries the Iceberg provider and S3 warehouse, so the same FQN transparently reaches an Iceberg table whose data sits on object storage, while a cat_pg FQN would reach Postgres — the caller does not change anything but the name.
3. SQL engines use the trailing three parts (catalog.schema.table) within the connected metalake, so a Trino or Spark query names the entity the same way a human does.
4. The CLI and governance operations key on the identical FQN, so there is exactly one naming scheme across query, administration, classification, and access — no translation layer between "what the engine calls it" and "what the catalog calls it."
5. Because lineage events also identify datasets by this FQN, an event emitted by Spark and one emitted by Trino for cat_iceberg.sales.orders land on the same node — the mechanical reason a unified name enables cross-engine lineage.

Output.

Access surface	How it names the entity	Same name?
SQL (Spark/Trino)	cat_iceberg.sales.orders	yes
CLI / SDK	company.cat_iceberg.sales.orders	yes
Tag / grant	company.cat_iceberg.sales.orders	yes
Lineage event	company.cat_iceberg.sales.orders	yes

Rule of thumb. Treat the fully-qualified name metalake.catalog.schema.entity as the one primitive everything keys on — query, CLI, tags, grants, and lineage all use it, so there is a single naming scheme across the platform. When the name is the same everywhere, governance and lineage compose for free.

Senior interview question on the Gravitino namespace model

A senior interviewer might ask: "Design the metadata model for a company with Iceberg lakehouse tables, an operational Postgres, raw filesets on two clouds, and Kafka topics — using Gravitino. Show the metalake/catalog/schema layout, which provider each catalog uses, how a relational table, a fileset, and a topic are each addressed, and where Gravitino keeps its own metadata versus what it reads from the backends."

Solution Using a metalake, provider-typed catalogs, and one FQN scheme

# 1. One metalake; one catalog per backend, each with its correct provider.
gcli metalake create --metalake company

gcli catalog create --metalake company --name cat_iceberg \
  --provider lakehouse-iceberg \
  --properties catalog-backend=jdbc,uri=jdbc:postgresql://pg:5432/ice,warehouse=s3a://lake/wh
gcli catalog create --metalake company --name cat_pg \
  --provider jdbc-postgresql --properties jdbc-url=jdbc:postgresql://pg:5432/app
gcli catalog create --metalake company --name cat_s3 \
  --provider hadoop --properties location=s3a://lake/
gcli catalog create --metalake company --name cat_gcs \
  --provider hadoop --properties location=gs://lake/
gcli catalog create --metalake company --name cat_events \
  --provider kafka --properties bootstrap.servers=kafka:9092

# 2. One addressing scheme across every entity kind and both clouds.
company.cat_iceberg.sales.orders        # relational table  (data on S3 warehouse)
company.cat_pg.app.customers            # operational table (federated Postgres)
company.cat_s3.landing.raw_events       # fileset  (s3a://lake/landing/raw_events)
company.cat_gcs.landing.raw_events_gcs  # fileset  (gs://lake/landing/raw_events)
company.cat_events.streams.clickstream  # Kafka topic

# 3. What Gravitino stores itself vs what it reads from the backend.
Gravitino entity store (its OWN metadata):
  - the metalake, catalogs, provider + connection properties
  - schemas/filesets/topics it manages, tags, roles/privileges
Read live FROM the backend (Gravitino does NOT copy):
  - the authoritative column schema of a federated Hive/Iceberg/JDBC table
  - current Kafka topic partitions, fileset directory contents

Step-by-step trace.

Level	Object	Backing
metalake	company	Gravitino entity store
catalog	cat_iceberg … cat_events	provider + properties (entity store)
schema	sales, app, landing, streams	backend namespace (federated)
table	orders, customers	backend metastore/DB (read live)
fileset	raw_events	path handle (governed by Gravitino)
topic	clickstream	Kafka (federated)

After the design, the metalake company holds one catalog per backend, each with the provider that matches it; a table, a Postgres table, two clouds' filesets, and a topic are all addressed by the same metalake.catalog.schema.entity scheme; and Gravitino persists only its own registrations, tags, and roles in the entity store while reading authoritative schemas from the backends on demand. The addressing is unified; the ownership of table data stays with the source systems.

Output:

Metric	Per-source modelling	Gravitino model
Naming schemes	one per system	one FQN
Entity kinds unified	no	tables + filesets + topics
Clouds spanned	separate configs	one namespace
Gravitino stores table data?	—	no (reads live)
Governance vocabulary	per system	one (tags/RBAC)

Why this works — concept by concept:

• Four-level namespace — metalake → catalog → schema → entity encodes both what a dataset is called and how to reach it, so the name resolves through the tree to the right backend without the caller knowing the backend.
• Provider-typed catalogs — each catalog wraps exactly one provider (hive, lakehouse-iceberg, jdbc-*, hadoop, kafka), so a relational table, a fileset, and a topic all join the same namespace while delegating to their true source.
• Entity store vs federated read — Gravitino persists its own registrations, tags, and roles but reads authoritative backend schemas live, so it governs and addresses without becoming a stale copy of every source.
• One FQN, one governance vocabulary — because everything shares metalake.catalog.schema.entity, a single tag/role/lineage scheme spans tables, files, topics, and both clouds instead of three per-system schemes.
• Cost — one server holding lightweight registrations plus live backend reads, versus per-system metadata modelling and per-cloud reconfiguration. The eliminated cost is the mental and operational overhead of many naming schemes — O(1) addressing across the estate instead of O(sources).

Design
Topic — design
Design problems on namespace modelling and catalog layout

Practice →

Data processing Topic — data-processing Data processing problems on tables, filesets, and topics

Practice →

---

3. Federation — one Iceberg REST endpoint over many backends

Engines speak one Iceberg REST URI; Gravitino routes to the real metastore behind it

The mental model in one line: Gravitino ships an Iceberg REST catalog service that implements the open Iceberg REST catalog spec, so any Iceberg-REST-compatible engine (Spark, Trino, Flink, StarRocks) points at one URI and Gravitino federates that single endpoint to whatever backend actually holds the metadata — a Hive metastore, a JDBC catalog, or an in-memory catalog — which is the federated catalog idea made concrete: instead of every engine carrying a bespoke config per Iceberg backend, they all speak one standard protocol to one address, and the metadata stays in place behind it rather than being copied into a new store. One endpoint, many backends, zero migration — the engine cannot tell whether the table's metadata lives in Hive or Postgres, and it does not need to.

What the Iceberg REST service is.

• A standard protocol. The Iceberg REST catalog spec is an open HTTP contract for listing namespaces, loading tables, and committing snapshots. Any engine that speaks it can use any compliant server — Gravitino is one such server.
• A single address. Engines configure one type = rest Iceberg catalog with one uri; they need no knowledge of the backend metastore, its host, or its protocol.
• A federating router. Behind the endpoint, Gravitino maps the requested Iceberg catalog to a configured backend — hive, jdbc, or memory — and forwards the operation, so one service can front several backends at once.
• Credential-aware. The Iceberg REST service can also vend short-lived storage credentials with the table-load response, so engines read data without static cloud keys (covered in section 5).

Backends the endpoint can front.

• Hive backend. Iceberg tables whose metadata lives in a Hive metastore — the most common brownfield case — are exposed over REST without touching the HMS.
• JDBC backend. Iceberg's JDBC catalog (metadata in Postgres/MySQL) is fronted the same way, so a newer JDBC-catalog lakehouse and an old Hive one both answer on the same protocol.
• Memory backend. An in-memory catalog for tests/dev, useful for spinning up a throwaway REST endpoint.
• One service, many. A single Gravitino Iceberg REST deployment can expose multiple backend catalogs, selected by the catalog name in the request path.

Federation, not migration.

• Metadata stays put. Fronting a Hive metastore over REST does not copy its tables into Gravitino; it routes to it. The HMS remains the store of record for those tables.
• Incremental adoption. You can put the REST endpoint in front of existing catalogs and migrate engines one at a time, because each engine only ever changes its catalog config to the single URI.
• Protocol decoupling. Engines are decoupled from backend churn: swap a Hive backend for a JDBC one behind the endpoint and the engine config does not change.

The failure modes senior engineers pre-empt.

• Assuming REST means "new store." Teams fear a migration that is not required. Mitigation: emphasise the endpoint fronts the existing metastore; nothing is copied.
• Mixing Gravitino access surfaces. Using the generic Gravitino connector and the Iceberg REST service interchangeably without deciding which. Mitigation: pick per engine — Iceberg REST for pure Iceberg workloads, the Gravitino connector for multi-provider access.
• Ignoring credential scope. Vending broad credentials from the REST response. Mitigation: scope vended tokens to the table's path and a short TTL.

Common interview probes on federation.

• "How does one endpoint serve many metastores?" — the Iceberg REST service routes each catalog request to its configured backend (hive/jdbc/memory).
• "Is this a migration?" — no; metadata stays in the backend, the endpoint fronts it.
• "Which engines can use it?" — any Iceberg-REST-compatible engine: Spark, Trino, Flink, StarRocks.
• "How do engines get storage credentials?" — the REST load response can vend short-lived, path-scoped tokens.

Worked example — stand up the Iceberg REST service over a Hive backend

Detailed explanation. The canonical federation setup: run Gravitino's Iceberg REST service configured with a Hive backend, then point Spark at the single REST URI. The Hive metastore is untouched; Spark just speaks REST.

• The backend. An existing Hive metastore at thrift://hms:9083.
• The service. Gravitino Iceberg REST, catalog-backend = hive.
• The engine. Spark with a type = rest Iceberg catalog pointing at the endpoint.

Question. Configure the Iceberg REST service over a Hive backend and connect Spark to it with one URI.

Input.

Piece	Value
Backend	Hive metastore thrift://hms:9083
Warehouse	s3a://lake/warehouse
REST endpoint	http://gravitino:9001/iceberg/
Engine catalog	Spark rest catalog lake

Code.

# Gravitino Iceberg REST service — front an existing Hive metastore (no migration).
gravitino.iceberg-rest.catalog-backend = hive
gravitino.iceberg-rest.uri             = thrift://hms:9083
gravitino.iceberg-rest.warehouse       = s3a://lake/warehouse
# service listens at http://<host>:9001/iceberg/

# Spark points at ONE Iceberg REST URI — it never knows a Hive metastore is behind it.
spark = (SparkSession.builder
    .config("spark.sql.catalog.lake", "org.apache.iceberg.spark.SparkCatalog")
    .config("spark.sql.catalog.lake.type", "rest")
    .config("spark.sql.catalog.lake.uri", "http://gravitino:9001/iceberg/")
    .config("spark.sql.catalog.lake.warehouse", "s3a://lake/warehouse")
    .getOrCreate())

# Standard Iceberg SQL — resolved over REST, executed against the Hive-backed tables.
spark.sql("SELECT region, sum(revenue_cents) FROM lake.sales.orders GROUP BY region").show()
spark.sql("INSERT INTO lake.sales.orders VALUES ('EU', 4200, DATE'2026-08-26')")

Step-by-step explanation.

1. The service config sets catalog-backend = hive and the HMS thrift URI, so Gravitino's Iceberg REST server translates every incoming REST call (list, load, commit) into an operation against the existing Hive metastore — the metastore is fronted, not replaced.
2. Spark configures a standard Iceberg rest catalog: type = rest and one uri. There is no Hive config in Spark at all — from Spark's side this is a vanilla Iceberg REST catalog, which is the whole point of using an open protocol.
3. spark.sql("... FROM lake.sales.orders ...") triggers a loadTable REST call; Gravitino resolves it against the Hive backend, returns the Iceberg table metadata, and Spark reads the data from s3a://lake/warehouse directly.
4. The INSERT commits a new Iceberg snapshot through the REST updateTable call, which Gravitino applies to the Hive-backed table — so writes federate too, not just reads.
5. Nothing was migrated: the Hive metastore is still the store of record, Spark carries one URI instead of a thrift config plus catalog wiring, and swapping the backend later (to JDBC) would leave the Spark config untouched.

Output.

Spark operation	REST call	Executed against
SELECT ... FROM lake.sales.orders	loadTable	Hive-backed Iceberg table
INSERT INTO lake.sales.orders	updateTable (commit)	same, new snapshot
Spark's Hive config	—	none (only a REST URI)
Hive metastore	untouched	still store of record

Rule of thumb. Front an existing Hive (or JDBC) Iceberg catalog with Gravitino's Iceberg REST service and give engines a single type = rest URI — no migration, no per-engine metastore config. The open protocol is what lets you swap the backend later without touching a single engine.

Worked example — one REST URI, multiple backend catalogs

Detailed explanation. Federation's real payoff is one endpoint fronting several backends. Configure the Iceberg REST service to expose a Hive-backed catalog and a JDBC-backed catalog, and reach both from the same engine URI.

• Backend A. warehouse_hive — Iceberg on a Hive metastore.
• Backend B. warehouse_jdbc — Iceberg on a JDBC catalog.
• The engine. One REST URI; the catalog name in the path selects the backend.

Question. Expose two Iceberg backends behind one REST endpoint and query a table from each with the same engine configuration.

Input.

Backend catalog	catalog-backend	Store of record
warehouse_hive	hive	Hive metastore
warehouse_jdbc	jdbc	Postgres JDBC catalog
endpoint	—	http://gravitino:9001/iceberg/

Code.

# Two named Iceberg backends behind ONE Gravitino Iceberg REST service.
# Backend 1 — Hive
gravitino.iceberg-rest.warehouse_hive.catalog-backend = hive
gravitino.iceberg-rest.warehouse_hive.uri             = thrift://hms:9083
gravitino.iceberg-rest.warehouse_hive.warehouse       = s3a://lake/hive-wh
# Backend 2 — JDBC
gravitino.iceberg-rest.warehouse_jdbc.catalog-backend = jdbc
gravitino.iceberg-rest.warehouse_jdbc.uri             = jdbc:postgresql://pg:5432/ice
gravitino.iceberg-rest.warehouse_jdbc.warehouse       = s3a://lake/jdbc-wh

# ONE engine config; the catalog name in the path picks the backend.
for backend in ("warehouse_hive", "warehouse_jdbc"):
    spark.conf.set(f"spark.sql.catalog.{backend}", "org.apache.iceberg.spark.SparkCatalog")
    spark.conf.set(f"spark.sql.catalog.{backend}.type", "rest")
    spark.conf.set(f"spark.sql.catalog.{backend}.uri",
                   f"http://gravitino:9001/iceberg/{backend}/")

# Same URI host, different backend behind it — the engine can't tell them apart.
spark.sql("SELECT count(*) FROM warehouse_hive.sales.orders").show()   # -> Hive backend
spark.sql("SELECT count(*) FROM warehouse_jdbc.mart.daily").show()     # -> JDBC backend

Step-by-step explanation.

1. The service is configured with two named backends (warehouse_hive, warehouse_jdbc), each with its own catalog-backend and connection — so a single Gravitino Iceberg REST deployment fronts a Hive metastore and a JDBC catalog simultaneously.
2. The engine adds one Iceberg rest catalog per backend, and the only thing that differs is the catalog name in the REST path (/iceberg/warehouse_hive/ vs /iceberg/warehouse_jdbc/) — same host, same protocol, same driver.
3. A query against warehouse_hive.sales.orders routes to the Hive backend; a query against warehouse_jdbc.mart.daily routes to the JDBC backend — the routing happens inside Gravitino, invisible to Spark.
4. This is the N × M → N + M collapse made physical: adding a third backend is a service-side registration, and every engine that already speaks the REST URI reaches it by adding one catalog name — not a new bespoke integration.
5. The engine genuinely cannot tell the backends apart: both are Iceberg-over-REST, so an operator can migrate a table's metadata from Hive to JDBC behind the endpoint and the engine keeps working — the decoupling that makes brownfield modernisation safe.

Output.

Engine query	REST path	Backend hit
warehouse_hive.sales.orders	/iceberg/warehouse_hive/	Hive metastore
warehouse_jdbc.mart.daily	/iceberg/warehouse_jdbc/	JDBC catalog
adding a 3rd backend	service-side registration	reachable by all engines
engine awareness of backend	none	fully decoupled

Rule of thumb. Run one Iceberg REST service with several named backends and let the catalog name in the path select the store — engines add a catalog name, not a new integration. One endpoint fronting many backends is what turns per-engine catalog sprawl into a single, standard address.

Worked example — the N×M → N+M integration collapse

Detailed explanation. Make the federation economics undeniable by counting integrations before and after. This is the number that justifies the metadata lake to a skeptical staff engineer.

• Before. Each engine integrates each Iceberg backend with its own config/driver.
• After. Each engine speaks one REST URI; each backend is registered once.
• The count. E engines and B backends.

Question. For 4 engines and 3 Iceberg backends, count integrations before and after federation and state the growth law.

Input.

Scenario	Integrations	Formula
Per-engine, per-backend	4 × 3 = 12	E × B
One REST endpoint	4 + 3 = 7	E + B
Add 1 engine (before)	+3	+B
Add 1 engine (after)	+1	+1

Code.

Before federation — every engine wires every backend.
=====================================================
              hive_backend   jdbc_backend   memory_backend
  Spark            X              X               X
  Trino            X              X               X
  Flink            X              X               X
  StarRocks        X              X               X
integrations = E x B = 4 x 3 = 12    (each X is a bespoke config)
adding one engine  -> +B  (=3) new integrations
adding one backend -> +E  (=4) new integrations

After federation — one Iceberg REST endpoint.
=============================================
  Spark ----\
  Trino -----\        one URI            hive_backend
  Flink ------>  [ Gravitino REST ] ---> jdbc_backend
  StarRocks -/                           memory_backend
integrations = E + B = 4 + 3 = 7    (E engine URIs + B backend registrations)
adding one engine  -> +1  (point it at the URI)
adding one backend -> +1  (register it once; all engines reach it)

Step-by-step explanation.

1. In the per-engine world, the integration count is the product E × B, because each engine independently configures each backend — twelve bespoke surfaces for four engines and three backends.
2. Worse than the count is its growth: adding one engine adds B integrations, and adding one backend adds E — the cost of every new component scales with the size of the estate, which is why sprawl compounds.
3. Federation makes the count the sum E + B: each engine has one URI (its single integration) and each backend is registered once behind the endpoint — seven surfaces instead of twelve.
4. The growth law flips to additive: a new engine is +1 (point it at the URI) and a new backend is +1 (register it once, and every engine reaches it) — so the marginal cost of a component is constant, not proportional to the estate.
5. This is the entire economic argument for a federated metadata layer: it converts a multiplicative integration surface into an additive one, which is the difference between a platform that gets harder to change as it grows and one that does not.

Output.

Change	Per-engine cost	Federated cost
Baseline (4 eng × 3 back)	12	7
Add an engine	+3 (+B)	+1
Add a backend	+4 (+E)	+1
Growth law	E × B	E + B

Rule of thumb. Justify a federated metadata layer with the growth law: per-engine integration is E × B and grows multiplicatively, federation is E + B and grows additively. When the marginal cost of a new engine or backend is constant instead of proportional to the estate, the platform stops fighting its own success.

Senior interview question on Iceberg REST federation

A senior interviewer might ask: "You have Iceberg tables split across an old Hive metastore and a newer JDBC catalog, and four engines each wired to both. You cannot migrate the metastores right now. How do you give every engine one way in, front both backends, keep the option to migrate later, and avoid the per-engine integration explosion — all without copying any metadata?"

Solution Using Gravitino's Iceberg REST service fronting multiple backends

# 1. One Iceberg REST service, two named backends — metadata stays in place.
gravitino.iceberg-rest.hive_wh.catalog-backend = hive
gravitino.iceberg-rest.hive_wh.uri             = thrift://hms:9083
gravitino.iceberg-rest.hive_wh.warehouse       = s3a://lake/hive-wh

gravitino.iceberg-rest.jdbc_wh.catalog-backend = jdbc
gravitino.iceberg-rest.jdbc_wh.uri             = jdbc:postgresql://pg:5432/ice
gravitino.iceberg-rest.jdbc_wh.warehouse       = s3a://lake/jdbc-wh

# 2. Every engine: ONE standard Iceberg REST catalog per backend name, same host.
def rest_catalog(spark, name):
    spark.conf.set(f"spark.sql.catalog.{name}", "org.apache.iceberg.spark.SparkCatalog")
    spark.conf.set(f"spark.sql.catalog.{name}.type", "rest")
    spark.conf.set(f"spark.sql.catalog.{name}.uri", f"http://gravitino:9001/iceberg/{name}/")

for n in ("hive_wh", "jdbc_wh"):
    rest_catalog(spark, n)      # Trino/Flink/StarRocks configure the analogous rest catalog

# 3. Integration count and the migration path.
integrations : 4 engines + 2 backends = 6      (not 4 x 2 = 8; grows as E + B)
migrate later: move a table's metadata hive_wh -> jdbc_wh BEHIND the endpoint;
               engines keep the SAME URI and SQL — zero engine changes.
no copies    : Gravitino ROUTES to hms:9083 / pg:5432; each stays store of record.

Step-by-step trace.

Requirement	Mechanism	Result
One way in per engine	single type=rest URI	Spark/Trino/Flink/StarRocks all speak REST
Front both backends	named backends in the service	one endpoint, two stores
No migration	Gravitino routes, never copies	HMS & JDBC stay authoritative
Migrate later	move metadata behind the endpoint	engine config unchanged
No explosion	E + B integrations	6, not 8 (and additive growth)

After deployment, the Iceberg REST service fronts both the Hive metastore and the JDBC catalog under one host; each of the four engines carries one standard Iceberg REST catalog per backend name; nothing is copied — Gravitino routes each request to hms:9083 or pg:5432, which remain the stores of record; and a future migration of a table from the Hive backend to the JDBC backend happens behind the endpoint with no engine changes. The integration surface is E + B = 6 and grows by one per new component.

Output:

Metric	Per-engine wiring	Iceberg REST federation
Ways into the lakehouse	one per (engine, backend)	one URI per engine
Metadata copied	—	none (routed)
Integrations (4 eng, 2 back)	8 (E × B)	6 (E + B)
Migrate a backend later	touch every engine	zero engine changes
Store of record	scattered	unchanged (HMS/JDBC)

Why this works — concept by concept:

• Open Iceberg REST protocol — engines speak a standard HTTP catalog contract, so one URI replaces per-engine metastore configuration and any compliant engine (Spark, Trino, Flink, StarRocks) integrates the same way.
• Named backends behind one endpoint — the service fronts a Hive metastore and a JDBC catalog at once, selected by the catalog name in the path, so a single deployment federates several stores.
• Route, don't copy — Gravitino forwards list/load/commit calls to the real backend, which stays the store of record, so federation requires no migration and no duplicate metadata to keep in sync.
• Backend decoupling — because the engine only knows the protocol, a table's metadata can move between backends behind the endpoint without changing a single engine config or query.
• Cost — one endpoint plus one URI per engine and one registration per backend, versus a bespoke integration per (engine, backend) pair. The eliminated cost is the multiplicative integration surface — O(E + B) to connect an estate instead of O(E × B).

Optimization
Topic — optimization
Optimization problems on integration surface and endpoint federation

Practice →

Data processing Topic — data-processing Data processing problems on lakehouse tables and Iceberg catalogs

Practice →

---

4. Connecting engines — Spark, Trino & Flink

Point every engine at one URI and metalake; the same fully-qualified name means the same table everywhere

The mental model in one line: beyond the Iceberg REST door, Gravitino ships native connectors for Spark, Trino, and Flink that each take the same two settings — the Gravitino uri and a metalake — and then expose every catalog in that metalake to the engine at once, so a Spark job, a Trino cluster, and a Flink pipeline all resolve the identical catalog.schema.table name through one server, giving true multi-cloud, multi-engine portability where a table registered once is queryable from any engine without re-registration, and a unified catalog becomes something you compute against, not just browse. Configure the engine once, and the whole metalake — Iceberg, Hive, JDBC, filesets — appears under names that match every other engine.

The Spark connector.

• Two settings plus a plugin. Register the Gravitino Spark plugin and set spark.sql.gravitino.uri and spark.sql.gravitino.metalake; every catalog in the metalake becomes a Spark catalog.
• Cross-catalog SQL. A single Spark session can JOIN a table in an Iceberg catalog with one in a JDBC catalog, because both are catalogs of the same connected metalake.
• Iceberg passthrough. For Iceberg catalogs, the connector delegates to the native Iceberg Spark integration, so snapshots, time travel, and writes work as usual — Gravitino handles naming and governance, Iceberg handles the table.

The Trino connector.

• One connector, many catalogs. The Gravitino Trino connector dynamically exposes each Gravitino catalog as a Trino catalog, so SHOW CATALOGS lists them without a properties file per catalog.
• Federated SQL. Trino's whole value is querying across catalogs; with Gravitino it queries across every registered backend under one connector — Iceberg joined to Postgres joined to Hive in one statement.
• Live registration. Register a new catalog in Gravitino and it appears in Trino without editing Trino's static catalog files and restarting per catalog.

The Flink connector and consistent naming.

• Catalog store integration. The Flink connector plugs Gravitino in as a catalog so streaming SQL addresses the same tables and topics as batch, under the same names.
• Same FQN, all engines. catalog_iceberg.sales.orders is the same entity in Spark, Trino, and Flink — no per-engine alias, no translation, one name.
• Portable across clouds. Because the name resolves through the server, an engine on cloud A and an engine on cloud B reach the same registered table without re-registering it locally.

The failure modes senior engineers pre-empt.

• Per-engine aliases. Letting each engine name catalogs differently defeats the unified namespace. Mitigation: use the Gravitino catalog names verbatim in every engine.
• Ignoring engine capability differences. Assuming every engine supports every provider identically. Mitigation: verify the engine's connector supports the catalog's provider (e.g. writes, streaming) before promising it.
• Static Trino files when dynamic is available. Re-introducing per-catalog config. Mitigation: let the Gravitino connector expose catalogs dynamically.

Common interview probes on engine connectivity.

• "How does Spark connect?" — the Gravitino Spark plugin plus uri and metalake; all catalogs appear.
• "How does Trino avoid a file per catalog?" — the Gravitino connector exposes every catalog dynamically.
• "Can one query span catalogs?" — yes; every registered catalog is a catalog in the engine, so cross-catalog joins work.
• "How is naming kept consistent?" — the same catalog.schema.table resolves through Gravitino for every engine.

Worked example — Spark connector config and a cross-catalog query

Detailed explanation. The Spark integration exposes the whole metalake to one session. Configure it, then run a query that joins an Iceberg table to a JDBC table — two backends, one SQL statement.

• The config. Plugin + uri + metalake.
• The catalogs. cat_iceberg (Iceberg) and cat_pg (Postgres).
• The query. Join orders (Iceberg) to customers (Postgres) in one statement.

Question. Configure the Gravitino Spark connector and join an Iceberg table to a Postgres table in a single query.

Input.

Setting	Value
Plugin	GravitinoSparkPlugin
spark.sql.gravitino.uri	http://gravitino:8090
spark.sql.gravitino.metalake	company
Catalogs seen	cat_iceberg, cat_pg

Code.

# Register the Gravitino Spark plugin and point it at ONE server + metalake.
spark = (SparkSession.builder
    .config("spark.plugins", "org.apache.gravitino.spark.connector.plugin.GravitinoSparkPlugin")
    .config("spark.sql.gravitino.uri", "http://gravitino:8090")
    .config("spark.sql.gravitino.metalake", "company")
    .config("spark.sql.gravitino.enableIcebergSupport", "true")
    .getOrCreate())

# Every catalog in the metalake is now a Spark catalog — no per-catalog wiring.
spark.sql("SHOW CATALOGS").show()   # -> cat_iceberg, cat_pg, cat_files, cat_events ...

-- One statement joins an ICEBERG table and a POSTGRES table — two backends, one query.
SELECT c.name, sum(o.revenue_cents) AS lifetime_cents
FROM cat_iceberg.sales.orders  AS o          -- Iceberg (data on S3)
JOIN cat_pg.app.customers      AS c          -- Postgres (federated JDBC)
  ON o.customer_id = c.id
WHERE o.region = 'EU'
GROUP BY c.name
ORDER BY lifetime_cents DESC;

Step-by-step explanation.

1. Registering GravitinoSparkPlugin and setting the uri and metalake is the entire integration; Spark now asks Gravitino for the catalog list rather than being told about catalogs one by one in config.
2. SHOW CATALOGS returns every catalog registered in the company metalake — Iceberg, Postgres, filesets, Kafka — so a single session can reach the whole estate without a per-catalog properties block.
3. The join names cat_iceberg.sales.orders and cat_pg.app.customers, which Gravitino resolves to an Iceberg table on S3 and a Postgres table respectively; Spark executes the join across the two backends in one plan.
4. enableIcebergSupport = true makes the connector delegate Iceberg operations to Spark's native Iceberg integration, so features like time travel and snapshot writes behave exactly as they would with a directly-configured Iceberg catalog — Gravitino adds naming and governance, not a reimplementation.
5. The senior observation: the same query text works in Trino next, because the catalog names are Gravitino's, not Spark-specific aliases — the unified namespace is what makes the SQL portable between engines.

Output.

Aspect	Without Gravitino	With the Spark connector
Catalog wiring	one config block each	uri + metalake only
Catalogs visible	those you configured	all in the metalake
Iceberg ⋈ Postgres join	two sessions/tools	one statement
Query portability	Spark-specific	same names in Trino/Flink

Rule of thumb. Give Spark the Gravitino plugin plus a uri and metalake, and the whole metalake becomes queryable — including cross-catalog joins in one statement. Use Gravitino's catalog names verbatim so the same SQL runs unchanged on Trino and Flink.

Worked example — Trino connector and one SQL across two catalogs

Detailed explanation. Trino's superpower is federated SQL, and the Gravitino connector supplies it every registered catalog dynamically. Configure one connector and query across catalogs without a file per catalog.

• The connector. A single gravitino Trino catalog with uri + metalake.
• The effect. Each Gravitino catalog appears as a Trino catalog.
• The query. Join across two of them.

Question. Configure the Gravitino Trino connector so every metalake catalog is a Trino catalog, then query across two.

Input.

Piece	Value
Trino catalog file	gravitino.properties
connector.name	gravitino
gravitino.uri	http://gravitino:8090
gravitino.metalake	company

Code.

# etc/catalog/gravitino.properties — ONE file exposes EVERY Gravitino catalog to Trino.
connector.name=gravitino
gravitino.uri=http://gravitino:8090
gravitino.metalake=company

-- Each Gravitino catalog now appears as a Trino catalog (no file-per-catalog, no restart).
SHOW CATALOGS;                 -- -> cat_iceberg, cat_pg, cat_hive, ...

-- Federated SQL across two backends in one Trino statement:
SELECT p.region, count(*) AS orders, avg(c.credit_limit) AS avg_limit
FROM cat_iceberg.sales.orders AS p         -- Iceberg backend
JOIN cat_pg.app.customers     AS c         -- Postgres backend
  ON p.customer_id = c.id
GROUP BY p.region;

-- Register a NEW catalog in Gravitino and it appears here with NO Trino restart:
--   gcli catalog create --metalake company --name cat_doris --provider jdbc-doris ...
--   SHOW CATALOGS;  -> now also lists cat_doris

Step-by-step explanation.

1. A single gravitino.properties with connector.name=gravitino and the uri/metalake is the only Trino config — the connector then asks Gravitino for the catalog list and surfaces each one as a Trino catalog.
2. SHOW CATALOGS lists every Gravitino catalog, so the classic Trino chore of writing and shipping one properties file per data source (and restarting) is replaced by one connector that discovers them.
3. The federated join reads cat_iceberg.sales.orders from the Iceberg backend and cat_pg.app.customers from Postgres in a single Trino query — Trino's cross-catalog planner works because both are catalogs of the same connected metalake.
4. Registering a new catalog in Gravitino (say a Doris JDBC catalog) makes it appear in Trino dynamically, without editing Trino's static catalog directory or restarting the coordinator per source — the operational win over file-based catalogs.
5. The names are identical to Spark's, so the earlier Spark join and this Trino join reference the same entities; a query authored against the unified namespace is engine-agnostic by construction.

Output.

Trino task	File-per-catalog Trino	Gravitino connector
Add a data source	new file + restart	register in Gravitino (dynamic)
Catalogs available	those with files	all in the metalake
Cross-catalog join	yes (if configured)	yes (all registered)
Names vs Spark	may differ	identical

Rule of thumb. Use the single Gravitino Trino connector so every registered catalog appears dynamically — no properties file per source, no restart to add one. Because the catalog names match Spark and Flink, federated SQL written once runs on any engine.

Worked example — the same FQN across Spark, Trino, and Flink

Detailed explanation. The portability claim is only real if the same name addresses the same table in every engine. Show the identical fully-qualified name used from all three, including a Flink streaming write into a topic and a batch read of the resulting table.

• The entity. company.cat_iceberg.sales.orders and a topic company.cat_events.streams.clickstream.
• The engines. Flink writes streaming, Spark/Trino read batch.
• The invariant. One name, three engines, zero re-registration.

Question. Address the same Iceberg table and Kafka topic from Flink, Spark, and Trino using the identical Gravitino names.

Input.

Engine	Role	Names it uses
Flink	streaming read/write	cat_events.streams.clickstream, cat_iceberg.sales.orders
Spark	batch ETL	same names
Trino	interactive SQL	same names

Code.

-- FLINK SQL (Gravitino catalog registered as the current catalog):
-- read a Kafka topic and write into an Iceberg table — both Gravitino-named.
INSERT INTO cat_iceberg.sales.orders
SELECT region, revenue_cents, order_date
FROM   cat_events.streams.clickstream            -- Kafka topic, same namespace
WHERE  event_type = 'purchase';

-- SPARK SQL — the SAME Iceberg table name, no re-registration, batch read:
SELECT region, sum(revenue_cents) FROM cat_iceberg.sales.orders GROUP BY region;

-- TRINO SQL — again the SAME name, interactive:
SELECT count(*) FROM cat_iceberg.sales.orders WHERE region = 'EU';

# One entity, three engines, one name — resolved through Gravitino:
company.cat_iceberg.sales.orders
   Flink  : streaming INSERT   ->\
   Spark  : batch aggregate    -->  SAME table, SAME snapshot lineage node
   Trino  : interactive count  ->/
# No engine holds its own registration; the metalake is the single source of names.

Step-by-step explanation.

1. Flink writes into cat_iceberg.sales.orders by reading cat_events.streams.clickstream — both are Gravitino names, so a streaming pipeline addresses a topic and a table under the same scheme it would use for anything else.
2. Spark reads the identical cat_iceberg.sales.orders for a batch aggregate with no separate registration; it resolves the name through the same metalake Flink used, so it sees the rows Flink wrote.
3. Trino queries the same name interactively; all three engines converge on one entity because the name is resolved by Gravitino, not by per-engine config that could drift.
4. The portability is structural: no engine keeps a private catalog registration, so there is nothing to keep in sync between engines or clouds — the metalake is the single source of names, and pointing a new engine at the uri/metalake is the whole onboarding.
5. Because all three engines name the table identically, their lineage and audit events reference the same node — a Flink write, a Spark aggregate, and a Trino read all attach to company.cat_iceberg.sales.orders, which is what makes cross-engine lineage (section 5) actually line up.

Output.

Engine	Operation	Name used	Re-registration
Flink	streaming INSERT	cat_iceberg.sales.orders	none
Spark	batch aggregate	cat_iceberg.sales.orders	none
Trino	interactive count	cat_iceberg.sales.orders	none
all	—	identical FQN	metalake is source of names

Rule of thumb. Verify portability by using the identical Gravitino name from every engine — Flink, Spark, and Trino should all address catalog.schema.table the same way with no per-engine registration. One source of names is what lets a table cross engines and clouds, and what makes lineage events from different engines land on the same node.

Senior interview question on multi-engine connectivity

A senior interviewer might ask: "You want Spark for ETL, Trino for interactive SQL, and Flink for streaming — all over the same Gravitino metalake spanning Iceberg, Hive, and JDBC catalogs across two clouds. Show how each engine connects, how you avoid per-catalog config and per-engine aliases, how a query or pipeline can span catalogs, and how you guarantee the same table name means the same table in every engine."

Solution Using the Gravitino Spark, Trino, and Flink connectors over one metalake

# 1. Spark — plugin + uri + metalake; all catalogs appear, cross-catalog joins work.
spark = (SparkSession.builder
    .config("spark.plugins", "org.apache.gravitino.spark.connector.plugin.GravitinoSparkPlugin")
    .config("spark.sql.gravitino.uri", "http://gravitino:8090")
    .config("spark.sql.gravitino.metalake", "company")
    .config("spark.sql.gravitino.enableIcebergSupport", "true")
    .getOrCreate())

# 2. Trino — ONE connector exposes EVERY Gravitino catalog dynamically (no file-per-source).
# etc/catalog/gravitino.properties
connector.name=gravitino
gravitino.uri=http://gravitino:8090
gravitino.metalake=company

-- 3. Flink — Gravitino catalog for streaming; SAME names as Spark/Trino.
--    (catalog store configured with the same uri + metalake=company)
INSERT INTO cat_iceberg.sales.orders
SELECT region, revenue_cents, order_date
FROM   cat_events.streams.clickstream WHERE event_type = 'purchase';

-- 4. The SAME federated SQL runs on Spark AND Trino — names are Gravitino's, not aliases.
SELECT c.name, sum(o.revenue_cents)
FROM cat_iceberg.sales.orders o JOIN cat_pg.app.customers c ON o.customer_id = c.id
GROUP BY c.name;

Step-by-step trace.

Requirement	Mechanism	Result
Spark connects	plugin + uri + metalake	all catalogs as Spark catalogs
Trino connects	one gravitino connector	all catalogs dynamically
Flink connects	Gravitino catalog store	streaming over same names
No per-catalog config	server supplies the catalog list	one config per engine
No per-engine alias	Gravitino names used verbatim	identical FQN everywhere
Span catalogs	every catalog is an engine catalog	cross-catalog joins/pipelines

After the rollout, each engine carries one configuration — a plugin plus uri/metalake for Spark, one connector for Trino, one catalog store for Flink — and every catalog in company appears in all three under identical names. A Flink pipeline writes an Iceberg table from a Kafka topic, Spark and Trino read it by the same name, and a federated join across Iceberg and Postgres runs unchanged on both Spark and Trino. No engine holds a private registration, so names cannot drift across engines or clouds.

Output:

Metric	Per-engine wiring	Gravitino connectors
Config per engine	one block per catalog	one (uri + metalake)
New catalog reaches engines	edit + restart each	register once (dynamic)
Cross-catalog query	if separately configured	native (all registered)
Same name across engines	not guaranteed	guaranteed
Cloud portability	re-register per cloud	none (server resolves)

Why this works — concept by concept:

• One config per engine — Spark's plugin, Trino's connector, and Flink's catalog store each take only the Gravitino uri and metalake, so the entire metalake becomes available without a per-catalog block per engine.
• Dynamic catalog exposure — the server supplies the catalog list at runtime, so registering a catalog once in Gravitino makes it appear in every engine without restarts or new files.
• Gravitino names verbatim — using the server's catalog names instead of per-engine aliases means catalog.schema.table addresses the same entity in Spark, Trino, and Flink, which is what makes SQL portable and lineage events align.
• Cross-catalog and cross-cloud reach — because every registered catalog is an engine catalog and names resolve through the server, one statement spans backends and a table registered once is reachable from any engine on any cloud.
• Cost — one integration per engine plus one registration per catalog, versus a config block per (engine, catalog) and re-registration per cloud. The eliminated cost is per-engine catalog maintenance and name drift — O(engines + catalogs) to run the estate instead of O(engines × catalogs).

ETL
Topic — etl
ETL problems on multi-engine pipelines over a shared catalog

Practice →

Data processing Topic — data-processing Data processing problems on Spark, Trino, and Flink over one namespace

Practice →

---

5. Governance & lineage — tags, access, credential vending

Classify, authorize, vend short-lived credentials, and trace lineage — once, in the metadata lake

The mental model in one line: because every dataset in a metadata lake has one governed name, governance moves into the metadata layer and is defined once — tags classify and make datasets discoverable across every catalog, a role-based access model grants privileges on securable objects (optionally pushed down to Apache Ranger so the underlying engines enforce it too), credential vending hands engines short-lived, path-scoped cloud tokens instead of static keys, and lineage (via OpenLineage) rides the unified namespace so events from different engines refer to the same nodes — turning classification, authorization, secrets, and provenance from four per-engine reimplementations into four properties of one layer. Govern the name once, and every engine that reaches the data through that name inherits the governance.

Tags — classification and discovery.

• Attach anywhere. A tag can sit on a catalog, schema, table, fileset, topic, or column, so pii, gold, or deprecated mean the same thing across every backend.
• Discovery by tag. List everything carrying a tag to answer "where is all our PII?" across the whole estate in one query — impossible when each system tags differently or not at all.
• Properties on tags. Tags carry properties (owner, policy id), so classification can drive downstream policy without hard-coding it per system.

Access control — RBAC on securable objects.

• Roles and privileges. A role bundles privileges (USE_CATALOG, CREATE_TABLE, SELECT_TABLE, MODIFY_TABLE) on securable objects (a metalake, catalog, schema, or table), granted to users and groups.
• Least privilege by name. Because securables are FQNs, a grant is as narrow as SELECT_TABLE on cat_iceberg.sales.* — scoped to exactly the objects it should cover.
• Owners. Every object has an owner with implicit rights, so administration has a clear chain rather than scattered ad-hoc grants.

Ranger pushdown and credential vending.

• Apache Ranger pushdown. Gravitino's authorization plugin can translate its grants into Ranger policies, so the underlying engines (Hive, HDFS, and others) enforce the same rules Gravitino expresses — defence in depth, not a single choke point.
• Credential vending. For fileset and Iceberg catalogs on cloud storage, Gravitino issues short-lived, path-scoped credentials (AWS STS tokens, GCS tokens, ADLS SAS, OSS STS) in the load response, so engines never hold static long-lived keys.
• Multi-cloud secrets. Vending is per-cloud, so a request for an S3 table gets an STS token and a request for a GCS fileset gets a GCS token — one model, every cloud, no static credentials in engine configs.

Lineage over the unified namespace.

• Same nodes across engines. Engines emit OpenLineage run events; because a dataset has one FQN, a Spark write and a Trino read of the same table attach to the same lineage node.
• Cross-catalog edges. A pipeline reading a Kafka topic and writing an Iceberg table records an edge between two different catalogs under one namespace — the cross-catalog lineage sprawl cannot produce.
• Provenance for governance. Lineage plus tags answers "what downstream tables inherit PII from this column?" — classification and provenance composing because both key on the FQN.

The failure modes senior engineers pre-empt.

• Governance only in Gravitino, engines wide open. Expressing grants in Gravitino but letting engines hit the storage directly. Mitigation: push down to Ranger and vend credentials so the enforcement is real, not advisory.
• Broad or long-lived credentials. Vending a token for a whole bucket for hours. Mitigation: scope to the table's path and a short TTL.
• Lineage with inconsistent names. Engines naming the same table differently break the graph. Mitigation: use Gravitino names everywhere (section 4) so events align.

Common interview probes on governance.

• "Where do tags/access live?" — in the metadata layer, on securable objects addressed by FQN, applied across catalogs.
• "How is it enforced in the engines?" — RBAC in Gravitino, optionally pushed down to Ranger for engine-side enforcement.
• "How do engines get storage credentials?" — credential vending: short-lived, path-scoped, per-cloud tokens.
• "How does cross-engine lineage work?" — OpenLineage events keyed on the shared FQN land on the same nodes.

Worked example — tag a PII column once, discover it everywhere

Detailed explanation. Classification is only useful if it spans the estate. Tag a column pii in one catalog and query for every object carrying the tag across all catalogs.

• The tag. pii, with an owner property.
• The target. A column of cat_pg.app.customers.
• The discovery. List all pii-tagged objects across every catalog.

Question. Create a pii tag, attach it to a column, and list everything tagged pii across the metalake.

Input.

Piece	Value
Tag	pii (property owner=privacy)
Attach to	cat_pg.app.customers.email (column)
Also on	cat_iceberg.sales.orders.billing_email
Discovery	list objects with tag pii

Code.

# 1. Create a tag once, with a property that can drive downstream policy.
gcli tag create --metalake company --tag pii --properties owner=privacy

# 2. Attach it to columns in DIFFERENT catalogs — same vocabulary across backends.
gcli tag set --metalake company --name cat_pg.app.customers.email          --tag pii
gcli tag set --metalake company --name cat_iceberg.sales.orders.billing_email --tag pii

# 3. Discover across the WHOLE estate in one query — "where is all our PII?"
gcli tag list-objects --metalake company --tag pii
-> cat_pg.app.customers.email            (Postgres backend)
-> cat_iceberg.sales.orders.billing_email (Iceberg backend, data on S3)

# The tag means the SAME thing on a Postgres column and an Iceberg column,
# because both are addressed as securables in one namespace.

Step-by-step explanation.

1. Creating the pii tag once, with an owner property, establishes a single classification vocabulary; the property lets a downstream policy engine key on the tag rather than hard-coding column lists.
2. Attaching the tag to a Postgres column and an Iceberg column shows the tag is backend-agnostic — the securable is an FQN, so tagging works identically whether the column lives in a JDBC database or an object-store table.
3. tag list-objects answers the governance question that sprawl makes unanswerable: "where is all our PII?" returns hits from every catalog in one query, because classification lives in the metadata layer, not per system.
4. This is discovery, not just labelling: an auditor or a policy job can enumerate every pii securable across the estate and act on it, which is only possible when one place knows about all catalogs.
5. Combined with lineage (below), a pii tag on a source column can be traced to downstream tables — classification and provenance compose because both are expressed against the same FQN.

Output.

Object	Backend	Tagged pii
cat_pg.app.customers.email	Postgres	yes
cat_iceberg.sales.orders.billing_email	Iceberg/S3	yes
tag list-objects pii	across all catalogs	both returned
Vocabulary	one (pii)	spans backends

Rule of thumb. Define classification tags once in the metadata lake and attach them by FQN, so pii means the same thing on a Postgres column and an Iceberg column and a single query finds all of it. Classification is only governance when it spans every catalog — which is exactly what a unified namespace gives you.

Worked example — roles, privileges, and Ranger pushdown

Detailed explanation. Authorization lives on securable objects and can be pushed to Ranger so engines enforce it too. Create a read-only analyst role scoped to one catalog and push it down.

• The role. analyst_ro — read-only on cat_iceberg.sales.*.
• The privileges. USE_CATALOG + SELECT_TABLE.
• The pushdown. Translate the grant into Ranger policies for engine-side enforcement.

Question. Create a least-privilege analyst role scoped to one catalog, grant it to a group, and enable Ranger pushdown.

Input.

Piece	Value
Role	analyst_ro
Privileges	USE_CATALOG, SELECT_TABLE
Securable	cat_iceberg, cat_iceberg.sales.*
Grantee	group analysts

Code.

# 1. A least-privilege role: read-only, scoped to ONE catalog by FQN.
gcli role create --metalake company --role analyst_ro \
  --privilege USE_CATALOG:cat_iceberg \
  --privilege SELECT_TABLE:cat_iceberg.sales

# 2. Grant it to a group (users inherit via group membership).
gcli group grant --metalake company --group analysts --role analyst_ro

# 3. Push authorization DOWN to Apache Ranger so the ENGINES enforce it too.
#    Gravitino translates its grants into Ranger policies (defence in depth).
gravitino.authorization.enable = true
gravitino.authorization.provider = ranger
gravitino.authorization.ranger.admin.url = http://ranger:6080
gravitino.authorization.ranger.service.name = hive_prod

# Result — one grant, two layers of enforcement:
#   Gravitino  : analyst_ro can SELECT cat_iceberg.sales.* and nothing else
#   Ranger     : the SAME rule as a Hive/HDFS policy -> engines refuse other access
# A user in 'analysts' running Trino or Spark can read sales tables, cannot MODIFY,
# cannot USE other catalogs — enforced in the metadata layer AND at the engine.

Step-by-step explanation.

1. analyst_ro bundles exactly two privileges on FQN securables — USE_CATALOG on cat_iceberg and SELECT_TABLE on cat_iceberg.sales — so the role is least-privilege by construction: it cannot write, and it cannot touch other catalogs.
2. Granting to the group analysts rather than individuals means membership drives access, so onboarding a new analyst is a group add, not a new grant — the standard scalable RBAC pattern.
3. Enabling the Ranger provider makes Gravitino translate its grants into Ranger policies, so the underlying Hive/HDFS enforcement matches what Gravitino expresses — the rule is enforced at the engine, not only advisory in the metadata layer.
4. This is defence in depth: even a path that bypasses Gravitino and hits Hive directly is still checked by the Ranger policy that mirrors the grant, closing the "governance only in the catalog, storage wide open" hole.
5. The senior framing: express authorization once against FQN securables, push it down so every enforcement point agrees, and keep grants scoped to names — the unified namespace is what lets one grant cover exactly the right objects and be mirrored consistently downstream.

Output.

Actor / action	Gravitino verdict	Ranger (engine) verdict
analysts SELECT cat_iceberg.sales.orders	allow	allow
analysts MODIFY cat_iceberg.sales.orders	deny	deny
analysts USE cat_pg	deny	deny
direct Hive read bypassing Gravitino	—	deny (mirrored policy)

Rule of thumb. Model access as least-privilege roles granting FQN-scoped privileges to groups, and push the grants down to Ranger so engines enforce the same rules Gravitino expresses. Authorization that lives only in the catalog while storage stays open is advisory; pushdown makes it real.

Worked example — credential vending for a cloud fileset

Detailed explanation. Static cloud keys in engine configs are the classic leak. Credential vending replaces them with short-lived, path-scoped tokens issued at load time. Configure vending for an S3-backed Iceberg catalog.

• The problem. Engines holding long-lived S3 keys.
• The fix. Gravitino vends an STS token scoped to the table path, valid minutes.
• The flow. Engine loads the table → Gravitino returns metadata plus a temporary credential.

Question. Enable credential vending on an S3-backed catalog so engines read data with short-lived, path-scoped tokens instead of static keys.

Input.

Piece	Value
Catalog	cat_iceberg (warehouse s3a://lake/wh)
Provider of creds	AWS STS (assume-role)
Scope	the requested table's path
TTL	short (e.g. 15 min)

Code.

# Enable credential vending on the catalog — Gravitino assumes a role and vends STS tokens.
gcli catalog create --metalake company --name cat_iceberg \
  --provider lakehouse-iceberg \
  --properties \
catalog-backend=jdbc,uri=jdbc:postgresql://pg:5432/ice,warehouse=s3a://lake/wh,\
credential-providers=s3-token,\
s3-region=eu-west-1,\
s3-role-arn=arn:aws:iam::123456789012:role/gravitino-vending,\
credential-cache-ttl=900

# Load flow — the engine never holds a static key.
Engine loadTable(cat_iceberg.sales.orders)
  -> Gravitino: return Iceberg metadata
             +  vend an STS token: { path: s3a://lake/wh/sales/orders/*, ttl: 15m, read }
  -> Engine reads the data files with the TEMPORARY, PATH-SCOPED token
  -> token expires in 15m; next load vends a fresh one

# Contrast:
#   static keys  : long-lived, broad (whole bucket), copied into every engine config  -> leak risk
#   vended token : short-lived, scoped to the table path, issued per request           -> minimal blast radius

Step-by-step explanation.

1. The catalog is configured with credential-providers=s3-token and an assume-role ARN, so instead of storing S3 keys, Gravitino is authorised to mint temporary credentials via STS for the warehouse path.
2. When an engine loads a table, Gravitino returns the Iceberg metadata and a freshly vended STS token scoped to that table's path with a short TTL — the credential arrives with the metadata, so the engine never needs a static key.
3. The token is path-scoped: it grants access to s3a://lake/wh/sales/orders/*, not the whole bucket, so even if it leaked its blast radius is one table for fifteen minutes.
4. Expiry is automatic: the credential-cache-ttl bounds the token's life, and the next load vends a fresh one — there is no long-lived secret to rotate or accidentally commit.
5. The same mechanism generalises across clouds — a GCS fileset vends a GCS token, an ADLS one vends a SAS — so a multi-cloud estate has one credential model (vend short-lived, path-scoped tokens) instead of static keys per cloud scattered through engine configs.

Output.

Property	Static keys	Credential vending
Lifetime	long-lived	short (e.g. 15 min)
Scope	whole bucket	the table's path
Where stored	every engine config	nowhere (issued per load)
Leak blast radius	large, lasting	one path, minutes

Rule of thumb. Turn on credential vending so engines receive short-lived, path-scoped cloud tokens at table-load time instead of carrying static keys — and let the same model cover S3, GCS, and ADLS. A vended credential's blast radius is one path for minutes; a static key's is the whole bucket forever.

Senior interview question on federated governance and lineage

A senior interviewer might ask: "Govern a multi-cloud Gravitino estate end to end. Show how you classify sensitive data across every catalog, how you grant least-privilege access and make engines actually enforce it, how engines read cloud data without static keys, and how you get lineage that spans Spark, Trino, and Flink — all defined once rather than per engine."

Solution Using tags, RBAC with Ranger pushdown, credential vending, and OpenLineage

# 1. Classify once, across every catalog (tags on FQN securables).
gcli tag create --metalake company --tag pii --properties owner=privacy
gcli tag set --metalake company --name cat_pg.app.customers.email --tag pii
gcli tag set --metalake company --name cat_iceberg.sales.orders.billing_email --tag pii

# 2. Least-privilege RBAC scoped by name, granted to a group.
gcli role create --metalake company --role analyst_ro \
  --privilege USE_CATALOG:cat_iceberg --privilege SELECT_TABLE:cat_iceberg.sales
gcli group grant --metalake company --group analysts --role analyst_ro

# 3. Push authorization to Ranger (engine-side enforcement) + vend cloud credentials.
gravitino.authorization.enable = true
gravitino.authorization.provider = ranger
gravitino.authorization.ranger.admin.url = http://ranger:6080
# catalog property: vend short-lived, path-scoped S3 tokens (no static keys in engines)
#   credential-providers=s3-token, s3-role-arn=arn:aws:iam::...:role/gravitino-vending, credential-cache-ttl=900

# 4. Lineage over the unified namespace — same nodes across engines.
Flink INSERT cat_iceberg.sales.orders  FROM cat_events.streams.clickstream
Spark  aggregate cat_iceberg.sales.orders -> cat_iceberg.mart.region_daily
Trino  read cat_iceberg.mart.region_daily
# OpenLineage events (all keyed by the SAME FQN) form ONE graph:
#   cat_events.streams.clickstream -> cat_iceberg.sales.orders -> cat_iceberg.mart.region_daily
# tag 'pii' on a source column is now TRACEABLE to every downstream node.

Step-by-step trace.

Governance concern	Mechanism	Enforced / visible where
Classification	tags on FQN securables	across all catalogs
Authorization	roles/privileges on securables	Gravitino + Ranger pushdown
Enforcement in engines	Ranger policies mirror grants	Hive/HDFS/engines
Cloud secrets	credential vending	short-lived, path-scoped tokens
Provenance	OpenLineage on shared FQN	one cross-engine graph
Composition	tags + lineage	PII traced downstream

After the rollout, sensitive columns across Postgres and Iceberg both carry pii and are discoverable in one query; analyst_ro grants read on cat_iceberg.sales.* only, mirrored into Ranger so engines enforce it and direct storage access is refused; engines read S3 data with vended, path-scoped, 15-minute tokens instead of static keys; and Flink, Spark, and Trino operations on identically-named datasets form one OpenLineage graph, so the pii tag on a source column can be traced to every downstream table. Every one of these is defined once in the metadata layer.

Output:

Metric	Per-engine governance	Gravitino (one layer)
PII discovery	per system, partial	one query, all catalogs
Access rules	duplicated per engine	one role, pushed to Ranger
Storage credentials	static keys everywhere	vended, short-lived, scoped
Lineage across engines	fragmented/none	one graph (shared FQN)
Tag→downstream tracing	impossible	tags + lineage compose

Why this works — concept by concept:

• Tags on FQN securables — classification attached by fully-qualified name means one vocabulary spans every backend, so "where is all our PII?" is a single query across catalogs rather than a per-system hunt.
• RBAC with Ranger pushdown — least-privilege roles granting privileges on named securables, translated into Ranger policies, make the same rule enforced in the metadata layer and at the engine, closing the advisory-only gap.
• Credential vending — issuing short-lived, path-scoped cloud tokens at load time removes static keys from engine configs entirely, shrinking a leak's blast radius from a whole bucket forever to one path for minutes.
• Lineage over one namespace — because engines name datasets identically, OpenLineage events land on the same nodes, so provenance spans Spark, Trino, and Flink and composes with tags to trace sensitive data downstream.
• Cost — one definition each of classification, access, secrets, and lineage in the metadata layer, versus four reimplementations per engine. The eliminated cost is duplicated, drift-prone governance — O(1) policy surfaces across the estate instead of O(engines × catalogs).

Data validation
Topic — data-validation
Data validation problems on classification, access, and lineage

Practice →

Design
Topic — design
Design problems on governance, RBAC, and credential vending

Practice →

---

Cheat sheet — Apache Gravitino

• The metadata-lake distinction. A metastore stores one system's metadata; a data catalog (DataHub/OpenMetadata) indexes many for humans to search; a metadata lake (Gravitino) federates many into one read/write namespace that engines operate through. Gravitino sits over metastores and can feed a data catalog — different layer, not a clone.
• Federation, not migration. Register each existing catalog once; metadata stays in its backend. The coupling law flips from engines × catalogs (multiplicative, per-source wiring) to engines + catalogs (additive, one config per engine + one registration per catalog).
• The namespace. metalake → catalog → schema → table/fileset/topic/model. A catalog wraps a provider: hive, lakehouse-iceberg, lakehouse-paimon, jdbc-mysql|postgresql|doris, hadoop (fileset), kafka, or model. Everything is addressed by the FQN metalake.catalog.schema.entity.
• Server + entity store. Gravitino is a service with a REST API and its own entity store (registrations, tags, roles) — it reads authoritative backend schemas live rather than copying them.
• Iceberg REST federation. Gravitino's Iceberg REST service implements the open spec, so any Iceberg-REST engine (Spark/Trino/Flink/StarRocks) points at one URI; the service fronts named backends (catalog-backend = hive | jdbc | memory) and routes by catalog. One endpoint, many metastores, zero copies.
• Spark connector. spark.plugins = ...GravitinoSparkPlugin; spark.sql.gravitino.uri + spark.sql.gravitino.metalake; enableIcebergSupport=true. Every catalog in the metalake becomes a Spark catalog; cross-catalog joins in one statement.
• Trino connector. One etc/catalog/gravitino.properties with connector.name=gravitino, gravitino.uri, gravitino.metalake — every Gravitino catalog appears dynamically as a Trino catalog (no file-per-source, no restart to add one).
• Flink connector. Gravitino as a catalog store (same uri + metalake) so streaming SQL uses the same names as batch. The FQN catalog.schema.table is identical across Spark, Trino, and Flink — no per-engine aliases.
• Tags. Attach pii / gold / deprecated to any securable (catalog→column) by FQN; one vocabulary across backends; list-objects answers "where is all our PII?" across the estate.
• RBAC + Ranger. Roles bundle privileges (USE_CATALOG, SELECT_TABLE, CREATE_TABLE, MODIFY_TABLE) on FQN securables, granted to users/groups. Push down to Apache Ranger so engines enforce the same rule (defence in depth), not just the catalog.
• Credential vending. For cloud filesets/Iceberg catalogs, vend short-lived, path-scoped tokens (S3 STS / GCS / ADLS SAS / OSS) at load time — no static keys in engine configs. Blast radius: one path for minutes, per cloud, one model.
• Lineage. Engines emit OpenLineage keyed on the shared FQN, so a Flink write, a Spark aggregate, and a Trino read land on the same nodes — cross-engine, cross-catalog lineage that composes with tags to trace sensitive data downstream.
• Multi-cloud portability. Register once; reach from any engine on any cloud without re-registration, because names resolve through the server and credentials are vended per cloud.

Frequently asked questions

What is Apache Gravitino and what is a metadata lake?

Apache Gravitino is an open-source, federated metadata lake: a single server that sits above the catalogs an organisation already runs — Hive metastores, Iceberg catalogs, JDBC databases, Kafka registries, and object-store filesets — and exposes them as one governed, read/write namespace. A "metadata lake" is distinct from a metastore (which stores one system's metadata) and from a data catalog like DataHub (which ingests metadata into a search index for humans): the metadata lake is in the operational path, so engines create, query, and govern tables through it. Its value is a single fully-qualified name metalake.catalog.schema.entity for every dataset regardless of backend, which is what makes unified governance, credential handling, and cross-engine lineage possible. Crucially, it federates the stores you have rather than migrating them — the metadata stays where it lives, and Gravitino routes to it.

How is Gravitino different from a Hive metastore or a data catalog like DataHub or Unity Catalog?

A Hive metastore is the store of record for one system family and stops at its protocol boundary, which is why estates accumulate several of them. A data discovery catalog (DataHub, OpenMetadata, Amundsen) ingests metadata from many systems into a search index so people can find and document datasets — but it is read-mostly and out of the query path; you do not create or query a table through it. Gravitino is a third thing: a federated, read/write metadata layer that engines operate through, so a table created in Spark is immediately visible to Trino under the same name with the same governance. It is complementary to the other two — Gravitino can sit over your metastores and feed your data catalog. Compared with a single-vendor governance catalog, its emphasis is open, multi-engine, multi-cloud federation via open protocols (its own REST plus the Iceberg REST spec) rather than lock-in to one platform.

How does the single Iceberg REST endpoint federation actually work?

Gravitino ships an Iceberg REST catalog service that implements the open Iceberg REST catalog specification. Any engine that speaks that spec — Spark, Trino, Flink, StarRocks — configures one type = rest Iceberg catalog with a single uri and needs no knowledge of the backend. Behind the endpoint, Gravitino is configured with one or more named backends, each with a catalog-backend of hive, jdbc, or memory, and it routes each incoming request (list namespaces, load table, commit snapshot) to the appropriate backend. So one endpoint can front an old Hive metastore and a newer JDBC catalog at once, selected by the catalog name in the request path, and the engine cannot tell them apart. Nothing is copied — the metastore remains the store of record — which means you can adopt the endpoint incrementally and even migrate a table's metadata between backends later without changing any engine's configuration. The service can also vend short-lived storage credentials in the load response.

How do Spark, Trino, and Flink connect to Gravitino?

Each engine has a native connector that takes the same two settings — the Gravitino uri and a metalake — and then exposes every catalog in that metalake at once. Spark registers the GravitinoSparkPlugin and sets spark.sql.gravitino.uri and spark.sql.gravitino.metalake; all catalogs become Spark catalogs and a session can join across them, with Iceberg operations delegated to Spark's native Iceberg support. Trino uses a single gravitino connector (connector.name=gravitino plus uri and metalake) that dynamically surfaces each Gravitino catalog as a Trino catalog, so you avoid a properties file per source and can add a catalog without restarting the coordinator. Flink plugs Gravitino in as a catalog store so streaming SQL addresses the same tables and topics as batch. Because all three use Gravitino's catalog names verbatim, catalog.schema.table is the same entity in every engine — no per-engine aliases, and a table registered once is reachable from any engine on any cloud.

Where does governance live — tags, RBAC, or Apache Ranger?

All three, layered, and defined once in the metadata lake. Tags provide classification and discovery: attach pii or gold to any securable by fully-qualified name and one query lists every tagged object across every catalog. Access control is role-based: roles bundle privileges (USE_CATALOG, SELECT_TABLE, CREATE_TABLE, MODIFY_TABLE) on securable objects (metalake, catalog, schema, table) granted to users and groups, scoped precisely because securables are FQNs. Apache Ranger is the enforcement pushdown: Gravitino can translate its grants into Ranger policies so the underlying engines (Hive, HDFS, and others) enforce the same rules, which is defence in depth rather than an advisory-only catalog. The pattern is to express classification and authorization once against fully-qualified names, push authorization down so every enforcement point agrees, and let tags plus lineage compose so sensitive data can be traced downstream.

How do credential vending and multi-cloud portability work?

Credential vending replaces static cloud keys in engine configs with short-lived, path-scoped tokens issued at table-load time. You configure a fileset or Iceberg catalog with a credential provider (for example s3-token with an assume-role ARN and a short cache TTL); when an engine loads a table, Gravitino returns the metadata plus a temporary credential — an AWS STS token, a GCS token, an ADLS SAS, or an OSS STS token — scoped to that table's path and valid for minutes. The engine reads the data with that token and never holds a long-lived key, so a leak's blast radius shrinks from a whole bucket forever to one path for minutes. The same model works per cloud, which is the backbone of multi-cloud portability: because a dataset has one governed name that resolves through the server and credentials are vended per request per cloud, the same table is reachable from any engine on any cloud without re-registration or scattered static secrets. Register once, reach it everywhere, with a fresh scoped credential each time.

Practice on PipeCode

• Drill the ETL practice library → for the multi-engine pipeline, cross-catalog join, and streaming-to-table problems that a federated metadata lake makes concrete.
• Rehearse platform patterns on the data processing practice library → for the Spark/Trino/Flink-over-one-namespace, Iceberg, and fileset scenarios where the unified catalog earns its keep.
• Sharpen the architecture axis with the system design practice library → for the federation, namespace-modelling, governance-placement, and credential-vending trade-offs a metadata platform must get right.
• Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the federation, RBAC, credential-vending, and lineage patterns against real graded inputs — metalakes, catalogs, Iceberg REST, and multi-cloud access.

Lock in federated-metadata muscle memory

Docs explain what Apache Gravitino is. PipeCode drills explain the decision — when to federate instead of migrate, why one `unified catalog` namespace is the precondition for `lineage`, when a single `Iceberg REST` endpoint beats per-engine config, and when credential vending has to replace static keys. Pipecode.ai is Leetcode for Data Engineering — metadata-platform practice tuned for the production trade-offs senior data engineers actually face.

Practice system design problems →
Practice data processing problems →