Gowtham Potureddi

Posted on Aug 29

Confluent Tableflow & Kafka-to-Iceberg: Streaming Topics Straight Into the Lakehouse

#python #sql #interview #dataengineering

Confluent Tableflow is the feature that finally lets a Kafka topic stop being a stream you have to drain into the lakehouse with a bespoke pipeline and start being an Apache Iceberg table you can simply query — materialized continuously from the topic, its schema taken straight from the registry, its small files compacted and its rows deduplicated without a single job you wrote yourself. The hard problem was never "get the bytes out of Kafka"; it was everything that came after. A topic is an append-only log tuned for low-latency consumers reading a few records at a time, and a lakehouse table is columnar Parquet tuned for engines scanning billions of rows — and for years the only bridge was a Kafka Connect sink connector you ran, plus a compaction job you scheduled, plus snapshot expiry you babysat, plus schema-mapping glue you maintained, plus commit coordination you hoped got exactly-once right.

This guide is the senior-data-engineering walkthrough for closing that gap — for turning a Kafka topic into a governed, queryable Iceberg table through materialization rather than a hand-built ingestion service — framed the way interviewers actually probe it: why a topic is not a table, how Tableflow represents a topic as an Iceberg (or Delta) table in your object store and catalog, how the schema registry drives the table schema so nobody writes DDL, how the Kafka-to-Iceberg mechanics work under the hood (serialization and type mapping, partitioning, compaction, and exactly-once commits), how one table in a shared catalog is queried unchanged from Spark, Trino, and Athena, and how Tableflow compares to the Kafka Connect Iceberg sink and Flink on cost and ops. Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the streaming practice library →, rehearse serving patterns on the real-time analytics practice library →, and sharpen the architecture axis with the system design practice library →.

On this page

Why stream Kafka topics into the lakehouse
Confluent Tableflow — materialize a topic as an Iceberg table
Kafka-to-Iceberg mechanics — serialization, partitioning, compaction, exactly-once
Schema Registry to Iceberg — query from Spark, Trino, and Athena
Tableflow vs the Kafka Connect Iceberg sink and Flink — cost and ops
Cheat sheet — Kafka-to-Iceberg with Tableflow
Frequently asked questions
Practice on PipeCode

1. Why stream Kafka topics into the lakehouse

A topic is an append-only log; an Iceberg table is a scannable dataset — and nothing native bridges them

The one-sentence invariant: a Kafka topic and a lakehouse table are two different physical representations of the same events — a topic is a partitioned, append-only, row-oriented log tuned for milliseconds-fresh streaming consumers, and an Iceberg table is a columnar, partitioned, catalog-tracked dataset tuned for cheap analytical scans and time travel — so the whole job of materialization is to keep one continuously in sync with the other, which historically meant a sink connector, a compaction job, snapshot expiry, schema mapping, and commit coordination you built and operated, and which Confluent Tableflow collapses into a topic-to-table configuration. Point a batch analytics engine at raw Kafka and you get no columnar pruning, no time travel, and a consumer that falls behind; hand-roll the bridge and you inherit a pipeline that breaks every time the schema changes or the small files pile up.

The two representations, and why the gap is real.

The topic side. A topic is an ordered log per partition, retained for hours or days, read sequentially by consumers tracking offsets. It is optimised for throughput and recency, not for "scan last quarter's orders grouped by region." There is no column pruning, no statistics, no schema catalog a SQL engine can plan against.
The table side. An Iceberg table is Parquet data files plus metadata (manifests, snapshots, a schema with field IDs) registered in a catalog. It is optimised for analytical scans: partition pruning, predicate pushdown, time travel, and multi-engine reads. But it expects files written and committed in Iceberg's transactional protocol.
The gap. Nothing in Kafka natively writes Iceberg, and nothing in Iceberg natively reads a topic. Something must continuously read the log, serialize records into partitioned Parquet, commit Iceberg snapshots, evolve the schema, and compact small files — correctly, forever.

The old bridge — a pipeline you build and babysit.

A sink connector. A Kafka Connect cluster running an Iceberg sink connector, consuming the topic and writing data files. You size it, monitor it, and restart it.
A compaction job. Streaming writes produce many small files; a scheduled rewrite_data_files job compacts them so queries stay fast. Another thing to run.
Snapshot expiry + orphan cleanup. Iceberg accumulates snapshots and orphaned files; you schedule expire_snapshots and remove_orphan_files or storage grows unbounded.
Schema + commit glue. You map the registry's Avro/Protobuf schema to Iceberg types, keep them in step through evolutions, and make sure commits are idempotent so a connector restart does not duplicate rows.

The 2026 reality — materialization as a managed feature.

Tableflow represents a Kafka topic as an Iceberg (or Delta) table continuously: it reads the topic, writes partitioned Parquet, commits snapshots, and publishes the table to a catalog — with compaction and other table maintenance handled for you.
The schema comes from the registry. The table schema is the topic's registered Avro/Protobuf/JSON schema; there is no hand-written DDL and no drift between the stream contract and the table contract.
Exactly-once is a property, not a hope. Tableflow tracks Kafka offsets and commits Iceberg snapshots idempotently, so a restart or replay materializes each record once — no duplicate rows.
The catalog is the interop point. Once the table is in a shared catalog (AWS Glue, a REST catalog, Snowflake's Open Catalog), Spark, Trino, Athena, and Snowflake read the same table — no per-engine copy.

What interviewers listen for.

Do you explain that a topic is not a table and name the columnar/append-only mismatch unprompted? — senior signal.
Do you enumerate the maintenance a hand-built sink inherits (compaction, snapshot expiry, schema mapping, commit idempotency)? — senior signal.
Do you say the schema comes from the registry, not hand-written DDL? — required answer.
Do you treat exactly-once / no duplicate rows as a correctness property of the ingestion, not an afterthought? — required answer.
Do you name the catalog as what makes one table queryable by many engines? — senior signal.

Worked example — the topic-vs-table decision table

Detailed explanation. The most useful artifact for a Kafka-to-lakehouse interview is a memorised mapping of access pattern → representation. Every senior discussion converges on it: given a consumer, do you read the live topic, materialize an Iceberg table, or both? Walk through building the table for an orders event stream feeding several consumers.

The consumers. A real-time fraud check (needs each event in milliseconds), a BI dashboard over the last 90 days (needs cheap columnar scans), a data-science feature backfill (needs full history plus time travel).
The tension. The topic serves recency but cannot scan history cheaply; the Iceberg table serves scans but is seconds-to-minutes behind the log.
The rule. Match the representation to the consumer's freshness and scan needs — and let Tableflow keep the table in sync so you are not choosing either/or.

Question. For each consumer, name the representation it reads and why the other one is the wrong tool.

Input.

Consumer	Freshness need	Access shape	Representation
Real-time fraud check	milliseconds	per-event	the Kafka topic (stream)
BI dashboard (90 days)	minutes OK	columnar scan/aggregate	Iceberg table (materialized)
DS feature backfill	historical	full scan + time travel	Iceberg table + snapshots
Ops "orders last 1 min"	seconds	small recent range	topic or stream-fed table

Code.

Two representations of ONE order event, kept in sync by Tableflow.

  Kafka topic `orders`  (append-only log, row-oriented)
    partition 0: [off 100][off 101][off 102] ...   <- consumers read by OFFSET
    - great for: per-event, milliseconds-fresh, streaming joins
    - bad  for: "sum(amount) by region for 90 days"  (no columns, no pruning)

        |  Tableflow materializes continuously
        v

  Iceberg table `orders`  (columnar Parquet + snapshots, in a catalog)
    data/  region=EU/  day=2026-08-26/  part-0001.parquet ...
    - great for: analytical scans, partition pruning, time travel, multi-engine
    - bad  for: "tell me about THIS event 5 ms after it happened"

Rule: don't force one representation to do the other's job.
      Read the topic for recency; read the table for scans; Tableflow keeps them consistent.

Step-by-step explanation.

The topic stores each order as a row in an append-only log addressed by offset — perfect for the fraud check that must see event 102 milliseconds after it is produced, and useless for a 90-day aggregate because there are no columns to prune and no statistics to plan a scan.
The Iceberg table stores the same orders as columnar Parquet partitioned by region/day, registered in a catalog — perfect for the dashboard's sum(amount) scan because the engine prunes partitions and reads only the amount column, and wrong for the fraud check because it is seconds-to-minutes behind the log.
Tableflow is the arrow between them: it continuously materializes the topic into the table, so you do not pick either streaming or lakehouse — you keep both, consistent, from one source of truth.
The DS backfill wants full history and the ability to read the table as it was last Tuesday; Iceberg snapshots give time travel, which the raw topic (bounded by retention) cannot.
The mistake is forcing one representation to do the other's job: scanning 90 days off the topic melts your consumers, and polling the table every 5 ms for a single event is both slow and stale. Name the access pattern; pick the representation; let materialization keep them in sync.

Output.

Access pattern	Right representation	Wrong representation (common mistake)
Per-event, ms-fresh	Kafka topic	scan the Iceberg table per event
Analytical scan/aggregate	Iceberg table	replay the whole topic each query
Historical + time travel	Iceberg snapshots	rely on topic retention
Recent small range	topic or stream-fed table	full table scan

Rule of thumb. A topic and a table are two representations of the same events; read the topic for recency and the Iceberg table for scans, and use materialization to keep them in sync instead of forcing one to do the other's job. Name the consumer's freshness and scan needs first — the representation follows.

Worked example — what maintenance a hand-built sink actually costs

Detailed explanation. Interviewers love to ask "so you'd just run the Iceberg sink connector?" because the naive answer stops at ingestion and forgets the operational tail. The senior answer enumerates everything you inherit the moment you own the bridge yourself. Walk the full list for a self-managed Kafka-to-Iceberg pipeline.

The visible part. A Connect cluster + the Iceberg sink connector consuming the topic and writing files.
The invisible part. Compaction, snapshot expiry, orphan cleanup, schema mapping, and commit idempotency — each a job or a guarantee you must provide.
The point. Ingestion is maybe 20% of the work; the maintenance tail is the other 80%, and it never stops.

Question. List the operational responsibilities a self-managed Iceberg sink inherits beyond "consume the topic and write files," and say what Tableflow does with each.

Input.

Responsibility	Self-managed (you own it)	Tableflow (managed)
Ingestion	run Connect + sink connector	built in
Small files	schedule `rewrite_data_files`	managed compaction
Snapshot growth	schedule `expire_snapshots`	managed maintenance
Orphan files	schedule `remove_orphan_files`	managed maintenance
Schema mapping	hand-map registry → Iceberg, keep in step	schema-registry-driven
Duplicate rows	make commits idempotent	offset-tracked exactly-once

Code.

Self-managed Kafka -> Iceberg: the jobs you now own (and page on)
================================================================

[1] INGEST      Connect cluster + Iceberg sink connector
                 -> size it, monitor lag, restart on failure

[2] COMPACT     spark.sql("CALL sys.rewrite_data_files('db.orders')")
                 -> nightly/hourly; streaming writes make thousands of small files

[3] EXPIRE      spark.sql("CALL sys.expire_snapshots('db.orders', older_than => ...)")
                 -> or snapshot metadata + storage grows forever

[4] ORPHANS     spark.sql("CALL sys.remove_orphan_files('db.orders')")
                 -> failed writes leave files the table never references

[5] SCHEMA      map Avro/Protobuf -> Iceberg types by hand; re-map on every evolution
                 -> drift here = ingestion breaks or columns silently drop

[6] EXACTLY-ONCE  ensure connector commits are idempotent on restart/rebalance
                 -> get it wrong and a replay DOUBLES rows

Tableflow: items [1]-[6] are the product. You configure the topic; it owns the tail.

Step-by-step explanation.

Item [1] is the only part most people picture: a Connect cluster and the sink connector. But a connector that only ingests leaves a table that degrades — the remaining items are why "just run the connector" is a junior answer.
Items [2]–[4] are the Iceberg table maintenance trio. Streaming ingestion writes small files constantly, so without periodic rewrite_data_files your query engines open thousands of tiny Parquet files and slow to a crawl; without expire_snapshots and remove_orphan_files, metadata and storage grow without bound. These are recurring jobs on their own schedule and compute.
Item [5] is the schema tax: you map the registry's Avro/Protobuf types to Iceberg types, and every time a producer evolves the schema you must keep the mapping in step or ingestion breaks (or, worse, silently drops a column).
Item [6] is the correctness tax: on a connector restart or a consumer-group rebalance, a naive sink can re-process offsets and duplicate rows in the table. Making commits idempotent (so a re-processed batch is a no-op) is subtle and easy to get wrong.
Tableflow's pitch is precisely that items [1]–[6] are the managed product: you declare which topic to materialize, and compaction, maintenance, schema propagation, and exactly-once commits are the service's job — so the 80% operational tail stops being yours.

Output.

Cost bucket	Self-managed	Tableflow
Clusters to run	Connect (+ Spark for maintenance)	none
Recurring jobs	compaction, expiry, orphan cleanup	managed
Schema drift risk	hand-mapped, per evolution	registry-driven
Duplicate-row risk	your idempotency logic	offset-tracked EOS
Where the eng time goes	the maintenance tail	the config

Rule of thumb. When someone says "just run the Iceberg sink connector," name the tail it drags in — compaction, snapshot expiry, orphan cleanup, schema mapping, and commit idempotency. Those recurring jobs and guarantees are the real cost, and they are exactly what a managed materialization like Tableflow absorbs.

Worked example — the 5-minute senior serving answer

Detailed explanation. The Kafka-to-lakehouse interview has a predictable escalation: an ambiguous opener ("get this topic into the warehouse"), then narrowing follow-ups on schema, small files, duplicates, and query access. The candidates who pre-empt all four score highest. Draft the monologue.

Ambiguous opener. "We need orders from Kafka queryable in the lakehouse. How?"
Follow-up 1. "Where does the table schema come from?" — probes schema registry.
Follow-up 2. "Streaming writes make tons of tiny files. Now what?" — probes compaction.
Follow-up 3. "A connector restart duplicated rows once. Why?" — probes exactly-once.
Follow-up 4. "Which engines can read it?" — probes catalog interop.

Question. Draft a 5-minute senior answer that pre-empts schema, compaction, duplicates, and query access without waiting to be asked.

Input.

Interview signal	Weak answer	Senior answer
Get topic into lakehouse	"dump it to files nightly"	"materialize the topic as an Iceberg table"
Schema source	"we'll write the DDL"	"the registered Avro/Protobuf schema drives it"
Small files	"we'll deal with it later"	"compaction — managed by Tableflow, or a rewrite job"
Duplicates	"hope the connector is fine"	"offset-tracked idempotent commits = exactly-once"
Query access	"copy it into Snowflake"	"one Iceberg table in a catalog, read by all engines"

Code.

Senior Kafka -> Iceberg answer template (5 minutes)
===================================================

Minute 1 — topic is not a table
  "A topic is an append-only log for streaming consumers; the lakehouse needs
   a columnar Iceberg table. I'd MATERIALIZE the topic into an Iceberg table —
   with Tableflow that's a config on the topic, not a pipeline I run."

Minute 2 — schema from the registry
  "The table schema comes straight from the Schema Registry subject (Avro/
   Protobuf). No hand-written DDL, no drift between the stream and the table."

Minute 3 — small files + maintenance
  "Streaming writes create many small files, so compaction is non-negotiable.
   Tableflow runs it (plus snapshot expiry) for me; self-managed, that's a
   rewrite_data_files job on a schedule."

Minute 4 — exactly-once
  "Correctness = no duplicate rows. Tableflow tracks Kafka offsets and commits
   Iceberg snapshots idempotently, so a restart or replay materializes each
   record once. Kafka's EOS (transactional producer, read_committed) upstream."

Minute 5 — query anywhere
  "It's ONE Iceberg table in a shared catalog (Glue/REST/Snowflake), so Spark,
   Trino, Athena, and Snowflake read it unchanged — no per-engine copy, and it
   time-travels for backfills and audits."

Step-by-step explanation.

Minute 1 frames the whole answer around representation: naming "a topic is not a table" and reaching for materialization signals you understand the architecture, not just that Kafka and Iceberg both exist.
Minute 2 pre-empts the schema follow-up. Saying the schema comes from the registry — not hand-written DDL — is the difference between an engineer who has run this and one who will discover schema drift in production.
Minute 3 volunteers compaction before the interviewer raises small files, which is the single most common way a naive Kafka-to-Iceberg pipeline degrades. Naming managed compaction (or the rewrite job you'd otherwise run) shows you know the maintenance tail.
Minute 4 states exactly-once as a correctness property with a mechanism (offset-tracked idempotent commits) rather than a wish — and correctly attributes upstream EOS to Kafka's transactional producer and read_committed consumers.
Minute 5 closes on the catalog as the interop point: one table, many engines, plus time travel. That is the sentence that separates a platform engineer from someone who would copy the data into every tool.

Output.

Grading criterion	Weak score	Senior score
Topic-vs-table framing	rare	mandatory
Schema from registry	occasional	mandatory
Compaction named	rare	senior signal
Exactly-once with a mechanism	rare	senior signal
Catalog interop + time travel	rare	senior signal

Rule of thumb. The senior Kafka-to-lakehouse answer is a 5-minute monologue covering topic-vs-table, schema-from-registry, compaction, exactly-once, and catalog interop — delivered before the follow-ups. Rehearse it once; deploy it every interview.

Senior interview question on the Kafka-to-lakehouse boundary

A senior interviewer often opens with: "Your platform produces an orders stream to Kafka, and analysts want it queryable in the lakehouse with cheap 90-day scans, while a fraud service still needs each event in milliseconds. You currently have only the topic. Design the boundary: how you make it an Iceberg table, where the schema comes from, what happens to small files and duplicate rows, and how you keep from building and operating a bespoke ingestion pipeline — and explain why the topic and the table coexist rather than one replacing the other."

Solution Using topic-to-table materialization, a registry-driven schema, managed maintenance, and a shared catalog

# 1. Two representations, one source of truth — the topic stays; the table is materialized.
Kafka topic `orders`  --->  Confluent Tableflow  --->  Iceberg table `orders`
   (fraud reads here,          (materialize:              (analysts read here,
    ms-fresh)                   schema-driven, EOS,         90-day columnar scans,
                                compaction managed)         time travel)

// 2. The schema is NOT hand-written — it is the registered subject `orders-value`.
{
  "type": "record", "name": "Order",
  "fields": [
    {"name": "id",        "type": "long"},
    {"name": "region",    "type": "string"},
    {"name": "amount",    "type": {"type": "bytes", "logicalType": "decimal", "precision": 12, "scale": 2}},
    {"name": "event_time","type": {"type": "long",  "logicalType": "timestamp-millis"}}
  ]
}

# 3. Tableflow config on the topic — materialize to Iceberg, your bucket, into a catalog.
tableflow:
  topic: orders
  table_format: ICEBERG              # or DELTA
  storage: { bucket: s3://lake/orders, region: us-east-1 }   # your object store
  catalog:
    type: GLUE                        # or REST / SNOWFLAKE_OPEN_CATALOG
    database: analytics
  partitioning: [ "day(event_time)" ] # hidden partitioning for pruning
  # compaction, snapshot expiry, and exactly-once commits are managed by Tableflow

-- 4. Analysts query the ONE Iceberg table from any engine via the catalog.
SELECT region, sum(amount) AS revenue
FROM analytics.orders
WHERE event_time >= current_date - INTERVAL '90' DAY   -- prunes day() partitions
GROUP BY region;

Step-by-step trace.

Decision	Before (topic only)	After (Tableflow)
Analytical scans	replay the whole topic	columnar Iceberg scan, partition-pruned
Table schema	hand-written DDL, drifts	registry subject, no drift
Small files	pile up (if self-built)	managed compaction
Duplicate rows	risk on restart	offset-tracked exactly-once
Query engines	none can read a topic	Spark/Trino/Athena via catalog
Fraud path	reads the topic	still reads the topic (unchanged)

After the rollout, the orders topic is untouched for the fraud service, while Tableflow continuously materializes it into the analytics.orders Iceberg table whose schema is exactly the registered orders-value subject; compaction and snapshot expiry are the service's job, offset-tracked idempotent commits guarantee no duplicate rows on a restart, and the table lives in a Glue catalog so Spark, Trino, and Athena all scan the same partitioned Parquet. Analysts get 90-day columnar scans with partition pruning; nobody built or operates an ingestion pipeline.

Output:

Metric	Before (topic only)	After (Tableflow)
90-day aggregate	full topic replay (slow)	pruned columnar scan
Pipeline to operate	a bespoke sink + jobs	none (config)
Schema drift risk	hand-mapped DDL	zero (registry-driven)
Duplicate-row risk	restart-dependent	zero (exactly-once)
Engines that can query	none (it's a log)	all (shared catalog)
Fraud latency	milliseconds	milliseconds (unchanged)

Why this works — concept by concept:

Topic-to-table materialization — the topic remains the streaming source of truth while Tableflow keeps a columnar Iceberg mirror in sync, so recency-sensitive and scan-sensitive consumers each read the representation built for them instead of one being forced to do the other's job.
Registry-driven schema — the Iceberg schema is the registered Avro/Protobuf subject, so there is no hand-written DDL to drift and the stream contract and table contract can never diverge.
Managed maintenance — compaction, snapshot expiry, and orphan cleanup are the service's responsibility, so the small-file degradation and unbounded-metadata growth that sink a self-built pipeline simply do not happen.
Exactly-once commits — offset-tracked idempotent Iceberg snapshots mean a restart, rebalance, or replay materializes each record once, so a correctness property is enforced by the ingestion rather than hoped for.
Cost — one managed materialization plus object storage, versus a Connect cluster, a maintenance-job fleet, and the engineering time to keep schema mapping and idempotency correct. The eliminated cost is an entire ingestion service and its operational tail — O(config) to publish a queryable table instead of O(engineers) to build and babysit one.

Design
Topic — design
Design problems on the Kafka-to-lakehouse boundary

Practice →

Streaming Topic — streaming Streaming problems on topics, logs, and materialization

Practice →

2. Confluent Tableflow — materialize a topic as an Iceberg table

Point Tableflow at a topic and it keeps an Iceberg table in your bucket and catalog in sync

The mental model in one line: Confluent Tableflow is a managed feature that represents a Kafka topic as an Apache Iceberg (or Delta) table — you enable it on a topic, it reads the registered schema from the schema registry to define the table, writes the topic's records as partitioned Parquet into object storage you designate, commits Iceberg snapshots, publishes the table to a catalog (Glue, a REST catalog, or Snowflake's Open Catalog), and keeps that table continuously in sync as new records arrive — so instead of building an ingestion service you declare a topic-to-table materialization and query the result. You describe which topic becomes a table and where it lives; Tableflow owns the reading, writing, committing, and maintenance.

What Tableflow produces from a topic.

A real Iceberg table. Not an export or a copy job — a live Iceberg table (Parquet data files + manifests + snapshots) that grows as the topic does, readable by any Iceberg-aware engine through the catalog.
Delta as an alternative format. The same materialization can emit Delta Lake instead of Iceberg where your query stack prefers it; the topic is the source, the table format is a config choice.
Continuous sync, not a batch dump. Records flow from the topic into the table on an ongoing basis, so the table trails the log by a small, bounded lag rather than being a nightly snapshot.
Managed maintenance underneath. Compaction of small files and other Iceberg housekeeping happen without you scheduling jobs, so the table stays query-fast as it grows.

Schema-registry-driven — the table schema is the topic's contract.

The subject defines the columns. Tableflow reads the topic's value schema (typically <topic>-value under TopicNameStrategy) from the schema registry and uses it as the Iceberg table schema — the registered Avro/Protobuf/JSON schema is the DDL.
No hand-written DDL. You never CREATE TABLE by hand; adding the topic to Tableflow with a registered schema produces the table definition automatically, eliminating the stream-vs-table drift a hand-mapped schema invites.
Evolution follows compatibility. When the subject evolves under the registry's compatibility rules (e.g. BACKWARD), Tableflow propagates the change into the Iceberg schema by field ID — covered in depth in section 4.
A schema is a prerequisite. A topic with no registered schema (raw bytes) has nothing to materialize into columns; registering a schema is step zero.

The config surface — a handful of decisions.

Enable on a topic. Materialization is opt-in per topic; you turn it on for the topics that should become tables, not the whole cluster.
Table format. Iceberg or Delta — pick the one your query engines and catalog support best.
Storage. Which object-storage bucket holds the data files (your own bucket for bring-your-own-storage, or a managed store), plus the credentials/role to write it.
Catalog integration. Which catalog the table is registered in — AWS Glue, a REST catalog, or Snowflake's Open Catalog — because the catalog is what makes the table discoverable and queryable by external engines.

The failure modes senior engineers pre-empt.

No registered schema. Enabling Tableflow on a raw-bytes topic with no subject gives it nothing to build columns from. Mitigation: register an Avro/Protobuf/JSON schema for the topic first; treat the schema as the ingestion contract.
Wrong subject strategy. If the registry uses a non-default subject-name strategy, Tableflow may not find the schema it expects. Mitigation: confirm the subject name strategy and that <topic>-value (or the configured subject) resolves.
Catalog not wired. A materialized table nobody registered in a catalog is data files no engine can find. Mitigation: configure catalog integration so Spark/Trino/Athena can discover the table.
Expecting transforms. Tableflow mirrors a topic; it is not a stream processor. Mitigation: if you need joins, filters, or enrichment before landing, do them upstream (e.g. in Flink/ksqlDB) into a derived topic, then materialize that.

Common interview probes on Tableflow.

"Where does the Iceberg schema come from?" — the topic's registered schema-registry subject, not hand-written DDL.
"Is it a copy job or a live table?" — a continuously synced Iceberg table, not a periodic export.
"What do you configure?" — the topic, table format, storage, and catalog; maintenance is managed.
"Can Tableflow transform the data?" — no; it mirrors the topic. Transform upstream into a derived topic, then materialize.

Worked example — enable Tableflow on a topic and read the Iceberg table

Detailed explanation. The canonical Tableflow setup: a topic with a registered schema, materialization enabled with a storage and catalog target, and a query against the resulting Iceberg table. Turn the orders topic into a queryable table end to end.

Precondition. orders has a registered orders-value Avro schema.
Enable. Materialize to Iceberg, into s3://lake/orders, registered in a Glue database.
Read. Query analytics.orders from any Iceberg engine.

Question. Enable Tableflow on orders and show the SQL a downstream engine runs against the produced Iceberg table.

Input.

Piece	Value
Topic	`orders`
Schema subject	`orders-value` (Avro, registered)
Table format	Iceberg
Storage	`s3://lake/orders`
Catalog	Glue database `analytics`

Code.

# Tableflow: enable materialization on the topic (config, not a pipeline you run).
tableflow:
  topic: orders
  enabled: true
  table_format: ICEBERG
  storage:
    bucket: s3://lake/orders
    provider: AWS_S3
    # a role/credential Tableflow uses to write data files
    credential: arn:aws:iam::123456789012:role/tableflow-writer
  catalog:
    type: GLUE
    database: analytics
    table: orders          # -> analytics.orders becomes queryable
  partitioning:
    - "day(event_time)"    # hidden partitioning so engines prune by date

-- Downstream: the materialized Iceberg table is just a table now.
SELECT
    order_date,
    region,
    count(*)          AS orders,
    sum(amount)       AS revenue
FROM analytics.orders                         -- the Tableflow-materialized table
WHERE event_time >= TIMESTAMP '2026-08-01 00:00:00'   -- prunes day() partitions
GROUP BY order_date, region
ORDER BY order_date;

Step-by-step explanation.

enabled: true on the orders topic is the entire "build the pipeline" step: Tableflow starts consuming the topic and writing Iceberg data files, using the registered orders-value schema to shape the columns — there is no connector to deploy and no DDL to write.
storage names your bucket and the credential Tableflow uses to write it, so the data files live in object storage you own and govern; the table format (ICEBERG) decides the on-disk metadata layout.
catalog registers the table as analytics.orders in Glue, which is what makes it discoverable: an engine asks the catalog "where is analytics.orders?" and gets the current Iceberg metadata pointer, then reads the data files.
partitioning: day(event_time) tells the table to lay data out by day so the downstream WHERE event_time >= ... prunes to only the relevant date partitions — the query never scans August if it asks for September.
The downstream SQL treats analytics.orders as an ordinary table; nothing in the query knows or cares that it is a continuously materialized Kafka topic — which is exactly the point: the stream is now a table.

Output.

Step	Result
Enable Tableflow	topic starts materializing to Iceberg
Schema	`orders-value` subject → table columns
Storage	Parquet + metadata under `s3://lake/orders`
Catalog	`analytics.orders` discoverable
Query	ordinary partition-pruned SQL scan

Rule of thumb. Materialization is opt-in per topic: register a schema, enable Tableflow with a storage bucket and a catalog target, choose a partitioning transform, and the topic becomes an ordinary Iceberg table your engines query. There is no connector to run and no DDL to write — the config is the pipeline.

Worked example — the schema registry subject becomes the table schema

Detailed explanation. The feature that removes the most toil is that the registered schema defines the table — no hand-mapped DDL. Trace how an Avro subject turns into an Iceberg schema so nothing drifts between the stream contract and the table.

The subject. orders-value (Avro), the topic's value schema in the registry.
The mapping. Each Avro field becomes an Iceberg column with a stable field ID.
The guarantee. The table cannot disagree with the stream, because they are the same schema.

Question. Show the Avro subject and the Iceberg table schema Tableflow derives from it, and explain why this eliminates schema drift.

Input.

Avro field	Avro type	Iceberg column
`id`	`long`	`id bigint`
`region`	`string`	`region string`
`amount`	`bytes` + decimal logical	`amount decimal(12,2)`
`event_time`	`long` + timestamp-millis	`event_time timestamptz`

Code.

// Registered subject `orders-value` (Avro) — the SINGLE source of the table schema.
{
  "type": "record",
  "name": "Order",
  "namespace": "com.shop.events",
  "fields": [
    {"name": "id",         "type": "long"},
    {"name": "region",     "type": "string"},
    {"name": "amount",     "type": {"type": "bytes", "logicalType": "decimal",
                                     "precision": 12, "scale": 2}},
    {"name": "event_time", "type": {"type": "long",  "logicalType": "timestamp-millis"}}
  ]
}

-- The Iceberg table Tableflow derives — you never wrote this DDL.
CREATE TABLE analytics.orders (
    id          bigint          COMMENT 'field-id 1',
    region      string          COMMENT 'field-id 2',
    amount      decimal(12,2)   COMMENT 'field-id 3',
    event_time  timestamptz     COMMENT 'field-id 4'
)
USING iceberg
PARTITIONED BY (day(event_time));
-- ^ generated FROM the registered subject; the registry is the source of truth.

Step-by-step explanation.

The registered orders-value subject is the only place the schema is authored: producers serialize records against it, and Tableflow reads it to define the table — so there is exactly one schema, not a stream schema plus a hand-written table DDL that can fall out of step.
Each Avro field maps to an Iceberg column by a deterministic type mapping: long → bigint, string → string, the decimal logical type → decimal(12,2), and the timestamp-millis logical type → timestamptz (covered in detail in section 3).
Every Iceberg column gets a stable field ID, which is what makes later evolution safe: renames and additions are tracked by ID, not by position or name, so old Parquet files remain readable after the schema changes.
The PARTITIONED BY (day(event_time)) clause is the one thing you do choose (in the Tableflow config), because partitioning is a physical-layout decision the logical schema does not imply — everything else is derived.
The drift elimination is structural: because the table schema is generated from the registry subject rather than mirrored by hand, there is no manual step to forget when the schema evolves — the contract the producers already honour is the contract the table exposes.

Output.

Concern	Hand-mapped DDL	Registry-driven (Tableflow)
Source of schema	two (stream + table)	one (the subject)
Drift on evolution	manual re-map needed	propagated by field ID
Field identity	by name/position	by stable field ID
DDL you write	the whole table	none

Rule of thumb. Let the registered schema be the table schema: authoring the subject once and deriving the Iceberg table from it means the stream contract and the table contract are the same object, so there is no hand-mapped DDL to drift. The only physical decision left to you is partitioning.

Worked example — choosing storage and catalog so every engine can see the table

Detailed explanation. A materialized table nobody can find is just files in a bucket. The storage target decides where the bytes live and who governs them, and the catalog decides how engines discover the table. Wire both so Spark, Trino, and Athena can all read analytics.orders.

Storage. Your S3 bucket (bring-your-own-storage) so you own governance and cost.
Catalog. A shared catalog (Glue or REST) that every engine already trusts.
The result. One table, discovered through the catalog, read by all engines.

Question. Configure storage and catalog so the same Tableflow Iceberg table is queryable from Spark, Trino, and Athena without copying it.

Input.

Decision	Choice	Why
Storage	your S3 bucket	you own governance + cost
Catalog	Glue (or REST)	engines already integrate
Table identity	`analytics.orders`	one name everywhere
Copies	none	catalog is the single pointer

Code.

# Tableflow storage + catalog: the two settings that decide who can read the table.
tableflow:
  topic: orders
  table_format: ICEBERG
  storage:
    bucket: s3://lake/orders        # YOUR bucket — you own retention, encryption, cost
    provider: AWS_S3
    credential: arn:aws:iam::123456789012:role/tableflow-writer
  catalog:
    type: GLUE                      # a catalog every AWS engine already reads
    database: analytics
    table: orders                   # discoverable as analytics.orders

# One table, three engines — all point at the SAME catalog entry + storage.

Glue catalog: analytics.orders  ->  s3://lake/orders/{metadata,data}/...

  Spark   : spark.table("glue_catalog.analytics.orders")
  Trino   : SELECT * FROM iceberg.analytics.orders
  Athena  : SELECT * FROM analytics.orders        (Athena reads Glue-registered Iceberg)

No per-engine copy. The catalog resolves the current Iceberg metadata; each engine
reads the same Parquet files. Add Snowflake by pointing its catalog integration
at the same table.

Step-by-step explanation.

The storage bucket is yours, so retention, encryption, and storage cost are governed by your account — bring-your-own-storage keeps the data under your control rather than locked inside a vendor store, and it is the same object storage your lakehouse engines already read.
The catalog entry is the discovery mechanism: registering the table as analytics.orders in Glue means any engine that integrates with Glue can resolve the table's current Iceberg metadata pointer and read it — no engine needs to know it came from Kafka.
Because all three engines resolve the same catalog entry pointing at the same files, there is exactly one physical copy of the data; Spark, Trino, and Athena differ only in how they name the catalog, not in what they read.
Athena reads Glue-registered Iceberg tables natively, Trino uses its Iceberg connector against the same catalog, and Spark uses the Iceberg Spark runtime — three engines, one table, zero copies, which is the whole promise of an open table format in a shared catalog.
Adding a new engine (Snowflake, DuckDB, Flink for reads) is just pointing its catalog integration at the same entry — the marginal cost of another consumer is a config line, not another ingestion pipeline or another copy of the data.

Output.

Engine	How it reads `analytics.orders`	Extra copy?
Spark	Iceberg Spark runtime via Glue	no
Trino	Iceberg connector via Glue	no
Athena	native Glue Iceberg	no
new engine	point its catalog at the same entry	no

Rule of thumb. Store the data files in a bucket you govern and register the table in a catalog every engine already trusts; the catalog is the single pointer that lets Spark, Trino, Athena, and Snowflake read one physical table without copies. Adding a consumer is a config line, not a pipeline.

Senior interview question on standing up Tableflow

A senior interviewer might ask: "Stand up an Iceberg table over a Kafka orders topic using Tableflow, with zero bespoke ingestion code. Cover where the table schema comes from, which storage and catalog you target and why, how downstream engines discover and query it, and what you would do if the product needed the data filtered or enriched before it lands — and be explicit about what Tableflow does and does not do."

Solution Using a registered schema, opt-in materialization, bring-your-own-storage, and catalog integration

// 1. Prerequisite: a registered value schema — the table schema comes from HERE.
{ "type": "record", "name": "Order", "fields": [
  {"name":"id","type":"long"},
  {"name":"region","type":"string"},
  {"name":"amount","type":{"type":"bytes","logicalType":"decimal","precision":12,"scale":2}},
  {"name":"event_time","type":{"type":"long","logicalType":"timestamp-millis"}}
]}

# 2. Enable Tableflow on the topic — Iceberg, your bucket, Glue catalog, partitioned.
tableflow:
  topic: orders
  enabled: true
  table_format: ICEBERG
  storage: { bucket: s3://lake/orders, provider: AWS_S3,
             credential: arn:aws:iam::123456789012:role/tableflow-writer }
  catalog: { type: GLUE, database: analytics, table: orders }
  partitioning: [ "day(event_time)" ]
  # maintenance (compaction, snapshot expiry) + exactly-once commits: managed

-- 3. Downstream discovery + query — ordinary Iceberg SQL, any engine, one copy.
SELECT region, sum(amount) AS revenue
FROM analytics.orders
WHERE event_time >= current_date - INTERVAL '30' DAY
GROUP BY region;

# 4. Need filtering/enrichment BEFORE landing? Tableflow mirrors a topic — so
#    transform UPSTREAM into a derived topic, then materialize THAT.
flink_sql: |
  CREATE TABLE orders_enriched AS
  SELECT o.*, c.tier AS customer_tier
  FROM orders o JOIN customers c ON o.customer_id = c.id
  WHERE o.amount > 0;          -- filter + enrich in flight
tableflow:
  topic: orders_enriched       # materialize the DERIVED topic
  enabled: true
  table_format: ICEBERG

Step-by-step trace.

Layer	Component	Purpose
Contract	registered `orders-value` subject	defines table columns (no DDL)
Materialize	Tableflow on the topic	topic → Iceberg, continuously
Storage	your S3 bucket	you govern retention/cost
Discovery	Glue catalog entry	engines find `analytics.orders`
Layout	`day(event_time)` partitioning	downstream pruning
Transforms	derived topic upstream	Tableflow mirrors, doesn't transform

After deployment, the registered orders-value subject defines the table; enabling Tableflow materializes orders into analytics.orders as partitioned Iceberg in your S3 bucket, registered in Glue so Spark, Trino, and Athena all query one copy; maintenance and exactly-once commits are managed. When the product needs filtered/enriched data, the transform happens upstream in Flink into orders_enriched, and Tableflow materializes that derived topic — because Tableflow mirrors a topic faithfully rather than transforming it.

Output:

Metric	Bespoke ingestion	Tableflow
Ingestion code to maintain	a sink service + jobs	none (config)
Schema source	hand-mapped DDL	registered subject
Copies of the data	often one per engine	one (shared catalog)
Storage governance	vendor-dependent	your bucket
Transform model	in the sink (coupled)	upstream, explicit topic

Why this works — concept by concept:

Registered-schema-as-DDL — deriving the Iceberg table from the topic's registry subject means the schema is authored once and the table can never disagree with the stream, removing the hand-mapped DDL that is the usual source of drift.
Opt-in materialization — enabling Tableflow per topic replaces an entire ingestion service with a configuration, so the topics that should be tables become tables without a connector to deploy or operate.
Bring-your-own-storage — writing data files into a bucket you own keeps retention, encryption, and cost under your governance while still producing a standard Iceberg table.
Catalog integration — registering the table in a shared catalog is what turns files-in-a-bucket into a discoverable table that Spark, Trino, Athena, and Snowflake read as one copy, and it makes adding a consumer a config line.
Cost — a managed materialization plus object storage and one catalog entry, versus a Connect cluster, maintenance jobs, and a copy per engine. The eliminated cost is the ingestion service and the data duplication — O(config) per topic instead of O(pipeline) per topic and O(copies) per engine.

Streaming
Topic — streaming
Streaming problems on topic-to-table materialization

Practice →

Event processing Topic — event-processing Event-processing problems on schemas and derived topics

Practice →

3. Kafka-to-Iceberg mechanics — serialization, partitioning, compaction, exactly-once

Records land as partitioned Parquet; compaction keeps files healthy; offset-tracked snapshots commit each record once

The mental model in one line: turning a Kafka stream into a healthy Iceberg table is four mechanics working together — serialization (schema-registry Avro/Protobuf records are deserialized and written as columnar Parquet under a deterministic Iceberg type mapping), partitioning (Iceberg's hidden partitioning lays data out by a transform like day(event_time) so engines prune), compaction (streaming writes make many small files, so the many-small-files problem is fixed by rewriting them into fewer large ones), and exactly-once (Kafka offsets are tracked and Iceberg snapshots committed idempotently so a replay never duplicates rows) — and understanding these four is what lets you reason about correctness and performance instead of treating Tableflow as a black box. Get the type mapping and partitioning right and queries fly; ignore compaction and the table rots; skip idempotent commits and a restart doubles your data.

Serialization and type mapping.

Deserialize against the schema. A record on the topic is Avro/Protobuf/JSON bytes plus a schema ID; the materializer deserializes it using the registry schema, then writes the fields as columnar Parquet — row-oriented log to column-oriented file.
A deterministic type map. Avro/Protobuf primitives map to Iceberg types by fixed rules: integer widths, decimals (via the decimal logical type/precision-scale), timestamps (millis/micros → timestamptz), and bytes/strings — so the table's column types are predictable.
Nested and repeated types. Avro records → Iceberg structs, arrays → lists, maps → maps; Protobuf messages and repeated fields map the same way, so nested event shapes survive into the table.
Optional via unions. An Avro ["null", T] union (or a Protobuf optional) becomes a nullable Iceberg column — the rule that makes additive schema evolution safe.

Partitioning for pruning.

Hidden partitioning. Iceberg partitions by a transform of a column (day(event_time), bucket(16, id)), and stores the partition values in metadata — so queries filtering on the column prune partitions without the writer or reader spelling out partition paths.
Choose the transform by the query. Time-series analytics almost always partition by day or hour of the event time; high-cardinality keys use bucket(N, key) to spread data evenly.
Avoid over-partitioning. Partitioning by a very high-cardinality raw column (e.g. per-user) creates a partition per value and a small-file storm; a bucket transform bounds the count.
Partition evolution. Iceberg can change partitioning without rewriting old data, so a table can start day and add hour later — old files keep their scheme, new files use the new one.

Compaction — the small-file problem.

Why it happens. Streaming ingestion commits frequently to keep the table fresh, and frequent commits mean many small Parquet files — each a separate open/seek at query time.
The cost of ignoring it. Thousands of tiny files inflate metadata, defeat vectorised reads, and make even a pruned scan slow — the classic "why is my Iceberg table slow" answer.
What compaction does. rewrite_data_files merges small files into larger, well-sized ones (and can re-sort/cluster them), so a scan opens a handful of big files instead of thousands of small ones.
Managed vs self-run. Tableflow runs compaction (and snapshot expiry) for you; a self-managed sink means scheduling these Iceberg maintenance procedures yourself on their own compute.

Exactly-once — no duplicate rows.

Upstream: Kafka EOS. An idempotent/transactional producer plus read_committed consumers give exactly-once semantics on the topic, so the source stream itself has no duplicates in a committed transaction.
Downstream: idempotent commits. The materializer tracks which Kafka offsets have been committed to which Iceberg snapshot, so re-processing a range (after a restart or rebalance) is a no-op rather than a second insert.
Snapshots are atomic. An Iceberg commit is atomic — a batch of records either becomes a snapshot or it doesn't — so a reader never sees a half-written commit and a failed write leaves no partial rows.
The failure it prevents. Without offset-tracked idempotency, a connector that restarts mid-batch re-reads and re-writes the same offsets, doubling rows — the single most common Kafka-to-Iceberg correctness bug.

Common interview probes on the mechanics.

"How does an Avro record become table columns?" — deserialize against the registry schema, write columnar Parquet under a fixed Iceberg type mapping.
"Why is my streamed Iceberg table slow?" — small-file explosion; fix with compaction (rewrite_data_files).
"How do you partition a streamed table?" — hidden partitioning on a transform (day(event_time)), bucket for high-cardinality keys.
"How do you guarantee no duplicate rows?" — Kafka EOS upstream + offset-tracked idempotent Iceberg snapshot commits.

Worked example — Avro/Protobuf to Iceberg type mapping and the optional-field rule

Detailed explanation. The correctness of every downstream query depends on the type mapping being right, especially decimals, timestamps, and nullability. Walk the mapping for a realistic order event and show why the union-to-nullable rule is what makes evolution safe.

The tricky types. Decimal (precision/scale), timestamp (millis vs micros), and optional (["null", T]).
The safe default. Model additive fields as unions so they land as nullable columns.
The payoff. Old rows (without the field) and new rows (with it) coexist in one table.

Question. Map an Avro order schema to an Iceberg table schema, getting decimal, timestamp, and optional right, and explain the union-to-nullable rule.

Input.

Avro	Iceberg	Note
`long` / `int`	`bigint` / `int`	integer width preserved
`bytes` + decimal(12,2)	`decimal(12,2)`	precision/scale from logical type
`long` + timestamp-millis	`timestamptz`	epoch millis → timestamp
`["null","string"]`	nullable `string`	union → optional column

Code.

// Avro subject with the three tricky cases: decimal, timestamp, and an optional field.
{
  "type": "record", "name": "Order", "namespace": "com.shop.events",
  "fields": [
    {"name": "id",         "type": "long"},
    {"name": "amount",     "type": {"type": "bytes", "logicalType": "decimal",
                                     "precision": 12, "scale": 2}},
    {"name": "event_time", "type": {"type": "long",  "logicalType": "timestamp-millis"}},
    {"name": "coupon",     "type": ["null", "string"], "default": null},   // OPTIONAL
    {"name": "items",      "type": {"type": "array", "items": "string"}}   // repeated
  ]
}

-- The Iceberg schema derived from it — types chosen so queries are correct.
CREATE TABLE analytics.orders (
    id          bigint         NOT NULL,   -- required
    amount      decimal(12,2)  NOT NULL,   -- exact money, never a float
    event_time  timestamptz    NOT NULL,   -- epoch millis interpreted as a timestamp
    coupon      string,                    -- NULLABLE  <- from ["null","string"]
    items       array<string>  NOT NULL    -- repeated -> list
)
USING iceberg
PARTITIONED BY (day(event_time));

Step-by-step explanation.

amount uses Avro's decimal logical type with precision 12 and scale 2, which maps to Iceberg decimal(12,2) — money must be a decimal, never a float, or aggregates accumulate rounding error; the precision/scale carry over exactly so sum(amount) is correct to the cent.
event_time is an Avro long tagged timestamp-millis, so the epoch-millis integer is interpreted as a timestamptz — without the logical type it would land as a meaningless bigint, and time-based partition pruning (day(event_time)) would be impossible.
coupon is a ["null", "string"] union with default: null, which maps to a nullable Iceberg column. This is the crucial rule: an optional field is a union, and a union becomes a column that can be null.
The union-to-nullable rule is what makes additive evolution safe — when you later add an optional field, old rows (serialized before it existed) simply have null there, and both old and new rows live in one table without a rewrite.
items is an Avro array, which maps to an Iceberg list; nested records would map to structs and maps to maps — so the full nested shape of an event survives into the table rather than being flattened or stringified.

Output.

Field	Iceberg type	Correctness point
`amount`	`decimal(12,2)`	exact money, no float drift
`event_time`	`timestamptz`	enables time partition pruning
`coupon`	nullable `string`	old rows null, new rows valued
`items`	`array<string>`	nested shape preserved

Rule of thumb. Get decimals (precision/scale), timestamps (logical type → timestamptz), and nullability (union ["null", T] → nullable column) right, because those three are where a sloppy mapping corrupts money, breaks partition pruning, or blocks safe evolution. Model every additive field as an optional union so old and new rows coexist.

Worked example — partition the Iceberg output and prove pruning

Detailed explanation. Partitioning is the single biggest lever on scan cost for a streamed table. Iceberg's hidden partitioning lets a day(event_time) transform prune whole days of data without the query mentioning partitions. Show the layout and prove the pruning.

The transform. day(event_time) — one partition per calendar day.
The query. A 7-day filter that should touch only 7 partitions.
The proof. The scan reads 7 days of files, not the whole table.

Question. Partition a streamed orders table by day(event_time) and show that a date-bounded query prunes to only the matching partitions.

Input.

Aspect	Unpartitioned	`day(event_time)`
Layout	one flat set of files	files grouped per day
7-day query scans	the whole table	7 day-partitions
Cost at 2 years	O(all data)	O(7 days)
Pruning	none	metadata prunes days

Code.

-- Hidden partitioning: the writer partitions by a TRANSFORM of event_time.
CREATE TABLE analytics.orders (
    id bigint, region string, amount decimal(12,2), event_time timestamptz
)
USING iceberg
PARTITIONED BY (day(event_time));      -- one partition per day, stored in metadata

-- A 7-day query. Note: it filters on event_time, NOT on a partition column —
-- Iceberg derives the day() partitions to prune from the metadata.
EXPLAIN
SELECT region, sum(amount)
FROM analytics.orders
WHERE event_time >= TIMESTAMP '2026-08-19 00:00:00'
  AND event_time <  TIMESTAMP '2026-08-26 00:00:00'
GROUP BY region;

# What the planner does (conceptually):
#   event_time in [2026-08-19, 2026-08-26)
#     -> day(event_time) in { 2026-08-19 ... 2026-08-25 }   (7 partitions)
#     -> read ONLY those 7 days' data files; skip the other ~720 days
#
# Physical layout Tableflow writes:
#   data/ day=2026-08-19/ *.parquet
#   data/ day=2026-08-20/ *.parquet
#   ...                                  <- the query opens 7 of these folders

Step-by-step explanation.

PARTITIONED BY (day(event_time)) tells Iceberg to store, per data file, which day of event_time it holds — that partition value lives in the table metadata, so the writer never has to build partition paths by hand and the reader never has to know them.
The query filters on event_time directly, not on a partition column — this is the "hidden" in hidden partitioning: Iceberg translates the event_time range predicate into the set of day() partitions it overlaps and prunes the rest.
For a 7-day window, the planner computes 7 matching day-partitions and reads only their data files, skipping the ~720 days of a two-year table — the scan cost is proportional to the window, not the table size.
This is why the transform choice matters: day suits day-grained analytics; a fraud query scanning the last hour would benefit from hour(event_time); a lookup by a high-cardinality id would use bucket(N, id) so the partitions are balanced instead of one-per-value.
The failure mode is an unpartitioned streamed table: every query scans all files regardless of its filter, so the table gets slower every day it ingests — partitioning is what keeps a growing streamed table's scan cost flat per query.

Output.

Query window	Partitions read (2-year table)	Data scanned
1 day	1	~1/730
7 days	7	~7/730
90 days	90	~90/730
unpartitioned equivalent	all	100%

Rule of thumb. Partition a streamed table by a transform that matches how it is queried — day/hour of the event time for time-series analytics, bucket(N, key) for high-cardinality lookups — so filters prune whole partitions from metadata. An unpartitioned streamed table scans everything on every query and only gets slower as it grows.

Worked example — exactly-once via offset-tracked idempotent commits

Detailed explanation. The scariest Kafka-to-Iceberg bug is silent row duplication after a restart. Exactly-once materialization prevents it by tying each Iceberg snapshot to a range of Kafka offsets and committing idempotently. Trace a restart that would double rows in a naive sink but does not here.

The naive bug. A sink re-reads uncommitted offsets after a crash and re-inserts them.
The fix. Record "offsets 100–149 → snapshot s7"; on replay, skip already-committed offsets.
The guarantee. Each record becomes exactly one row, even across restarts.

Question. Show how offset-tracked idempotent commits make a mid-batch restart produce no duplicate rows, and contrast with a naive sink.

Input.

Event	Naive sink	Exactly-once (offset-tracked)
Write offsets 100–149	insert rows	commit snapshot s7, record 100–149
Crash before ack	offsets look unprocessed	commit already durable
Restart, re-read 100–149	insert AGAIN (duplicates)	offsets ≤ 149 already committed → skip
Result	rows doubled	each record once

Code.

# Exactly-once materialization: bind Iceberg snapshots to Kafka offset ranges.

commit snapshot s7  <=>  { topic: orders, partition: 0, offsets: [100, 150) }
   (Iceberg commit is ATOMIC: the whole batch becomes snapshot s7, or nothing does)

  ---- crash right here, before the consumer offset is externally acked ----

restart:
  read committed watermark -> "partition 0 committed through offset 149"
  poll returns offsets starting at 100 (consumer thinks it hasn't committed)
     -> materializer sees 100..149 are ALREADY in snapshot s7
     -> SKIPS them (idempotent) and resumes at 150
  => zero duplicate rows

# Upstream, Kafka's own EOS keeps the SOURCE clean:
#   producer: enable.idempotence=true, transactional.id set
#   consumer: isolation.level=read_committed   (never see aborted txns)

-- Reader's view: an atomic snapshot means you never see a half-written batch.
SELECT count(*) FROM analytics.orders;         -- before s7 commits: excludes 100..149
-- ... s7 commits atomically ...
SELECT count(*) FROM analytics.orders;         -- after: includes 100..149 EXACTLY once

Step-by-step explanation.

Each Iceberg commit is bound to a Kafka offset range: snapshot s7 is "offsets 100–149 of partition 0." Because an Iceberg commit is atomic, that batch either becomes snapshot s7 in full or not at all — a reader never sees a partial batch.
When the materializer crashes after committing s7 but before the consumer offset is externally acknowledged, the durable truth is the Iceberg commit: offsets 100–149 are already in the table, regardless of what the consumer's offset bookkeeping thinks.
On restart, the materializer reads the committed watermark from the table metadata ("committed through 149") and, even though the poll re-delivers offsets from 100, recognises that 100–149 are already materialized and skips them — the idempotency that turns a re-read into a no-op.
This is exactly the point where a naive sink fails: it re-reads 100–149 and inserts them again, doubling those rows. The difference is not luck; it is the explicit offset-to-snapshot binding plus the skip-already-committed check.
Upstream, Kafka's own exactly-once semantics keep the source clean — an idempotent/transactional producer plus read_committed consumers mean aborted transactions never appear — so end to end, each business event maps to exactly one row in the Iceberg table across producer retries, consumer rebalances, and materializer restarts.

Output.

Scenario	Naive sink	Exactly-once
Happy path	each record once	each record once
Restart mid-batch	rows duplicated	each record once
Producer retry	possible dup upstream	deduped by EOS
Reader sees partial batch	possible	never (atomic snapshot)

Rule of thumb. Exactly-once is not a hope — it is offset ranges bound to atomic Iceberg snapshots plus a skip-already-committed check on restart, backed by Kafka EOS upstream. If a design cannot explain what happens when the writer restarts mid-batch, assume it duplicates rows.

Senior interview question on Kafka-to-Iceberg correctness and performance

A senior interviewer might ask: "Walk me through what actually happens when a Kafka orders topic is materialized into an Iceberg table. Cover how an Avro record becomes columns and where the tricky types (decimal, timestamp, optional) go, how you partition so 90-day scans stay cheap, why the table would get slow without maintenance and what fixes it, and how you guarantee a writer restart never duplicates rows."

Solution Using a deterministic type map, hidden partitioning, managed compaction, and offset-tracked commits

-- 1. Serialization + type map: the registered schema becomes correct column types.
CREATE TABLE analytics.orders (
    id bigint, region string,
    amount decimal(12,2),          -- decimal logical type -> exact money
    event_time timestamptz,        -- timestamp-millis -> timestamptz (enables pruning)
    coupon string,                 -- ["null","string"] -> nullable (safe evolution)
    items array<string>            -- array -> list (nested shape preserved)
) USING iceberg
PARTITIONED BY (day(event_time));  -- 2. hidden partitioning for pruning

# 3. Compaction: streaming commits make MANY small files -> reads slow.
#    Managed (Tableflow) OR self-run rewrite:
CALL catalog.system.rewrite_data_files(
  table => 'analytics.orders',
  options => map('target-file-size-bytes','536870912')   -- ~512 MB targets
);
CALL catalog.system.expire_snapshots('analytics.orders', now() - interval '7' day);

# 4. Exactly-once: snapshots bound to offset ranges + idempotent skip on restart.
snapshot s7 <=> offsets [100,150)   (atomic commit)
restart: committed-through=149 -> re-read of 100..149 is SKIPPED -> no duplicates
upstream Kafka EOS: producer idempotence + transactions, consumer read_committed

-- 5. The payoff: a 90-day scan prunes to 90 day-partitions of well-sized files.
SELECT region, sum(amount) AS revenue
FROM analytics.orders
WHERE event_time >= current_date - INTERVAL '90' DAY
GROUP BY region;

Step-by-step trace.

Mechanic	What it does	Failure it prevents
Type mapping	Avro → correct Iceberg types	corrupt money, broken timestamps
Hidden partitioning	`day(event_time)` prunes	full-table scans
Compaction	small files → large	slow reads / metadata bloat
Snapshot expiry	drop old snapshots	unbounded metadata/storage
Exactly-once	offset↔snapshot + skip	duplicate rows on restart
Upstream EOS	idempotent txn producer	duplicates at the source

After deployment, each orders record is deserialized against the registered schema into correctly typed columns (exact-decimal money, timestamp for pruning, nullable optionals); the table is partitioned by day(event_time) so a 90-day scan reads 90 partitions, not the whole table; managed compaction rewrites the streaming small files into ~512 MB files and snapshot expiry bounds metadata; and offset-bound atomic snapshots plus a skip-already-committed check make a mid-batch restart a no-op — no duplicate rows — with Kafka EOS keeping the source clean.

Output:

Metric	Naive sink (no maintenance)	Tableflow mechanics
Money accuracy	float drift risk	exact decimal
90-day scan cost	whole table	90 partitions
Files per scan	thousands (small)	dozens (compacted)
Metadata growth	unbounded	bounded (expiry)
Rows on restart	duplicated	each record once

Why this works — concept by concept:

Deterministic type mapping — deserializing against the registry schema and applying fixed Avro/Protobuf→Iceberg rules gives correct column types, so money stays exact decimal, timestamps enable pruning, and optional unions become nullable columns that make evolution safe.
Hidden partitioning — partitioning by a transform of the event time lets Iceberg prune whole days from metadata when a query filters on the raw column, so scan cost tracks the query window rather than the ever-growing table.
Managed compaction and expiry — rewriting the many small files streaming ingestion produces into well-sized files, and expiring old snapshots, keeps reads fast and metadata bounded — the maintenance a self-built sink must schedule itself.
Offset-tracked idempotent commits — binding atomic Iceberg snapshots to Kafka offset ranges and skipping already-committed offsets on restart makes materialization exactly-once, so no restart, rebalance, or replay can double a row, with Kafka EOS guaranteeing a clean source.
Cost — correctly typed, partitioned, compacted files scanned by window, versus a small-file, unpartitioned, possibly-duplicated table scanned in full. The eliminated cost is both the query bill (partition pruning + compaction) and the correctness incidents (exactly-once) — O(window) pruned reads versus O(all-data) scans, and zero duplicate-row cleanups.

ETL
Topic — etl
ETL problems on serialization, partitioning, and compaction

Practice →

Event processing Topic — event-processing Event-processing problems on exactly-once and idempotency

Practice →

4. Schema Registry to Iceberg — query from Spark, Trino, and Athena

The registered schema drives the Iceberg schema; one catalog table is queried unchanged by every engine

The mental model in one line: the schema registry is the contract that makes materialization safe and multi-engine — a subject's compatibility mode (BACKWARD, FORWARD, FULL) governs which schema evolutions are legal, Tableflow propagates each legal change into the Iceberg table schema by field ID so old Parquet files stay readable, and because the result is one table in a shared catalog, Spark, Trino, Athena, and Snowflake all read it unchanged — the same partitioned files, the same evolving schema, the same time-travel snapshots, with no per-engine copy. The registry keeps the stream and the table honest; the catalog keeps every query engine reading one source of truth.

The registry as the evolution contract.

Compatibility modes. BACKWARD (new schema reads old data — the common default), FORWARD (old schema reads new data), and FULL (both) decide which changes the registry will accept, so an incompatible change is rejected before it can break the table.
What is safe additively. Adding an optional field (with a default) is backward-compatible: old data lacks it and reads as null, new data carries it — the everyday evolution.
What is not. Removing a required field, changing a type incompatibly, or renaming without an alias breaks compatibility and is refused — the registry is the guardrail.
The contract holds both sides. Because producers serialize against the subject and Tableflow builds the table from it, the registry enforcing compatibility means the table's evolution is automatically safe too.

Field-ID resolution — why old files still read.

Columns are identified by ID, not name. Iceberg tracks each column by a stable field ID; a rename changes the name but not the ID, and an addition allocates a new ID — so reads resolve columns by ID, not by position in the file.
Old files are not rewritten. When the schema gains a column, existing Parquet files are untouched; a read of an old file simply returns null for the new column because that field ID is absent.
Type promotion. Iceberg permits safe widening (e.g. int → long, float → double, decimal precision increase) without rewriting data — the read path promotes on the fly.
The result. A table can evolve for years while every historical file stays readable, so materialization never forces a full rewrite on a schema change.

One table, many engines — the catalog is the interop point.

A shared catalog. Glue, a REST catalog, or Snowflake's Open Catalog holds the table's current metadata pointer, so any engine that speaks the catalog resolves the same table.
Identical SQL surface. SELECT ... FROM analytics.orders is the same query in Spark, Trino, and Athena — the engines differ only in catalog naming, not in the data or schema they see.
Snapshot isolation. Every engine reading through the catalog sees a consistent snapshot, so a query does not observe a half-committed batch even as materialization writes continuously.
Time travel. Because Iceberg keeps snapshots, any engine can query the table as of a snapshot or timestamp — for backfills, audits, and reconciling late corrections.

The failure modes senior engineers pre-empt.

Incompatible change rejected. A producer tries a breaking schema change and the registry refuses it. Mitigation: this is working as intended — evolve additively (optional fields), and version the subject deliberately for genuine breaks.
Engine without the Iceberg runtime. An engine configured without the Iceberg connector/runtime cannot read the table. Mitigation: use the Iceberg-capable engine version and point it at the shared catalog.
Querying files, not the catalog. Reading the raw Parquet paths bypasses snapshot isolation and schema resolution. Mitigation: always query through the catalog so you get the current schema and a consistent snapshot.

Common interview probes on schema and query.

"What makes a schema change safe?" — registry compatibility (additive/optional) + Iceberg field-ID resolution so old files still read.
"How do old rows read after you add a column?" — by field ID; the missing column reads as null, no rewrite.
"How can Athena and Spark read the same table?" — one Iceberg table in a shared catalog; no per-engine copy.
"How do you read the table as it was yesterday?" — Iceberg time travel to a snapshot/timestamp.

Worked example — evolve the Avro schema and query old and new rows together

Detailed explanation. The everyday evolution is adding a field. Do it the safe way — an optional field under BACKWARD compatibility — and show that a single query returns old rows (null for the new field) and new rows (valued) with no rewrite.

The change. Add optional channel to orders-value.
The compatibility. BACKWARD: old data (no channel) is still readable.
The read. One query spans pre- and post-change rows; channel is null for the old ones.

Question. Add an optional channel field to the order schema and write a query that returns both old and new rows correctly.

Input.

Step	Action	Effect
v1 schema	`id, amount, event_time`	table has 3 columns
Evolve	add `channel` optional (default null)	BACKWARD-compatible
Iceberg	new field ID for `channel`	old files untouched
Query	`SELECT ... channel ...`	old rows null, new valued

Code.

// v2 subject: add an OPTIONAL field. BACKWARD-compatible -> the registry accepts it.
{
  "type": "record", "name": "Order", "namespace": "com.shop.events",
  "fields": [
    {"name": "id",         "type": "long"},
    {"name": "amount",     "type": {"type":"bytes","logicalType":"decimal","precision":12,"scale":2}},
    {"name": "event_time", "type": {"type":"long","logicalType":"timestamp-millis"}},
    {"name": "channel",    "type": ["null","string"], "default": null}   // NEW, optional
  ]
}

-- Iceberg gains a NEW field-id for `channel`; old data files are NOT rewritten.
-- One query spans rows written before AND after the change:
SELECT
    id,
    amount,
    channel,                                    -- NULL for pre-v2 rows, valued for v2+
    coalesce(channel, 'unknown') AS channel_norm
FROM analytics.orders
WHERE event_time >= current_date - INTERVAL '30' DAY
ORDER BY id;

# Why it just works:
#   old parquet file (v1)  -> has field-ids {1,2,3}; asked for field-id 4 (channel) -> NULL
#   new parquet file (v2)  -> has field-ids {1,2,3,4}; channel read directly
#   NO rewrite of old files; NO downtime; the registry blocked any UNSAFE change.

Step-by-step explanation.

The v2 subject adds channel as a ["null", "string"] union with a null default, which is a backward-compatible change: a reader using v2 can still read v1 data (the missing channel defaults to null), so the registry accepts it.
Tableflow propagates the change into the Iceberg schema by allocating a new field ID for channel; crucially, it does not touch the existing data files — evolution is a metadata operation, not a rewrite.
A read of an old (v1) Parquet file asks for field IDs {1,2,3,4}, finds {1,2,3} present and field ID 4 absent, and returns null for channel — the field-ID resolution that lets old and new files coexist in one logical table.
The single query therefore returns pre-change rows with channel = null and post-change rows with channel valued, and coalesce(channel, 'unknown') normalises them — no separate handling for "old" versus "new" data.
Had the producer attempted an unsafe change (dropping amount, or retyping it incompatibly), the registry would have rejected it under BACKWARD compatibility — so the table is protected from breaking changes at the source, before any file is written.

Output.

Row era	`channel` value	Rewrite needed
pre-v2 (old)	null (→ 'unknown')	no
v2+ (new)	actual channel	no
unsafe change	rejected by registry	n/a
downtime	none	—

Rule of thumb. Evolve by adding optional fields under a compatibility mode (BACKWARD is the common default); the registry rejects unsafe changes and Iceberg's field-ID resolution lets old files read as null for new columns with no rewrite. One query spans every schema era — you never fork "old table" and "new table."

Worked example — query the same table from Spark, Trino, and Athena

Detailed explanation. The payoff of an open table in a shared catalog is that the same table is read by every engine with essentially the same SQL. Show one Iceberg table read from Spark, Trino, and Athena — one copy, three engines.

The table. analytics.orders, materialized by Tableflow, in a Glue catalog.
The engines. Spark (batch/ML), Trino (interactive), Athena (serverless).
The point. No copies; the catalog resolves one set of files for all three.

Question. Show the query that returns revenue-by-region from the same Iceberg table in Spark, Trino, and Athena, and explain why there is only one physical copy.

Input.

Engine	Catalog binding	Query style
Spark	Iceberg Spark runtime + Glue	`spark.sql(...)`
Trino	Iceberg connector + Glue	ANSI SQL
Athena	native Glue Iceberg	ANSI SQL
copies	—	one (shared catalog)

Code.

# Spark (Iceberg runtime, Glue catalog) — batch / ML feature reads.
spark.sql("""
  SELECT region, sum(amount) AS revenue
  FROM glue_catalog.analytics.orders
  WHERE event_time >= current_date - INTERVAL 30 DAYS
  GROUP BY region
""").show()

-- Trino (iceberg connector over the SAME Glue catalog) — interactive analytics.
SELECT region, sum(amount) AS revenue
FROM iceberg.analytics.orders
WHERE event_time >= current_date - INTERVAL '30' DAY
GROUP BY region;

-- Athena (native Iceberg via Glue) — serverless, IDENTICAL SQL, same files.
SELECT region, sum(amount) AS revenue
FROM analytics.orders
WHERE event_time >= current_date - INTERVAL '30' DAY
GROUP BY region;

# One table, three readers — the catalog is the single pointer.
#
#   Glue: analytics.orders -> current metadata.json -> data/ *.parquet (ONE copy)
#            ^                    ^                        ^
#          Spark               Trino                    Athena
#          reads it            reads it                 reads it
#
# Differences are cosmetic: the catalog PREFIX (glue_catalog / iceberg / <none>).
# The schema, partitioning, snapshots, and files are identical for all three.

Step-by-step explanation.

All three engines are pointed at the same Glue catalog, so each resolves analytics.orders to the identical current metadata pointer and the identical set of Parquet data files — there is one physical copy of the data, not one per engine.
The SQL is essentially identical across Trino and Athena (both ANSI) and only trivially different in Spark; the sole real difference is the catalog prefix (glue_catalog. / iceberg. / none), which is naming, not data.
Each engine independently applies the same partition pruning (day(event_time)) and reads the same schema (including any evolved columns by field ID), because those live in the shared table metadata — the engine does not carry its own notion of the schema.
Snapshot isolation means each query reads a consistent snapshot even while Tableflow keeps writing, so an interactive Trino query and a nightly Spark job see coherent data rather than a half-written batch.
Adding a fourth engine (Snowflake via catalog integration, DuckDB, a BI tool's Iceberg reader) costs a config binding, not a new pipeline or a new copy — the open-table-in-a-catalog pattern makes consumers cheap and keeps them all consistent.

Output.

Engine	Rows read	Physical copy
Spark	revenue by region	shared
Trino	revenue by region	shared
Athena	revenue by region	shared
all	identical (same snapshot)	one

Rule of thumb. An Iceberg table in a shared catalog is read by Spark, Trino, Athena, and Snowflake with the same SQL surface and one physical copy — the catalog is the single pointer. Never copy the data per engine; add a consumer by binding its catalog, and every engine automatically shares the schema, partitioning, and snapshots.

Worked example — time travel to reconcile a late-arriving correction

Detailed explanation. Streaming data gets corrections — a late event, a fixed amount. Iceberg snapshots let you read the table as of a point in time to see exactly what a downstream report consumed, and to reconcile before and after a correction. Use time travel to audit a revenue change.

The situation. A report ran at snapshot s40; a correction landed as snapshot s41.
The tool. Query AS OF each snapshot to diff what changed.
The value. Reproduce yesterday's numbers and quantify the correction.

Question. Use Iceberg time travel to compare revenue as the report saw it (snapshot s40) with the corrected table (s41), and explain how this is possible.

Input.

Aspect	Value
Report snapshot	s40 (what the report consumed)
Corrected snapshot	s41 (after a late fix)
Tool	`FOR SYSTEM_VERSION AS OF` / timestamp
Output	before/after diff, reproducible

Code.

-- Revenue as the report SAW it (the snapshot it ran against).
SELECT region, sum(amount) AS revenue
FROM analytics.orders FOR SYSTEM_VERSION AS OF 40      -- snapshot s40
GROUP BY region;

-- Revenue AFTER the late correction committed (a later snapshot).
SELECT region, sum(amount) AS revenue
FROM analytics.orders FOR SYSTEM_VERSION AS OF 41      -- snapshot s41
GROUP BY region;

-- You can also travel by wall-clock time:
SELECT count(*) FROM analytics.orders
FOR TIMESTAMP AS OF TIMESTAMP '2026-08-25 23:59:59';

# Why time travel exists for free here:
#   Iceberg keeps a lineage of SNAPSHOTS; each commit (a Tableflow batch, or a
#   correction) creates a new snapshot pointing at the files valid at that moment.
#   Reading "AS OF s40" resolves the metadata as it was at s40 -> the exact files
#   the report consumed. Nothing was overwritten; the correction is a NEW snapshot.
#
#   (snapshot expiry eventually drops very old snapshots — keep a window that
#    covers your audit/repro needs.)

Step-by-step explanation.

Every commit — each Tableflow materialization batch and the late correction — creates a new Iceberg snapshot that references the exact set of files valid at that instant, so the table has a lineage rather than a single mutable state.
FOR SYSTEM_VERSION AS OF 40 resolves the table metadata as it was at snapshot s40, returning precisely the files the report consumed — which is how you reproduce a past report's numbers exactly, even after more data arrived.
Querying AS OF 41 shows the table after the correction, so diffing the two results quantifies exactly what the late fix changed, per region — an audit that would be impossible if the correction had overwritten data in place.
You can also travel by wall-clock time (FOR TIMESTAMP AS OF), which is convenient when you know when a report ran but not the snapshot ID — Iceberg maps the timestamp to the snapshot that was current then.
The one caveat is snapshot expiry: time travel works only for snapshots you have retained, so the expiry window (a maintenance setting) must cover your audit and reproducibility needs — keep enough history to reconcile the periods you care about.

Output.

Query	Sees	Use
`AS OF 40`	report-time data	reproduce the report
`AS OF 41`	corrected data	current truth
diff s40 vs s41	what the fix changed	audit the correction
`TIMESTAMP AS OF`	data at a wall-clock time	when you know the time, not the id

Rule of thumb. Use Iceberg time travel (AS OF a snapshot or timestamp) to reproduce exactly what a report consumed and to diff before/after a late correction — every commit is a new snapshot, so nothing is overwritten. Just keep your snapshot-expiry window wide enough to cover the periods you must audit.

Senior interview question on schema evolution and multi-engine querying

A senior interviewer might ask: "Your Tableflow Iceberg orders table must evolve as producers add fields, and it is read by Spark, Trino, and Athena plus an occasional audit. Explain how you evolve the schema without breaking old data or downstream engines, how one table serves all those engines without copies, and how you reproduce a report's numbers after a late correction — and be specific about what the registry and Iceberg each guarantee."

Solution Using registry compatibility, field-ID evolution, a shared catalog, and snapshot time travel

// 1. Evolve safely: add an OPTIONAL field. Registry compatibility (BACKWARD) gates it.
{ "type":"record","name":"Order","fields":[
  {"name":"id","type":"long"},
  {"name":"amount","type":{"type":"bytes","logicalType":"decimal","precision":12,"scale":2}},
  {"name":"event_time","type":{"type":"long","logicalType":"timestamp-millis"}},
  {"name":"channel","type":["null","string"],"default":null}   // NEW: old data reads null
]}

-- 2. Iceberg evolves by FIELD ID; old files are not rewritten. One query spans eras.
SELECT id, amount, coalesce(channel,'unknown') AS channel
FROM analytics.orders
WHERE event_time >= current_date - INTERVAL '30' DAY;

-- 3. One table, every engine (shared Glue catalog) — same SQL surface, one copy.
--    Spark : glue_catalog.analytics.orders
--    Trino : iceberg.analytics.orders
--    Athena: analytics.orders

-- 4. Reproduce a report after a late correction via time travel.
SELECT region, sum(amount) FROM analytics.orders FOR SYSTEM_VERSION AS OF 40 GROUP BY region; -- report-time
SELECT region, sum(amount) FROM analytics.orders FOR SYSTEM_VERSION AS OF 41 GROUP BY region; -- corrected

Step-by-step trace.

Concern	Mechanism	Guaranteed by
Safe evolution	optional field, compatibility check	Schema Registry
Old files still read	field-ID resolution, no rewrite	Iceberg
Type widening	int→long, precision bump on read	Iceberg
One table, many engines	shared catalog pointer	catalog (Glue/REST)
Consistent reads	snapshot isolation	Iceberg
Reproduce/audit	time travel `AS OF`	Iceberg snapshots

After deployment, producers evolve the subject additively and the registry rejects anything that would break BACKWARD compatibility; Tableflow applies each accepted change to the Iceberg schema by field ID, so old Parquet reads null for new columns without a rewrite; the one table lives in Glue so Spark, Trino, and Athena query it with the same SQL and one physical copy; and Iceberg snapshots let an auditor reproduce a report AS OF its snapshot and diff it against the corrected snapshot. The registry guarantees the evolution is safe; Iceberg guarantees the files and snapshots stay coherent.

Output:

Metric	Fragile pipeline	Registry + Iceberg + catalog
Breaking schema change	reaches the table	rejected at the registry
Old data after evolution	rewrite or break	reads null, no rewrite
Engines per copy	one copy each	one copy, all engines
Reproduce a past report	impossible (overwritten)	exact via time travel
Downstream breakage on change	common	avoided (compatibility + field ID)

Why this works — concept by concept:

Registry compatibility — the subject's compatibility mode rejects unsafe changes at the source, so only evolutions the table can absorb (additive/optional, safe widening) ever reach it, and the stream and table contracts stay aligned by construction.
Field-ID evolution — Iceberg identifies columns by stable IDs, so adding or renaming a column is a metadata change; old files read null for absent IDs and are never rewritten, making evolution instant and downtime-free.
Shared catalog — one catalog entry is the single pointer every engine resolves, so Spark, Trino, Athena, and Snowflake read the same files, schema, and partitioning with one physical copy instead of a fan-out of exports.
Snapshot time travel — because every commit is a new snapshot over an immutable file set, any engine can read the table as of a past snapshot or timestamp, making report reproduction and correction audits exact rather than best-effort.
Cost — additive metadata-only evolution, one shared copy, and free time travel, versus table rewrites, per-engine copies, and lost history. The eliminated cost is the rewrite-and-recopy churn of a schema change and the storage of N per-engine copies — O(metadata) evolution and O(1) copies instead of O(data) rewrites and O(engines) copies.

Real-time analytics
Topic — real-time-analytics
Real-time analytics problems on querying streamed tables

Practice →

ETL Topic — etl ETL problems on schema evolution and multi-engine reads

Practice →

5. Tableflow vs the Kafka Connect Iceberg sink and Flink — cost and ops

Tableflow mirrors a topic with no ops; Connect and Flink add control and transforms at the cost of running them

The mental model in one line: there are three ways to get a Kafka topic into an Iceberg table — Confluent Tableflow (a managed materialization that mirrors a topic with zero clusters and maintenance handled), the Kafka Connect Iceberg sink connector (a self-run connector plus the compaction, snapshot-expiry, and commit-coordination jobs you inherit), and Flink SQL (a self-run stream processor that can transform — join, filter, enrich — before landing) — and the senior choice is not "which tool is best" but "how much transform flexibility do I need, and how much operational surface am I willing to own to get it," because Tableflow trades in-flight transforms for no ops, while Connect and Flink trade ops for control. Pick Tableflow to mirror a clean topic; pick Flink when the landing data must be reshaped; pick Connect when you need the connector's control and already run the platform.

The three approaches.

Tableflow — managed materialization. Enable on a topic; it mirrors the topic to an Iceberg/Delta table, schema-registry-driven, with compaction, snapshot expiry, and exactly-once commits managed. No cluster, no maintenance jobs, no transform.
Kafka Connect Iceberg sink — self-run connector. A Connect cluster runs the Iceberg sink connector; you own the connector config, the commit coordination, and separately the compaction and snapshot-expiry jobs. Maximum connector-level control, maximum ops.
Flink SQL — self-run stream processor. A Flink job reads the topic, applies transforms (joins, filters, aggregations, enrichment), and writes Iceberg; you own the Flink cluster, checkpointing, and table maintenance. The only option that reshapes data in flight.
The common endpoint. All three land an Iceberg table in a shared catalog, so the read side (Spark/Trino/Athena) is identical regardless of how the data got there.

The decision axis — transforms vs ops.

Need a faithful mirror of a topic? Tableflow. If the topic already holds the shape you want to query, materialization with zero ops is the cheapest correct answer.
Need to transform before landing? Flink. Joins, filters, enrichment, or reshaping belong in a stream processor upstream — either write Iceberg from Flink directly, or transform into a derived topic and let Tableflow mirror that.
Need connector-level control or already run Connect? The Connect sink. If you have a Connect platform and want its knobs, the sink connector fits — but you accept the maintenance tail.
The hybrid. A common senior pattern is Flink (transform) → derived topic → Tableflow (mirror) — reshaping upstream while still getting managed materialization and maintenance.

The cost model.

Tableflow cost. The feature plus object storage for the table; no cluster to run, no maintenance-job compute, and engineering time goes to config, not operations.
Connect/Flink cost. Cluster infra (Connect or Flink) plus the compute for compaction and snapshot-expiry jobs plus the engineering time to build and keep them correct — a standing operational line item.
The hidden cost. The maintenance tail (compaction, expiry, orphan cleanup, commit idempotency) is easy to under-budget; it is recurring work and recurring compute, and it is exactly what a managed option removes.
When self-run pays off. When you genuinely need in-flight transforms (Flink) or deep connector control, the ops cost buys capability; when you only need a mirror, it buys nothing.

The failure modes senior engineers pre-empt.

Choosing Connect for a plain mirror. Running a connector plus maintenance jobs to mirror a clean topic is ops you did not need. Mitigation: use Tableflow for faithful mirrors; reserve Connect/Flink for control/transforms.
Trying to transform in Tableflow. Expecting Tableflow to join or filter fails — it mirrors. Mitigation: transform upstream (Flink/ksqlDB) into a derived topic, then materialize.
Under-budgeting maintenance. Adopting the Connect sink without staffing compaction/expiry leads to a slow, bloated table. Mitigation: budget the maintenance tail explicitly, or pick managed materialization.

Common interview probes on the trade-off.

"Tableflow or the Connect sink?" — Tableflow to mirror a topic with no ops; Connect for control if you already run the platform and accept maintenance.
"Where do transforms go?" — Flink (or ksqlDB) upstream; Tableflow does not transform.
"What's the real cost of self-managing?" — the maintenance tail: compaction, snapshot expiry, orphan cleanup, commit idempotency — recurring compute and engineering.
"Do the read engines care how it landed?" — no; all three produce an Iceberg table in a catalog.

Worked example — the mirror-vs-transform-vs-self-managed decision table

Detailed explanation. The senior artifact is a decision table mapping requirement → approach. Build it for three realistic requirements so the choice is mechanical, not a matter of taste.

Requirement A. Mirror a clean orders topic for analytics. → Tableflow.
Requirement B. Join orders with customers and land the enriched table. → Flink.
Requirement C. You already run Connect and want its control for a mirror. → Connect sink (eyes open on ops).

Question. For each requirement, pick the approach and justify it by transform need and ops appetite.

Input.

Requirement	Transform needed?	Ops appetite	Approach
A: mirror clean topic	no	none	Tableflow
B: join/enrich before landing	yes	some	Flink (or Flink → derived topic → Tableflow)
C: control, already run Connect	no	high (have it)	Kafka Connect Iceberg sink
D: mirror, but want no clusters	no	none	Tableflow

Code.

Kafka -> Iceberg: pick by (transform need) x (ops appetite)
==========================================================

A) Mirror a clean topic, zero ops
   -> TABLEFLOW.  enable on topic; compaction/expiry/EOS managed. Done.

B) Transform before landing (join orders + customers, filter, enrich)
   -> FLINK SQL.  reshape in flight, write Iceberg
      OR hybrid:  Flink -> derived topic `orders_enriched` -> TABLEFLOW mirrors it
      (keeps the transform explicit AND the materialization managed)

C) You already operate Connect and want the connector's control
   -> KAFKA CONNECT Iceberg sink.  accept the maintenance tail:
        + compaction job (rewrite_data_files)
        + snapshot expiry + orphan cleanup
        + commit coordination / idempotency

Default: if you only need a MIRROR, choose the managed one. Ops you don't need is waste.

Step-by-step explanation.

Requirement A is a faithful mirror of a topic that already has the right shape, so there is no transform to justify running a cluster — Tableflow is the cheapest correct answer, with maintenance and exactly-once handled.
Requirement B needs a join and enrichment before the data lands, which is a stream-processing job — Flink SQL does it, either writing Iceberg directly or (the senior hybrid) transforming into a derived topic that Tableflow then mirrors, keeping the transform explicit and the materialization managed.
Requirement C is the legitimate case for the Connect sink: an org that already operates a Connect platform and wants the connector's control. The key is choosing it eyes open — accepting the compaction, expiry, and idempotency tail rather than discovering it in production.
Requirement D is a trap that looks like C: wanting a mirror but not wanting clusters. The answer is Tableflow, not Connect — do not adopt operational surface for a job that has none.
The default rule falls out of the table: if the requirement is only a mirror, pick the managed option; reserve the self-run options for genuine transform needs or existing platform investment, because unneeded ops is pure cost.

Output.

Requirement	Approach	Why
Mirror clean topic	Tableflow	no transform, no ops
Join/enrich	Flink (± Tableflow)	in-flight reshape
Control, run Connect	Connect sink	control, accepts tail
Mirror, no clusters	Tableflow	don't buy ops you don't need

Rule of thumb. Choose by transform need and ops appetite: Tableflow to mirror a topic with zero ops, Flink when the data must be reshaped before landing, and the Connect sink only for connector-level control you'll actually use. If all you need is a mirror, the managed option wins — unneeded operations are pure cost.

Worked example — what you delete when you move from a Connect sink to Tableflow

Detailed explanation. The clearest way to see Tableflow's value is to migrate an existing self-managed pipeline and count what disappears. Take a working Connect-sink + cron-compaction setup and replace it, listing every deleted component.

Before. Connect cluster + Iceberg sink + a compaction cron + an expiry cron + schema-mapping glue.
After. One Tableflow config on the topic.
The delta. Everything except the topic, the schema, and the table.

Question. Enumerate the components a Connect-sink pipeline runs and show which survive the move to Tableflow.

Input.

Component (before)	Survives?
Kafka topic + registered schema	yes (unchanged)
Connect cluster + Iceberg sink	deleted
Compaction cron (`rewrite_data_files`)	deleted (managed)
Snapshot-expiry / orphan cron	deleted (managed)
Schema-mapping glue	deleted (registry-driven)
The Iceberg table + catalog	yes (unchanged)

Code.

# BEFORE — the self-managed pipeline (multiple moving parts you operate).
connect_cluster:            # <-- run + monitor + scale
  connector: iceberg-sink
  config: { table: analytics.orders, "iceberg.catalog": glue, ... }
cron_compaction:            # <-- schedule + compute
  schedule: "0 * * * *"
  run: "CALL sys.rewrite_data_files('analytics.orders')"
cron_expiry:                # <-- schedule + compute
  schedule: "0 3 * * *"
  run: "CALL sys.expire_snapshots('analytics.orders', now() - interval '7' day)"
schema_glue:                # <-- hand-maintained mapping
  map: "avro -> iceberg types, re-check on every evolution"

# AFTER — one config replaces all of the above. Same topic, same table, same catalog.
tableflow:
  topic: orders
  enabled: true
  table_format: ICEBERG
  storage: { bucket: s3://lake/orders, provider: AWS_S3 }
  catalog:  { type: GLUE, database: analytics, table: orders }
  partitioning: [ "day(event_time)" ]
  # DELETED: connect_cluster, cron_compaction, cron_expiry, schema_glue
  # (compaction, expiry, schema propagation, exactly-once are managed)

Step-by-step explanation.

The topic and its registered schema survive untouched — they are the source of truth in both worlds — as does the Iceberg table and its catalog entry, so downstream Spark/Trino/Athena queries do not change at all.
The Connect cluster and Iceberg sink connector are deleted: the ingestion they performed becomes Tableflow's managed materialization, removing a cluster to run, monitor, and scale.
The compaction cron disappears because Tableflow runs compaction itself; this is often the biggest operational relief, since a mis-scheduled or under-provisioned compaction job is a classic cause of a slow Iceberg table.
The snapshot-expiry/orphan crons and the hand-maintained schema-mapping glue also go: expiry is managed, and the schema is registry-driven, so the two most error-prone recurring tasks (metadata growth and schema drift) stop being yours.
The net is that a five-component pipeline collapses to one config block, and the components that remain (topic, schema, table, catalog) are exactly the ones that represent what you want, not how you keep it running — the managed option deletes the "how."

Output.

Category	Before	After
Clusters	Connect (+ compute for crons)	none
Cron jobs	compaction + expiry	none
Hand-maintained glue	schema mapping	none
Config to own	connector + 2 crons + glue	one Tableflow block
Downstream queries	unchanged	unchanged

Rule of thumb. Migrating a Connect-sink pipeline to Tableflow deletes the cluster, the compaction and expiry crons, and the schema-mapping glue — everything that was "how we keep it running" — while the topic, schema, table, and catalog (the "what we want") stay identical. Count the deleted components to size the operational win.

Worked example — the end-to-end managed serving architecture

Detailed explanation. Put the whole picture together: a topic materialized by Tableflow into a cataloged Iceberg table that every engine reads, with the optional Flink-transform lane for reshaping. Sketch the reference architecture a senior candidate draws on the whiteboard.

The spine. topic → Tableflow → Iceberg table (in your bucket) → catalog → engines.
The optional lane. Flink transform → derived topic → Tableflow (for enrichment).
The properties. Managed maintenance, exactly-once, one copy, multi-engine.

Question. Draw the end-to-end architecture from Kafka topic to query engines, showing where transforms fit and which properties hold.

Input.

Layer	Component	Property
Source	Kafka topic (+ schema)	streaming truth, EOS
Transform (opt)	Flink → derived topic	reshape in flight
Materialize	Tableflow	managed, exactly-once
Store	Iceberg in your bucket	one copy, partitioned
Catalog	Glue/REST	discovery, interop
Serve	Spark/Trino/Athena/Snowflake	multi-engine reads

Code.

End-to-end managed Kafka -> lakehouse serving architecture
==========================================================

   producers (idempotent/transactional)         # Kafka EOS at the source
        |
        v
   Kafka topic `orders`  <----- Schema Registry (orders-value)   # the contract
        |    \
        |     \  (optional transform lane)
        |      v
        |   Flink SQL: join/filter/enrich -> derived topic `orders_enriched`
        |      |
        v      v
   Confluent Tableflow   # materialize: schema-driven, compaction+expiry managed, EXACTLY-ONCE
        |
        v
   Iceberg table (Parquet + snapshots) in s3://lake/orders   # ONE copy, day()-partitioned
        |
        v
   Shared catalog (Glue / REST / Snowflake Open Catalog)      # the interop point
        |
   +----+----+-----------+-------------+
   v         v           v             v
 Spark     Trino       Athena       Snowflake        # same table, same SQL surface
 (ML)      (interactive) (serverless) (warehouse)     # no per-engine copy; time travel

Step-by-step explanation.

The spine is topic → Tableflow → Iceberg → catalog → engines: producers write to the topic with Kafka EOS, Tableflow materializes it schema-driven with managed maintenance and exactly-once, the table lands as one partitioned copy in your bucket, and the catalog exposes it to every engine.
The Schema Registry sits beside the topic as the contract, feeding both the producers (serialization) and Tableflow (the table schema), which is what keeps the stream and table aligned and evolution safe.
The optional Flink lane is where transforms live: when the raw topic is not the shape you want to query, Flink joins/filters/enriches into a derived topic, and Tableflow mirrors that — so you get in-flight reshaping and managed materialization instead of choosing between them.
The Iceberg table is one physical copy in object storage you govern, partitioned for pruning; the catalog is the single interop point, so adding Snowflake or a BI tool is a binding, not a new pipeline.
The properties that hold end to end are the senior talking points: exactly-once (EOS + idempotent commits), managed maintenance (compaction/expiry), one copy (open table in a shared catalog), multi-engine reads, and time travel — the whole reason to prefer this over a fan-out of bespoke pipelines and per-engine copies.

Output.

Property	How the architecture delivers it
Exactly-once	Kafka EOS + Tableflow offset-tracked commits
Managed maintenance	Tableflow compaction + snapshot expiry
In-flight transforms	optional Flink → derived topic lane
One copy, many engines	Iceberg in a shared catalog
Reproducibility	Iceberg snapshot time travel

Rule of thumb. The reference architecture is topic → Tableflow → Iceberg → catalog → engines, with an optional Flink → derived topic lane when data must be reshaped before landing. It delivers exactly-once, managed maintenance, one copy, multi-engine reads, and time travel — everything a fan-out of bespoke sinks and per-engine copies makes you build and duplicate by hand.

Senior interview question on choosing and operating the Kafka-to-Iceberg path

A senior interviewer might ask: "You own the Kafka-to-lakehouse platform. Some topics need a faithful Iceberg mirror, some need joins and enrichment before landing, and one team already runs Kafka Connect. Lay out how you'd choose among Tableflow, the Connect Iceberg sink, and Flink for each case, what operational surface each choice adds, how the cost model differs, and how you keep the read side identical regardless of the path."

Solution Using managed materialization by default, Flink for transforms, and Connect only for control

# 1. Route each topic by transform need and existing platform investment.
mirror-only topics          -> TABLEFLOW  (managed, no ops, exactly-once)
transform-before-landing    -> FLINK SQL  -> derived topic -> TABLEFLOW (hybrid)
team already running Connect -> CONNECT Iceberg sink (accept the maintenance tail)

# 2. Default path: Tableflow mirrors clean topics. No cluster, no cron.
tableflow:
  topic: orders
  enabled: true
  table_format: ICEBERG
  storage: { bucket: s3://lake/orders, provider: AWS_S3 }
  catalog:  { type: GLUE, database: analytics, table: orders }
  partitioning: [ "day(event_time)" ]

-- 3. Transform path: reshape upstream in Flink, then materialize the DERIVED topic.
CREATE TABLE orders_enriched AS
SELECT o.*, c.tier AS customer_tier
FROM orders o JOIN customers c ON o.customer_id = c.id
WHERE o.amount > 0;
-- then: tableflow { topic: orders_enriched, ... }   (managed materialization again)

# 4. Read side is IDENTICAL regardless of path — all land Iceberg in one catalog.
#    Spark / Trino / Athena / Snowflake  ->  analytics.orders  (one copy)
#
# Cost model:
#   Tableflow : feature + storage           (no cluster, no cron compute)
#   Connect   : Connect infra + compaction/expiry compute + eng time
#   Flink     : Flink infra + checkpoint tuning + (maintenance if writing Iceberg direct)

Step-by-step trace.

Case	Choice	Ops surface added	Cost driver
Mirror clean topic	Tableflow	none	feature + storage
Enrich before landing	Flink → topic → Tableflow	Flink job	Flink infra + storage
Existing Connect team	Connect sink	connector + maintenance crons	infra + eng time
Read side (all cases)	Iceberg + catalog	none extra	one copy

After deployment, mirror-only topics use Tableflow with zero operational surface; topics needing joins or filters are reshaped in Flink into a derived topic that Tableflow then materializes (managed again); and the one team already invested in Connect keeps the sink connector, accepting the compaction/expiry/idempotency tail. Every path lands an Iceberg table in the shared catalog, so Spark, Trino, Athena, and Snowflake read one copy with identical SQL — the ingestion choice is invisible downstream, and the cost model makes the managed default obviously cheaper for the common mirror case.

Output:

Metric	Connect-everywhere	Routed (Tableflow default)
Clusters for mirror topics	Connect + cron compute	none
Transform capability	in-connector (limited)	Flink (full) where needed
Maintenance ownership	every topic	only the Connect team's
Copies of the data	risk of per-engine	one (shared catalog)
Eng time	operating pipelines	config + the few transforms

Why this works — concept by concept:

Managed-by-default routing — sending mirror-only topics to Tableflow removes clusters and maintenance for the common case, so operational surface is spent only where a transform or existing investment justifies it.
Flink for transforms — putting joins, filters, and enrichment in a stream processor upstream (into a derived topic Tableflow mirrors) keeps reshaping explicit and testable while still getting managed materialization — the best of both.
Connect only for control — reserving the self-run sink for teams that already operate Connect and want its knobs avoids adopting a maintenance tail for jobs that do not need it.
Identical read side — because every path lands an Iceberg table in a shared catalog, the ingestion decision is invisible to Spark/Trino/Athena/Snowflake, so you can route per topic without fragmenting the query experience.
Cost — a managed feature plus storage for mirrors, Flink infra only where transforms are required, and Connect maintenance only where a team already owns it, versus running clusters and maintenance crons for every topic. The eliminated cost is the standing operational line item of self-managing ingestion you did not need — O(config) for mirrors and O(transform) only where reshaping is real.

Design
Topic — design
Design problems on Kafka-to-Iceberg architecture choices

Practice →

Optimization
Topic — optimization
Optimization problems on ingestion cost and maintenance

Practice →

Cheat sheet — Kafka-to-Iceberg with Tableflow

Topic vs table. A Kafka topic is an append-only, row-oriented log for streaming consumers; an Iceberg table is columnar Parquet + snapshots in a catalog for analytical scans. They are two representations of the same events — read the topic for recency, the table for scans, and use materialization to keep them in sync. Never scan history off the topic; never poll the table for a single fresh event.
The gap Tableflow closes. Getting a topic into Iceberg used to mean a sink connector + a compaction job + snapshot expiry + orphan cleanup + schema-mapping glue + commit idempotency. Confluent Tableflow collapses all of that into a topic-to-table config: enable on a topic and it materializes, maintains, and exactly-once-commits for you.
Tableflow enable template. tableflow: { topic, enabled: true, table_format: ICEBERG|DELTA, storage: {bucket, provider, credential}, catalog: {type: GLUE|REST|SNOWFLAKE, database, table}, partitioning: [ "day(event_time)" ] }. Compaction, snapshot expiry, and exactly-once are managed — you do not schedule them.
Schema is the contract. The Iceberg table schema is the registered <topic>-value subject (Avro/Protobuf/JSON) — no hand-written DDL, no stream-vs-table drift. A topic with no registered schema has nothing to materialize; register one first. Confirm the subject-name strategy resolves.
Type mapping. long/int → bigint/int; decimal logical type → decimal(p,s) (exact money, never float); timestamp-millis/micros → timestamptz (enables time pruning); Avro record → struct, array → list, map → map; ["null", T] union → nullable column (the rule that makes additive evolution safe).
Partitioning. Iceberg hidden partitioning on a transform: day(event_time)/hour(event_time) for time-series, bucket(N, key) for high-cardinality keys. Filters on the raw column prune partitions from metadata. Never leave a streamed table unpartitioned (it scans everything, forever) and never partition on a raw high-cardinality column (small-file storm).
Compaction. Streaming commits make many small files → slow reads + metadata bloat. Managed by Tableflow; self-run it is rewrite_data_files (target ~512 MB) + expire_snapshots + remove_orphan_files on a schedule. "Why is my Iceberg table slow?" is almost always missing compaction.
Exactly-once. Bind atomic Iceberg snapshots to Kafka offset ranges + skip already-committed offsets on restart → no duplicate rows across restarts/rebalances/replays. Upstream, Kafka EOS = idempotent/transactional producer + read_committed consumers. If a design can't answer "what happens on a mid-batch restart," assume it duplicates.
Schema evolution. Registry compatibility (BACKWARD default) rejects unsafe changes at the source; add optional fields to evolve. Iceberg evolves by field ID, so old Parquet files are never rewritten (absent field → null) and safe widening (int→long, decimal precision) is a read-time promotion. One query spans every schema era.
Query anywhere. One Iceberg table in a shared catalog (Glue/REST/Snowflake Open Catalog) is read by Spark, Trino, Athena, and Snowflake with the same SQL surface and one physical copy — the catalog is the single pointer. Add an engine by binding its catalog; never copy the data per engine. Time-travel (AS OF snapshot/timestamp) reproduces past reports and audits corrections.
Tableflow vs Connect vs Flink. Tableflow = managed mirror, zero ops, no transforms. Kafka Connect Iceberg sink = self-run connector + the maintenance tail (compaction/expiry/idempotency), for control you already operate. Flink SQL = self-run processor that transforms (join/filter/enrich) before landing. Default to Tableflow for mirrors; Flink → derived topic → Tableflow for transforms; Connect only for existing-platform control.
Cost model. Tableflow = feature + object storage (no cluster, no cron compute). Connect/Flink = infra + compaction/expiry compute + engineering time for the maintenance tail. Unneeded ops is pure cost — pay for self-managing only when you need transforms or connector control.

Frequently asked questions

What is Confluent Tableflow?

Confluent Tableflow is a managed feature that represents a Kafka topic as an Apache Iceberg (or Delta Lake) table continuously — it reads the topic, uses the registered schema-registry schema to define the table, writes the records as partitioned columnar Parquet into object storage you designate, commits Iceberg snapshots, and publishes the table to a catalog so query engines can find it. Instead of building and operating an ingestion pipeline — a sink connector plus compaction, snapshot-expiry, and schema-mapping jobs — you enable materialization on a topic and the table stays in sync with the log, with compaction and exactly-once commits handled for you. The result is that a streaming topic becomes an ordinary lakehouse table that Spark, Trino, Athena, and Snowflake can query, without you writing DDL or running a cluster.

How does Tableflow get the table schema — do I write DDL?

You do not write DDL. Tableflow reads the topic's registered value schema (typically the <topic>-value subject under the default subject-name strategy) from the Schema Registry and uses that Avro, Protobuf, or JSON schema as the Iceberg table schema — the registered contract is the table definition. Each field maps to an Iceberg column by a deterministic type mapping (integers preserve width, the decimal logical type becomes decimal(p,s), timestamp logical types become timestamptz, records become structs, arrays become lists, and ["null", T] unions become nullable columns). Because the table is derived from the registry rather than mapped by hand, the stream contract and the table contract are the same object and cannot drift; the one physical decision you still make is the partitioning transform. A topic with no registered schema has nothing to build columns from, so registering a schema is the prerequisite.

Does Tableflow guarantee exactly-once / no duplicate rows?

Yes — exactly-once is a correctness property of the materialization, not a hope. Tableflow tracks which Kafka offsets have been committed to which Iceberg snapshot and commits idempotently, so if the writer restarts or a consumer group rebalances mid-batch, re-reading the same offsets is a no-op rather than a second insert — each record becomes exactly one row. Iceberg commits are atomic, so a reader never sees a half-written batch and a failed write leaves no partial rows. Upstream, Kafka's own exactly-once semantics keep the source clean: an idempotent/transactional producer plus read_committed consumers mean aborted transactions never appear in the stream. End to end, each business event maps to one row in the table across producer retries, rebalances, and restarts — which is exactly the guarantee a naive sink connector fails to provide when it re-processes offsets after a crash and doubles rows.

Tableflow vs the Kafka Connect Iceberg sink connector — which do I pick?

Pick Tableflow when you want a faithful, managed mirror of a topic with zero operational surface: it materializes, compacts, expires snapshots, and commits exactly-once without a cluster or cron jobs to run. Pick the Kafka Connect Iceberg sink connector when you want connector-level control and already operate a Connect platform — but go in eyes open, because you inherit the maintenance tail: a compaction job (rewrite_data_files), snapshot expiry and orphan cleanup, hand-maintained schema mapping, and commit-idempotency logic, each on its own schedule and compute. If you need to transform the data (joins, filters, enrichment) before it lands, neither is the answer to that part — do the transform upstream in Flink (or ksqlDB) into a derived topic, then let Tableflow mirror the derived topic. The rule of thumb: managed materialization for mirrors, Flink for transforms, and the Connect sink only for control you already run — because operations you do not need are pure cost.

Can I query the Tableflow Iceberg table from Athena / Trino / Snowflake?

Yes — that is the point of landing an open Iceberg table in a shared catalog. Once the table is registered (in AWS Glue, a REST catalog, or Snowflake's Open Catalog), any engine that speaks the catalog resolves the same table and reads the same partitioned Parquet files: Spark via the Iceberg Spark runtime, Trino via its Iceberg connector, Athena natively over Glue-registered Iceberg, and Snowflake through a catalog integration. The SQL surface is essentially identical (the only difference is the catalog prefix in the table name), there is one physical copy of the data rather than a per-engine export, and every engine sees a consistent snapshot even while Tableflow keeps writing. Because Iceberg retains snapshots, any of those engines can also time-travel — querying the table as of a past snapshot or timestamp — which is how you reproduce a report's exact numbers or audit a late correction. Adding another consumer is a catalog binding, not a new pipeline.

What happens to small files and schema changes over time?

Both are handled so the table stays healthy for years. Streaming ingestion commits frequently to keep the table fresh, which naturally produces many small Parquet files; left alone, that inflates metadata and slows every scan, so Tableflow runs compaction (rewriting small files into well-sized ones) and snapshot expiry as managed maintenance — the recurring jobs a self-built sink would have to schedule itself. Schema changes are governed by the Schema Registry's compatibility mode: additive, optional-field evolutions are accepted and unsafe changes (dropping a required field, an incompatible retype) are rejected at the source, so the table is protected from breaking changes before any file is written. Tableflow applies each accepted change to the Iceberg schema by stable field ID, so existing data files are never rewritten — a read of an old file simply returns null for a newly added column, and safe type widening is a read-time promotion. The upshot is a table that compacts itself, evolves without downtime, and keeps every historical file readable.

Practice on PipeCode

Drill the streaming practice library → for the topic-vs-table, materialization, and exactly-once problems that Tableflow makes concrete.
Rehearse serving patterns on the real-time analytics practice library → for the query-the-streamed-table, freshness, and time-travel scenarios a lakehouse serving layer has to get right.
Work the ingestion mechanics on the ETL practice library → for the serialization, partitioning, compaction, and schema-evolution patterns that decide whether an Iceberg table stays fast.
Sharpen the architecture axis with the system design practice library → for the Tableflow-vs-Connect-vs-Flink, catalog-interop, and cost/ops trade-offs a Kafka-to-lakehouse platform must weigh.
Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the topic-vs-table, schema-registry, and compaction/exactly-once patterns against real graded inputs — Kafka, Iceberg, partitioning, and multi-engine reads.

Lock in Kafka-to-Iceberg muscle memory

Docs explain Tableflow and Iceberg. PipeCode drills explain the decision — when a topic must become a table instead of a scan target, when the schema has to come from the registry, when `exactly-once` beats "the connector is probably fine," and when a managed mirror beats a self-run sink plus a compaction cron. Pipecode.ai is Leetcode for Data Engineering — streaming and lakehouse practice tuned for the production trade-offs senior data engineers actually face.

Practice streaming problems →
Practice system design problems →