AWS S3 Tables & S3 Metadata: Fully-Managed Iceberg on Object Storage

S3 Tables are AWS's answer to a question every lakehouse team eventually hits: how do you get a real, ACID, time-travelling table on top of cheap object storage without also signing up to run the catalog, babysit the small-files problem, and cron a fleet of maintenance jobs forever? For years the pattern was raw Parquet on S3 plus Apache Iceberg for the table format — which gets you snapshots, schema evolution, and hidden partitioning, but leaves you owning a metadata catalog, a compaction pipeline, snapshot expiration, and orphan-file cleanup as your operational problem. A pile of Parquet files in a prefix is not a table; Iceberg makes it one, and then hands you the maintenance bill.

This guide is the senior-data-engineering walkthrough for the managed alternative — running Iceberg on object storage as a service instead of a self-operated pipeline. It is framed the way interviewers actually probe it: why Iceberg needs a catalog at all, how a table bucket differs from the general-purpose bucket you already know, how S3 Tables exposes a managed Iceberg REST endpoint that Athena, EMR, Spark, and Redshift can all read, how S3 Metadata turns the object metadata of an ordinary bucket into queryable Iceberg tables, and how the automatic maintenance — compaction, snapshot expiration, and unreferenced-file removal — is the part you no longer have to build. Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the data processing practice library →, rehearse pipeline patterns on the ETL practice library →, and sharpen the lakehouse architecture axis with the system design practice library →.

---

On this page

• Why S3 Tables exist — managed Iceberg on object storage
• S3 Tables — table buckets, namespaces, and the catalog
• S3 Metadata — object metadata as queryable Iceberg tables
• Automatic maintenance — compaction, snapshot expiration, cleanup
• Architecture, cost, and when vs self-managed Iceberg
• Cheat sheet — S3 Tables & S3 Metadata
• Frequently asked questions
• Practice on PipeCode

---

1. Why S3 Tables exist — managed Iceberg on object storage

The lakehouse gap — files on S3 are not a table; Iceberg makes them one, then hands you the maintenance

The one-sentence invariant: S3 Tables are a purpose-built S3 bucket type that stores your data as Apache Iceberg tables behind an AWS-managed Iceberg catalog and runs the table maintenance — compaction of small files, snapshot expiration, and unreferenced-file cleanup — automatically, so the reason they exist is that a lakehouse on object storage needs three things beyond raw Parquet — a table format for ACID/snapshots/schema evolution, a catalog that is the source of truth for which files are the current table, and continuous maintenance to keep scans fast — and self-managing all three is exactly the operational tax S3 Tables removes. Drop ten thousand Parquet files into a prefix and query engines will scan garbage; put Iceberg over them and you get a table, but you now own the catalog and the maintenance — and that ownership is the gap S3 Tables closes.

The four axes interviewers actually probe.

• Table format vs storage layer. Do you know the difference? Object storage (S3) is the storage layer — durable bytes in buckets. A table format (Iceberg, Delta, Hudi) is a metadata layer on top that tells an engine which files constitute the table right now, what the schema is, and how to time-travel. The senior answer never conflates "data in S3" with "a table"; the table format is what makes files behave like a table.
• Catalog ownership. Where does an engine learn the current metadata pointer for a table? Iceberg needs a catalog to atomically swap the pointer to the latest metadata file on commit. Self-managed, that catalog is Glue, a Hive Metastore, Nessie, or a JDBC catalog you run. The senior answer names the catalog as the source of truth and knows S3 Tables ships a managed one.
• Table maintenance. What keeps a streaming/CDC Iceberg table fast over time? Frequent small writes create thousands of tiny files and pile up snapshots; without compaction, snapshot expiration, and orphan-file cleanup, scans slow down and storage bloats. The senior answer treats maintenance as mandatory, not optional — and knows S3 Tables runs it for you.
• Cost and operability. What is the total cost — storage, requests, and the maintenance you either run yourself or pay AWS to run? The senior answer counts the compute and engineer-time of a self-managed maintenance pipeline against the managed-maintenance line item on the S3 Tables bill.

The 2026 reality — the AWS-managed Iceberg stack.

• S3 Tables introduce a new bucket type — the table bucket — alongside general-purpose and directory buckets. A table bucket stores tabular data as Iceberg tables, organised into namespaces, and exposes a managed Iceberg REST catalog endpoint that any Iceberg-compatible engine can use.
• Automatic maintenance is the headline: S3 Tables continuously compacts small files toward a target size, expires old snapshots, and removes unreferenced files — the three jobs you would otherwise schedule and monitor yourself. AWS positions this as delivering materially higher query throughput than the same data hand-managed on general-purpose S3.
• S3 Metadata is the sibling feature: it writes the object metadata of a general-purpose bucket (keys, sizes, tags, versions, create/delete events) into managed Iceberg tables — a journal table of change events and a live inventory table snapshot — so you can query "what is in my bucket / what changed" with Athena or Spark.
• Integration is through the Glue Data Catalog and Lake Formation: registering a table bucket with Glue lets Athena, Redshift, and EMR query S3 Tables with ordinary SQL and centralised access control, while Spark can also talk to the Iceberg REST endpoint directly.

What interviewers listen for.

• Do you say files on S3 are not a table and name the table format as what makes them one? — senior signal.
• Do you name the catalog as the atomic source of truth for the current metadata pointer, not an afterthought? — required answer.
• Do you treat compaction / snapshot expiration / orphan cleanup as mandatory maintenance, and know S3 Tables automates it? — senior signal.
• Do you count maintenance as a cost — either compute you run or a managed line item — rather than pretending Iceberg is free after ingest? — required answer.
• Do you place S3 Metadata as observability over ordinary buckets, distinct from S3 Tables as a data store? — senior signal.

Worked example — the "is this a table?" decision map

Detailed explanation. The most useful thing to carry into an S3 Tables interview is a crisp map from what you have to what you get. Every senior discussion starts by separating the storage layer, the table format, the catalog, and the maintenance — because conflating them is the most common junior mistake. Walk the map for four states of the same sales data on S3.

• The states. Raw Parquet in a prefix; Iceberg over that Parquet with a self-run Glue catalog; the same data in an S3 Tables table bucket; and S3 Metadata over the raw bucket.
• The tension. Each step up adds a capability (atomic commits, time travel, managed maintenance) and a cost (catalog ops, maintenance compute, or a managed fee).
• The rule. Name which layer gives which capability, and who operates it.

Question. For each state, say whether you have a real table, who owns the catalog, and who runs maintenance.

Input.

State	Real table?	Catalog owner	Maintenance owner
Raw Parquet in an S3 prefix	no (just files)	none	you (glue crawler hacks)
Iceberg on general-purpose S3	yes	you (Glue/Hive/Nessie)	you (Spark jobs)
S3 Tables (table bucket)	yes	AWS-managed	AWS-managed
S3 Metadata over a bucket	metadata table	AWS-managed	AWS-managed

Code.

The lakehouse stack, bottom to top — who owns each layer?
========================================================

  Query engine     (Athena / Spark / EMR / Redshift / Trino)
       |
  Catalog          <- "which metadata file IS the table right now?"
       |               self-managed: Glue / Hive / Nessie (you run it)
       |               S3 Tables:    managed Iceberg REST catalog (AWS runs it)
       |
  Table format     <- Apache Iceberg: snapshots, schema evolution, ACID commits
       |               (manifest lists -> manifests -> data files)
       |
  Storage layer    <- S3 object storage: durable Parquet bytes in a bucket
                       general-purpose bucket  OR  table bucket

Maintenance (orthogonal, continuous):
  compaction (small files -> target size), snapshot expiration, orphan cleanup
  self-managed: your Spark/cron jobs      S3 Tables: automatic

Step-by-step explanation.

1. The storage layer is just S3 — durable Parquet bytes. Raw Parquet in a prefix stops here: an engine can scan the files, but there is no atomic notion of "the table," no snapshots, and no safe concurrent writes. It is data, not a table.
2. The table format (Iceberg) adds a metadata tree — a current metadata file pointing at manifest lists, manifests, and data files — so an engine knows exactly which files are live, can time-travel to an old snapshot, and can commit changes atomically. This is the step that turns files into a table.
3. The catalog is what makes the commit atomic: it holds the pointer to the current metadata file and swaps it in one operation, so two writers cannot corrupt the table. Self-managed, you run that catalog (Glue, Hive, Nessie); with S3 Tables the table bucket is a managed Iceberg catalog.
4. Maintenance is orthogonal to all three and never stops mattering: streaming writes create small files and snapshots that must be compacted and expired or scans decay. Self-managed, you schedule Spark maintenance actions; S3 Tables runs them automatically.
5. S3 Metadata is a different animal — it does not store your analytical data; it stores metadata about objects in an ordinary bucket as an Iceberg table, so it is observability, not the lakehouse store. Keeping those two straight is the tell of someone who has actually used both.

Output.

Capability	Raw Parquet	Iceberg (self-managed)	S3 Tables
Atomic commits / ACID	no	yes	yes
Time travel / snapshots	no	yes	yes
Managed catalog	no	no (you run it)	yes
Automatic maintenance	no	no (you run it)	yes

Rule of thumb. Separate the four layers out loud — storage (S3), table format (Iceberg), catalog (the atomic pointer), and maintenance (compaction/expiry/cleanup). Raw files give you none of the table properties; Iceberg gives you the format but leaves catalog and maintenance to you; S3 Tables manages the catalog and the maintenance so you own only the data and the schema.

Worked example — what interviewers actually probe

Detailed explanation. The senior S3 Tables interview escalates predictably: an innocuous opener ("just put the data in S3"), then progressive narrowing to test whether you understand the table format, the catalog, and the maintenance burden. Candidates who volunteer compaction, the catalog-as-source-of-truth, and the managed-vs-self trade score highest.

• Ambiguous opener. "Store the events in S3 and let Athena query them. Done?"
• Follow-up 1. "Streaming writes 50k tiny files a day — queries are crawling. Why?" — probes small files / compaction.
• Follow-up 2. "Two jobs write the same table concurrently. How is that safe?" — probes catalog / atomic commit.
• Follow-up 3. "Who deletes old snapshots and orphaned files?" — probes maintenance ownership.
• Follow-up 4. "Build it managed or self-managed?" — probes S3 Tables vs DIY Iceberg.

Question. Draft a senior answer that pre-empts all four follow-ups without waiting to be asked.

Input.

Interview signal	Weak answer	Senior answer
Storage	"dump Parquet in a prefix"	"an Iceberg table, not loose files"
Small files	"add more compute"	"compaction to a target file size"
Concurrency	"hope they don't collide"	"the catalog commits atomically"
Maintenance	"we'll script it later"	"expire snapshots + remove orphans on a schedule"
Build	"whatever's fastest"	"S3 Tables unless a reason forces DIY"

Code.

Senior S3 Tables answer template (5 minutes)
============================================

Minute 1 — files are not a table
  "Raw Parquet in a prefix isn't a table — no atomic commits, no
   time travel. I'd store it as an Iceberg table so engines see one
   consistent snapshot and writers commit atomically."

Minute 2 — the small-files problem
  "Streaming writes make thousands of tiny files; scans then pay
   per-file overhead and LIST/GET cost. Compaction rewrites them into
   target-sized (~512 MB) files so scans stay fast."

Minute 3 — the catalog
  "Concurrent writers are safe because the catalog owns the pointer to
   the current metadata file and swaps it atomically on commit — that
   catalog is the table's source of truth, not the file listing."

Minute 4 — maintenance ownership
  "Old snapshots and unreferenced files must be expired and deleted or
   storage bloats and time-travel gets expensive. Someone has to run
   compaction, snapshot expiration, and orphan cleanup continuously."

Minute 5 — managed vs self-managed
  "S3 Tables gives a table-bucket with a managed Iceberg catalog and
   runs all that maintenance automatically, integrated with Glue and
   Athena. I'd default to it and only self-manage for a specific reason."

Step-by-step explanation.

1. Minute 1 frames everything around files-are-not-a-table. Weak candidates stop at "put it in S3"; naming the Iceberg table and atomic commits signals you understand the format layer, not just storage.
2. Minute 2 shows you know the single most common lakehouse performance killer — the small-files problem — and its fix, compaction toward a target size. This is the most senior operational thing you can say about Iceberg.
3. Minute 3 pre-empts the concurrency follow-up by naming the catalog as the atomic source of truth. "The file listing is the table" is the wrong mental model; "the catalog pointer is the table" is right.
4. Minute 4 pre-empts the maintenance follow-up. Volunteering snapshot expiration and orphan cleanup before being asked shows you have operated an Iceberg table past day one, not just created one in a demo.
5. Minute 5 closes on the managed-vs-self trade and lands S3 Tables as the default — with the maturity to note that a specific requirement (an existing catalog, an engine S3 Tables doesn't support, strict portability) can still justify DIY.

Output.

Grading criterion	Weak score	Senior score
Files vs table distinction	rare	mandatory
Compaction for small files	occasional	mandatory
Catalog as atomic source of truth	rare	senior signal
Maintenance ownership named	rare	senior signal
Managed-vs-self reasoning	rare	senior signal

Rule of thumb. The senior S3 Tables answer is a 5-minute monologue: files are not a table, the small-files problem needs compaction, the catalog commits atomically, maintenance must run continuously, and S3 Tables manages the catalog and maintenance for you. Rehearse it once; deploy it every interview.

Worked example — S3 Tables vs self-managed Iceberg vs raw S3

Detailed explanation. A frequent trap is "why not just use Iceberg on normal S3, or even plain Parquet?" The weak answer picks by familiarity. The senior answer picks by who operates the catalog and maintenance and what the workload needs. Compare three ways to hold the same fact table.

• Raw Parquet on S3. Cheapest bytes, but not a table: no ACID, no time travel, no safe concurrent writes.
• Self-managed Iceberg on general-purpose S3. A real table, full control and portability — but you run the catalog (Glue/Hive/Nessie) and the maintenance jobs yourself.
• S3 Tables. A real Iceberg table with an AWS-managed catalog and automatic maintenance — less to operate, at the cost of a managed fee and AWS-specific setup.

Question. Contrast the three on table semantics, catalog ownership, maintenance, and portability.

Input.

Dimension	Raw Parquet	Self-managed Iceberg	S3 Tables
ACID / time travel	none	yes	yes
Catalog	none	you run it	AWS-managed
Compaction / expiry	manual scripts	your Spark jobs	automatic
Portability	high (just files)	high (open format)	high (open format)
Ops burden	low until it breaks	high	low

Code.

Same fact table, three ways — who does the work?
================================================

Raw Parquet on S3
  s3://lake/events/dt=2026-08-26/part-0001.parquet ...
  -> engine scans files; concurrent writes can corrupt; no snapshot.
     cheap bytes, but NOT a table. Fine only for append-only staging.

Self-managed Iceberg on general-purpose S3
  catalog: Glue/Hive/Nessie  (you run + secure + scale it)
  maintenance: nightly Spark  rewrite_data_files / expire_snapshots /
               remove_orphan_files  (you schedule + monitor + pay compute)
  -> full control + portability; you own the catalog AND the maintenance.

S3 Tables (table bucket)
  arn:aws:s3tables:us-east-1:111122223333:bucket/analytics
    namespace: sales -> table: orders (Iceberg)
  catalog: managed Iceberg REST endpoint (AWS runs it)
  maintenance: compaction + snapshot expiration + orphan cleanup (automatic)
  -> you own only the data + schema; AWS owns catalog + maintenance.

Step-by-step explanation.

1. Raw Parquet is the floor: it is the cheapest way to hold bytes, but it gives you none of the table guarantees. It is acceptable only for append-only staging where an engine reads whole partitions and no one needs snapshots or concurrent-writer safety.
2. Self-managed Iceberg turns those files into a real table with ACID commits and time travel — the open format keeps you portable across engines and clouds. The price is operational: you stand up and secure a catalog and you run maintenance jobs (rewrite_data_files, expire_snapshots, remove_orphan_files) on a schedule, paying their compute and owning their failures.
3. S3 Tables keeps the same open Iceberg format and portability but moves the catalog and the maintenance to AWS: the table bucket exposes a managed Iceberg REST catalog and continuously compacts, expires, and cleans up for you. You own only the data and the schema.
4. Portability is a wash across the two table options — both are Iceberg, so any Iceberg engine can read them and you can migrate out. The real axis is ops burden: self-managed trades your engineering time for control; S3 Tables trades a managed fee for that time back.
5. The senior move is not "managed is always better" but "default to S3 Tables and justify DIY": you self-manage when you already run a battle-tested catalog, need an engine or a feature S3 Tables doesn't yet support, or must avoid the managed-maintenance cost profile — otherwise the automatic maintenance is worth more than the fee.

Output.

Question	Raw Parquet	Self-managed Iceberg	S3 Tables
"Safe concurrent writes?"	no	yes	yes
"Who runs the catalog?"	nobody	you	AWS
"Who compacts small files?"	you (scripts)	you (Spark)	AWS (auto)
"Portable to another engine?"	yes	yes	yes

Rule of thumb. Use raw Parquet only for append-only staging, self-managed Iceberg when you need full control of the catalog and maintenance or already run one, and S3 Tables when you want a real Iceberg table without owning the catalog or the maintenance. All three keep the open format, so portability is not the deciding factor — operational ownership is.

Senior interview question on choosing a lakehouse table layer on AWS

A senior interviewer often opens with: "A team is landing streaming clickstream and CDC data on S3 and wants Athena, EMR/Spark, and Redshift to all query it as one governed set of tables, with time travel and safe concurrent writes, and without a data engineer permanently assigned to babysit compaction. You have plain S3 today. Design the table layer: what makes the files a table, where the catalog lives, who runs maintenance, and whether you reach for S3 Tables or self-managed Iceberg — and why it's a lakehouse, not a folder of Parquet."

Solution Using an Iceberg table bucket, a managed catalog, automatic maintenance, and Glue integration

-- Step 1 — decide the table layer. Files -> Iceberg table; managed catalog + maintenance.
Requirement: multi-engine reads (Athena/EMR/Redshift), time travel, safe concurrent
writes, no dedicated compaction owner  ->  S3 Tables (managed Iceberg), not DIY.

# Step 2 — a table bucket is the managed Iceberg catalog + storage in one.
aws s3tables create-table-bucket --name analytics --region us-east-1
# -> arn:aws:s3tables:us-east-1:111122223333:bucket/analytics

# A namespace groups related tables (like a database/schema).
aws s3tables create-namespace \
  --table-bucket-arn arn:aws:s3tables:us-east-1:111122223333:bucket/analytics \
  --namespace clickstream

# Step 3 — maintenance is automatic and configured as metadata, not cron jobs.
table_bucket: analytics
maintenance:
  iceberg_compaction:        { status: enabled, target_file_size_mb: 512 }
  iceberg_snapshot_management:{ status: enabled, min_snapshots: 5, max_age_hours: 168 }
  iceberg_unreferenced_file_removal: { status: enabled, non_current_hours: 72 }
# AWS runs compaction, snapshot expiry, and orphan cleanup continuously — no Spark cron.

-- Step 4 — one governed contract for every engine via the Glue Data Catalog.
-- Register the table bucket with Glue + Lake Formation, then EVERY engine reads it:
SELECT event_date, count(*) AS events
FROM   "s3tablescatalog/analytics".clickstream.events   -- Athena, federated catalog
WHERE  event_date = DATE '2026-08-26'
GROUP  BY event_date;
-- EMR/Spark can also hit the Iceberg REST endpoint directly; Redshift reads via Glue.

Step-by-step trace.

Decision	Before (plain S3)	After (S3 Tables)
Is it a table?	loose Parquet files	Iceberg table (ACID, snapshots)
Concurrent writes	can corrupt	atomic commit via managed catalog
Small files	pile up, scans slow	compacted to ~512 MB automatically
Snapshots / orphans	grow forever	expired + cleaned on a schedule
Multi-engine access	per-engine glue hacks	one Glue-federated catalog
Maintenance owner	a data engineer	AWS (managed)

After the rollout, the clickstream and CDC streams land as Iceberg tables inside the analytics table bucket; the managed Iceberg catalog makes every writer's commit atomic so concurrent Spark and Firehose writes never corrupt the table; automatic compaction rewrites the flood of tiny streaming files into ~512 MB files so Athena scans stay fast; snapshot expiration and orphan removal keep storage and time-travel cost bounded; and Glue Data Catalog federation means Athena, EMR/Spark, and Redshift all read the same governed tables. No engineer is assigned to compaction.

Output:

Metric	Plain S3 (files)	S3 Tables (managed Iceberg)
Table semantics	none (files)	ACID + time travel
Concurrent-writer safety	corruption risk	atomic (catalog)
Scan performance over time	degrades (small files)	stable (auto-compaction)
Maintenance engineering	ongoing (Spark cron)	zero (managed)
Engines that can query it	ad-hoc per engine	Athena/EMR/Redshift via Glue

Why this works — concept by concept:

• Iceberg table format — a metadata tree (current metadata file → manifests → data files) turns loose Parquet into a table with atomic commits, snapshots, and schema evolution, so every engine sees one consistent version and can time-travel.
• Managed catalog — the table bucket is an Iceberg REST catalog that owns the pointer to the current metadata file and swaps it atomically on commit, so concurrent writers are safe without you running Glue/Hive/Nessie yourself.
• Automatic maintenance — compaction toward a target file size, snapshot expiration, and unreferenced-file removal run continuously as managed operations, so the small-files problem and storage bloat never accumulate and no Spark maintenance cron exists to fail.
• Glue Data Catalog federation — registering the table bucket into Glue and Lake Formation exposes one governed contract that Athena, EMR/Spark, and Redshift all query, so access control and discovery are central rather than per-engine.
• Cost — you pay storage, requests, and a managed-maintenance line item instead of the compute and engineer-time of a self-run catalog and nightly compaction jobs. The eliminated cost is an entire maintenance pipeline and its on-call — O(managed fee) instead of O(engineers) to keep an Iceberg table healthy.

Design
Topic — design
Design problems on lakehouse and table-layer architecture

Practice →

Data processing Topic — data-processing Data processing problems on Iceberg tables and batch scans

Practice →

---

2. S3 Tables — table buckets, namespaces, and the catalog

A table bucket groups namespaces of Iceberg tables behind a managed Iceberg REST endpoint

The mental model in one line: S3 Tables add a new bucket type — the table bucket — that is simultaneously an S3 storage container and a managed Apache Iceberg REST catalog: inside it you create namespaces (logical groupings, like databases) that hold Iceberg tables, each identified by an ARN, and the bucket exposes an Iceberg REST endpoint so Spark, EMR, Flink, and Trino can create/read/write tables through the standard Iceberg catalog API, while registering the bucket with the AWS Glue Data Catalog lets Athena and Redshift query the same tables with ordinary SQL and Lake Formation governance — so one table bucket becomes a multi-engine, governed catalog you did not have to operate. You provision a table bucket, drop namespaces and tables into it, and every Iceberg-capable engine can reach them.

The three levels of the hierarchy.

• Table bucket → the catalog + storage. A table bucket is a first-class resource with its own ARN (arn:aws:s3tables:region:account:bucket/name). It is not a general-purpose bucket — you cannot PutObject arbitrary keys into it; you interact with it through the S3 Tables API and Iceberg operations. It is the unit of the managed catalog.
• Namespace → a database/schema. Inside a table bucket, a namespace groups related tables (clickstream, sales, ops). It is the middle level of the three-part name bucket.namespace.table, the same shape engines expect from a catalog.
• Table → an Iceberg table. Each table is a full Apache Iceberg table: Parquet data files, a metadata tree of manifests and snapshots, schema evolution, hidden partitioning, and time travel. The table bucket manages its metadata pointer.

How engines connect.

• The Iceberg REST endpoint. The table bucket implements the Iceberg REST Catalog spec, so Spark/EMR/Flink/Trino configure it as a REST catalog (endpoint + sigv4 auth) and then run ordinary Iceberg SQL/DataFrame operations — no bespoke connector.
• Glue Data Catalog integration. Registering (federating) the table bucket into Glue surfaces its namespaces and tables as Glue databases/tables, which is what lets Athena and Redshift query them by name with SQL and what puts them under Lake Formation access control.
• The AWS analytics services integration. With the Glue integration in place, Athena, Redshift, EMR, and QuickSight treat S3 Tables like any other Glue-catalogued data — one governed surface, many engines.

Working with tables.

• Create via API or SQL. You can create a table with the S3 Tables API/CLI (create-table) or by issuing CREATE TABLE through an engine pointed at the Iceberg REST endpoint or the Glue-federated catalog.
• Write with any Iceberg writer. Spark, Flink, and Firehose (via its Iceberg destination) append/merge into the table; every commit goes through the managed catalog atomically.
• Standard Iceberg superpowers. MERGE INTO for upserts, snapshot time travel (FOR TIMESTAMP AS OF), schema evolution (ADD COLUMN), and partition evolution all work because it is real Iceberg underneath.

The failure modes senior engineers pre-empt.

• Treating a table bucket like a general-purpose bucket. You cannot browse it with s3 ls or PutObject keys; access is through the S3 Tables/Iceberg APIs. Mitigation: use the s3tables client and the Iceberg REST/Glue path, and get the ARN and region right.
• No Glue integration, then "Athena can't see it." Athena and Redshift need the table bucket federated into Glue; without it they have no catalog entry. Mitigation: enable the Glue Data Catalog integration for the table bucket before expecting SQL engines to find the tables.
• Engine/Iceberg version mismatch. An old Spark or Iceberg runtime may not speak the REST catalog or a required Iceberg spec version. Mitigation: use a supported EMR/Spark release and the AWS-provided Iceberg runtime, and match the catalog client version.

Common interview probes on S3 Tables.

• "What is a table bucket?" — a new S3 bucket type that is a managed Iceberg catalog plus storage; it holds namespaces of Iceberg tables.
• "How does Spark talk to it?" — via the Iceberg REST catalog endpoint the bucket exposes (SigV4-authenticated), using standard Iceberg operations.
• "How does Athena query it?" — through the Glue Data Catalog integration that federates the table bucket into Glue.
• "Is it still Iceberg?" — yes; it is real Apache Iceberg, so it is portable and supports time travel, MERGE, and schema evolution.

Worked example — create a table bucket, namespace, and table with the CLI and boto3

Detailed explanation. The canonical bootstrap: provision a table bucket, add a namespace, and create an Iceberg table — first with the AWS CLI, then with boto3 for automation. This is the "hello world" of S3 Tables and the thing an interviewer may ask you to whiteboard.

• The bucket. analytics table bucket → an ARN.
• The namespace. clickstream.
• The table. events with an Iceberg schema (partitioned by day).

Question. Provision a table bucket, a namespace, and a partitioned Iceberg events table using the S3 Tables API.

Input.

Piece	Value
Table bucket	analytics
Namespace	clickstream
Table	events (Iceberg)
Partition	day(event_ts)

Code.

# 1. Create the table bucket (the managed Iceberg catalog + storage).
aws s3tables create-table-bucket --name analytics --region us-east-1
#   -> "arn":"arn:aws:s3tables:us-east-1:111122223333:bucket/analytics"

# 2. Create a namespace (a database/schema inside the bucket).
aws s3tables create-namespace \
  --table-bucket-arn arn:aws:s3tables:us-east-1:111122223333:bucket/analytics \
  --namespace clickstream

# 3. Create an Iceberg table with a schema (metadata describes the Iceberg fields).
aws s3tables create-table \
  --table-bucket-arn arn:aws:s3tables:us-east-1:111122223333:bucket/analytics \
  --namespace clickstream \
  --name events \
  --format ICEBERG \
  --metadata '{"iceberg":{"schema":{"fields":[
      {"name":"event_ts","type":"timestamp","required":true},
      {"name":"user_id","type":"string"},
      {"name":"event_type","type":"string"},
      {"name":"payload","type":"string"}]}}}'

# Same three steps in boto3 for a provisioning script / IaC glue.
import boto3
s3t = boto3.client("s3tables", region_name="us-east-1")

resp = s3t.create_table_bucket(name="analytics")
bucket_arn = resp["arn"]                                   # capture the ARN

s3t.create_namespace(tableBucketARN=bucket_arn, namespace=["clickstream"])

s3t.create_table(
    tableBucketARN=bucket_arn,
    namespace="clickstream",
    name="events",
    format="ICEBERG",
    metadata={"iceberg": {"schema": {"fields": [
        {"name": "event_ts",  "type": "timestamp", "required": True},
        {"name": "user_id",   "type": "string"},
        {"name": "event_type","type": "string"},
        {"name": "payload",   "type": "string"},
    ]}}},
)
# The table now exists in the managed catalog; any Iceberg engine can write to it.

Step-by-step explanation.

1. create-table-bucket provisions the managed catalog and returns its ARN — the identity you pass to every later call. This bucket is not browsable with s3 ls; it is addressed through the s3tables API, which is the first thing that surprises people coming from general-purpose buckets.
2. create-namespace adds the middle level of the bucket.namespace.table name. Namespaces are cheap organisational units; you typically make one per data domain (clickstream, sales, finance) so access control and discovery map to teams.
3. create-table with --format ICEBERG and an Iceberg schema materialises a real Iceberg table: the metadata describes the fields, and the table bucket begins managing its metadata pointer and snapshots. The schema here declares event_ts as a required timestamp — the column you will partition on.
4. The boto3 version is byte-for-byte the same three operations, which matters because you provision table buckets from IaC/CI, not by hand — capturing resp["arn"] and threading it through is the automation pattern.
5. At this point the table exists in the managed catalog with zero data files; the next step is pointing a writer (Spark/Flink/Firehose) at the Iceberg REST endpoint or the Glue-federated catalog to append data — every commit of which the table bucket records atomically.

Output.

Step	Creates	Addressed by
create-table-bucket	managed catalog + storage	bucket ARN
create-namespace	a database/schema	bucket.namespace
create-table	an Iceberg table	bucket.namespace.table
(next) writer append	data files + snapshot	Iceberg REST / Glue

Rule of thumb. Provision the three levels in order — table bucket (the catalog), namespace (the domain), table (the Iceberg schema) — from the s3tables API or IaC, and remember a table bucket is not a general-purpose bucket: you never PutObject into it, you write through an Iceberg engine against the managed catalog.

Worked example — query an S3 Table from Spark and from Athena

Detailed explanation. Two engines, two connection paths, one table. Spark/EMR connects to the table bucket's Iceberg REST endpoint directly; Athena reads the same table through the Glue Data Catalog integration. Wire both against the clickstream.events table.

• Spark path. Configure an Iceberg REST catalog pointed at the S3 Tables endpoint, SigV4-authenticated.
• Athena path. The table bucket federated into Glue appears as a catalog you select with a three-part name.
• The point. Same Iceberg table, portable across engines.

Question. Read and write clickstream.events from Spark via the Iceberg REST catalog, and query it from Athena via the Glue integration.

Input.

Engine	Connection	Auth
Spark / EMR	Iceberg REST catalog endpoint	SigV4 (IAM)
Athena	Glue Data Catalog federation	Lake Formation
Table	analytics.clickstream.events	Iceberg

Code.

# Spark: configure the S3 Tables bucket as an Iceberg REST catalog, then use SQL.
spark = (SparkSession.builder
  .config("spark.sql.catalog.s3t", "org.apache.iceberg.spark.SparkCatalog")
  .config("spark.sql.catalog.s3t.type", "rest")
  .config("spark.sql.catalog.s3t.uri",
          "https://s3tables.us-east-1.amazonaws.com/iceberg")
  .config("spark.sql.catalog.s3t.warehouse",
          "arn:aws:s3tables:us-east-1:111122223333:bucket/analytics")
  .config("spark.sql.catalog.s3t.rest.sigv4-enabled", "true")
  .config("spark.sql.catalog.s3t.rest.signing-name", "s3tables")
  .getOrCreate())

# Write (atomic commit through the managed catalog):
spark.sql("""
  INSERT INTO s3t.clickstream.events
  SELECT event_ts, user_id, event_type, payload FROM staging_events
""")

# Read with a standard Iceberg query — and TIME TRAVEL, because it's real Iceberg:
spark.sql("SELECT count(*) FROM s3t.clickstream.events").show()
spark.sql("""
  SELECT * FROM s3t.clickstream.events
  FOR TIMESTAMP AS OF TIMESTAMP '2026-08-25 00:00:00'
""").show()

-- Athena: the SAME table via the Glue Data Catalog integration (federated catalog).
-- The three-part name is  <federated-catalog>/<namespace>.<table>.
SELECT event_type, count(*) AS n
FROM   "s3tablescatalog/analytics".clickstream.events
WHERE  event_ts >= TIMESTAMP '2026-08-26 00:00:00'
GROUP  BY event_type
ORDER  BY n DESC;

Step-by-step explanation.

1. The Spark config registers a catalog named s3t of type = rest pointed at the S3 Tables Iceberg REST URI, with the table bucket ARN as the warehouse and SigV4 signing turned on — that is the entire wiring; from there s3t.clickstream.events is an ordinary Iceberg table to Spark.
2. The INSERT INTO commits through the managed catalog, so even if a Flink job and this Spark job write concurrently, the catalog serialises the metadata-pointer swap and neither corrupts the table — the atomic-commit guarantee you would otherwise run a catalog to get.
3. FOR TIMESTAMP AS OF proves it is genuine Iceberg: time travel reads a prior snapshot without any special S3 Tables feature, because snapshots are a table-format property the managed catalog preserves.
4. The Athena query hits the same table with zero Spark involvement, via the Glue Data Catalog integration that federates the table bucket — the three-part catalog/namespace.table name is how SQL engines address it, and Lake Formation governs who may read it.
5. The lesson is portability through one open format: Spark uses the REST endpoint, Athena uses the Glue integration, Redshift and Trino can join in — all reading one Iceberg table, so you are never locked to a single engine even though AWS operates the catalog.

Output.

Engine	Path	Capability shown
Spark write	Iceberg REST catalog	atomic INSERT/MERGE
Spark read	Iceberg REST catalog	scan + time travel
Athena	Glue federation	ANSI SQL over the same table
Redshift/Trino	Glue / Iceberg	same table, more engines

Rule of thumb. Point compute engines (Spark/EMR/Flink) at the table bucket's Iceberg REST endpoint with SigV4, and point SQL services (Athena/Redshift) at the Glue Data Catalog integration — both read one open Iceberg table, so time travel, MERGE, and multi-engine portability all come for free.

Worked example — upserts with MERGE INTO on an S3 Table

Detailed explanation. CDC pipelines need upserts, not just appends. Because an S3 Table is real Iceberg, MERGE INTO works exactly as it does on any Iceberg table — the managed catalog handles the atomic commit. Apply a CDC batch of inserts/updates/deletes to a dimension table.

• The target. sales.customers (an S3 Table, Iceberg).
• The source. A CDC batch with an op column (I/U/D).
• The operation. One MERGE INTO that inserts, updates, and deletes atomically.

Question. Apply a CDC change set to an S3 Table with a single MERGE INTO that upserts and deletes, committed atomically.

Input.

CDC op	Meaning	MERGE clause
I	new row	WHEN NOT MATCHED THEN INSERT
U	changed row	WHEN MATCHED ... THEN UPDATE
D	deleted row	WHEN MATCHED ... THEN DELETE

Code.

-- One atomic MERGE upserts and deletes a CDC batch into an S3 Table (real Iceberg).
MERGE INTO s3t.sales.customers      AS t
USING staging.customer_changes      AS s          -- CDC batch with an `op` column
ON t.customer_id = s.customer_id
WHEN MATCHED AND s.op = 'D' THEN DELETE
WHEN MATCHED AND s.op = 'U' THEN UPDATE SET
    name = s.name, tier = s.tier, updated_at = s.updated_at
WHEN NOT MATCHED AND s.op IN ('I','U') THEN INSERT
    (customer_id, name, tier, updated_at)
    VALUES (s.customer_id, s.name, s.tier, s.updated_at);

# Why this is safe on S3 Tables:
#   - the MERGE is ONE Iceberg transaction -> ONE atomic metadata commit.
#   - the managed catalog serialises it against any concurrent writer.
#   - a new snapshot is created; the previous one is still time-travellable
#     (until snapshot expiration reclaims it — see section 4).

Step-by-step explanation.

1. MERGE INTO matches the CDC batch against the target on customer_id and, in one statement, deletes rows flagged D, updates rows flagged U, and inserts rows that do not yet exist — the standard upsert-plus-delete a dimension table needs from CDC.
2. The entire MERGE is a single Iceberg transaction, so it produces exactly one new snapshot and one atomic metadata commit — there is no window where the table is half-updated, even though many rows changed.
3. The managed catalog serialises this commit against any other writer, so a concurrent append from a streaming job cannot interleave and corrupt the metadata — the concurrency guarantee is the catalog's job, and S3 Tables runs the catalog.
4. Every MERGE creates a new snapshot while keeping the prior one, so you can time-travel to the pre-MERGE state for audit or rollback — until snapshot expiration (managed maintenance) reclaims old snapshots per your retention policy.
5. Nothing here is S3-Tables-specific SQL: it is ordinary Iceberg MERGE, which is exactly the point — the managed service does not fork the API, so your CDC logic is portable to any Iceberg catalog.

Output.

CDC row	Matched?	Effect on the table
I customer 42	no	inserted
U customer 7	yes	updated in place
D customer 9	yes	deleted
commit	—	one atomic snapshot

Rule of thumb. Use ordinary Iceberg MERGE INTO for CDC upserts on S3 Tables — one statement inserts, updates, and deletes atomically as a single snapshot, and the managed catalog serialises it against concurrent writers. The SQL is standard Iceberg, so your pipeline stays portable.

Senior interview question on S3 Tables structure and multi-engine access

A senior interviewer might ask: "Stand up an S3 Tables lakehouse for a CDC + streaming workload that Spark writes and Athena and Redshift read. Cover the table-bucket/namespace/table hierarchy, how a writer connects versus how a SQL engine connects, how you apply CDC upserts atomically with concurrent writers, and how the whole thing stays portable open-format Iceberg rather than a proprietary store."

Solution Using a table bucket, the Iceberg REST endpoint for writers, and Glue federation for readers

# 1. Provision the hierarchy: table bucket -> namespace -> table.
aws s3tables create-table-bucket --name analytics --region us-east-1
aws s3tables create-namespace  --table-bucket-arn $ARN --namespace sales
aws s3tables create-table      --table-bucket-arn $ARN --namespace sales \
    --name customers --format ICEBERG --metadata "$SCHEMA_JSON"

# 2. Writers (Spark/EMR/Flink) connect via the Iceberg REST catalog + SigV4.
spark = (SparkSession.builder
  .config("spark.sql.catalog.s3t", "org.apache.iceberg.spark.SparkCatalog")
  .config("spark.sql.catalog.s3t.type", "rest")
  .config("spark.sql.catalog.s3t.uri", "https://s3tables.us-east-1.amazonaws.com/iceberg")
  .config("spark.sql.catalog.s3t.warehouse", ARN)
  .config("spark.sql.catalog.s3t.rest.sigv4-enabled", "true")
  .getOrCreate())

-- 3. CDC upsert: one atomic MERGE, serialised by the managed catalog.
MERGE INTO s3t.sales.customers t USING staging.customer_changes s
ON t.customer_id = s.customer_id
WHEN MATCHED AND s.op='D' THEN DELETE
WHEN MATCHED AND s.op='U' THEN UPDATE SET name=s.name, tier=s.tier
WHEN NOT MATCHED AND s.op IN ('I','U') THEN INSERT (customer_id,name,tier)
     VALUES (s.customer_id,s.name,s.tier);

-- 4. Readers (Athena/Redshift) hit the SAME table via the Glue Data Catalog integration.
SELECT tier, count(*) FROM "s3tablescatalog/analytics".sales.customers
GROUP BY tier;                     -- Lake Formation governs access; open Iceberg underneath

Step-by-step trace.

Layer	Component	Purpose
Hierarchy	bucket → namespace → table	catalog + domain + Iceberg table
Writer path	Iceberg REST endpoint (SigV4)	Spark/Flink atomic commits
Reader path	Glue Data Catalog federation	Athena/Redshift ANSI SQL
Concurrency	managed catalog	serialised atomic commits
Upserts	Iceberg MERGE INTO	insert/update/delete in one snapshot
Portability	open Iceberg format	any engine, no lock-in

After deployment, the analytics table bucket holds the sales.customers Iceberg table; Spark writes to it through the Iceberg REST endpoint while Athena and Redshift read it through the Glue integration; the CDC MERGE INTO applies each change set as one atomic snapshot that the managed catalog serialises against the streaming appends; and because it is genuine Iceberg, any engine can join in and you can migrate out. Writers and readers never fight, and no one runs a catalog server.

Output:

Metric	Ad-hoc Parquet + per-engine setup	S3 Tables (managed catalog)
Writer connection	custom per engine	one Iceberg REST endpoint
Reader connection	per-engine Glue crawlers	one Glue-federated catalog
CDC upserts	rewrite partitions by hand	atomic MERGE INTO
Concurrent-writer safety	corruption risk	serialised by the catalog
Lock-in	varies	none (open Iceberg)

Why this works — concept by concept:

• Table-bucket hierarchy — bucket → namespace → table gives engines the three-part catalog name they expect, so a table bucket behaves as a real multi-database catalog rather than a flat prefix of files.
• Iceberg REST endpoint for writers — Spark/EMR/Flink configure the table bucket as a standard REST catalog with SigV4, so writes are ordinary Iceberg commits with no proprietary connector, and every commit is atomic.
• Glue federation for readers — federating the bucket into Glue lets Athena and Redshift query the same tables by name under Lake Formation governance, so the reader surface is centralised and access-controlled.
• Atomic MERGE via the managed catalog — CDC upserts run as a single Iceberg transaction that the managed catalog serialises against concurrent writers, so the table is never half-updated and streaming plus batch writers coexist safely.
• Cost — one managed catalog and its maintenance versus a self-run catalog, per-engine crawlers, and hand-rolled upsert jobs. The eliminated cost is the catalog server and its HA/on-call — O(open format) portability with O(managed) operations instead of O(engineers) to wire every engine.

ETL
Topic — etl
ETL problems on CDC upserts and multi-engine tables

Practice →

Data processing Topic — data-processing Data processing problems on Spark and Iceberg reads/writes

Practice →

---

3. S3 Metadata — object metadata as queryable Iceberg tables

Turn "what is in my bucket / what changed" into a SQL question over managed Iceberg tables

The mental model in one line: S3 Metadata is a feature you enable on a general-purpose bucket that makes S3 automatically write that bucket's object metadata into managed Apache Iceberg tables — a journal table that captures every object change event (create, update-metadata, delete) in near-real-time and a live inventory table that maintains a current snapshot of all objects — both stored as S3 Tables you query with Athena, Spark, or any Iceberg engine, so questions like "which objects are over 1 GB," "what got deleted last night," "which files are untagged," or "how is storage split by class" become ordinary SQL instead of a paginated ListObjectsV2 crawl. It is observability for your object storage, delivered through the same managed-Iceberg machinery as S3 Tables.

Two metadata tables, two shapes.

• The journal table — an event stream. Append-only rows, one per object change, with a record_type of CREATE, UPDATE_METADATA, or DELETE, plus a record_timestamp, the key, version, size, storage class, ETag, tags, user metadata, and requester context. It answers what changed and when and is populated within minutes of the event.
• The live inventory table — a current snapshot. A fully-managed, continuously-updated table of the objects that exist right now — key, size, last-modified, storage class, tags, encryption status. It answers what is in the bucket right now without a full LIST.
• Both are Iceberg S3 Tables. They live in a managed table bucket, so you query them with Athena/Spark exactly like your data tables, and they benefit from the same automatic maintenance.

How it is wired.

• Enable a metadata configuration. You turn on a metadata configuration on the general-purpose bucket (journal and/or inventory); S3 provisions the backing table(s) and begins populating them. No Lambda, no manual inventory job.
• Query with any Iceberg engine. Point Athena or Spark at the metadata table via the Glue integration or the Iceberg REST endpoint and run SQL — filter, aggregate, and join it to your own tables.
• Near-real-time journal, snapshot inventory. The journal reflects changes within minutes; the inventory is kept current — you pick the shape by whether you need the change history or the present state.

What it replaces.

• ListObjectsV2 crawls. Enumerating a bucket with millions of objects via paginated LIST is slow, rate-limited, and awkward; a SQL scan of the inventory table is far faster and expressible.
• S3 Inventory CSV/Parquet reports. The older S3 Inventory feature drops periodic report files you then have to catalog; S3 Metadata gives you a live, always-queryable Iceberg table instead of a batch report.
• Bespoke event pipelines. Wiring S3 Event Notifications → SQS/Lambda → a database to track object changes is now a built-in journal table.

The failure modes senior engineers pre-empt.

• Reading the journal as a snapshot. The journal is an event log: an object can have CREATE then DELETE rows. Summing sizes across all journal rows double-counts. Mitigation: use the inventory table for current state, or reduce the journal to the latest event per key.
• Expecting instant population. Metadata appears within minutes, not synchronously with the PUT. Mitigation: do not build strict read-after-write logic on the metadata table; it is for analytics/observability, not the write path.
• Scanning cost on huge buckets. The metadata table for a billion-object bucket is large; unfiltered scans cost. Mitigation: filter on partitioned/clustered columns, and let compaction (managed) keep the metadata table scan-efficient.

Common interview probes on S3 Metadata.

• "What does S3 Metadata give you?" — object metadata as queryable Iceberg tables (a change journal and a live inventory).
• "Journal vs inventory?" — journal is an append-only event stream of changes; inventory is a current snapshot of all objects.
• "How do you query it?" — Athena/Spark over the managed S3 Table, joinable to your own data.
• "What did it replace?" — LIST crawls, periodic S3 Inventory reports, and DIY event pipelines.

Worked example — enable a metadata configuration and query the journal in Athena

Detailed explanation. The setup is one API call, then it is just SQL. Enable a journal metadata configuration on a data-lake bucket and answer an auditing question — "what objects were deleted in the last 24 hours, and by whom?" — with Athena.

• The bucket. raw-events (general-purpose, millions of objects).
• The config. A journal metadata table.
• The query. Deletes in the last day from the journal.

Question. Enable S3 Metadata journaling on a bucket and query the journal table in Athena for recent deletes.

Input.

Piece	Value
Source bucket	raw-events
Metadata type	journal (event stream)
Key columns	record_type, record_timestamp, key, requester
Question	deletes in the last 24h

Code.

# 1. Enable a journal metadata configuration on the general-purpose bucket.
#    S3 provisions the backing Iceberg table and starts populating it.
aws s3api create-bucket-metadata-configuration \
  --bucket raw-events \
  --metadata-configuration '{"journalTableConfiguration":{
      "recordExpiration":{"expiration":"ENABLED","days":90}}}'

-- 2. Query the journal like any table: recent DELETE events and who caused them.
--    The journal is an EVENT LOG: one row per change, record_type in
--    (CREATE, UPDATE_METADATA, DELETE).
SELECT record_timestamp,
       key,
       version_id,
       requester,            -- account/principal that made the change
       source_ip_address
FROM   "s3tablescatalog/raw-events-metadata".s3metadata.journal
WHERE  record_type = 'DELETE'
  AND  record_timestamp >= current_timestamp - interval '24' hour
ORDER  BY record_timestamp DESC;

Step-by-step explanation.

1. create-bucket-metadata-configuration with a journalTableConfiguration turns on journaling; S3 creates the backing Iceberg table and starts appending a row per object change within minutes — no Lambda, no event plumbing. recordExpiration bounds how long journal rows are retained so the table does not grow without limit.
2. The query treats the journal as an ordinary table via the Glue-federated catalog. The crucial columns are record_type (the kind of change) and record_timestamp (when) — filtering record_type = 'DELETE' isolates deletions.
3. requester and source_ip_address turn the journal into an audit trail: you can see which principal deleted which key from which IP, which is exactly the forensic question an incident or compliance review asks.
4. Because it is an append-only event log, this query is correct by construction for "what changed" — you are counting events, not current state, so DELETE rows are precisely what you want here.
5. The contrast to keep straight: if the question were "how much storage do we have now," the journal is the wrong table (it would double-count created-then-deleted objects); you would use the live inventory table instead — the subject of the next example.

Output.

record_type	record_timestamp	key	requester
DELETE	2026-08-26 04:12	events/dt=…/p-8.parquet	role/etl-cleanup
DELETE	2026-08-26 03:59	tmp/scratch-42.json	user/alice
DELETE	2026-08-26 01:20	events/dt=…/p-3.parquet	role/etl-cleanup
(CREATE/UPDATE rows)	filtered out	—	—

Rule of thumb. Enable a journal metadata configuration for change history and audit questions, and query it as an append-only event log filtered by record_type and record_timestamp. Set recordExpiration so the journal is bounded, and never use it for current-state totals — that is the inventory table's job.

Worked example — join the live inventory to find cost and governance risks

Detailed explanation. The live inventory table is a current snapshot, so it answers present-state questions and, crucially, joins to your own data. Use it to find governance and cost risks — large untagged objects and cold data in an expensive storage class.

• The table. The live inventory snapshot of raw-events.
• The questions. Objects > 1 GB with no owner tag; total bytes by storage class.
• The payoff. Governance and cost analysis in SQL, no LIST crawl.

Question. Query the live inventory table for large untagged objects and for storage-class cost distribution.

Input.

Column	Use
key, size	find large objects
object_tags	detect missing ownership
storage_class	cost distribution
last_modified_date	detect cold data

Code.

-- 1. Governance: large objects with NO owner tag (present-state, so use INVENTORY).
SELECT key,
       size,
       storage_class,
       last_modified_date
FROM   "s3tablescatalog/raw-events-metadata".s3metadata.inventory
WHERE  size > 1073741824                       -- > 1 GB
  AND  ( object_tags IS NULL
         OR NOT map_keys(object_tags) @> ARRAY['owner'] )  -- missing owner tag
ORDER  BY size DESC
LIMIT  100;

-- 2. Cost: current bytes and object counts per storage class (a snapshot aggregate).
SELECT storage_class,
       count(*)                       AS objects,
       round(sum(size)/1e12, 2)       AS terabytes
FROM   "s3tablescatalog/raw-events-metadata".s3metadata.inventory
GROUP  BY storage_class
ORDER  BY terabytes DESC;

Step-by-step explanation.

1. The inventory table is a current snapshot, so aggregating over it gives true present-state answers — unlike the journal, summing size here is correct because each live object appears once.
2. The first query filters size > 1 GB and checks object_tags for a missing owner key, surfacing exactly the large, unowned objects that are both a governance gap and a cost risk — the kind of list you would otherwise assemble by crawling millions of keys with LIST.
3. The second query groups by storage_class to show where your bytes and money actually sit (Standard vs Infrequent Access vs Glacier tiers), which is the input to a lifecycle-policy or tiering decision — computed in one SQL scan instead of a batch inventory report.
4. Both queries are ordinary Iceberg scans, so they join to your own tables: you could join inventory.key to a manifest of expected files to find objects nobody references, or to a catalog of datasets to attribute storage cost to teams.
5. The senior framing: S3 Metadata collapses a whole category of custom tooling — LIST crawlers, inventory-report ETL, tag-audit scripts — into SQL over a managed, always-current Iceberg table, and the managed compaction keeps that table fast to scan even for very large buckets.

Output.

Query	Answers	Right table
large untagged objects	governance risk list	inventory (snapshot)
bytes by storage class	cost distribution	inventory (snapshot)
deletes last night	audit / forensics	journal (events)
objects created per hour	ingest rate	journal (events)

Rule of thumb. Use the live inventory table for present-state questions — current size, tags, storage class, cold data — and aggregate freely because each object appears once. Reserve the journal for change history. Join either to your own tables to turn object metadata into governance and cost analytics without a single LIST crawl.

Worked example — reduce the journal to current state (event log → snapshot)

Detailed explanation. Sometimes you only have the journal (or want a point-in-time reconstruction) and need current state from it. The pattern is the classic event-log-to-snapshot reduction: keep the latest event per key and drop keys whose latest event is a DELETE. Reconstruct "objects that exist now" from the journal.

• The trap. Summing all journal rows double-counts created-then-deleted objects.
• The fix. Window by key, keep the latest event, exclude DELETEs.
• The payoff. A correct snapshot from an event stream.

Question. From the append-only journal, compute the set of objects that currently exist and their total size, correctly excluding deleted keys.

Input.

Step	Operation
rank	latest event per key by record_timestamp
filter	keep rank = 1
exclude	drop rows where latest record_type = DELETE
aggregate	sum size of survivors

Code.

-- Reduce the event log to current state: latest event per key, drop DELETEs.
WITH ranked AS (
  SELECT key, size, record_type, record_timestamp,
         row_number() OVER (PARTITION BY key
                            ORDER BY record_timestamp DESC) AS rn
  FROM   "s3tablescatalog/raw-events-metadata".s3metadata.journal
),
current_objects AS (
  SELECT key, size
  FROM   ranked
  WHERE  rn = 1                       -- latest event per key
    AND  record_type <> 'DELETE'      -- object still exists
)
SELECT count(*)                 AS live_objects,
       round(sum(size)/1e12, 2) AS terabytes
FROM   current_objects;

Step-by-step explanation.

1. The ranked CTE assigns row_number() per key ordered by record_timestamp descending, so rn = 1 marks each key's most recent event — the standard latest-record-per-group window.
2. current_objects keeps only rn = 1 and drops keys whose latest event is a DELETE, so a key that was created then deleted contributes nothing, and a key created then had its metadata updated contributes its latest size — exactly the current-state semantics.
3. The final aggregate sums size over survivors, giving a correct live-object count and total bytes derived from the event log — the reconstruction you need when only the journal is available or when you want state as-of a chosen timestamp (add a WHERE record_timestamp <= :as_of in the CTE).
4. This is the same event-sourcing reduction that appears across data engineering — CDC change logs, Kafka compacted topics, SCD tables — so recognising the journal as "just another change stream" is the senior insight that makes S3 Metadata immediately familiar.
5. The practical guidance: if you need current state often, query the live inventory table directly (S3 maintains it for you); reserve this reduction for as-of reconstructions or when you deliberately enabled only the journal.

Output.

Approach	Created-then-deleted key	Correct total?
sum all journal rows	counted twice	no (double count)
latest-per-key, drop DELETE	excluded	yes
live inventory table	excluded	yes (managed)
as-of reconstruction	via timestamp filter	yes (point-in-time)

Rule of thumb. Treat the journal as a change stream: to get current state, window to the latest event per key and drop DELETEs — never sum raw rows. For frequent current-state queries prefer the managed live inventory table; use the reduction for point-in-time (as-of) reconstructions the inventory cannot give.

Senior interview question on S3 Metadata for observability and governance

A senior interviewer might ask: "You run a data lake on general-purpose S3 with hundreds of millions of objects and no good way to answer 'what's in here, what changed, and what's it costing us.' Design an object-observability layer with S3 Metadata: which tables you enable, how you answer a change-audit question versus a current-state cost question, how you correctly derive current state from the change log, and how you keep the metadata tables cheap to query at that scale."

Solution Using a journal table for audit, a live inventory table for state, and joins for governance

# 1. Enable BOTH metadata tables: journal (events) + live inventory (snapshot).
aws s3api create-bucket-metadata-configuration --bucket lake-raw \
  --metadata-configuration '{
     "journalTableConfiguration":  {"recordExpiration":{"expiration":"ENABLED","days":90}},
     "inventoryTableConfiguration":{"configurationState":"ENABLED"}}'

-- 2. Audit (change history) -> JOURNAL: who deleted what in the last day.
SELECT record_timestamp, key, requester, source_ip_address
FROM   "s3tablescatalog/lake-raw-metadata".s3metadata.journal
WHERE  record_type = 'DELETE'
  AND  record_timestamp >= current_timestamp - interval '24' hour;

-- 3. Current-state cost/governance -> LIVE INVENTORY (snapshot), aggregated + joined.
SELECT i.storage_class,
       round(sum(i.size)/1e12, 2) AS tb,
       count(*)                   AS objects
FROM   "s3tablescatalog/lake-raw-metadata".s3metadata.inventory i
LEFT   JOIN governance.dataset_owners o ON o.prefix = regexp_extract(i.key, '^[^/]+')
WHERE  o.owner IS NULL                 -- unowned prefixes = governance gap
GROUP  BY i.storage_class;

-- 4. Derive current state from the journal when needed (as-of reconstruction).
WITH ranked AS (
  SELECT key, size, record_type,
         row_number() OVER (PARTITION BY key ORDER BY record_timestamp DESC) rn
  FROM "s3tablescatalog/lake-raw-metadata".s3metadata.journal)
SELECT count(*) FROM ranked WHERE rn = 1 AND record_type <> 'DELETE';

Step-by-step trace.

Question type	Table used	Why
"who deleted what?"	journal	append-only event history
"how much storage now?"	live inventory	one row per live object
"unowned data?"	inventory ⋈ owners	join snapshot to governance
"state as-of a time?"	journal (windowed)	reduce events to snapshot
scale / cost	managed compaction	keep metadata tables scannable

After deployment, the lake-raw bucket has both a journal and a live inventory table; audit questions read the journal's DELETE/CREATE events with requester context, current-state cost questions aggregate the inventory (one row per live object) and join it to a governance table to surface unowned prefixes, and when a point-in-time reconstruction is needed the journal is reduced to the latest event per key. Managed compaction keeps both tables fast to scan even at hundreds of millions of objects — and none of it required a LIST crawl or a custom event pipeline.

Output:

Metric	DIY (LIST + inventory reports + Lambda)	S3 Metadata
Answer "what changed"	build event pipeline	query the journal
Answer "what exists now"	paginated LIST / batch report	query the inventory
Governance joins	export + ETL	SQL join to your tables
Freshness	hours (batch)	minutes (journal) / live (inventory)
Scan performance	your problem	managed compaction

Why this works — concept by concept:

• Journal table — an append-only event log of object changes (CREATE/UPDATE_METADATA/DELETE) with requester and timestamp turns audit and forensics into filtered SQL, replacing a hand-built S3-events-to-database pipeline.
• Live inventory table — a continuously-maintained current snapshot means present-state cost and governance questions are correct one-row-per-object aggregates, replacing paginated LIST crawls and periodic inventory reports.
• Event-log reduction — windowing the journal to the latest event per key and dropping DELETEs reconstructs current or as-of state from the change stream, the same event-sourcing pattern used across CDC and streaming.
• Metadata as Iceberg S3 Tables — because both tables are managed Iceberg, they join to your own data, are queryable by any Iceberg engine, and are kept scan-efficient by the same automatic compaction as your data tables.
• Cost — enable a configuration versus building and operating LIST crawlers, inventory ETL, and event pipelines. The eliminated cost is a whole observability codebase — O(one config) instead of O(engineers) to know what is in your buckets, with managed compaction keeping query cost bounded at scale.

Indexing
Topic — indexing
Indexing problems on metadata scans and lookups

Practice →

ETL Topic — etl ETL problems on event-log reduction and snapshots

Practice →

---

4. Automatic maintenance — compaction, snapshot expiration, cleanup

Managed maintenance merges small files, expires old snapshots, and deletes orphans continuously

The mental model in one line: the reason S3 Tables are worth a managed fee is that an Iceberg table on object storage decays without three continuous maintenance jobs — compaction rewrites the flood of tiny files that streaming and CDC writes create into target-sized (~512 MB) files so scans stop paying per-file overhead, snapshot expiration drops old snapshots so metadata and storage stay bounded, and unreferenced-file removal deletes data files no live snapshot points to so orphaned bytes are reclaimed — and S3 Tables runs all three automatically as managed operations you configure with a target file size and retention windows, instead of scheduling, monitoring, and paying compute for rewrite_data_files / expire_snapshots / remove_orphan_files Spark jobs yourself. Skip maintenance on a self-managed table and query latency and storage cost climb every day; on S3 Tables it is handled.

The small-files problem — why compaction exists.

• How small files happen. Streaming ingestion, frequent micro-batches, and CDC each commit tiny data files — a Firehose delivery or a 1-minute Spark micro-batch might write hundreds of small Parquet files an hour. Over a day that is tens of thousands of files.
• Why they hurt. Every file adds open/seek overhead, more S3 GET/LIST requests, larger manifest metadata, and worse compression and column-stat pruning — so a scan of many small files is dramatically slower and more expensive than the same bytes in a few large files.
• What compaction does. It rewrites many small files into fewer files at a target size (default ~512 MB), preserving the data and updating the metadata, so subsequent scans read fewer, larger, better-compressed files.

Snapshot expiration and orphan cleanup.

• Snapshots accumulate. Every commit (append, MERGE, compaction) creates a snapshot. Kept forever, snapshots pin old data files (blocking their deletion) and bloat metadata. Expiration drops snapshots older than a retention window (while keeping a minimum count) so time travel stays bounded and old files become collectable.
• Orphaned files. Failed writes, aborted commits, and expired snapshots leave data files that no live snapshot references — bytes you pay for that no query can reach. Unreferenced-file removal deletes them after a safety window.
• The interaction. Compaction creates new files and dereferences old ones; expiration unpins the old snapshots; orphan removal deletes the now-unreferenced files. The three together keep both performance and storage healthy — which is why they must run as a coordinated, continuous set.

How S3 Tables runs it.

• Continuous and managed. S3 Tables performs compaction, snapshot management, and unreferenced-file removal automatically in the background — there is no Spark job for you to schedule or scale.
• Configured as policy, not code. You set the compaction target file size and the snapshot/orphan retention windows at the table or table-bucket level; AWS applies them. Defaults are sensible (e.g. ~512 MB target).
• A billed operation. Managed maintenance is a cost line (compaction is charged by objects/data processed) — the trade for not running and paying for your own maintenance compute and on-call.

The failure modes senior engineers pre-empt.

• Retention shorter than your time-travel need. If snapshot expiration is more aggressive than your rollback/audit window, you lose the ability to time-travel that far back. Mitigation: set max_age/min_snapshots to cover your actual recovery and audit requirements.
• Over-frequent tiny commits. Committing every few seconds creates enormous snapshot and small-file churn that even compaction must chase. Mitigation: batch writes to a reasonable cadence; let compaction handle the rest.
• Assuming maintenance is optional or off. On a self-managed table, forgetting maintenance is the classic "why did the lake get slow" outage. Mitigation: on S3 Tables verify maintenance is enabled and tuned; on DIY Iceberg, schedule it as first-class.

Common interview probes on maintenance.

• "What's the small-files problem?" — many tiny files add per-file overhead and requests; compaction merges them to a target size.
• "What does snapshot expiration do?" — drops old snapshots so metadata/storage stay bounded and old files become deletable.
• "What are orphan files?" — data files no live snapshot references; removed after a safety window.
• "Who runs this on S3 Tables?" — AWS, automatically, configured by policy — no Spark cron.

Worked example — configure compaction and snapshot expiration on a table bucket

Detailed explanation. Maintenance on S3 Tables is configuration, not a pipeline. Set a compaction target file size and snapshot/orphan retention on the table bucket so every table inherits sane policy, and reason about what each knob controls. Tune it for a streaming table with a 7-day time-travel requirement.

• Compaction. Target ~512 MB files.
• Snapshots. Keep ≥ 5 and anything newer than 7 days (168h).
• Orphans. Delete files unreferenced for > 72h.

Question. Configure managed maintenance for a streaming S3 Table that needs 7-day time travel and fast scans.

Input.

Knob	Setting	Controls
compaction target	512 MB	scan file size
min snapshots	5	never expire below this
max snapshot age	168h (7d)	time-travel window
orphan age	72h	orphan-deletion safety

Code.

// Table-bucket maintenance policy (applied by AWS; no Spark job).
{
  "iceberg": {
    "compaction": {
      "status": "enabled",
      "settings": { "targetFileSizeMB": 512 }        // merge small files toward 512 MB
    },
    "snapshotManagement": {
      "status": "enabled",
      "settings": {
        "minSnapshotsToKeep": 5,                      // safety floor
        "maxSnapshotAgeHours": 168                    // 7-day time-travel window
      }
    },
    "unreferencedFileRemoval": {
      "status": "enabled",
      "settings": { "nonCurrentDays": 3 }             // delete orphans after ~72h
    }
  }
}

# What each job does, in order, on a streaming table:
#   1) compaction:       50k tiny files/day -> a few hundred ~512 MB files -> fast scans
#   2) snapshotMgmt:     keep >=5 snapshots AND anything < 7d -> time travel stays bounded
#   3) orphanRemoval:    delete data files no live snapshot references (after 72h) -> reclaim bytes
# All three run continuously in the background; you set policy, AWS runs it.

Step-by-step explanation.

1. compaction.targetFileSizeMB = 512 tells the managed compactor to rewrite the day's flood of tiny streaming files into ~512 MB files. That single setting is what keeps scan latency and request count flat over time instead of climbing as files accumulate.
2. snapshotManagement keeps at least 5 snapshots and everything younger than 168 hours, so your 7-day time-travel/rollback requirement is guaranteed while older snapshots expire — bounding metadata size and unpinning old data files for deletion.
3. unreferencedFileRemoval.nonCurrentDays = 3 deletes data files that no live snapshot references after a 72-hour safety window, reclaiming the bytes left behind by failed writes and expired snapshots — the storage-cost half of maintenance.
4. The ordering matters: compaction dereferences old small files, snapshot expiration removes the old snapshots still pinning them, and only then can orphan removal safely delete the files — which is why S3 Tables coordinates all three rather than running them independently.
5. Setting this at the table-bucket level means every table inherits the policy, so a new table created next month is maintained correctly by default — the operational win over per-table Spark maintenance jobs that someone has to remember to schedule.

Output.

Job	Effect	Cost/perf impact
compaction (512 MB)	fewer, larger files	faster scans, fewer requests
keep ≥5, ≤7d snapshots	bounded time travel	smaller metadata
orphan removal (72h)	reclaim dead bytes	lower storage cost
all continuous	no accumulation	stable over time

Rule of thumb. Configure maintenance as policy at the table-bucket level: a ~512 MB compaction target for scan speed, a snapshot retention window that covers your real time-travel/rollback need, and an orphan-removal window with a safety margin. Set it once and every table inherits healthy maintenance — no cron, no Spark maintenance cluster.

Worked example — the before/after of compaction on scan cost

Detailed explanation. Make the small-files problem concrete. The same 100 GB of data as 50,000 tiny files versus as 200 compacted files produces wildly different scan cost and latency. Quantify why compaction is not optional for a streaming table.

• Before. 100 GB in 50,000 × 2 MB files.
• After. 100 GB in ~200 × 512 MB files.
• The metrics. Files opened, S3 requests, manifest size, scan time.

Question. Compare scanning 100 GB stored as 50,000 small files versus ~200 compacted files, on requests, metadata, and latency.

Input.

Metric	50,000 × 2 MB	200 × 512 MB
Files to open	50,000	200
S3 GET requests	~50,000+	~200
Manifest/metadata size	large	small
Effective scan speed	slow	fast

Code.

Scanning 100 GB — small files vs compacted, why the difference is huge
======================================================================

BEFORE compaction: 50,000 x 2 MB files
  - the engine must LIST + open 50,000 objects
  - ~50,000 S3 GET requests (per-request latency + $ per 1,000 requests)
  - manifests track 50,000 entries -> planning is slow, stats pruning weak
  - poor compression locality; column stats span tiny row groups
  => query planning + I/O dominated by per-FILE overhead, not by bytes

AFTER compaction: ~200 x 512 MB files
  - LIST + open ~200 objects
  - ~200 S3 GET requests
  - manifests track ~200 entries -> fast planning, strong min/max pruning
  - large row groups -> better compression + effective predicate pushdown
  => I/O dominated by BYTES actually needed; per-file overhead negligible

Same 100 GB. ~250x fewer files and requests. Compaction is the lever.

-- You can SEE the file counts via Iceberg metadata (files table) before/after.
SELECT count(*) AS data_files,
       round(avg(file_size_in_bytes)/1048576, 1) AS avg_mb
FROM   s3t.clickstream."events$files";     -- Iceberg metadata table
-- BEFORE: data_files ~ 50000, avg_mb ~ 2
-- AFTER : data_files ~ 200,   avg_mb ~ 512

Step-by-step explanation.

1. With 50,000 tiny files, the query engine spends its time on per-file work — listing and opening 50,000 objects, issuing ~50,000 GET requests, and planning over 50,000 manifest entries — so latency and request cost scale with the number of files, not the bytes you actually need.
2. Small files also defeat Iceberg's pruning: min/max column statistics over 2 MB row groups are coarse and numerous, so the engine cannot skip as much, and compression is weaker because dictionaries and encodings work better over larger row groups.
3. After compaction to ~200 × 512 MB files, the same 100 GB is opened in ~200 GETs with ~200 manifest entries, so planning is fast and I/O is dominated by the bytes the query truly needs — the per-file overhead becomes negligible.
4. The "events$files" Iceberg metadata table lets you observe the change directly: count(*) of data files and average file size before and after compaction, which is how you prove maintenance is working rather than assuming it.
5. The takeaway for a streaming table is stark: without compaction the file count grows every hour and scans get monotonically slower and pricier; with managed compaction the file count stays bounded, so performance and request cost are stable — the whole reason maintenance is continuous.

Output.

Metric	Before (50k files)	After (200 files)
Objects opened per scan	50,000	~200
S3 GET requests	~50,000	~200
Query planning	slow (huge manifests)	fast
Scan latency / cost	high	low

Rule of thumb. Compaction is the single biggest lever on Iceberg scan performance for streaming/CDC tables — it turns per-file overhead into per-byte work, cutting requests and latency by orders of magnitude for the same data. Watch the $files metadata table to confirm file count stays bounded; on S3 Tables the managed compactor keeps it there for you.

Worked example — snapshot expiration vs a time-travel requirement

Detailed explanation. Snapshot expiration is a trade: reclaim storage and bound metadata, but only keep time travel as far back as retention allows. Set expiration to satisfy a real audit/rollback window without hoarding snapshots forever. Reconcile a 30-day audit rule with cost.

• The requirement. Time-travel/rollback for 30 days (audit + recovery).
• The cost of forever. Snapshots pin data files → storage never shrinks.
• The setting. Keep 30 days (720h) and a minimum count.

Question. Configure snapshot expiration to guarantee 30-day time travel while still reclaiming storage from older snapshots.

Input.

Concern	Setting
Audit/rollback window	30 days
Max snapshot age	720h (30d)
Min snapshots kept	10 (safety floor)
Effect	files older than 30d become collectable

Code.

// Retention tuned to a 30-day audit/rollback requirement.
{ "iceberg": { "snapshotManagement": {
    "status": "enabled",
    "settings": { "minSnapshotsToKeep": 10, "maxSnapshotAgeHours": 720 } } } }

-- Within retention, time travel to any point in the last 30 days works:
SELECT * FROM s3t.sales.orders
FOR TIMESTAMP AS OF TIMESTAMP '2026-08-05 12:00:00';   -- ~21 days ago: OK

-- Beyond retention, the snapshot is expired and its files may be reclaimed:
-- FOR TIMESTAMP AS OF TIMESTAMP '2026-06-01 00:00:00'  -- > 30 days: no longer available

Step-by-step explanation.

1. maxSnapshotAgeHours = 720 keeps every snapshot from the last 30 days, so any FOR TIMESTAMP AS OF within that window resolves to a real snapshot — the audit and rollback requirement is met exactly, not approximately.
2. minSnapshotsToKeep = 10 is a safety floor: even if a table is quiet and its recent snapshots are all older than 30 days, the last 10 are retained so you are never left with zero recovery points — the belt-and-braces against an over-aggressive age rule.
3. Snapshots older than the window expire, which unpins the data files only they referenced; those files then become candidates for the unreferenced-file remover, and storage finally shrinks — the reclaim you forgo if you keep snapshots forever.
4. The explicit trade is visible in the second query: a timestamp inside 30 days works; one outside it does not, because the snapshot (and possibly its files) is gone. Retention is your time-travel horizon — there is no free lunch of infinite history at bounded cost.
5. The senior discipline is to set retention from a stated requirement (a compliance window, an RPO for rollback) rather than a guess — too short and you cannot recover/audit; too long and you pay to store history no one queries. On S3 Tables this is one policy value; on DIY Iceberg it is a tuned expire_snapshots job.

Output.

Time-travel target	Within 30d retention?	Result
2 days ago	yes	snapshot available
21 days ago	yes	snapshot available
45 days ago	no	expired (files reclaimable)
always keep ≥10	—	recovery floor guaranteed

Rule of thumb. Set snapshot retention to the maximum of your rollback RPO and your audit window, plus a minSnapshotsToKeep floor — retention is your time-travel horizon and the point past which storage is reclaimed. Do not keep snapshots forever "just in case"; that is storage you pay for indefinitely to store history nobody reads.

Senior interview question on keeping a streaming Iceberg table healthy

A senior interviewer might ask: "A streaming pipeline writes to an Iceberg table on S3 and, over weeks, queries have gone from seconds to minutes and storage keeps climbing even though row count is flat. Diagnose it and design the maintenance: what's causing the slowdown and the storage growth, which three jobs fix it, how you tune them against a 14-day time-travel requirement, and how S3 Tables changes who runs them."

Solution Using compaction, snapshot expiration, orphan removal, and S3 Tables managed maintenance

-- 1. Diagnosis — classic unmaintained streaming Iceberg table.
Symptom: scans slow over time         -> SMALL FILES (per-file overhead + requests)
Symptom: storage climbs, rows flat    -> old SNAPSHOTS pinning files + ORPHANS from
                                         failed/aborted writes never reclaimed
Root cause: no continuous maintenance running.

-- 2. Confirm with Iceberg metadata tables (the evidence, not a guess).
SELECT count(*) files, round(avg(file_size_in_bytes)/1048576,1) avg_mb
FROM s3t.stream."events$files";                    -- avg_mb tiny => small-files problem
SELECT count(*) AS snapshots FROM s3t.stream."events$snapshots";  -- huge => never expired

// 3. The three maintenance jobs as ONE S3 Tables managed policy (14-day time travel).
{ "iceberg": {
    "compaction":              { "status": "enabled", "settings": { "targetFileSizeMB": 512 } },
    "snapshotManagement":      { "status": "enabled", "settings": { "minSnapshotsToKeep": 7,
                                                                    "maxSnapshotAgeHours": 336 } },
    "unreferencedFileRemoval": { "status": "enabled", "settings": { "nonCurrentDays": 3 } } } }

# 4. The self-managed equivalent you AVOID with S3 Tables (a nightly Spark maintenance job):
#    spark.sql("CALL cat.system.rewrite_data_files(table => 'stream.events')")
#    spark.sql("CALL cat.system.expire_snapshots(table => 'stream.events', older_than => ...)")
#    spark.sql("CALL cat.system.remove_orphan_files(table => 'stream.events')")
#    ...scheduled, monitored, scaled, and PAID FOR by you. S3 Tables runs it instead.

Step-by-step trace.

Symptom	Cause	Fix (job)	S3 Tables
scans slow over weeks	small files pile up	compaction → 512 MB	automatic
storage climbs, rows flat	snapshots pin old files	snapshot expiration	automatic
dead bytes never freed	orphans from failed writes	unreferenced-file removal	automatic
time travel needed 14d	retention policy	keep ≥7 and ≤336h	policy value
who runs it	—	Spark cron (DIY)	AWS (managed)

After diagnosis, the metadata tables confirm tens of thousands of tiny files and thousands of never-expired snapshots. The fix is the three coordinated jobs: compaction rewrites small files to ~512 MB so scans speed back up, snapshot expiration (keep ≥7, ≤14 days) unpins old files and bounds metadata while preserving the 14-day time-travel window, and orphan removal reclaims the dead bytes. On S3 Tables all three are a single managed policy AWS executes continuously; the self-managed alternative is a scheduled Spark job you own end to end.

Output:

Metric	Unmaintained (self-run, neglected)	S3 Tables (managed)
Scan latency trend	degrades weekly	stable
Storage vs row count	grows unbounded	bounded (reclaimed)
Time travel	uncontrolled / costly	bounded to policy (14d)
Maintenance jobs	your Spark cron	none (managed)
On-call for maintenance	yes	no

Why this works — concept by concept:

• Compaction — rewriting many small files into ~512 MB files converts per-file scan overhead and request cost into per-byte work, so streaming-table query latency stays flat instead of degrading as files accumulate.
• Snapshot expiration — dropping snapshots older than the retention window (above a minimum count) bounds metadata size and unpins old data files, preserving exactly the time-travel horizon you require and no more.
• Unreferenced-file removal — deleting data files that no live snapshot references, after a safety window, reclaims the bytes left by failed writes and expired snapshots — the storage-cost half of a healthy table.
• Managed and coordinated — S3 Tables runs the three jobs continuously and in the right order (compact → expire → remove) as one policy, so you configure retention and a target file size instead of scheduling, scaling, and monitoring a Spark maintenance cluster.
• Cost — a managed-maintenance line item versus the compute, scheduling, and on-call of DIY rewrite_data_files / expire_snapshots / remove_orphan_files jobs. The eliminated cost is an entire maintenance pipeline — O(policy) instead of O(engineers) to keep a streaming Iceberg table fast and lean.

Optimization
Topic — optimization
Optimization problems on file compaction and scan cost

Practice →

Indexing Topic — indexing Indexing problems on file pruning and partition layout

Practice →

---

5. Architecture, cost, and when vs self-managed Iceberg

Writers land data, S3 Tables manages the catalog and maintenance, and any Iceberg engine reads it

The mental model in one line: a production S3 Tables lakehouse is a three-zone picture — writers (Spark/EMR/Flink/Firehose) commit through the managed Iceberg catalog, the table bucket stores the data and runs compaction/expiry/cleanup automatically, and readers (Athena/Redshift/Spark) query the same open tables via Glue Data Catalog federation and Lake Formation — with S3 Metadata as an observability side-car, and the central decision being managed (S3 Tables) versus self-managed Iceberg, resolved by whether you want to own a catalog and a maintenance pipeline or pay AWS to — where the cost model is storage + requests + a managed-maintenance line item traded against the compute and engineer-time you no longer spend. The catalog is the source of truth, maintenance is a first-class cost, and the open format keeps you portable either way.

The three zones of the architecture.

• Writers. Spark/EMR and Flink for batch and streaming transforms, and Firehose (Iceberg destination) for direct stream landing. All commit through the table bucket's managed Iceberg catalog, so every write is an atomic Iceberg commit regardless of engine.
• The table bucket (storage + catalog + maintenance). Holds the Iceberg tables, owns the atomic metadata pointer, and runs compaction/snapshot-expiration/orphan-removal continuously. This is the zone that is "managed" — the reason the architecture has no catalog server and no maintenance cluster.
• Readers. Athena and Redshift via the Glue Data Catalog integration (ANSI SQL, Lake Formation governance), Spark/Trino/EMR via the Iceberg REST endpoint. One set of open tables, many engines.

Governance and observability.

• Lake Formation + Glue. Federating the table bucket into Glue puts its tables under Lake Formation, so table/column/row access control is centralised across every engine — one governance model, not per-engine ACLs.
• S3 Metadata side-car. The journal and inventory tables give you object-level observability (what changed, what exists, what it costs) alongside the data tables, all in the same managed-Iceberg system.
• Open format = portability. Because it is Apache Iceberg, you are never locked in: another engine can read the tables, and you can migrate to a self-managed catalog if requirements change.

The cost model — what you pay for.

• Storage. Per-GB-month for the data (table buckets price storage similarly to S3, with a per-object component).
• Requests. GET/PUT/list-style operations from queries and writes — which compaction reduces by cutting file counts.
• Managed maintenance. Compaction and cleanup are billed by the data/objects processed — the explicit price of not running your own maintenance compute.
• The offset. Against that you delete a self-run catalog (its HA, scaling, on-call) and a maintenance cluster (its compute and engineering) — often the larger real cost.

Managed vs self-managed — the decision.

• Default to S3 Tables when. You want an Iceberg lakehouse on AWS without operating a catalog or maintenance, you read from Athena/Redshift/EMR, and the managed-maintenance cost is acceptable.
• Self-manage Iceberg when. You already run a mature catalog (Glue/Nessie) across clouds, need an engine/feature S3 Tables doesn't yet support, require multi-cloud or on-prem portability of the catalog (not just the data), or must control maintenance timing/cost precisely.
• Either way. The data is open Iceberg, so the choice is reversible and about operational ownership, not lock-in.

The failure modes senior engineers pre-empt.

• Serving OLTP/low-latency point lookups from it. S3 Tables is an analytical lakehouse store, not a request/response serving DB. Mitigation: front user-facing low-latency reads with a serving store; use S3 Tables for analytics.
• Ignoring maintenance cost in the estimate. Teams model storage and forget managed-maintenance (or, self-managed, the maintenance compute). Mitigation: put maintenance in the cost model explicitly on both sides of the decision.
• Assuming lock-in. Fearing S3 Tables is proprietary. Mitigation: it is open Iceberg — portability is preserved; the decision is about who operates the catalog and maintenance.

Common interview probes on architecture and cost.

• "Draw the S3 Tables architecture." — writers → table bucket (managed catalog + maintenance) → readers via Glue; S3 Metadata for observability.
• "What do you pay for?" — storage, requests, and managed maintenance, offset by no catalog/maintenance ops.
• "Managed or self-managed?" — default managed; self-manage for an existing catalog, unsupported engine, or strict catalog portability.
• "Is it lock-in?" — no; it is open Iceberg, so the data and the decision are portable.

Worked example — an SLA/cost decision table for S3 Tables vs self-managed

Detailed explanation. The architecture question usually reduces to "managed or DIY?" The senior answer is a decision table keyed on operational ownership and cost, not preference. Place four realistic scenarios.

• Scenarios. A small team on AWS; a multi-cloud platform; a team with an existing Nessie catalog; a cost-sensitive petabyte lake.
• The axes. Who operates the catalog, who runs maintenance, portability need, cost profile.
• The output. A defensible choice per scenario.

Question. For four scenarios, choose S3 Tables or self-managed Iceberg and justify it by ownership and cost.

Input.

Scenario	Key constraint	Choice
Small team, all-AWS	no ops capacity for a catalog	S3 Tables
Multi-cloud platform	catalog must span clouds	self-managed (Nessie/Glue)
Existing Nessie catalog	already operate it well	self-managed
Cost-sensitive PB lake	maintenance $ dominates	model both, likely S3 Tables

Code.

Managed (S3 Tables) vs self-managed Iceberg — decide by OWNERSHIP + COST
=======================================================================

Small team, all-AWS, Athena/EMR readers
  -> S3 TABLES. No catalog to run, no maintenance cron; managed fee < an engineer.

Multi-cloud (AWS + GCP + on-prem) reading one catalog
  -> SELF-MANAGED (Nessie/Glue). You need a catalog that spans clouds;
     S3 Tables' catalog is AWS-side. Data stays open Iceberg either way.

Already run a mature Nessie catalog + Spark maintenance well
  -> SELF-MANAGED. You have the ops muscle; switching buys little.

Cost-sensitive petabyte lake, streaming writes (heavy compaction)
  -> MODEL BOTH. Managed-maintenance billed by data processed; compare it to
     the compute + on-call of your own compaction cluster. Usually S3 Tables
     wins on TCO once you price engineer-time honestly.

Invariant: it's open Iceberg on both sides -> the choice is REVERSIBLE and about
who OPERATES the catalog + maintenance, not about lock-in.

Step-by-step explanation.

1. The small all-AWS team has no operational capacity to run and secure a catalog or a maintenance cluster, so S3 Tables' managed catalog and automatic maintenance are worth far more than the fee — the fee is less than a fraction of an engineer.
2. The multi-cloud platform's constraint is that the catalog must be reachable and consistent across clouds; S3 Tables' catalog lives on the AWS side, so a cloud-neutral catalog (Nessie or a self-run Glue federation) is the fit — while the data itself stays open Iceberg on any object store.
3. The team already operating Nessie well has sunk the ops cost and has the muscle; switching to managed buys little and disrupts a working system — a case where self-managed remains correct.
4. The cost-sensitive petabyte lake is the one that demands actual modelling: heavy streaming means heavy compaction, and managed compaction is billed by data processed, so you compare that line against the compute and on-call of your own compaction cluster — and once engineer-time is priced honestly, managed usually wins TCO.
5. The invariant across all four is that both sides are open Iceberg, so the decision is reversible and is fundamentally about who operates the catalog and maintenance, never about lock-in — which is what lets you choose on cost and ownership rather than fear.

Output.

Scenario	Choice	Deciding factor
Small all-AWS team	S3 Tables	no ops capacity
Multi-cloud catalog	self-managed	cross-cloud catalog
Mature Nessie shop	self-managed	existing ops muscle
Cost-sensitive PB lake	usually S3 Tables	TCO incl. engineer-time

Rule of thumb. Decide managed-vs-self-managed on operational ownership and honest TCO, not preference: default to S3 Tables unless you need a cross-cloud catalog, already run one well, or hit an unsupported engine. Both are open Iceberg, so the choice is reversible — price in engineer-time, and managed maintenance usually beats a self-run compaction cluster.

Worked example — the reference architecture with governance and observability

Detailed explanation. Assemble the whole picture: streaming and batch writers, the table bucket with managed maintenance, multi-engine readers under Lake Formation, and S3 Metadata for observability. Sketch the end-to-end lakehouse.

• Ingest. Firehose (stream) + Spark/EMR (batch/CDC) → S3 Tables.
• Store/manage. Table bucket: Iceberg tables + automatic maintenance.
• Consume/govern. Athena/Redshift/Spark via Glue + Lake Formation; S3 Metadata side-car.

Question. Lay out the reference architecture: where data enters, how it is stored and maintained, how it is queried and governed, and where observability comes from.

Input.

Zone	Component	Role
Ingest	Firehose, Spark/EMR, Flink	write Iceberg commits
Store	table bucket	data + catalog + maintenance
Serve	Athena, Redshift, Spark	multi-engine reads
Govern	Glue + Lake Formation	central access control
Observe	S3 Metadata	object journal + inventory

Code.

Reference S3 Tables lakehouse
=============================

  [ Firehose ]      [ Spark / EMR ]      [ Flink ]        <- WRITERS
       \                  |                  /            (atomic Iceberg commits
        \                 |                 /              via the managed catalog)
         v                v                v
   +----------------------------------------------+
   |            S3 TABLES  (table bucket)          |
   |  namespaces -> Iceberg tables (Parquet)       |
   |  managed catalog (atomic commits)             |
   |  AUTO maintenance: compaction / expiry /       |
   |                    orphan removal              |
   +----------------------------------------------+
        |                  |                 |
     Glue Data Catalog federation + Lake Formation (governance)
        |                  |                 |
        v                  v                 v
   [ Athena ]         [ Redshift ]       [ Spark/Trino ]   <- READERS

  Side-car:  S3 METADATA over the raw landing bucket
             -> journal (events) + live inventory (snapshot)
             -> "what landed / what changed / what it costs" in SQL

-- Governance is central: one Lake Formation grant covers every engine.
GRANT SELECT ON "s3tablescatalog/analytics".sales.orders
  TO ROLE analyst;             -- Athena, Redshift, EMR all honour this one grant

Step-by-step explanation.

1. Writers are heterogeneous — Firehose lands streams directly, Spark/EMR runs batch and CDC transforms, Flink handles low-latency streaming — but all of them commit through the same managed catalog, so the table has one consistent, atomically-committed history regardless of who wrote.
2. The table bucket is the center of gravity: it stores the Iceberg tables, owns the atomic metadata pointer, and runs compaction/expiry/cleanup continuously — so the diagram has no separate catalog server box and no maintenance-cluster box, which is exactly what "managed" buys.
3. Readers attach through Glue federation, so Athena, Redshift, and Spark/Trino all see the same tables by name, and a single Lake Formation grant governs them — one access model instead of per-engine ACLs, shown by the one GRANT that every engine honours.
4. S3 Metadata sits to the side over the raw landing bucket, giving object-level observability (what landed, what changed, current inventory and cost) in the same SQL surface — so operational questions about the lake are answerable without leaving the analytics stack.
5. The whole thing is open Iceberg, so the architecture is portable and each zone is independently swappable — you could change a writer engine, add a reader, or (in extremis) migrate the catalog — which is the property that makes this a durable design rather than a bet on one product.

Output.

Concern	Where it lives	Managed by
Atomic writes	managed catalog	AWS
Fast scans over time	auto-compaction	AWS
Multi-engine reads	Glue federation	you configure, AWS serves
Access control	Lake Formation	central, one grant
Object observability	S3 Metadata	AWS

Rule of thumb. Design the lakehouse as three zones — heterogeneous writers committing through one managed catalog, a table bucket that stores and maintains the Iceberg tables, and multi-engine readers under one Lake Formation governance model — with S3 Metadata as the observability side-car. The absence of a catalog server and a maintenance cluster in the diagram is the whole value of the managed approach.

Worked example — building the total cost picture

Detailed explanation. Estimating an S3 Tables lakehouse means counting the managed line items and the self-managed costs you avoid. Build a like-for-like TCO comparison for a streaming lake so the decision is defensible. Compare managed vs DIY for the same workload.

• The workload. Heavy streaming writes → lots of compaction.
• Managed costs. Storage + requests + managed maintenance.
• Avoided costs. Catalog HA/ops + maintenance compute + on-call.

Question. Lay out the TCO line items for S3 Tables vs self-managed Iceberg on the same streaming workload.

Input.

Line item	S3 Tables	Self-managed
Storage	per-GB (+per-object)	per-GB
Requests	GET/PUT (fewer, compacted)	GET/PUT (you compact)
Maintenance	managed fee (data processed)	your compute (Spark)
Catalog	included	you run Glue/Nessie (HA)
Engineer-time	~0 for maintenance	on-call + tuning

Code.

TCO — same streaming lake, honest accounting
=============================================

S3 TABLES (managed)
  + storage      : $/GB-month (+ small per-object component)
  + requests     : GET/PUT  (LOWER — compaction cuts file count -> fewer requests)
  + maintenance  : managed fee, billed by data/objects processed by compaction
  + catalog      : included (no server to run)
  + engineers    : ~0 for catalog/maintenance ops
  ---------------------------------------------------------------
  = predictable managed bill; no maintenance on-call

SELF-MANAGED ICEBERG (DIY)
  + storage      : $/GB-month
  + requests     : GET/PUT  (you must run compaction to keep this down)
  + maintenance  : Spark cluster compute for rewrite/expire/remove (recurring)
  + catalog      : Glue/Nessie you run + secure + make HA
  + engineers    : scheduling, tuning, on-call for maintenance failures
  ---------------------------------------------------------------
  = lower line-item fees, but real compute + significant engineer-time

Decision: compare the MANAGED FEE against (maintenance compute + engineer-time +
catalog ops). At small/medium scale managed almost always wins; at very large
scale, model the compaction fee explicitly — it's the one line that scales with write volume.

Step-by-step explanation.

1. On the managed side, storage and requests are ordinary S3-like costs — and requests are lower than a neglected DIY table because compaction cuts file count, so there are fewer objects to GET/LIST per scan. The distinctive line is managed maintenance, billed by the data compaction processes.
2. On the DIY side the per-item fees look cheaper, but two big costs appear that managed hides: the recurring compute of a Spark maintenance cluster running rewrite/expire/remove, and the engineer-time to schedule, tune, and be on-call for those jobs — the cost teams routinely forget.
3. The honest comparison is therefore the managed-maintenance fee versus the sum of maintenance compute, catalog operations (running Glue/Nessie with HA), and engineer-time — not storage against storage, which is roughly a wash.
4. At small and medium scale the managed fee is almost always less than even a fraction of an engineer plus a maintenance cluster, so S3 Tables wins TCO comfortably; the honest place to actually model is very large, write-heavy scale where the compaction fee scales with write volume.
5. The senior framing is to make maintenance a line item on both sides of the estimate: the mistake that skews these comparisons is pricing storage and requests but treating maintenance as free on the DIY side, when it is precisely the cost S3 Tables is charging you to remove.

Output.

Cost driver	S3 Tables	Self-managed
Storage	similar	similar
Requests	lower (compacted)	you must compact
Maintenance	managed fee	compute + on-call
Catalog ops	included	you run it
TCO at small/mid scale	usually lower	usually higher

Rule of thumb. Build the TCO with maintenance as an explicit line item on both sides: managed maintenance fee vs your compaction compute plus catalog ops plus engineer-time. Storage and requests roughly wash; the real comparison is who pays for maintenance, and once engineer-time is priced honestly, managed usually wins except at very large, write-heavy scale where you should model the compaction fee directly.

Senior interview question on the end-to-end S3 Tables architecture and cost

A senior interviewer might ask: "Design the full lakehouse for a multi-engine analytics platform on AWS with streaming and batch ingest. Cover the write path and the read path, where the catalog and maintenance live, how governance and observability work, how you decide S3 Tables versus self-managed Iceberg, and how you build a defensible cost estimate — all while keeping the data portable open-format."

Solution Using a managed table bucket, Glue-governed multi-engine reads, S3 Metadata, and a TCO-driven build decision

-- 1. Write path + read path + where each concern lives.
--    Firehose / Spark / Flink  --atomic Iceberg commits-->  TABLE BUCKET
--      table bucket: Iceberg tables + MANAGED CATALOG + AUTO maintenance
--        --Glue Data Catalog federation + Lake Formation-->  Athena / Redshift / Spark
--      S3 METADATA (side-car over the landing bucket): journal + live inventory
--    Data is open Iceberg end to end -> portable, reversible decision.

# 2. Provision the managed store; maintenance is policy, not a Spark cluster.
aws s3tables create-table-bucket --name analytics --region us-east-1
# maintenance policy (bucket-level): compaction 512MB + snapshot expiry + orphan removal

-- 3. Governance: ONE Lake Formation grant covers every engine (central access control).
GRANT SELECT ON "s3tablescatalog/analytics".sales.orders TO ROLE analyst;

-- 4. Observability: S3 Metadata answers "what landed / what it costs" in SQL.
SELECT storage_class, round(sum(size)/1e12,2) tb
FROM "s3tablescatalog/landing-metadata".s3metadata.inventory GROUP BY storage_class;

-- 5. Build decision (TCO): managed fee  vs  (maintenance compute + catalog ops + on-call).
--    Default S3 Tables; self-manage only for cross-cloud catalog / unsupported engine /
--    existing mature catalog. Open Iceberg both ways -> reversible, not lock-in.

Step-by-step trace.

Layer	Component	Responsibility
Ingest	Firehose / Spark / Flink	atomic Iceberg commits
Store + catalog	table bucket	data, atomic pointer, maintenance
Maintenance	managed (policy)	compaction / expiry / cleanup
Serve	Athena / Redshift / Spark	multi-engine reads via Glue
Govern	Lake Formation	one grant, every engine
Observe	S3 Metadata	journal + inventory in SQL

After deployment, streaming and batch writers commit atomically into the analytics table bucket; the managed catalog serialises those commits and automatic maintenance keeps scans fast and storage lean; Athena, Redshift, and Spark read the same open tables under a single Lake Formation governance model; S3 Metadata answers object-level "what landed / what it costs" questions in SQL; and the managed-vs-self decision is made on honest TCO with the data portable either way. The architecture has no catalog server and no maintenance cluster to operate — that is the whole point.

Output:

Metric	DIY Iceberg lakehouse	S3 Tables lakehouse
Catalog to operate	Glue/Nessie (HA)	none (managed)
Maintenance to run	Spark cron + on-call	none (managed)
Multi-engine governance	per-engine ACLs	one Lake Formation model
Object observability	DIY pipeline	S3 Metadata (SQL)
Portability	open Iceberg	open Iceberg
Decision basis	preference/inertia	TCO + ownership

Why this works — concept by concept:

• Managed catalog as source of truth — every writer commits through one Iceberg REST catalog that owns the atomic metadata pointer, so heterogeneous streaming and batch engines share one consistent, corruption-free table history without a catalog server you operate.
• Maintenance as managed policy — compaction, snapshot expiration, and orphan removal run continuously from a bucket-level policy, so scan performance and storage stay healthy with no maintenance cluster or cron in the architecture.
• Central governance via Lake Formation — Glue federation puts every table under one access-control model, so a single grant governs Athena, Redshift, and Spark rather than per-engine ACLs drifting out of sync.
• S3 Metadata observability + open format — object-level journal and inventory tables answer operational questions in SQL, and because everything is open Iceberg the whole design is portable and the managed-vs-self decision is reversible.
• Cost — a predictable managed bill (storage + requests + maintenance fee) versus DIY line items plus catalog HA, maintenance compute, and on-call. The eliminated cost is the catalog server and the maintenance pipeline — O(managed policy) instead of O(engineers) to operate a portable lakehouse.

Design
Topic — design
Design problems on lakehouse architecture and cost trade-offs

Practice →

Optimization
Topic — optimization
Optimization problems on storage cost and query efficiency

Practice →

---

Cheat sheet — S3 Tables & S3 Metadata

• Files are not a table. Raw Parquet in an S3 prefix is bytes, not a table. A table format (Iceberg) adds snapshots, schema evolution, and atomic commits; a catalog owns the current-metadata pointer; maintenance keeps it fast. S3 Tables manages the catalog and the maintenance so you own only the data and schema.
• The hierarchy. table bucket (a new S3 bucket type = managed Iceberg catalog + storage) → namespace (a database/schema) → table (a full Apache Iceberg table). ARN: arn:aws:s3tables:region:account:bucket/name. You never PutObject into a table bucket — write through an Iceberg engine.
• Create it. aws s3tables create-table-bucket --name X → create-namespace → create-table --format ICEBERG --metadata '{...schema...}' (or boto3 s3tables client). Provision from IaC and thread the bucket ARN through.
• Writers connect via the Iceberg REST endpoint. Spark/EMR/Flink configure a REST catalog (type=rest, the S3 Tables URI, the bucket ARN as warehouse, sigv4-enabled=true). Then it is ordinary Iceberg: INSERT, MERGE INTO (CDC upserts), FOR TIMESTAMP AS OF (time travel), ADD COLUMN.
• Readers connect via Glue. Register/federate the table bucket into the Glue Data Catalog and Athena/Redshift query it by three-part name (catalog/namespace.table) under Lake Formation governance. Same open Iceberg table, many engines, one access model.
• S3 Metadata = object metadata as Iceberg tables. Enable on a general-purpose bucket. A journal table is an append-only event log (record_type CREATE/UPDATE_METADATA/DELETE + timestamp + requester) for audit/change history; a live inventory table is a current snapshot (key/size/tags/storage_class) for present-state cost/governance. Query with Athena/Spark; join to your data.
• Journal vs inventory. Never sum raw journal rows (created-then-deleted double-counts); for current state use the inventory table, or reduce the journal (row_number() latest-per-key, drop DELETE) for as-of reconstruction. Metadata populates in minutes — it is observability, not the write path.
• Maintenance = three continuous jobs. Compaction rewrites small files toward ~512 MB (fixes the small-files problem — fewer requests, faster scans); snapshot expiration drops old snapshots (bounds metadata, sets your time-travel horizon); unreferenced-file removal deletes orphaned data files (reclaims storage). S3 Tables runs all three automatically; DIY is rewrite_data_files / expire_snapshots / remove_orphan_files Spark jobs you own.
• Maintenance config. Policy, not code: targetFileSizeMB (~512), minSnapshotsToKeep + maxSnapshotAgeHours (= your rollback/audit window), orphan nonCurrentDays. Set at the table-bucket level so every table inherits it. Watch the "table$files" / "table$snapshots" metadata tables to verify health.
• The small-files math. 100 GB as 50k × 2 MB files ≈ 50k GETs + huge manifests + slow planning; as ~200 × 512 MB files ≈ 200 GETs + fast pruning. Same bytes, orders-of-magnitude fewer files/requests. Compaction is the single biggest scan-performance lever.
• Architecture. Writers (Spark/EMR/Flink/Firehose) → table bucket (managed catalog + auto-maintenance) → readers (Athena/Redshift/Spark) via Glue + Lake Formation; S3 Metadata as the observability side-car. No catalog server, no maintenance cluster in the diagram — that is the value.
• Cost model. Storage (+ per-object) + requests (lower thanks to compaction) + a managed-maintenance fee (billed by data processed), offset by no catalog HA/ops and no maintenance compute/on-call. Price maintenance on both sides of the estimate.
• Managed vs self-managed. Default to S3 Tables. Self-manage Iceberg for a cross-cloud catalog, an engine/feature S3 Tables doesn't support, or an existing mature catalog. Both are open Iceberg — the choice is about who operates the catalog and maintenance, and it is reversible, not lock-in.
• When NOT to use it. It is an analytical lakehouse store, not an OLTP/low-latency serving DB — front user-facing point lookups with a serving store and keep S3 Tables for analytics.

Frequently asked questions

What are AWS S3 Tables?

S3 Tables are a purpose-built Amazon S3 bucket type — the table bucket — that stores your data as Apache Iceberg tables behind an AWS-managed Iceberg catalog, and that runs the table maintenance (compaction of small files, snapshot expiration, and unreferenced-file cleanup) for you automatically. Inside a table bucket you create namespaces (like databases) that hold Iceberg tables, and the bucket exposes an Iceberg REST catalog endpoint so Spark, EMR, Flink, and Trino can read and write it, while a Glue Data Catalog integration lets Athena and Redshift query the same tables with ordinary SQL under Lake Formation governance. The point is to get a real, ACID, time-travelling table on cheap object storage without also having to operate the catalog and a maintenance pipeline yourself — you own the data and the schema; AWS owns the catalog and the maintenance.

S3 Tables vs Iceberg on general-purpose S3 — what's the difference?

Both give you genuine Apache Iceberg tables — snapshots, schema evolution, atomic commits, time travel, and full portability across Iceberg engines — so the data format and lock-in profile are the same. The difference is operational ownership. With self-managed Iceberg on a general-purpose bucket, you stand up and secure a catalog (Glue, Hive Metastore, Nessie, or a JDBC catalog) and you schedule, scale, monitor, and pay for the maintenance jobs (rewrite_data_files, expire_snapshots, remove_orphan_files). With S3 Tables, the table bucket is a managed Iceberg catalog and AWS runs that maintenance continuously as a managed operation you configure by policy. So the choice is not about the table format — it is about whether you want to operate the catalog and the maintenance pipeline or pay AWS to, and because both are open Iceberg the decision is reversible.

What is S3 Metadata and how does it relate to S3 Tables?

S3 Metadata is a feature you enable on a general-purpose bucket that makes S3 automatically write that bucket's object metadata into managed Iceberg tables — so it uses the same managed-Iceberg machinery as S3 Tables, but stores metadata about your objects rather than your analytical data. It produces two table shapes: a journal table, an append-only event log of object changes (create, update-metadata, delete) with timestamps and requester context for audit and change history; and a live inventory table, a continuously-maintained snapshot of the objects that exist right now with their size, tags, and storage class for present-state cost and governance analysis. Both are queryable with Athena, Spark, or any Iceberg engine and join to your own tables, so questions like "what got deleted last night," "which objects are over 1 GB and untagged," or "how is storage split by class" become SQL instead of a paginated LIST crawl or a batch inventory report.

Do I still need Glue and Athena with S3 Tables?

You need the Glue Data Catalog integration if you want SQL engines like Athena and Redshift to query your S3 Tables by name — federating the table bucket into Glue is what surfaces its namespaces and tables to those services and puts them under Lake Formation access control. Compute engines like Spark, EMR, Flink, and Trino do not strictly need Glue; they can connect to the table bucket's Iceberg REST catalog endpoint directly with SigV4 auth and operate the tables as standard Iceberg. In practice most platforms enable the Glue integration anyway, because it gives one governed catalog and one Lake Formation access model that every engine honours — Athena for ad-hoc SQL, Redshift for BI, EMR/Spark for transforms — rather than wiring access per engine. So Glue is the governance and SQL-discovery layer; Athena is one of several readers on top of it.

What maintenance does S3 Tables run automatically?

Three continuous jobs. Compaction rewrites the many small Parquet files that streaming and CDC writes create into fewer, target-sized files (around 512 MB by default), which fixes the small-files problem — fewer S3 requests, smaller manifests, faster query planning, and better compression and pruning. Snapshot expiration drops snapshots older than your retention window (while keeping a configured minimum), which bounds metadata size, sets your time-travel horizon, and unpins old data files so they can be reclaimed. Unreferenced-file removal deletes data files that no live snapshot references — the orphans left by failed writes and expired snapshots — after a safety window, reclaiming storage. You configure these by policy (target file size, retention hours, minimum snapshots, orphan age) at the table or table-bucket level, and AWS runs them in the background in the right order, so there is no Spark maintenance cluster or cron for you to operate.

When should I NOT use S3 Tables?

Do not reach for S3 Tables when the workload is not analytical: it is a lakehouse store built for scans and batch/streaming analytics, not an OLTP or low-latency request/response serving database, so front user-facing point lookups with a proper serving store and keep S3 Tables for analytics. Self-managed Iceberg is the better fit when you need a catalog that spans multiple clouds or on-prem (S3 Tables' catalog is AWS-side), when you already operate a mature catalog like Nessie or a tuned Glue setup and switching buys little, or when you need an engine or an Iceberg feature that S3 Tables does not yet support. And at very large, write-heavy scale you should explicitly model the managed-compaction fee (billed by data processed) against your own maintenance compute before defaulting to managed — usually managed still wins on total cost once engineer-time is priced honestly, but that is the one place to check the numbers rather than assume.

Practice on PipeCode

• Drill the data processing practice library → for the Iceberg-table, batch-scan, and Spark read/write problems that S3 Tables and compaction make concrete.
• Rehearse pipeline patterns on the ETL practice library → for the CDC upsert, event-log-to-snapshot, and multi-engine loading scenarios where the journal-vs-inventory and MERGE decisions earn their keep.
• Sharpen the lakehouse architecture axis with the system design practice library → for the catalog-placement, maintenance-ownership, governance, and managed-vs-self-managed trade-offs a table layer must get right.
• Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the table-format, compaction, snapshot-expiration, and metadata-observability patterns against real graded inputs — Iceberg, Athena, Spark, and object storage.

Lock in S3 Tables and Iceberg muscle memory

Docs explain S3 Tables and S3 Metadata. PipeCode drills explain the decision — when files are not a table, when `compaction` is the difference between a fast scan and a request storm, when the journal is the wrong table for current state, and when managed maintenance beats a self-run Spark cluster. Pipecode.ai is Leetcode for Data Engineering — lakehouse practice tuned for the production trade-offs senior data engineers actually face.

Practice data processing problems →
Practice system design problems →