Amazon S3 Tables Alternatives for Production Iceberg

Amazon S3 Tables solves a real problem. If you have ever spent a weekend debugging a failed compaction job, or watched a table accumulate 40,000 small files because nobody remembered to schedule rewrite_data_files, the pitch is compelling: create a table bucket, write data, and AWS handles compaction, snapshot expiration, and orphan cleanup automatically. No Spark clusters. No Airflow DAGs. No on-call for maintenance failures.

For greenfield deployments with straightforward requirements — especially AWS-native stacks using Athena, EMR, and Redshift — that trade-off works well and can save weeks of infrastructure setup. But production Iceberg estates tend to outgrow what S3 Tables can deliver, and the constraints become apparent exactly when you need flexibility most.

This article breaks down what S3 Tables does well, where it falls short, and what alternatives exist for teams that need more from their Iceberg operations.

What S3 Tables actually gives you

S3 Tables, generally available across most AWS regions since 2025, merges Iceberg storage and catalog into a single managed service. You create a "table bucket" (a dedicated S3 bucket type that only stores Iceberg tables), and AWS provides:

• Built-in Iceberg REST Catalog — each table bucket exposes a REST endpoint that any Iceberg-compatible engine can connect to. S3 Tables also integrates with the Glue Data Catalog through a federated s3tablescatalog, making tables discoverable alongside standard Glue-managed tables.
• Automatic compaction — background binpack, sort, or z-order compaction with configurable target file sizes (64 MB to 512 MB). As of July 2025, compaction costs were reduced by up to 90%.
• Snapshot expiration and orphan cleanup — managed automatically without any user-side scheduling.
• Higher write throughput — AWS reports up to 10x higher transactions per second compared to general-purpose S3 buckets. For high-concurrency streaming ingestion pipelines, this can be the deciding factor — Glue Catalog limits concurrent UPDATE TABLE calls to roughly 10 per second per table by default.
• Intelligent-Tiering — automatic storage cost optimization based on access patterns. Compaction runs only on files in the Frequent Access tier, reducing maintenance costs on cold data.
• SageMaker Lakehouse integration — S3 Tables integrates with SageMaker Lakehouse for unified analytics across S3 Tables, Redshift, and third-party sources through a single management experience.
• Cross-region replication — read-only replicas of tables across regions and accounts, preserving full snapshot history.

For a team of two or three engineers standing up their first Iceberg lakehouse on AWS, this eliminates weeks of infrastructure work. Credit where it is due: zero-configuration maintenance is a genuine step forward for the ecosystem.

Where S3 Tables falls short

The constraints surface as soon as requirements move beyond the simple case.

Black-box maintenance

S3 Tables handles compaction, but you cannot see what it did. There are no detailed compaction logs, no before/after file counts, and no way to correlate a compaction run with a change in query latency. The trigger conditions for auto-compaction are undocumented — understanding when and why compaction fired (or didn't) requires an AWS support case.

For streaming workloads with freshness SLAs, this opacity is a production risk. Compaction that runs during a peak query window can temporarily increase latency. Without visibility into the schedule, you cannot coordinate maintenance with critical workloads.

There is also a subtler issue: commit contention. S3 Tables runs maintenance operations continuously in the background, which can advance table metadata and conflict with your writes. Pipelines see CommitFailedException: branch main has changed — a retry-and-backoff situation that requires explicit handling in your ingestion code. AWS documents that maintenance jobs themselves can fail due to "out of date metadata" and get retried. For high-frequency writers, this contention is an operational reality that the "zero-ops" pitch does not surface.

Snapshot management has its own gotchas. If any user-defined tag or branch exists on a table, snapshot management fails for the entire table — no snapshots are expired or removed until you delete all tags and branches. Setting Iceberg snapshot retention properties (history.expire.max-snapshot-age-ms or history.expire.min-snapshots-to-keep) as table properties also causes snapshot management to fail silently. These are not edge cases for teams using Iceberg's branching features for testing or auditing.

The maintenance API lets you configure target file size and compaction strategy per table. That is the extent of your control.

Engine compatibility gaps

S3 Tables works natively with Athena, EMR, and Redshift. Through the REST Catalog API and the Glue Data Catalog federation, AWS also lists Spark, Trino, Flink, Snowflake, and DuckDB as compatible. Engine breadth has improved significantly since launch. But practical gaps remain:

dbt: dbt-athena 1.11.0 (July 2026) shipped end-to-end S3 Tables support via catalogs.yml v2. This is real progress. However, S3 Tables does not support ALTER TABLE RENAME, which breaks dbt's standard near-zero-downtime table swap. The materialization uses drop-and-recreate on every run, S3 cleanup is skipped (since S3 Tables manages storage), and tables must be lowercase-only. For teams where dbt is the primary transformation framework — which is most modern data teams — this is workable but not equivalent to standard Glue-backed Iceberg.

Views: CREATE VIEW is not supported against the S3 Tables catalog endpoint. Views are fundamental to most transformation and BI layers.

DDL restrictions: The REST endpoint does not support stage-create (blocking CTAS in some engine configurations), multi-level namespaces, or OAuth-based authentication. Metadata files over 50 MB return a 400 Bad Request.

Catalog separation: S3 table bucket catalogs surface as a federated s3tablescatalog in the Glue Data Catalog, which is an improvement over full isolation. But the hierarchical mapping (table bucket → child catalog → database → table) means cross-catalog joins between standard Glue tables and S3 Tables still require fully qualified references. It is more integrated than at launch, but not seamless.

Permission failure modes: A notable gotcha from production PoCs: when Lake Formation permissions are missing, SHOW TABLES returns empty results rather than an access-denied error. There is no way to distinguish between "table doesn't exist" and "you lack permissions." This was reported as the single biggest time sink in at least one published S3 Tables evaluation.

The structural concern remains: new tools in the data ecosystem integrate against the open Iceberg spec first. When a query engine ships Iceberg support, it works against a standard Glue-registered table on day one. S3 Tables has narrowed the gap significantly through its REST endpoint and Glue federation, but some tools still need specific adapter work for the s3tablescatalog path.

Security: Lake Formation support with caveats

S3 Tables now integrates with Lake Formation through the Glue Data Catalog federation, supporting column-level, row-level, and cell-level security. This is a significant improvement over the table-level-only IAM policies available at launch.

The caveat: when you enable "full table access" for external engines (third-party engines outside the AWS analytics stack), Lake Formation returns credentials without enforcing column or row filters. Fine-grained access control only applies when engines use Lake Formation credential vending without the full-table-access flag. For teams running mixed engine stacks — which is common in production — this means security enforcement depends on how each engine connects, not just what permissions are defined. Standard AWS engines (Athena, EMR, Redshift) enforce Lake Formation grants; external engines may bypass them depending on configuration.

AWS-only, exit requires rewrite

To be clear about what is and is not locked in: the data itself is standard Apache Iceberg on Parquet. The format is open. The lock-in is at the storage and catalog layer. S3 table buckets are not accessible via standard S3 object APIs — you cannot aws s3 ls the underlying files or point another catalog at the same path. Moving data out of S3 Tables requires a full data export (typically via Spark CTAS), not a catalog pointer swap.

For a small table, this is a manageable migration. For a lake at scale, it is a significant project. As one detailed comparison put it: "With self-managed Iceberg, your files are on standard S3, accessible via standard S3 APIs. The catalog is a pointer. Switching to any other catalog means re-registering the table path. Nothing moves. S3 Tables is a different commitment."

There is also a subtler ecosystem portability dimension. When your catalog is proprietary, ecosystem support depends on AWS's roadmap and release schedule. On the self-managed Iceberg path, integrations happen in the open, on the community's timeline. That structural difference matters when the ecosystem is moving as fast as it is now.

No cross-estate observability

S3 Tables provides basic CloudWatch metrics per table — object count, storage size, maintenance job status. There is no cross-table health dashboard, no severity-ranked alerting, and no way to see which tables are degrading before they affect query performance or cost.

At scale, the question is not "is compaction running?" — it is "which of my 500 tables has a small-file ratio above 40%, and what is the impact on query latency across the three engines that read it?"

Here's a preview of table health observability with a control plane like LakeOps:

Incomplete maintenance coverage

S3 Tables runs compaction, snapshot expiration, and orphan cleanup. It does not handle manifest rewriting or Puffin column-level statistics computation. Both directly affect query planning performance:

• Manifest bloat: Frequent commits accumulate hundreds of small manifest files. Each one must be opened during query planning, adding seconds of latency before any data is read.
• Missing statistics: Without fresh Puffin NDV, min/max, and null-count statistics, engines cannot prune files effectively. They scan more data than necessary.

These are not niche operations. They are part of the standard Iceberg maintenance lifecycle, and their absence means S3 Tables' "fully managed" maintenance is, in practice, partial.

The alternatives

When engineers search for S3 Tables alternatives, they are typically looking for one of three things: more engine flexibility, more operational control, or vendor neutrality. The answer depends on which constraint is binding.

Self-managed Iceberg on standard S3

The traditional approach: write Iceberg tables to standard S3 buckets, register them in AWS Glue Data Catalog (or a REST catalog like Apache Polaris or Nessie), and run maintenance yourself.

What you gain:

• Every Iceberg-compatible engine works on day one — Trino, Spark, Flink, DuckDB, Snowflake, Databricks, ClickHouse, dbt.
• Full Lake Formation support for column-level security and row-level filtering without the external-engine caveats that affect S3 Tables.
• Standard S3 object access — no proprietary storage layer, no exit rewrite. Switching catalogs means re-registering the table path; nothing moves.
• Complete maintenance control — you decide when compaction runs, what sort order to use, and how aggressively to expire snapshots.

What it costs:

• You own the entire maintenance lifecycle. Compaction is a Spark job or Athena OPTIMIZE call. Snapshot expiration and orphan cleanup are separate procedures. Getting the sequence wrong — for example, running orphan cleanup before snapshot expiration finishes, or skipping manifest rewrites after compaction — wastes compute or risks deleting files still referenced by active snapshots.
• At 10–50 tables, this is manageable with a few DAGs. At 500+ tables with different ingestion rates, it becomes the primary source of operational incidents.
• No built-in observability across the estate.

Self-managed Iceberg is the default for most production teams today. It works — but the operational burden scales linearly with the number of tables, and most teams underestimate the ongoing cost until they are deep into it.

Open-source REST catalogs (Apache Polaris, Lakekeeper, Nessie)

If the concern is catalog lock-in rather than maintenance automation, an open REST catalog can replace S3 Tables' built-in catalog without the proprietary coupling.

Apache Polaris (graduated to ASF Top-Level Project in 2026) implements the Iceberg REST Catalog spec with RBAC, credential vending, multi-engine support, and external catalog federation. It can be self-hosted on Kubernetes with PostgreSQL, or used through managed offerings like Snowflake Open Catalog or Dremio Catalog.

Lakekeeper is a Rust-based REST catalog focused on authorization — OPA-based policies, audit events, and fine-grained governance.

Nessie adds Git-like branching and tagging to the catalog layer, enabling table versioning and isolated experimentation.

What you gain:

• Vendor-neutral catalog that works on any cloud.
• Full Iceberg REST spec compliance — every engine integrates without custom adapters.
• Credential vending, RBAC, and (in Polaris) external catalog federation.

What they don't solve:

• None of these catalogs handle compaction, snapshot expiration, or orphan cleanup on their own. They are metadata management services, not maintenance automation. You still need a separate system for the operational lifecycle.
• Self-hosting requires infrastructure: PostgreSQL backend (Polaris) or Rust binary (Lakekeeper), Kubernetes deployment, monitoring, upgrades.

Emerging: open-source maintenance layers. Floe connects to Polaris and provides policy-driven maintenance (compaction, snapshot expiration, orphan cleanup, manifest optimization) executed via Spark or Trino. Firn is a Go-based maintenance daemon that works with Lakekeeper, Polaris, Nessie, or Glue — using DuckDB for compaction instead of Spark. Both are pre-1.0 but represent the open-source ecosystem's response to the maintenance gap.

An open catalog solves the portability problem. With Floe or Firn, you can start to solve the maintenance problem too — though both require more assembly than managed alternatives.

Lakehouse control plane (LakeOps)

A control plane complements the storage and catalog layers by handling the operational lifecycle that neither S3 Tables nor open catalogs address on their own.

LakeOps connects to any Iceberg catalog — Glue, S3 Tables, Polaris, Nessie, Gravitino — and provides autonomous maintenance, observability, and optimization across the entire estate. The distinction from S3 Tables is that LakeOps works with your existing infrastructure rather than replacing it, and provides the visibility and intelligence that managed services abstract away.

What this approach covers:

• Cross-estate observability — a single dashboard covers every table across every catalog (Glue, S3 Tables, REST) with severity-ranked insights. You see which tables are degrading before the impact reaches query dashboards or invoices.

• Full maintenance sequence in correct dependency order — snapshot expiration → orphan cleanup → compaction → manifest rewrite → statistics computation. Each step's output feeds the next, ensuring cleanup always precedes compaction and metadata optimization always follows it.

• Query-aware compaction — instead of blind binpack, the system analyzes which columns your queries actually filter, join, and group on, then physically sorts data to match those patterns. Engines skip entire file groups through min/max pruning, reducing I/O significantly. The sort strategy adapts as query patterns change.

• Purpose-built execution engine — compaction runs on a native Rust/DataFusion runtime that bypasses the JVM stack entirely. Production benchmarks show 7× higher throughput than Spark and over 20× faster than S3 Tables' built-in compaction on equivalent datasets.

• Health-driven triggers, not cron schedules — every table is scored along multiple structural dimensions (file fragmentation, delete-file depth, snapshot age, manifest bloat) and maintenance fires only when indicators cross actionable thresholds. Streaming tables might be compacted hourly, batch tables daily, and well-maintained tables are skipped entirely.

• Policy-driven governance — maintenance rules defined once at the namespace or catalog level, with per-table overrides where needed. Every execution is logged with before/after metrics for auditability.

For a deeper look at how each capability works in practice, the LakeOps platform page walks through the full system from observability through compaction, routing, and governance.

Comparison: S3 Tables vs. alternatives

Dimension	S3 Tables	Self-Managed (Glue + S3)	Open Catalog (Polaris)	Control Plane (LakeOps)
Compaction	Automatic, opaque	Manual (Spark/Athena)	Not included	Autonomous, query-aware, Rust engine
Snapshot / orphan mgmt	Automatic (fails if tags/branches exist)	Manual procedures	Not included	Autonomous, sequenced
Manifest rewrite	Not available	Manual	Not included	Automatic
Statistics (Puffin)	Not available	Manual	Not included	Automatic post-compaction
Engine support	Athena, EMR, Redshift, Spark, Trino, Flink, Snowflake, DuckDB via REST	All Iceberg engines	All Iceberg engines	All engines + routing
dbt	Supported (dbt-athena 1.11+, drop-recreate)	Full production support	Full	Full
Security model	Lake Formation via Glue (caveats for external engines)	Lake Formation (column, row, tag)	RBAC (Polaris-native)	Works with Lake Formation + operational policies
Observability	CloudWatch basics	Whatever you build	None	Health classification, insights, audit trails
Multi-cloud	AWS only	Any cloud (with catalog choice)	Any cloud	Any cloud
Vendor lock-in	Open format, proprietary storage layer, exit = data rewrite	Standard S3, portable	Fully open	No lock-in, works with existing stack
Maintenance control	Config only (target size, strategy)	Full control, full burden	N/A	Policy-driven with full visibility

What matters for your evaluation

The right alternative depends on which S3 Tables constraint is actually blocking you.

If engine compatibility is the issue

S3 Tables' engine compatibility has improved significantly — Spark, Trino, Flink, Snowflake, and DuckDB all connect through the REST endpoint. But if you need standard dbt table-swap semantics, full CREATE VIEW support, or CTAS via stage-create, self-managed Iceberg on standard S3 with Glue Data Catalog remains the straightforward path. Every engine works without adapter caveats. The trade-off is owning the maintenance lifecycle, which is well-understood and automatable — but requires sustained engineering investment.

If vendor lock-in is the concern

An open REST catalog like Apache Polaris gives you a vendor-neutral metadata layer that works on any cloud and any engine. You still need to solve maintenance separately. For teams that already have mature Spark-based maintenance pipelines and just want catalog portability, this can be the right layer to change.

If operational complexity is the bottleneck

This is where most teams land after running Iceberg in production for more than a few months. The problem is not any single capability — it is the combinatorial complexity of managing compaction schedules, monitoring health across hundreds of tables, keeping sort orders aligned with evolving query patterns, and ensuring the maintenance sequence runs correctly every time across different ingestion cadences.

S3 Tables addresses this by abstracting the problem away behind a black box. The control plane approach addresses it by automating the same operations with full visibility and policy governance — plus the query-aware optimization and cross-engine telemetry that no black-box service can provide, because they require understanding how your tables are actually queried.

If you are already using S3 Tables

LakeOps connects to S3 Tables as a catalog source alongside Glue and other catalogs. You do not have to migrate off S3 Tables to get cross-estate observability, health classification, or policy governance. For tables that work well in S3 Tables — high-throughput streaming writes that benefit from managed compaction and higher TPS, AWS-native stacks where the Glue federation provides sufficient engine access — you can keep them there. For tables that need standard dbt semantics, view support, or unrestricted Lake Formation enforcement across external engines, those live in standard S3 with Glue. LakeOps provides the unified operational layer above both.

The maintenance gap is the real problem

The deeper pattern behind the "S3 Tables alternative" search is not really about S3 Tables at all. It is about the gap between Iceberg's capabilities as a format and what it takes to operate Iceberg well in production.

Iceberg gives you ACID transactions, time travel, schema evolution, and engine-agnostic access. It does not give you a system that keeps tables healthy as data flows in. That requires compaction, cleanup, manifest optimization, statistics computation, health monitoring, and policy enforcement — coordinated correctly across every table in the estate.

S3 Tables fills part of this gap — and it fills it well for the use cases it targets. If you are building an AWS-native lakehouse with Athena and Redshift, want zero-ops maintenance, and can work within the DDL and view restrictions, S3 Tables is a strong starting point. The ecosystem has improved steadily since launch: Lake Formation integration, Glue federation, dbt support, and broader engine compatibility have addressed several early limitations.

But many production estates outgrow the managed-service model. They run multiple engines. They need views and standard dbt semantics. They need to understand what maintenance did and why. They need policies that work across catalogs, and observability that covers the entire lake — not just one storage layer.

Open catalogs like Apache Polaris fill the metadata portability piece. Emerging tools like Floe and Firn add open-source maintenance automation. Self-managed deployments give maximum control at the cost of operational burden.

A lakehouse control plane fills the gap end to end: the full maintenance lifecycle with correct sequencing, the observability to know what is happening, the policy layer to govern it at scale, and the execution engine to do it efficiently — across every catalog type, every engine, and every cloud. That is the architectural layer the Iceberg ecosystem has been missing, and it is the one most teams end up needing regardless of where their data physically sits.

If you are evaluating alternatives to S3 Tables, start by clarifying which constraint matters most — portability, engine support, security, or operational maturity. Then pick the combination of storage, catalog, and operational layers that matches. LakeOps handles the operational layer above any combination of the others.