Okay, we will be honest – data architecture is usually one of those topics that makes your head spin. I was there at AWS Community Day Chennai on 07-Mar-2026. There were a lot of good talks on many kinds of cloud technologies, but one speaker in particular caught my eye.
Then we had Vishali Sakthivel and Vikneshwara RB take the stage to discuss about transitioning from Data Lakes to Lakehouses with Amazon S3 Tables and AWS Glue. If you've ever wrestled with messy data pipelines, this workshop felt like a breath of fresh air. I want to break out exactly what I learnt.
The Headache of Traditional Data Architectures
Sound familiar? We did data analytics with on-prem data warehouses for the longest period. This old system has some significant baggage.
We had to work with strict concepts that were really hard to alter.
We were trapped with ETL heavy pipelines that had to be babysat constantly.
Scaling was brutally expensive and vendor lock-in was a significant risk.
Even as we migrated to the cloud there were still serious engineering problems. Data engineers have always struggled with the “small files problem" the presence of thousands of little files would significantly impact query performance.
Imagine reading a book where every sentence is printed on a separate piece of paper and spread out throughout a room. That’s how your database feels dealing with the little files problem. We also suffered schema evolution issues, slow analytical queries, difficult pipeline maintenance, late arriving data and duplicate records.
Enter the Lakehouse and Apache Iceberg
Here is when things become interesting. The industry responded to these problems by moving to the “Lakehouse” architecture. A Lakehouse can be thought of as the best of both worlds. It combines the vast storage scale of a Data Lake and the organized reliability of a Data Warehouse.
Apache Iceberg is the core of this transition. Iceberg is an open table format for data streaming and Lakehouses. You might think of it as a very efficient index for your data. It lets different compute engines like Apache Flink, Spark, Snowflake and Athena read the exact same data without duplicating it.
The Maintenance Trap
But running Apache Iceberg yourself on AWS isn’t a picnic. The speakers raised several serious challenges:
Operational Overhead: You need to keep a close watch on the system.
Metadata Explosion: Tracking data changes creates huge metadata files which bog things down.
Manual Maintenance Trap: Engineers run compaction jobs for hours at a time to maintain things healthy.
Catalog Consistency: It is hard to keep your metadata catalog fully in sync.
S3 Tables to the Rescue
The issue is: Amazon S3 Tables takes care of all that messy operational overhead. S3 Tables offers fully managed Iceberg tables with automatic table maintenance. Simply put, AWS conducts the heavy lifting behind the scenes, such as cleaning up small files and optimizing metadata, so you don't have to.
The architecture consists of few building blocks:
Amazon S3: Storage Buckets Table
AWS Glue: For data transformations.
Amazon Athena: For SQL-based analytics.
QuickSight (BI Reports): For visual dashboards.
Structuring Data: The Medallion Architecture
The workshop was designed to function seamlessly for that hence the talk was on the Medallion Architecture, best practice in arranging data into three layers:
Bronze Layer: This is for raw ingestion and history . It accepts CSV, JSON, and TXT files as they are. This was the raw orders and raw customers tables in their retail use case.
Silver Layer: Here the data is filtered, cleansed, and supplemented. The dataset gets cleaned up and silver_orders and silver_customers are considerably more readable.
Gold Layer: This is the aggregate layer of the business level. It has facts and dimensions like gold_fact_orders and gold_customer_metrics, which are suited for BI reporting and Machine Learning.
They showed a retail analytics use case where data was moving from an S3 bucket to ingestion jobs to Bronze S3 table, transformed through Glue to Silver and transformed again to Gold, and then pushed to QuickSight and SageMaker. Inside an S3 Table bucket in the AWS dashboard these managed tables were plainly visible, grouped neatly.
Getting Data In: Choose Your Weapon
You may be asking how we actually get data into these S3 tables? The speakers broke down three good intake approaches using user profile and data size. The best thing? They showed us the code itself.
1. Python (PyIceberg & PyArrow) for Small/Medium Data
If you are running lightweight data pipelines or single-threaded python programs, PyIceberg + PyArrow is your best friend.
First, you’ll need to connect to the S3 Tables catalog. Think of the catalog as the master index that points Python directly to where your data exists. Notice the usage of AWS SigV4 in the settings, which ensures that your connection is safely authenticated with your normal AWS credentials.
from pyiceberg.catalog import load_catalog
catalog = load_catalog(
"s3tables_catalog",
**{
"type": "rest",
"uri": f"https://s3tables.{region}.amazonaws.com/iceberg",
"warehouse": table_bucket_arn,
"rest.sigv4-enabled": "true",
"rest.signing-name": "s3tables",
"rest.signing-region": region,
}
)
Once we connect we are able to construct a namespace (think of it as a folder for our tables) and specify our schema using PyArrow . Imagine a schema as the column headers of an Excel spreadsheet, informing the database what kind of data to expect.
import pyarrow as pa
# Create a namespace
catalog.create_namespace("demo_ns")
# Define exactly what our data looks like
schema = pa.schema([
pa.field("order_id", pa.int32()),
pa.field("customer_name", pa.string()),
pa.field("product", pa.string()),
pa.field("quantity", pa.int32()),
pa.field("price", pa.float64()),
pa.field("status", pa.string()),
])
# Create the empty table
table = catalog.create_table("demo_ns.orders", schema=schema)
Now let’s insert some real data into our table and get it back. The very fascinating thing here is the "predictive pushdown” of the filtered scan. In layman's terms, this means that we are saying to the database, 'just give me the rows where the status is "shipped"' and so we avoid having to download large quantities of irrelevant data.
# Create some sample data
rows = [
{"order_id": 1, "customer_name": "Alice", "product": "Widget A", "quantity": 10, "price": 29.99, "status": "shipped"},
{"order_id": 2, "customer_name": "Bob", "product": "Widget B", "quantity": 5, "price": 49.99, "status": "pending"},
{"order_id": 3, "customer_name": "Charlie", "product": "Widget C", "quantity": 2, "price": 99.99, "status": "shipped"},
]
# Write data to the table
arrow_table = pa.Table.from_pylist(rows, schema=schema)
table.append(arrow_table)
# Filtered scan: Only pull data where status is "shipped"
from pyiceberg.expressions import EqualTo
df_shipped = table.scan(
row_filter=EqualTo("status", "shipped")
).to_pandas()
2. Spark (AWS Glue) for Big Data & ETL
If you want to do large scale heavy duty data transformation then AWS Glue running on Spark is your go-to service. You just inject the Iceberg extensions into your Spark Session and no hard workarounds are needed to make Spark connect to S3 Tables.
This is what that Glue Job configuration looks like:
# ----------------------------------------------------
# Spark Session Configuration for S3 Tables
# ----------------------------------------------------
spark = (
SparkSession.builder
.appName(job_name)
.config(
"spark.sql.extensions",
"org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions",
)
.config(
f"spark.sql.catalog.{catalog_name}",
"org.apache.iceberg.spark.SparkCatalog",
)
.config(
f"spark.sql.catalog.{catalog_name}.catalog-impl",
"software.amazon.s3tables.iceberg.S3TablesCatalog",
)
.config(
f"spark.sql.catalog.{catalog_name}.warehouse",
bucket_arn,
)
.getOrCreate()
)
3. SQL (Amazon Athena) for Analysts
If you are an analyst who prefers SQL, you can work with S3 Tables directly from the Athena console.
For example, running the famous “Time Travel” functionality we talked about above is as simple as adding FOR VERSION AS OF at the end of a normal SQL query. Here’s a sample from the session to show how you may query a historical snapshot of the data:
SELECT * FROM "daily_sales" FOR VERSION AS OF 2246846639951314761;
The "Wow" Moments: ACID, Time Travel, and Schema Evolution
To be fair there are a lot of tools that can shift data around. But S3 Tables adds real database like features to object storage.
ACID Transactions: Now you can execute
UPDATEandDELETEoperations straight in Athena. If a consumer deletes their account, you perform a normal SQL delete command on your data lake.Time Travel & Snapshot Isolation: Just think about finding out you've erased the wrong records by mistake. S3 Tables lets you query data as it was in the past, literally, with a single SQL command:
SELECT * FROM table FOR VERSION AS OF [snapshot_id].Schema Evolution: Business needs are always changing. To add a new column “City” to your database, just run an
ALTER TABLEcommand. It updates instantly without having to re-write the full history dataset.
Turning Data into Business Insights
After all, data is only valuable if it answers business questions. The speakers presented a highly optimized query layer on top of the Gold layer with Amazon Athena. Athena uses partition pruning and is highly optimized for Iceberg, so you only pay for the exact data you scan.
They ran some interesting analytical queries from the actual world:
Monthly Revenue Trend: Aggregating total orders and revenue by month.
Customer Lifetime Value & Segmentation: Classifying Customers to Platinum, Gold, Silver and Bronze levels depending on their total spend.
Repeat vs. One-Time Buyers: Behavior analysis to determine how many customers return vs. buy once.
These SQL queries were then wonderfully presented using Amazon QuickSight dashboards to generate pie charts and bar graphs that business stakeholders could truly utilize.
The Future is Here: Claude + MCP Server
Long story short, writing SQL can still be a bottleneck for non-technical people. The speakers thrilled the audience by displaying an integration with Anthropic’s Claude AI with an MCP (Model Context Protocol) server.
They set up a local MCP server called s3tablesagent which enabled Claude to securely access the S3 Tables data warehouse. In the chat they just wrote: "What are the top 10 customers by total revenue and what is their average order value?".
Claude got the intent, queried the Gold tables in the background, and sent back a neatly prepared markdown table with significant business insights, such as “Customer 92 combines both strategies well - fewer orders but a strong average order value”. It was like having a senior data analyst sitting inside the chat window!
Key Takeaways
Here’s a quick summary of what you need to remember based on the summary given at the end of the session:
The Best of Both Worlds: The Lakehouse architecture (Amazon S3 Tables and Apache Iceberg) combines the huge volume of a data lake with the dependability and performance of a traditional data warehouse.
Zero Maintenance: Iceberg administration is simple, because Amazon S3 Tables takes care of metadata, compaction and table optimization behind the scenes.
Massive Scale without Servers: AWS Glue lets you develop distributed, large-scale ETL pipelines without ever handling the underlying infrastructure.
Serverless SQL: Amazon Athena enables data analysts to query big datasets immediately using conventional SQL without having to manage clusters.
Better Organization: By organizing your data, the Medallion Architecture (Bronze, Silver, and Gold layers) will increase your data reliability, management, and ability to prepare for analytics.
Faster Decisions: You can also build Amazon Athena and QuickSight on top of handpicked datasets, to get powerful business insights and dashboards, much faster.
Conclusion
The conclusion is as follows: The new data architecture is evolving rapidly and AWS makes it very accessible. This session was a masterclass on updating your analytics platforms.
Vishali and Vikneshwara performed an amazing job solving complicated, headache producing data engineering difficulties and giving a clean, automated and highly scalable approach. Are you exhausted of wrestling with complicated data pipelines, strict schemas, and small file problems? Amazon S3 Tables might be the tool you have been waiting for.
About the Author
As an AWS Community Builder, I enjoy sharing the things I've learned through my own experiences and events, and I like to help others on their path. If you found this helpful or have any questions, don't hesitate to get in touch! 🚀
🔗 Connect with me on LinkedIn
References
Event: AWS Community Day Chennai
Topic: From Data Lake to Lakehouse: Building Modern Analytics Platforms on AWS with S3 Tables & AWS Glue
Date: March 7, 2026
Top comments (0)