DEV Community

AlpeshKumbhare
AlpeshKumbhare

Posted on

Building a Modern Data Lakehouse on AWS: S3, Iceberg, Glue, Athena, and Lake Formation

#aws #data #architecture #analytics

The data lakehouse has become the default architecture for analytics on AWS in 2026. It combines the best of both worlds: the low-cost, schema-flexible storage of a data lake (S3) with the performance, ACID transactions, and governance of a data warehouse — without actually running a warehouse for most workloads.

The enabling technology: Apache Iceberg — an open table format that brings SQL-like capabilities (INSERT, UPDATE, DELETE, time travel) to files sitting in S3. Combined with AWS Glue for ETL, Athena for queries, and Lake Formation for governance, you get a complete analytics platform without managing servers.

Data Lake vs Data Warehouse vs Lakehouse 

┌──────────────────┐  ┌──────────────────┐  ┌──────────────────────────┐
│   DATA LAKE      │  │  DATA WAREHOUSE  │  │     DATA LAKEHOUSE       │
│                  │  │                  │  │                          │
│ ✅ Cheap storage │  │ ✅ Fast queries   │  │ ✅ Cheap storage (S3)    │
│ ✅ Schema-on-read│  │ ✅ ACID txns     │  │ ✅ Fast queries (Iceberg)│
│ ✅ Any format    │  │ ✅ Governance    │  │ ✅ ACID transactions     │
│ ❌ No ACID       │  │ ❌ Expensive     │  │ ✅ Governance (LF)       │
│ ❌ No updates    │  │ ❌ Vendor lock-in│  │ ✅ Open format           │
│ ❌ Stale data    │  │ ❌ Schema-rigid  │  │ ✅ Schema evolution      │
│                  │  │                  │  │ ✅ Time travel           │
│  (S3 + Parquet)  │  │  (Redshift)      │  │ (S3 + Iceberg + Athena) │
└──────────────────┘  └──────────────────┘  └──────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

The AWS Lakehouse Architecture 

┌─────────────────────────────────────────────────────────────────────┐
│                         CONSUMPTION                                   │
│  Athena (SQL) | Redshift Spectrum | EMR (Spark) | QuickSight (BI)   │
├─────────────────────────────────────────────────────────────────────┤
│                         GOVERNANCE                                    │
│  Lake Formation (permissions, audit) | Glue Data Catalog (metadata) │
├─────────────────────────────────────────────────────────────────────┤
│                         TABLE FORMAT                                  │
│  Apache Iceberg (ACID, time travel, schema evolution, compaction)    │
├─────────────────────────────────────────────────────────────────────┤
│                         PROCESSING                                    │
│  Glue ETL (Spark) | Glue Streaming | EMR | Zero-ETL | Firehose     │
├─────────────────────────────────────────────────────────────────────┤
│                         INGESTION                                     │
│  Kinesis | DMS | AppFlow | S3 Transfer | Direct PUT                 │
├─────────────────────────────────────────────────────────────────────┤
│                         STORAGE                                       │
│  Amazon S3 (raw / curated / analytics zones)                         │
└─────────────────────────────────────────────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

Apache Iceberg: The Game Changer

Iceberg is an open table format that sits between your query engine and S3 files. It enables data warehouse capabilities on data lake storage:

What Iceberg Adds to S3

Capability Without Iceberg (Parquet on S3) With Iceberg
INSERT/UPDATE/DELETE ❌ Append-only ✅ Row-level operations
ACID transactions ❌ Partial writes possible ✅ Atomic commits
Schema evolution ❌ Break consumers ✅ Add/rename/drop columns safely
Time travel ❌ No history ✅ Query any past snapshot
Partition evolution ❌ Rewrite all data ✅ Change partitioning without rewrite
Hidden partitioning ❌ User must know partition scheme ✅ Engine handles automatically
Concurrent writes ❌ Last write wins ✅ Optimistic concurrency control

Iceberg on AWS: Service Support

Service Iceberg Support
Athena Full read/write (CREATE TABLE, INSERT, UPDATE, DELETE, MERGE)
Glue ETL Full read/write via Spark connector
EMR Full support (Spark, Trino, Flink)
Redshift Spectrum Read access to Iceberg tables
Glue Data Catalog Native Iceberg catalog integration
Lake Formation Fine-grained access control on Iceberg tables

Creating an Iceberg Table (Athena) 

CREATE TABLE analytics.orders (
    order_id STRING,
    customer_id STRING,
    amount DECIMAL(10,2),
    status STRING,
    order_date TIMESTAMP,
    region STRING
)
PARTITIONED BY (region, month(order_date))
LOCATION 's3://my-lakehouse/analytics/orders/'
TBLPROPERTIES ('table_type' = 'ICEBERG');
Enter fullscreen mode Exit fullscreen mode

Note: month(order_date) is hidden partitioning — queries don't need to know the partition scheme. Iceberg handles pruning automatically.

Row-Level Operations 

-- Update order status
UPDATE analytics.orders
SET status = 'shipped'
WHERE order_id = 'ORD-12345';

-- Delete cancelled orders older than 1 year
DELETE FROM analytics.orders
WHERE status = 'cancelled' AND order_date < current_timestamp - interval '1' year;

-- Merge (upsert) from staging
MERGE INTO analytics.orders target
USING staging.new_orders source
ON target.order_id = source.order_id
WHEN MATCHED THEN UPDATE SET status = source.status, amount = source.amount
WHEN NOT MATCHED THEN INSERT (order_id, customer_id, amount, status, order_date, region)
VALUES (source.order_id, source.customer_id, source.amount, source.status, source.order_date, source.region);
Enter fullscreen mode Exit fullscreen mode

Time Travel 

-- Query data as it was 24 hours ago
SELECT * FROM analytics.orders
FOR TIMESTAMP AS OF (current_timestamp - interval '24' hour);

-- Query a specific snapshot
SELECT * FROM analytics.orders
FOR VERSION AS OF 12345678901234;

-- View snapshot history
SELECT * FROM "analytics"."orders$snapshots";
Enter fullscreen mode Exit fullscreen mode

Data Lake Zones: Organizing Your S3 Storage 

s3://company-lakehouse/
├── raw/                    ← Landing zone (source format, immutable)
│   ├── crm/
│   ├── erp/
│   └── clickstream/
│
├── curated/                ← Cleaned, validated, Iceberg format
│   ├── customers/
│   ├── orders/
│   └── products/
│
└── analytics/              ← Aggregated, business-ready
    ├── daily_revenue/
    ├── customer_360/
    └── product_performance/
Enter fullscreen mode Exit fullscreen mode
Zone Format Purpose Access
Raw Source format (JSON, CSV, Parquet) Immutable landing zone, audit trail Data engineers only
Curated Iceberg (optimized Parquet) Cleaned, validated, single source of truth Data engineers + analysts
Analytics Iceberg (aggregated) Business-ready datasets, dashboards Analysts + BI tools

AWS Glue: The ETL Engine

Glue Components

Component Purpose
Glue Data Catalog Centralized metadata (databases, tables, schemas) — the "card catalog"
Glue ETL Jobs Spark-based data transformation (Python/Scala)
Glue Crawlers Auto-discover schemas from S3 data, populate catalog
Glue Studio Visual ETL designer (no-code/low-code)
Glue Streaming Near-real-time ETL from Kinesis/Kafka/MSK
Glue Data Quality Define and enforce quality rules on datasets

Glue ETL Job: Raw → Curated (Iceberg) 

import sys
from awsglue.transforms import *
from awsglue.context import GlueContext
from pyspark.context import SparkContext

sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session

# Read raw data
raw_df = spark.read.json("s3://company-lakehouse/raw/orders/2026/08/25/")

# Transform: clean, validate, enrich
curated_df = (raw_df
    .filter(raw_df.order_id.isNotNull())
    .withColumn("order_date", to_timestamp("order_date_str"))
    .withColumn("amount", col("amount").cast("decimal(10,2)"))
    .drop("order_date_str", "_metadata")
)

# Write to Iceberg table (append or merge)
curated_df.writeTo("glue_catalog.curated.orders") \
    .using("iceberg") \
    .append()
Enter fullscreen mode Exit fullscreen mode

Lake Formation: Governance Layer

Lake Formation provides fine-grained access control — column-level and row-level security on your lakehouse:

Permission Model 

Lake Formation Permissions:
├── Database-level: Who can see which databases?
├── Table-level: Who can query which tables?
├── Column-level: Who can see which columns? (mask PII)
└── Row-level: Who can see which rows? (data filters by region/team)
Enter fullscreen mode Exit fullscreen mode

Row-Level Security Example 

Policy: "EU Data Residency"
├── Principal: EU-Analytics-Team
├── Table: curated.customers
├── Filter: region IN ('eu-west-1', 'eu-central-1')
└── Effect: Team can ONLY see EU customers
Enter fullscreen mode Exit fullscreen mode

Tag-Based Access Control (LF-TBAC)

Instead of managing permissions per table, assign tags and grant access by tag:

Tag: classification = "pii"
  → Applied to: customers.email, customers.phone, orders.billing_address
  → Grant: Only "PII-Authorized" group can see these columns
  → Everyone else: columns masked or hidden
Enter fullscreen mode Exit fullscreen mode

Amazon Athena: Serverless Queries

Athena queries Iceberg tables directly on S3 — no infrastructure to manage:

  • Serverless — pay per query ($5 per TB scanned)
  • Federated queries — query RDS, DynamoDB, Redshift alongside S3 in one SQL statement
  • Prepared statements — parameterized queries for applications
  • Workgroups — separate teams with cost controls and query limits

Cost Optimization for Athena

Technique Savings
Columnar format (Parquet/ORC via Iceberg) 30-90% less data scanned
Partition pruning (Iceberg hidden partitioning) Scan only relevant partitions
Compression (Snappy/ZSTD) 50-70% less storage + scan cost
CTAS for materialized views Pre-compute expensive joins
Workgroup byte limits Prevent runaway queries

Zero-ETL Integrations

AWS is pushing "zero-ETL" — direct integration between operational databases and analytics without building ETL pipelines:

Source Destination What It Does
Aurora → Redshift Zero-ETL Near-real-time replication without Glue jobs
DynamoDB → OpenSearch Zero-ETL Automatic sync for search/analytics
DynamoDB → Redshift Zero-ETL Export DynamoDB data for analytics
RDS → S3 (via DMS) CDC Change data capture for lakehouse ingestion

When Zero-ETL vs Glue ETL

Use Zero-ETL when... Use Glue ETL when...
Source → destination without transformation Need data cleansing, validation, enrichment
Supported source/destination pair Custom transformation logic
Minimal latency requirements Complex multi-source joins
Simple replication Business rule application

Real-Time Lakehouse Pattern

For near-real-time analytics, combine streaming ingestion with Iceberg:

App Events → Kinesis Data Streams → Glue Streaming ETL → Iceberg Table
                                                              │
                                              Athena (queries latest data)
Enter fullscreen mode Exit fullscreen mode

Glue Streaming writes micro-batches to Iceberg every 1-5 minutes. Athena queries see near-real-time data without separate real-time infrastructure.

Table Maintenance (Iceberg Housekeeping)

Iceberg tables need periodic maintenance for optimal performance:

Operation What It Does Frequency
Compaction Merge small files into larger ones (better query performance) Daily
Expire snapshots Remove old snapshots (reduce metadata overhead + storage) Weekly
Remove orphan files Delete files not referenced by any snapshot Weekly
Rewrite manifests Optimize manifest file layout As needed

Athena OPTIMIZE (Compaction) 

-- Compact small files for better query performance
OPTIMIZE analytics.orders REWRITE DATA USING BIN_PACK;

-- Expire old snapshots (keep last 7 days)
ALTER TABLE analytics.orders SET TBLPROPERTIES (
    'vacuum_min_snapshots_to_keep' = '10',
    'vacuum_max_snapshot_age_seconds' = '604800'
);
VACUUM analytics.orders;
Enter fullscreen mode Exit fullscreen mode

Common Mistakes

Mistake Problem Fix
No partitioning strategy Full table scans on every query Use Iceberg hidden partitioning (by date, region)
Too many small files Slow queries, high S3 API costs Regular compaction (OPTIMIZE)
No snapshot expiry Storage costs grow forever VACUUM old snapshots weekly
Raw zone with no schema validation Bad data propagates downstream Glue Data Quality rules at ingestion
Everyone queries raw zone directly Inconsistent results, no governance Force queries through curated/analytics zones
Column-level security as afterthought PII exposed to unauthorized users Design Lake Formation tags from day one
Redshift for everything Expensive for exploratory queries Athena for ad-hoc, Redshift for heavy BI only

Summary

The modern AWS lakehouse in 2026:

  1. Storage: S3 (cheap, durable, infinitely scalable)
  2. Table format: Apache Iceberg (ACID, time travel, schema evolution, hidden partitioning)
  3. Catalog: Glue Data Catalog (centralized metadata for all engines)
  4. Processing: Glue ETL (batch + streaming) or Zero-ETL (for supported pairs)
  5. Query: Athena (serverless SQL, pay per scan) + Redshift Spectrum (for heavy BI)
  6. Governance: Lake Formation (column/row-level security, tag-based access)
  7. BI: QuickSight (serverless dashboards connected to Athena/Redshift)

The key shift: You don't need a data warehouse for most analytics anymore. Iceberg on S3 + Athena gives you warehouse-like capabilities at data lake prices. Add Redshift only when you need sub-second complex aggregations on petabytes.

Alpesh Kumbhare is an AWS Architect at Atos, specializing in AWS data architecture and cloud infrastructure automation. Connect on LinkedIn.

Top comments (0)