How I implemented ETL Pipeline Using AWS Glue

- Step 1: I considered:

1. Spark on EC2 (high control, high ops)
2. Databricks
3. AWS Glue I selected AWS Glue to minimize operational complexity.

- Step 2: Ingestion Strategy

1. Data lands in raw/
2. Glue Crawlers detect schema changes
3. Catalog updated automatically

- Step 3: Transformation Logic
Glue Jobs perform:

1. Type casting
2. Null handling
3. Deduplication
4. Format conversion (CSV → Parquet)

- Step 4: Performance Optimization

1. Enabled job bookmarks
2. Tuned DPUs
3. Used Parquet + Snappy compression

- Step 5: Output Strategy

1. Processed data written to S3