DEV Community

Muhammad Umair Ashraf
Muhammad Umair Ashraf

Posted on

AWS + Snowflake: Building Modern Data and AI Workflows with Glue, S3, Iceberg, and Cortex

#aws #snowflake #dataengineering #cloud

Modern data platforms are no longer just about moving data from one database to another. Organizations need scalable data pipelines, governed data access, open table formats, real-time analytics, and AI-powered insights.

Two platforms that play an important role in this ecosystem are Amazon Web Services (AWS) and Snowflake.

AWS provides a broad ecosystem for storage, data integration, security, analytics, and machine learning. Snowflake provides a cloud data platform for data warehousing, data sharing, analytics, and AI-powered applications.

The interesting part is how these platforms can work together without forcing organizations to move all their data into a single environment.

In this article, we'll explore important AWS and Snowflake integration capabilities and build a practical mini-project that moves sales data from Amazon S3 through AWS Glue into Snowflake.

1. Why Combine AWS and Snowflake?

Many organizations already store large volumes of data in Amazon S3 while using Snowflake for analytics and business intelligence.

Combining these platforms can help teams:

  • Build scalable ETL and ELT pipelines.
  • Process data using AWS Glue and Apache Spark.
  • Store raw and curated datasets in Amazon S3.
  • Analyze structured data in Snowflake.
  • Improve data governance and access control.
  • Work with open table formats such as Apache Iceberg.
  • Build AI-powered analytics workflows.

A simplified architecture looks like this:

        Data Sources
             |
             v
        Amazon S3
       (Raw Data)
             |
             v
         AWS Glue
      (ETL Processing)
             |
             v
        Snowflake
     (Data Warehouse)
             |
             v
      SQL Analytics
      BI / AI Apps
Enter fullscreen mode Exit fullscreen mode

Each service has a specific responsibility, allowing teams to build flexible data architectures.

2. AWS Glue Catalog Federation with Snowflake Horizon Catalog

One important development is the ability to access Snowflake Horizon Catalog metadata through AWS Glue catalog federation.

This is particularly useful when organizations work with Apache Iceberg tables and want to discover and query data across platforms.

Traditionally, integrating separate data catalogs can require additional metadata synchronization and operational complexity.

Catalog federation helps reduce that friction.

How it works

  1. Snowflake manages catalog metadata through Horizon Catalog.
  2. AWS Glue connects to the federated catalog.
  3. AWS Lake Formation can help manage permissions.
  4. Amazon Athena can query supported federated Iceberg data.

The key idea is that teams can access catalog metadata without treating every dataset as a separate copy that must be moved into another system.

Why it matters:

  • Less metadata duplication.
  • Better interoperability.
  • Centralized governance options.
  • More flexibility for analytics teams.
  • Easier access to Iceberg-based data.

Always verify the supported authentication method, catalog configuration, and table compatibility for your specific environment.

3. AWS Glue and Snowflake Connectivity

AWS Glue can connect to Snowflake to support ETL workloads.

A common pattern is:

  1. Read source data from Amazon S3.
  2. Transform records using AWS Glue and PySpark.
  3. Validate and clean the data.
  4. Write curated records into Snowflake.
  5. Run analytical SQL queries.

Authentication and credential management are important parts of this integration.

For production environments, prefer supported OAuth-based authentication or securely managed credentials over hardcoding passwords in scripts.

Keep credentials in an appropriate secret-management solution and grant only the permissions required by the job.

4. Amazon Quick and Snowflake Cortex AI

Another interesting direction is the integration of Amazon Quick with Snowflake Cortex AI through the Model Context Protocol (MCP).

This brings conversational analytics closer to business users.

Instead of manually writing every query, users can ask questions about their data using natural language.

For example:

  • Which products generated the highest revenue?
  • What were the monthly sales trends?
  • Which regions experienced declining sales?
  • Which customers contributed the most revenue?

Snowflake Cortex capabilities can support different AI workflows, including structured-data analysis, document search, and agent-based interactions.

Amazon Quick can provide a conversational interface for interacting with supported Snowflake capabilities.

This creates opportunities for more accessible analytics while keeping data access and governance important parts of the architecture.

5. Apache Iceberg: Connecting Open Data Across Platforms

Apache Iceberg is an open table format designed for large analytical datasets.

It provides capabilities such as:

  • Schema evolution.
  • Partition evolution.
  • Snapshot-based queries.
  • Time travel.
  • Reliable metadata management.

For organizations using both AWS and Snowflake, Iceberg can help reduce dependence on proprietary storage formats and improve interoperability.

A possible architecture is:

             Amazon S3
                 |
                 v
        Apache Iceberg Tables
                 |
          +------+------+
          |             |
          v             v
       AWS Glue      Snowflake
          |             |
          v             v
       Athena       Analytics
Enter fullscreen mode Exit fullscreen mode

The exact integration pattern depends on catalog ownership, permissions, supported table features, and the query engine being used.

6. Hands-On Mini Project: Build an S3-to-Snowflake Sales Pipeline

Let's build a simple data engineering project.

Project objective: Read sales data from Amazon S3, transform it using AWS Glue, load it into Snowflake, and run analytical queries.

Architecture 

       sales.csv
           |
           v
       Amazon S3
           |
           v
       AWS Glue Job
       (PySpark ETL)
           |
           v
       Snowflake
       SALES_DB
           |
           v
       SALES_ANALYTICS
Enter fullscreen mode Exit fullscreen mode

Step 1: Create a sample CSV file

Create a file named sales.csv:

order_id,customer,product,quantity,price,region
1001,Ali,Laptop,1,1200,Asia
1002,Sara,Mobile,2,600,Europe
1003,Ahmed,Tablet,1,400,Asia
1004,Ayesha,Laptop,2,1200,North America
1005,Usman,Mobile,1,600,Asia
1006,Hina,Monitor,2,300,Europe
Enter fullscreen mode Exit fullscreen mode

Step 2: Upload the file to Amazon S3

Create an S3 bucket and upload the CSV file.

Example S3 path:

s3://my-data-engineering-bucket/raw/sales.csv
Enter fullscreen mode Exit fullscreen mode

Your bucket should contain:

my-data-engineering-bucket/
    |
    └── raw/
         └── sales.csv
Enter fullscreen mode Exit fullscreen mode

Step 3: Create a Snowflake database and table

Run the following SQL in Snowflake:

CREATE DATABASE SALES_DB;

CREATE SCHEMA SALES_DB.ANALYTICS;

CREATE TABLE SALES_DB.ANALYTICS.SALES (
    ORDER_ID NUMBER,
    CUSTOMER VARCHAR(100),
    PRODUCT VARCHAR(100),
    QUANTITY NUMBER,
    PRICE NUMBER(10,2),
    REGION VARCHAR(100),
    TOTAL_AMOUNT NUMBER(12,2)
);
Enter fullscreen mode Exit fullscreen mode

Step 4: Configure AWS Glue

Before running the job:

  • Create an AWS Glue IAM role.
  • Grant the role access to the required S3 location.
  • Configure the supported Snowflake connector.
  • Provide Snowflake connection details securely.
  • Confirm network connectivity and permissions.
  • Ensure the Glue job has the required connector dependencies.

The exact connector configuration depends on your AWS Glue version and the Snowflake connector setup you choose.

Step 5: Transform the data using PySpark

The following example demonstrates the transformation logic.

from pyspark.sql import SparkSession
from pyspark.sql.functions import col

spark = SparkSession.builder \
    .appName("SalesETLPipeline") \
    .getOrCreate()

# Read CSV from Amazon S3
df = spark.read \
    .option("header", True) \
    .option("inferSchema", True) \
    .csv("s3://my-data-engineering-bucket/raw/sales.csv")

# Calculate total sales amount
df = df.withColumn(
    "TOTAL_AMOUNT",
    col("quantity") * col("price")
)

# Remove records with missing required values
df = df.dropna(
    subset=["order_id", "customer", "product", "quantity", "price"]
)

# Display transformed data
df.show()

# Inspect schema
df.printSchema()
Enter fullscreen mode Exit fullscreen mode

This example focuses on the transformation itself. To complete the pipeline, configure the Snowflake connector's supported Spark write integration and connection options, then write the transformed DataFrame into SALES_DB.ANALYTICS.SALES.

Do not assume that a generic JDBC or connector configuration will work unchanged across every Glue runtime. Follow the connector documentation for the selected runtime.

Step 6: Run analytical queries in Snowflake

Once the transformed records are loaded, run:

Total revenue

SELECT
    SUM(TOTAL_AMOUNT) AS TOTAL_REVENUE
FROM SALES_DB.ANALYTICS.SALES;
Enter fullscreen mode Exit fullscreen mode

Revenue by region

SELECT
    REGION,
    SUM(TOTAL_AMOUNT) AS REVENUE
FROM SALES_DB.ANALYTICS.SALES
GROUP BY REGION
ORDER BY REVENUE DESC;
Enter fullscreen mode Exit fullscreen mode

Top-selling products by revenue

SELECT
    PRODUCT,
    SUM(TOTAL_AMOUNT) AS TOTAL_SALES
FROM SALES_DB.ANALYTICS.SALES
GROUP BY PRODUCT
ORDER BY TOTAL_SALES DESC;
Enter fullscreen mode Exit fullscreen mode

Customer purchase analysis

SELECT
    CUSTOMER,
    COUNT(*) AS TOTAL_ORDERS,
    SUM(TOTAL_AMOUNT) AS TOTAL_SPENT
FROM SALES_DB.ANALYTICS.SALES
GROUP BY CUSTOMER
ORDER BY TOTAL_SPENT DESC;
Enter fullscreen mode Exit fullscreen mode

You now have a basic cloud data pipeline using Amazon S3, AWS Glue, PySpark, and Snowflake.

7. Production Architecture Considerations

A working demo is only the first step. Production data platforms require additional planning.

Security

  • Use IAM roles with least-privilege permissions.
  • Store credentials securely.
  • Prefer supported OAuth-based authentication where appropriate.
  • Encrypt data at rest and in transit.
  • Apply Snowflake role-based access control.
  • Use AWS Lake Formation where applicable.
  • Avoid exposing credentials in Glue scripts or logs.

Data Quality

  • Validate incoming schemas.
  • Handle null and duplicate records.
  • Track rejected records.
  • Implement data reconciliation checks.
  • Monitor failed and incomplete loads.

Performance

  • Use appropriate file formats and partitioning.
  • Avoid unnecessary Spark shuffles.
  • Tune Glue worker capacity based on workload.
  • Use Snowflake warehouse sizing appropriate to query demand.
  • Monitor query execution and data transfer patterns.

Cost Optimization

  • Avoid running oversized Glue jobs.
  • Use appropriate Snowflake warehouse auto-suspend settings.
  • Monitor S3 storage and request costs.
  • Track data movement between AWS and Snowflake.
  • Use incremental processing instead of repeatedly loading entire datasets.

8. When Should You Use Each Integration?

Requirement Possible approach
Process data in S3 AWS Glue
Query S3 data Amazon Athena
Centralize AWS data permissions AWS Lake Formation
Enterprise data warehousing Snowflake
Open table interoperability Apache Iceberg
Conversational analytics Amazon Quick + Snowflake Cortex AI
Cross-platform catalog discovery Glue catalog federation

The best architecture depends on where your data lives, which platform owns the catalog, your governance requirements, and the workloads you need to support.

9. Final Thoughts

AWS and Snowflake are not necessarily competing choices. They can complement each other in a modern data architecture.

AWS provides scalable infrastructure, data ingestion, transformation, storage, and governance capabilities. Snowflake provides a powerful environment for analytical workloads, data sharing, and AI-powered data experiences.

With catalog federation, Apache Iceberg interoperability, and conversational analytics integrations, organizations have more options for building connected data platforms.

The practical takeaway is simple: choose the right service for each workload, keep governance at the center, and avoid unnecessary data movement.

Start with a small pipeline, measure its performance and cost, and expand the architecture as your data requirements grow.

References and Further Reading

If you're building data pipelines with AWS and Snowflake, try the mini-project and extend it with incremental loads, data-quality checks, and automated orchestration.

Top comments (0)