DEV Community: Aki

Rethinking Lakehouse Architecture Through Data Ownership: AWS vs. Snowflake

Aki — Mon, 01 Jun 2026 13:31:04 +0000

Original Japanese article: データの主導権から考えるAWSとSnowflakeのレイクハウスアーキテクチャ

Introduction

I'm Aki, an AWS Community Builder (@jitepengin).
When designing a data platform, discussions about whether to lean toward AWS or Snowflake are still very common.

However, with the rise of Apache Iceberg, data and platforms can now be decoupled. Because of this shift, I believe we need to reconsider the question itself.

Rather than asking:

Should we build around AWS or Snowflake?

A more fundamental question is:

Who owns the data?

In this article, I'd like to define what I mean by data ownership and explore the architectural trade-offs of AWS-centric and Snowflake-centric lakehouse designs.

Why Data Ownership Matters

Apache Iceberg has made it possible to separate data from the platform that accesses it.

Today, an Iceberg table stored on Amazon S3 can be accessed from Athena, Snowflake, Spark, and many other engines. As a result, choosing a product is becoming less important than deciding who is responsible for managing the data.

Before diving into architectural patterns, let's first examine why this shift matters.

Defining Ownership Across Three Layers

In this article, I define data ownership through the following three layers:

Layer	Question	Example
Catalog Ownership	Who owns the metadata?	Glue Data Catalog / Snowflake Open Catalog
Write Ownership	Who can update or delete data?	Glue ETL / Snowflake DML
Governance Ownership	Who controls access policies?	Lake Formation / Snowflake Horizon

Only when these three layers are consistently controlled by the same authority can we truly say that ownership exists.

Conversely, when ownership is distributed or unclear, complexity tends to emerge in architecture, operations, and security.

The Reality of Vendor Lock-In

Even in the Iceberg era, platform dependencies have not disappeared—they have simply changed form.

Catalog dependency: Tables managed by Snowflake Open Catalog still rely operationally on a Snowflake-managed service, although external engines can access them through the REST Catalog API.
Write-engine dependency: Snowflake-managed Iceberg tables are primarily updated through Snowflake, though Horizon Catalog now supports external writes from engines such as Spark. The choice of write engine remains closely tied to catalog design.
Governance dependency: Lake Formation's fine-grained permissions are fundamentally tied to the AWS ecosystem.

Therefore, saying that "Iceberg eliminates vendor lock-in" is only partially true.

What Iceberg removes is storage-format lock-in.

Dependencies around catalog management, governance, and operational processes still remain. In practice, migrating a data platform involves challenges such as governance policies, access control, metadata management, and platform-specific features.

Extensibility and Strategic Flexibility

Data platforms are never finished.

The rapid evolution of AI technologies and the continuous changes in the Modern Data Stack mean that architectures must adapt over time.

Common examples include:

Adding or changing analytics tools
Athena may be sufficient initially, but business users may later request Snowflake access.
Introducing AI workloads
Integration with SageMaker or Snowflake Cortex AI may become necessary.
Cost optimization initiatives
As query volumes grow, Snowflake compute costs may become significant, leading teams to move batch processing to EMR.
Stronger governance requirements
Column masking or row-level security may need to be introduced later.

When ownership across the three layers is clearly defined from the beginning, these changes become easier to evaluate and implement.

Without that clarity, every change raises new questions about where responsibilities and controls should reside.

What Changed After Iceberg?

Historically, data and platforms were tightly coupled.

	Snowflake-Centric	AWS-Centric
Data Location	Inside Snowflake	Inside S3
Management Ownership	Snowflake owns everything	AWS owns everything
Access from Other Engines	Not possible	Snowflake could not access directly

Iceberg fundamentally changed this model.

Iceberg Tables on S3
        ↓
Shared by Multiple Engines

Athena / Glue / Snowflake / Spark / Redshift ...

Iceberg adds a metadata layer on top of Parquet files stored in object storage, enabling ACID transactions and schema evolution independent of any specific compute engine.

A catalog tracks metadata such as schemas and active data files, allowing multiple engines to safely access the same table.

Data files are now shareable.

However, ownership of the catalog, write operations, and governance still depends on architectural decisions.

In other words, deciding who manages the catalog effectively determines who owns the data.

Major Iceberg Catalog Options

Catalog Type Characteristics

AWS Glue Data Catalog AWS-managed Supports REST Catalog API and integrates with Lake Formation governance

Snowflake Open Catalog Snowflake-managed (based on Apache Polaris) REST Catalog compliant and accessible from Spark, Trino, and others

Snowflake Horizon Catalog Snowflake service Exposes Snowflake-managed Iceberg tables through APIs; differs from Open Catalog because it is not a standalone metadata store

Catalog	Type	Characteristics
AWS Glue Data Catalog	AWS-managed	Supports REST Catalog API and integrates with Lake Formation governance
Snowflake Open Catalog	Snowflake-managed (based on Apache Polaris)	REST Catalog compliant and accessible from Spark, Trino, and others
Snowflake Horizon Catalog	Snowflake service	Exposes Snowflake-managed Iceberg tables through APIs; differs from Open Catalog because it is not a standalone metadata store

Snowflake-Centric Architecture

Characteristics

In this approach, Snowflake becomes the center of catalog management, governance, and analytics, while data files remain in external object storage such as S3.

This model prioritizes simplicity and a streamlined analytics experience.

Ownership Model

Layer	Owner
Catalog Ownership	Snowflake Open Catalog or Horizon Catalog
Write Ownership	Primarily Snowflake DML
Governance Ownership	Snowflake Horizon

Although data files remain on S3, external engines can access Snowflake-managed Iceberg tables through two mechanisms:

Via Open Catalog: Snowflake-managed Iceberg tables are synced to Open Catalog and exposed through the REST Catalog API. In this sync scenario, external engines have read-only access. (Note: when Open Catalog itself is used as an internal catalog, read/write access is supported.)
Via Horizon Catalog: Tables are exposed directly through the Horizon Iceberg REST Catalog API without syncing to Open Catalog. External engines can both read and write, and existing Snowflake users and roles can be used for access control.

Benefits

Governance policies such as column masking and row-level security can be applied to Iceberg tables in the same way as native Snowflake tables. When external engines access tables through Horizon Catalog, the same policies are enforced at read time. Note, however, that writing to tables with masking policies or tags applied is not supported from external engines — this is an important constraint to be aware of.
Rich ecosystem support for BI tools such as Power BI makes Snowflake a convenient analytics front end.
External engines can access Iceberg tables through Open Catalog or Horizon Catalog while reusing Snowflake users and roles as the unit of access control.

Drawbacks

Snowflake warehouse compute costs can be significant for write-heavy workloads. When external engines such as Spark write through Horizon Catalog, Snowflake warehouses are not used — but Horizon Catalog API calls are billed at 0.5 credits per million requests, so cost planning is still required.
Coordination is needed when AWS services such as Glue ETL also write to the same datasets. Clearly defining who holds catalog ownership is essential.

Even with Iceberg, many enterprises ultimately converge on a Snowflake-centric operating model because governance, metadata, and write operations all remain concentrated within Snowflake.

In such cases, Iceberg provides openness in theory, but ownership remains firmly within the Snowflake ecosystem.

AWS-Centric Architecture

Characteristics

This architecture uses S3 for storage, Glue Data Catalog for metadata, and AWS-native services for ETL, analytics, and governance.

Its primary advantages are flexibility and service interoperability.

Ownership Model

Layer	Owner
Catalog Ownership	AWS Glue Data Catalog
Write Ownership	Glue ETL / EMR
Governance Ownership	Lake Formation

Because Glue Data Catalog supports the Iceberg REST Catalog API, external engines such as Snowflake and Databricks can access the same tables.

This enables AWS to retain ownership while allowing Snowflake to serve as an analytics front end.

Benefits

Tight integration across Athena, Glue, EMR, and Redshift with a shared catalog.
Fine-grained column- and row-level governance through Lake Formation, applicable to Iceberg tables.
Ability to optimize compute engines for different workloads — EMR for large-scale batch, Athena for interactive queries.

Drawbacks

Increased architectural and operational complexity due to the number of AWS services involved.
Additional design considerations for multi-cloud environments, as the catalog remains AWS-dependent.

Lake Formation is powerful, but troubleshooting permission issues can become challenging. Identifying why a specific user cannot access a specific table or row often takes considerable time, requiring mature operational practices and careful permission design.

Combining AWS and Snowflake

A realistic approach is not choosing one platform over the other, but assigning clear responsibilities to each.

The key is defining ownership boundaries upfront.

AWS Owns the Data, Snowflake Powers Analytics

This is one of the most common patterns.

The goal is to maintain data ownership within AWS while leveraging Snowflake's analytics capabilities and its rich ecosystem of BI connectors.

┌──────────────────────────────────────────────────┐
│                       AWS                        │
│  S3 (Iceberg data files)                         │
│  Glue Data Catalog (Catalog Ownership)           │
│  Lake Formation (Governance Ownership)           │
│  Glue / EMR (Write Ownership)                    │
└──────────────────────┬───────────────────────────┘
                       │ Iceberg REST Catalog API
        ┌──────────────┼───────────────────┐
        ▼              ▼                   ▼
     Athena          Glue              Snowflake
  (Interactive)     (ETL)            (Analytics)

In this model:

Layer	Owner
Catalog Ownership	AWS
Write Ownership	AWS
Governance Ownership	AWS

Snowflake acts primarily as an analytical interface.

Two variations exist:

1. Glue Catalog Integration (Read-Only)

Snowflake accesses AWS-managed Iceberg tables through External Iceberg Tables. Write ownership and governance remain entirely with AWS. Lake Formation can be used as the single source of truth for access control.

2. Catalog-Linked Database (Read/Write)

Snowflake can update Iceberg tables through the Iceberg REST Catalog API while the data remains stored on S3. This approach is attractive when analysts and AI workloads primarily operate in Snowflake.

However, governance responsibilities become shared between AWS and Snowflake. Both Lake Formation and Snowflake-side access controls must be configured carefully — a misconfiguration in either can become a security gap. If the read-only pattern (option 1) is sufficient, consolidating governance in Lake Formation is simpler.

For step-by-step implementation details of these patterns — including how to set up External Volumes, Catalog Integrations, and Catalog-Linked Databases — see this companion article:

AWS Snowflake Lakehouse: 2 Practical Apache Iceberg Integration Patterns

Comparison: Three Architectural Patterns

Dimension	Snowflake-Centric	AWS-Centric	Hybrid
Catalog Ownership	Snowflake	AWS	AWS
Write Ownership	Snowflake	AWS	AWS
Governance Ownership	Snowflake Horizon	Lake Formation	AWS primary (①) / AWS+SF (②)
Compute Cost	Tends to be higher	Optimizable by workload	Optimizable by workload
Operational Complexity	Low to medium	Medium to high	High
Multi-Engine Flexibility	Medium (via REST API)	High	High

Choosing the Right Pattern

Based on the patterns above, here is a simplified decision guide:

Snowflake-centric tends to fit when:

Analytics is BI-driven or led by non-engineers
Development speed and analytics experience take priority over data volume
Centralized governance through Snowflake Horizon is preferred

AWS-centric tends to fit when:

Data volumes are large and ETL is the dominant workload
A dedicated data engineering team is already working within the AWS ecosystem
Fine-grained access control through Lake Formation is a requirement

Hybrid tends to fit when:

Different teams use different tools (e.g., engineers on AWS, analysts on Snowflake)
Future extensibility for AI, ML, or multi-engine workloads is a priority
AWS retains data ownership while Snowflake's query performance is still needed

What Happens When Ownership Is Unclear

A common anti-pattern is building a platform that "works" without explicitly defining ownership.

Typical symptoms include:

Nobody knows who is responsible for schema changes. When both Glue and Snowflake have schema owners, it becomes unclear which definition is authoritative.
Data written from Snowflake is not visible in Athena. When two catalogs attempt to manage the same table, one may lose track of the latest snapshot, causing metadata inconsistencies.
Governance rules drift between Lake Formation and Snowflake Horizon. Maintaining access policies in two places creates risk — a gap in either becomes a security vulnerability.
Incident response slows down. When multiple engines can write, identifying what happened and where becomes difficult, delaying recovery.

These issues often evolve from technical challenges into organizational problems:

Teams blame each other over unclear responsibilities.
Audits become difficult because nobody can fully explain who has access to what.
Incident recovery is delayed due to unclear decision-making authority.

A running system is not necessarily a well-designed system.

Ownership becomes increasingly difficult to fix after the platform has already grown.

"AWS or Snowflake?" Is a Secondary Question

In practice, organizations often begin by debating whether to standardize on AWS or Snowflake.

In the Iceberg era, I believe that is the wrong starting point.

The first questions should be:

Who owns the catalog?
Who owns writes?
Who owns governance?

Once these three ownership layers are defined, the platform choice naturally follows.

Want all three owned by Snowflake? → Snowflake-centric architecture.
Want all three owned by AWS? → AWS-centric architecture.
Want AWS to own data while Snowflake provides analytics? → Hybrid architecture.

Iceberg has dramatically increased flexibility around where data lives.

As flexibility increases, architects must become more deliberate about defining responsibility.

Starting with product selection often leads to contradictions later. A configuration where Snowflake is used as the query interface, Glue handles writes, and Lake Formation controls governance — without intentional design — is a classic symptom of ownership being distributed and unclear from the start.

The hardest challenge is no longer connectivity.

It is ownership.

Conclusion

Apache Iceberg has significantly reduced storage-level vendor lock-in.

However, catalog ownership, write ownership, and governance ownership still require deliberate architectural decisions.

A useful decision-making sequence is:

Decide who owns the catalog. (Glue / Snowflake Open Catalog / Snowflake Horizon)
Decide who owns writes. (AWS-native services / Snowflake)
Decide who owns governance. (Lake Formation / Snowflake Horizon / both)

Once those three decisions are made, choosing between AWS and Snowflake becomes much easier. From there, you can design the architecture that best fits your requirements.

Ultimately, the hardest part of a modern lakehouse architecture is often not the technology itself. It is agreeing on ownership boundaries — deciding which team manages the catalog, who is responsible for data updates, and where governance policies are enforced.

Technology evolves. The challenge of people and processes remains.

I hope this article helps anyone evaluating lakehouse architectures built on AWS, Snowflake, and Apache Iceberg.

Exploring Snowpark While Comparing It with Apache Spark

Aki — Mon, 01 Jun 2026 04:00:00 +0000

Original Japanese article: Snowparkを動かしながらSparkとの違いを整理してみる

Introduction

Recently, I've had more opportunities to work with Snowflake when building data platforms.

When working with modern data platforms, Apache Spark is often used for distributed data processing. Snowflake also provides its own data processing framework called Snowpark.

If you're already familiar with Spark or AWS Glue, you may find yourself wondering:

"Wait... how is Snowpark actually different from Spark?"

In this article, I'd like to organize my own understanding while exploring Snowpark's behavior and comparing it with Spark.

For this experiment, everything was done entirely within Snowflake Notebooks in Snowsight.

One of the biggest advantages is that no local environment setup or connection configuration is required—you can start experimenting immediately.

What is Snowpark?

Snowpark is a data processing framework provided by Snowflake.

Its biggest feature is the ability to write code in Python, Java, or Scala and execute it directly inside Snowflake.

Traditionally, Snowflake workloads were primarily implemented using SQL. With Snowpark, however, you can use a DataFrame API similar to Spark or Pandas while keeping all processing within Snowflake.

In other words, you no longer need to pull data into a local environment or AWS Lambda for processing.

Some key characteristics include:

Managed execution environment – Processing runs on Snowflake warehouses with no infrastructure management required.
DataFrame API – Similar developer experience to Spark and Pandas.
Pushdown execution – Code is executed within Snowflake, eliminating data transfer overhead.
UDF and UDTF support – Custom functions can be defined and executed inside Snowflake.
Snowflake Notebook integration – Interactive development is supported directly in Snowsight.

Personally, I still like Scala, but these days I write most data processing code in Python.

While Scala often offers better performance, Python's simplicity and extensive ecosystem make it the more practical choice in many situations.

Differences Between Spark and Snowpark

Many people immediately think of Spark when they hear the name Snowpark.

The names are similar, and the DataFrame APIs feel very familiar.

However, there are several important differences.

Aspect	Spark	Snowpark
Execution Environment	Distributed cluster	Snowflake warehouse
Data Source	HDFS, S3, and others	Primarily Snowflake tables
Scaling	Cluster size managed by user	Warehouse size adjustment
Languages	Scala, Java, Python, R, etc.	Python, Java, Scala
External Data Support	Broad ecosystem support	Primarily Snowflake-centric
Infrastructure Management	Cluster management required	Fully managed

Spark requires awareness of distributed clusters and execution mechanics.

Snowpark, on the other hand, is fundamentally a processing framework that operates on top of the Snowflake platform.

DataFrame operations are internally converted into SQL execution plans and executed by Snowflake's SQL engine.

Unlike Spark, user code is not distributed across worker nodes.

Scaling is handled by Snowflake warehouses.

UDFs are an exception. UDF code is pushed into Snowflake and executed in parallel by Snowflake's infrastructure.

A useful mental model is:

DataFrame operations → SQL generation

UDFs → Server-side parallel execution

In either case, users do not need to manage clusters or DAG execution as they would in Spark.

If your data is already centralized in Snowflake, Snowpark provides a convenient way to write Spark-like code without worrying about infrastructure management.

Of course, AWS Glue also provides a largely serverless experience, making it another convenient option in the AWS ecosystem.

Getting Started

All examples in this article are executed within Snowflake Notebooks in Snowsight.

No local Python environment or connection configuration is required—the entire workflow runs directly in the browser.

Prerequisites

From the Snowsight menu:

Create → Notebooks

Create a new notebook.

Snowpark for Python is already installed, so there is no need to run pip install.

You can start coding immediately.

Obtaining a Session

In local environments, Snowpark sessions are typically created using:

Session.builder.configs(...).create()

In Snowflake Notebooks, an active session already exists.

You can simply retrieve it:

from snowflake.snowpark.context import get_active_session

session = get_active_session()
print("Session acquired successfully!")

One of the major advantages of Snowflake Notebooks is that connection details never need to be written manually.

Basic DataFrame Operations

Let's create and manipulate a DataFrame from a table.

df = session.table("MY_DB.MY_SCHEMA.SALES_DATA")

result_df = df.select("ORDER_ID", "AMOUNT", "REGION") \
              .filter(df["REGION"] == "Asia") \
              .sort(df["AMOUNT"].desc())

result_df.show()

One important point is that no SQL is actually executed until show() is called.

We'll discuss this in more detail when covering lazy evaluation.

Aggregations

GroupBy operations feel almost identical to Spark.

from snowflake.snowpark import functions as F

summary_df = df.group_by("REGION") \
               .agg(
                   F.sum("AMOUNT").alias("TOTAL_AMOUNT"),
                   F.count("ORDER_ID").alias("ORDER_COUNT"),
                   F.avg("AMOUNT").alias("AVG_AMOUNT")
               )

summary_df.show()

When executed, these DataFrame operations are translated into SQL and run within Snowflake.

You can inspect the generated SQL using:

print(summary_df.queries)

Writing Results Back to a Table

To save results into a Snowflake table:

summary_df.write.mode("overwrite").save_as_table(
    "MY_DB.MY_SCHEMA.SALES_SUMMARY"
)

Since save_as_table() does not return a result, it's often useful to reload the table to verify the output.

session.table("MY_DB.MY_SCHEMA.SALES_SUMMARY").show()

The overwrite mode replaces the existing table.

Use append if you want to add rows instead.

How Does Lazy Evaluation Work?

Anyone familiar with Spark has likely encountered lazy evaluation.

Sometimes it can even lead to unexpected behavior during debugging.

Snowpark adopts the same fundamental concept.

Understanding Lazy Evaluation

DataFrame transformations such as:

select
filter
group_by

are not executed immediately.

These operations merely build an execution plan.

Actual execution occurs only when an action is triggered.

Common action operations include:

show()
collect()
count()
to_pandas()
write.save_as_table()

Verifying Lazy Evaluation

A convenient way to inspect behavior is through df.queries.

from snowflake.snowpark import functions as F

df_filtered = session.table("MY_DB.MY_SCHEMA.LARGE_TABLE") \
                     .filter(F.col("STATUS") == "ACTIVE") \
                     .select("ID", "NAME", "STATUS", "CREATED_AT")

print(df_filtered.queries)

result = df_filtered.collect()
print(f"{len(result)} rows retrieved")

The generated SQL can be inspected before execution, but no query has actually been sent to Snowflake yet.

To verify execution timing precisely, we can use Query History in Snowsight.

Open:

Monitoring → Query History

Then perform the following steps:

Define the DataFrame.
Check Query History.
Confirm that no SELECT statement has been executed.
Execute collect().
Refresh Query History.
Observe that the SELECT statement now appears.

Define the DataFrame

No corresponding query appears yet.

Execute collect()

After execution, the query becomes visible.

This confirms that DataFrame definitions alone do not trigger execution.

The SQL is executed only when collect() is called.

Differences from Spark's Lazy Evaluation

In Spark, lazy evaluation constructs a DAG that is optimized and executed across a cluster.

In Snowpark, lazy evaluation ultimately produces SQL, which is then optimized and executed by Snowflake's query optimizer.

The concept is similar, but the execution engine is fundamentally different.

One particularly useful feature is that generated SQL can be inspected via df.queries, making it easier to validate execution plans.

Can We Use Caching?

If you're coming from Spark, your first instinct may be to use cache().

Snowpark provides a similar capability through cache_result().

Differences from Spark cache()

Aspect	Spark `cache()`	Snowpark `cache_result()`
Storage	Memory (and disk)	Temporary table
Lifetime	Until application ends	Until session ends
Cost	No additional write	INSERT into temporary table

Internally, cache_result() materializes results into a temporary table.

Subsequent operations reuse that table rather than re-running expensive transformations.

Example

df_heavy = session.table("MY_DB.MY_SCHEMA.LARGE_TABLE") \
                  .filter(F.col("STATUS") == "ACTIVE") \
                  .join(
                      session.table("MY_DB.MY_SCHEMA.MASTER"),
                      "ID"
                  )

cached_df = df_heavy.cache_result()

result1 = cached_df.filter(
    F.col("REGION") == "Asia"
).collect()

result2 = cached_df.group_by("REGION") \
                   .agg(F.count("*")) \
                   .collect()

cached_df.drop_table()

Using a with block is often more convenient because the temporary table is automatically dropped when the block exits.

with df_heavy.cache_result() as cached_df:
    result1 = cached_df.filter(
        F.col("REGION") == "Asia"
    ).collect()

    result2 = cached_df.group_by("REGION") \
                       .agg(F.count("*")) \
                       .collect()

Since cache_result() performs an INSERT into a temporary table, it can actually make things slower when the DataFrame is only used once.

It's most effective when the same expensive transformation is reused multiple times.

You can also observe this behavior in Snowsight.

Temporary table creation:

Subsequent SELECT from the temporary table:

Another query reusing the same temporary table:

Use Cases

Let's consider some practical scenarios where Snowpark can be useful.

ETL Pipelines

Traditionally, pipelines often look like:

S3 → Glue → Redshift

With Snowpark, many transformations can be performed entirely within Snowflake.

This reduces data movement and simplifies overall architecture.

Example:

raw_df = session.table("MY_DB.MY_SCHEMA.RAW_EVENTS")

cleaned_df = raw_df \
    .filter(raw_df["EVENT_TYPE"] != "test") \
    .with_column(
        "EVENT_DATE",
        F.to_date(raw_df["EVENT_TIMESTAMP"])
    ) \
    .drop_duplicates([
        "USER_ID",
        "EVENT_DATE",
        "EVENT_TYPE"
    ])

aggregated_df = cleaned_df \
    .group_by("EVENT_DATE", "EVENT_TYPE") \
    .agg(
        F.count("USER_ID")
         .alias("USER_COUNT")
    )

aggregated_df.write.mode("overwrite") \
             .save_as_table(
                 "MY_DB.MY_SCHEMA.DAILY_EVENT_SUMMARY"
             )

session.table(
    "MY_DB.MY_SCHEMA.DAILY_EVENT_SUMMARY"
).show()

Custom Transformations Using UDFs

Snowpark UDFs allow complex logic that would be cumbersome in SQL to be implemented in Python.

You can register UDFs using either the @udf decorator or session.udf.register().

from snowflake.snowpark.functions import udf
from snowflake.snowpark.types import StringType

@udf(return_type=StringType(),
     input_types=[StringType()])
def normalize_region(region: str) -> str:
    region_map = {
        "US": "North America",
        "JP": "Asia",
        "DE": "Europe",
    }
    return region_map.get(region, "Other")

df = session.table("MY_DB.MY_SCHEMA.RAW_EVENTS")

df_with_region = df.with_column(
    "NORMALIZED_REGION",
    normalize_region(df["REGION_CODE"])
)

If type hints are available, explicit type definitions can be omitted:

from snowflake.snowpark.functions import udf

@udf
def normalize_region(region: str) -> str:
    region_map = {
        "US": "North America",
        "JP": "Asia",
        "DE": "Europe",
    }
    return region_map.get(region, "Other")

That said, simple transformations are often faster when implemented using built-in SQL functions.

As always, benchmark before deciding.

Data Quality Validation

Snowpark can also be used for data quality checks before processing continues.

total_count = df.count()
null_count = df.filter(
    F.col("AMOUNT").is_null()
).count()

null_rate = null_count / total_count

if null_rate > 0.10:
    raise ValueError(
        f"NULL rate exceeds the threshold: {null_rate:.1%}"
    )

print(
    f"Data quality check passed "
    f"(NULL rate: {null_rate:.1%})"
)

Conclusion

In this article, we explored Snowpark's fundamentals, compared it with Spark, and examined its lazy evaluation behavior.

For engineers already familiar with Spark, Snowpark should feel quite approachable.

However, it's important to remember that execution occurs on Snowflake warehouses rather than a Spark cluster.

Reviewing generated SQL and understanding how Snowflake executes queries can help avoid unexpected full-table scans and other performance issues.

If your data is already centralized in Snowflake, keeping processing inside Snowflake rather than moving data to Lambda or Glue Python Shell can be a significant advantage.

Reducing infrastructure management overhead and consolidating ETL processing within Snowflake can also improve maintainability.

One final note: throughout this experiment, I frequently relied on Cortex Code whenever I encountered errors.

The workflow of iteratively fixing notebook errors through Cortex Code was surprisingly convenient.

That said, just like any AI-assisted coding workflow, it's still important to carefully validate the generated code rather than accepting it blindly.

I hope this article helps anyone considering Snowpark for data processing within Snowflake.

Organizing How to Use AWS Glue Workflow

Aki — Fri, 22 May 2026 13:38:32 +0000

Original Japanese article: AWS Glue Workflowの使い方について整理してみる

Introduction

I'm Aki, an AWS Community Builder (@jitepengin).

Previously, I wrote an article comparing when to use AWS Step Functions versus Glue Workflow.
Organizing the Use Cases of AWS Step Functions and Glue Workflow for ETL Processing with AWS Glue Jobs

As I mentioned there, I personally like Glue Workflow and consider it an excellent service that balances simplicity and low cost.

However, in recent years, Step Functions has become increasingly mainstream, and I get the impression that opportunities to work with Glue Workflow have decreased.
Because of that, I think many people are unsure about how they should actually use it in practice.

So in this article, I’d like to organize everything from the basics of Glue Workflow to practical usage patterns.
I hope this helps more people become interested in Glue Workflow.

What Is Glue Workflow?

AWS Glue Workflow is Glue’s native workflow orchestration feature.
It defines ETL pipelines by combining the following three elements:

Component	Role
Glue Job	The actual ETL processing using Spark or Python Shell
Glue Crawler	Scans data sources such as S3 and registers table definitions in the Data Catalog
Glue Trigger	Defines execution conditions for Jobs and Crawlers (schedule, event, conditional, etc.)

By connecting these components as a DAG (Directed Acyclic Graph), you can build ETL pipelines.

Another characteristic is that workflows can be visually configured through the Glue console GUI.

A major advantage is that workflows themselves incur no additional cost—you only pay for Job and Crawler execution.

Details of the Core Components

The core of Glue Workflow is the Trigger mechanism.
There are four trigger types, each with different roles.

Trigger Type	Execution Condition	Typical Use Case
SCHEDULED	Scheduled execution using cron expressions (UTC, minimum 5-minute interval)	Periodic execution such as daily ETL
ON_DEMAND	Manual execution or via API/SDK	Arbitrary execution timing
CONDITIONAL	Triggered based on the status of preceding Jobs/Crawlers	Chained execution between Jobs
EVENT	Triggered by EventBridge events	Event-driven pipelines

The typical pattern is:

Use SCHEDULED, ON_DEMAND, or EVENT as the workflow’s starting trigger
Use CONDITIONAL to connect downstream Jobs

By the way, AWS officially recommends keeping the number of elements included in a workflow (Jobs + Crawlers + Triggers) under 100.
Exceeding this recommendation can cause errors when resuming or stopping Workflow Runs.

CONDITIONAL Trigger Usage Patterns

CONDITIONAL Triggers are one of the key features that provide flexibility in Glue Workflow.
Here are several representative patterns.

1. Simple Sequential Pattern

This is the most basic usage pattern: “Run JobB after JobA succeeds.”

JobA (SUCCEEDED) → JobB

You can implement this simply by configuring a CONDITIONAL Trigger with the condition:
“Start when JobA reaches the SUCCEEDED state.”

2. Waiting for Multiple Jobs to Complete (AND Condition)

This pattern runs JobC only after both JobA and JobB succeed.

JobA (SUCCEEDED) ┐
                 ├→ JobC
JobB (SUCCEEDED) ┘

This can be implemented by setting Logical: AND in the CONDITIONAL Trigger predicate and listing multiple conditions.

This is useful in scenarios such as:
“Run an aggregation Job only after multiple data sources have finished loading.”

3. Triggering When Any Job Completes (ANY Condition)

This pattern runs JobC when either JobA or JobB succeeds.

The predicate of a CONDITIONAL Trigger has a Logical field where you can specify either AND or ANY.
Using ANY causes the trigger to fire as soon as any one of the specified conditions is satisfied.

Note that although this behavior is logically equivalent to “OR,” the actual Glue configuration value is ANY.
This is important when defining workflows using IaC or CLI because specifying OR will result in an error.

4. Failure Branching (Catching FAILED States)

CONDITIONAL Triggers can react not only to SUCCEEDED, but also to states such as:

FAILED
STOPPED
TIMEOUT
ERROR

(The supported states differ slightly between Jobs and Crawlers.)

Using this feature, you can create patterns such as launching a notification Job (for example, a Python Shell Job that publishes to SNS) when a Job fails.

JobA (SUCCEEDED) → Downstream Processing
JobA (FAILED)    → Notification Job

For relatively simple error handling, this approach allows you to avoid introducing Step Functions.

Managing Parameters with `default_run_properties`

Glue Workflow provides a property called default_run_properties, which acts like globally shared variables across the workflow.

How It Works

default_run_properties stores key-value pairs that can be referenced by all Jobs within the workflow.

It functions as the default set of parameters passed during Workflow execution and serves as the foundation for sharing information between Jobs.

One important note:
Run Property values may appear in logs, so you should avoid storing secrets directly in them.

Instead, retrieve secrets through services such as:

AWS Secrets Manager
Glue Connections

How to Configure It

There are three main configuration methods.

Configure from the Console

In the Glue console:
Workflow → Edit Properties

You can add key-value pairs there.

Configure via boto3

import boto3

glue = boto3.client('glue')

glue.create_workflow(
    Name='my-workflow',
    DefaultRunProperties={
        'env': 'production',
        'target_date': '2026-05-21'
    }
)

Configure via IaC (CloudFormation, etc.)

You can specify DefaultRunProperties in the AWS::Glue::Workflow resource.

Static Parameters vs Dynamic Parameters

Typical usage patterns include:

Static parameters
- Environment names (dev / prod)
- S3 bucket names
- Values that rarely change
Dynamic parameters
- Processing dates
- Execution IDs
- Values that change per execution

Dynamic parameters can be updated either by:

Passing RunProperties to start_workflow_run
Dynamically updating them later using put_workflow_run_properties

Passing Data Between Jobs

Using default_run_properties as the foundation, let’s look at how to exchange data between Jobs.

Dynamically Updating Run Properties

During Job execution, you can dynamically update Workflow Run properties by calling the put_workflow_run_properties API.

import boto3

glue = boto3.client('glue')

glue.put_workflow_run_properties(
    Name='my-workflow',
    RunId=workflow_run_id,
    RunProperties={
        'processed_records': '12345',
        'output_path': 's3://mybucket/output/2026-05-21/'
    }
)

This allows downstream Jobs to reference values calculated by upstream Jobs.

Retrieving Properties from a PySpark Job

Inside a PySpark Job, you first retrieve the Workflow Run ID and then call get_workflow_run_properties.

import sys
import boto3
from awsglue.utils import getResolvedOptions

args = getResolvedOptions(
    sys.argv,
    ['JOB_NAME', 'WORKFLOW_NAME', 'WORKFLOW_RUN_ID']
)

glue = boto3.client('glue')

response = glue.get_workflow_run_properties(
    Name=args['WORKFLOW_NAME'],
    RunId=args['WORKFLOW_RUN_ID']
)

properties = response['RunProperties']

target_date = properties.get('target_date')

WORKFLOW_NAME and WORKFLOW_RUN_ID are special arguments automatically passed when a Job is launched through a Workflow.

Retrieving Properties from a Python Shell Job

The basic approach is the same for Python Shell Jobs:

Retrieve arguments using getResolvedOptions
Access properties through boto3

Since Python Shell Jobs do not require SparkContext initialization, they can be written more lightweight.

They are also cheaper than Spark-based Glue Jobs, so depending on the requirements, they can be a good option.

Personally, I also like Python Shell Jobs, although I feel opportunities to use them in real-world projects have decreased, which is a bit unfortunate.

I’ve also written articles about Python Shell Jobs, so feel free to check them out.

S3 Triggers: How to Launch Glue Python Shell via AWS Lambda

Anti-Patterns for Data Passing

default_run_properties should only be used for metadata-like information exchange.

The following usage patterns should generally be avoided:

Passing large datasets directly
- Store the data itself in S3 and pass only the path
Storing secrets
- Use Secrets Manager or Glue Connections instead
Frequently rewriting properties
- This increases API calls and introduces race condition risks

Integrating with EventBridge and Other Services

Glue Workflow becomes much more flexible when combined with EventBridge.

EventBridge-Based Startup (EVENT Trigger)

Glue Workflow can be started directly by EventBridge events.

This is achieved by setting the Trigger Type to EVENT.

aws glue create-trigger \
  --workflow-name my-workflow \
  --type EVENT \
  --name s3-arrival-trigger \
  --actions JobName=my-job

By configuring an EventBridge rule with Glue Workflow as the target, the workflow starts automatically when an event occurs.

However, appropriate IAM permissions such as glue:notifyEvent are required.

Batch Event Startup

EVENT Triggers also support event batching.

Using EventBatchingCondition, you can configure the workflow to start when either:

N events arrive
M seconds pass since the first event arrived

aws glue create-trigger \
  --workflow-name my-workflow \
  --type EVENT \
  --name batch-trigger \
  --event-batching-condition BatchSize=10,BatchWindow=300 \
  --actions JobName=my-job

This enables patterns such as:
“Run ETL once 100 files have arrived.”

The maximum batch window is 900 seconds (15 minutes).

Starting from S3 Events (The Parameter Passing Limitation)

A common use case is:
“Start a workflow when a file is uploaded to S3.”

When starting Glue Workflow through EventBridge, the event IDs are automatically stored in a Run Property called aws:eventIds.

event_ids = glue_client.get_workflow_run_properties(
    Name=workflow_name,
    RunId=workflow_run_id
)['RunProperties']['aws:eventIds']

The returned value looks like:

'["abc-123", "def-456"]'

However, this is where one of Glue Workflow’s limitations becomes apparent.

The EventBridge event payload itself (such as the S3 object key or bucket name) is not automatically passed as Run Properties.

Only the event IDs are provided.

If you need the actual object details, your Job must retrieve the corresponding event contents from CloudTrail, which becomes somewhat cumbersome.

Because of this, many cases are easier to manage by placing Lambda in the middle and explicitly calling start_workflow_run with structured Run Properties.

S3 PUT → EventBridge → Lambda → start_workflow_run (pass parameters via RunProperties)

Example Lambda code:

import boto3

glue = boto3.client('glue')

def lambda_handler(event, context):
    bucket = event['detail']['bucket']['name']
    key = event['detail']['object']['key']

    glue.start_workflow_run(
        Name='my-workflow',
        RunProperties={
            'source_bucket': bucket,
            'source_key': key
        }
    )

With Step Functions, the EventBridge payload can be received directly using paths such as $.detail, so there is no need for an intermediate Lambda function.

This is one of the areas where Glue Workflow limitations become noticeable compared to Step Functions.

Detecting Workflow Completion via EventBridge

Glue Workflow status changes are emitted as EventBridge events.

You can use this to:

Send SNS notifications
Trigger downstream systems
Launch post-processing workflows

Glue Workflow (COMPLETED/FAILED)
    → EventBridge
        → SNS / Lambda

This is especially useful when you want separate processing for success and failure cases.

Calling Glue Workflow from Step Functions

It is also possible to launch Glue Workflow from Step Functions.

However, since there is no .sync integration pattern available, you must implement your own polling logic to detect completion.

I covered this in detail in the previous article, so feel free to refer to it if interested.

Organizing the Use Cases of AWS Step Functions and Glue Workflow for ETL Processing with AWS Glue Jobs

Operational Tips

Here are several practical points worth knowing when operating Glue Workflow in production.

Resuming Failed Workflows (`ResumeWorkflowRun`)

Glue Workflow provides a feature called ResumeWorkflowRun, which allows resuming execution from failed nodes.

In the console, you can:

Open the failed Workflow Run detail page
Select the nodes to resume
Enable the “Resume” checkbox

It is also available through CLI/API.

aws glue resume-workflow-run \
  --name my-workflow \
  --run-id wr_xxxx \
  --node-ids node_yyyy node_zzzz

When resumed:

The specified nodes
And all downstream nodes

are re-executed.

The resumed workflow is tracked using a new Run ID.

Compared with Step Functions Redrive, however, there are several limitations:

You must explicitly specify failed nodes
Retrieving node IDs requires calling: get-workflow-run --include-graph
Additional IAM permissions (glue:ResumeWorkflowRun) are required

For simple retry scenarios, restarting the entire workflow via ON_DEMAND execution is often easier.

For pipelines with more sophisticated recovery requirements, Step Functions tends to provide a smoother operational experience.

Monitoring with CloudWatch

Glue Workflow execution states can be viewed in the Glue console, but detailed Job and Crawler logs are output to CloudWatch Logs.

Typical monitoring targets include:

Workflow Run list
- Glue Console
Job execution logs
- /aws-glue/jobs/output
- /aws-glue/jobs/error
Metrics
- CloudWatch Metrics
- Execution duration
- DPU usage

In practice, monitoring and alerting strategies often combine:

Workflow-level statuses
Job-level metrics

depending on the situation.

Handling the 100-Object Recommendation Limit

As mentioned earlier, AWS recommends keeping the total number of objects in a workflow (Jobs + Crawlers + Triggers) below 100.

If a large pipeline approaches this limit, consider:

Splitting workflows
- Trigger downstream workflows after upstream completion
Consolidating common processing inside Jobs
- Merge smaller processing units into larger Jobs

In my experience, workflows approaching 100 objects are often already too complex from a design perspective.
At that point, it may be worth reconsidering the architecture itself or migrating to Step Functions.

Where Glue Workflow Fits — and Its Limitations

Glue Workflow shines in scenarios such as:

Simple ETL pipelines completed entirely within Glue Jobs and Crawlers
Periodic ingestion pipelines for data lakes
Lightweight small-to-medium scale pipelines that need to be launched quickly

On the other hand, Step Functions is generally better suited for:

Integrations involving Lambda, ECS, and other AWS services
Complex branching and dynamic parameter control
Large-scale development involving multiple teams

I discussed these decision criteria in more detail in the previous article, so feel free to refer to it.

Organizing the Use Cases of AWS Step Functions and Glue Workflow for ETL Processing with AWS Glue Jobs

Conclusion

In this article, I organized AWS Glue Workflow from the basics through practical usage patterns.

To summarize:

Four trigger types:
SCHEDULED / ON_DEMAND / CONDITIONAL / EVENT
CONDITIONAL Triggers:
Flexible flow control using AND / ANY conditions and failure branching
default_run_properties:
Shared workflow-wide parameter management
Data passing between Jobs:
Dynamic value propagation using put_workflow_run_properties
EventBridge integration:
Event-driven execution via EVENT Trigger
(although Lambda-based parameter passing is often easier)
ResumeWorkflowRun:
Partial restart functionality from failed nodes

After writing all of this, I still have to admit:
Step Functions is generally easier to use.

That said, Glue Workflow still provides meaningful value today because it allows you to build Glue-centric ETL pipelines in a very simple and cost-efficient way.

Rather than defaulting to Step Functions automatically, understanding Glue Workflow properly and knowing when to use it can broaden your architectural options.

I hope this article helps both people who are starting to use Glue Workflow and those already working with it.

Organizing the Use Cases of AWS Step Functions and Glue Workflow for ETL Processing with AWS Glue Jobs

Aki — Wed, 20 May 2026 12:09:21 +0000

Original Japanese article: Glue JobのETL処理におけるAWS Step FunctionsとGlue Workflowの使い分けを整理する

Introduction

I'm Aki, an AWS Community Builder (@jitepengin).

When building data pipelines on AWS, deciding which workflow orchestration tool to use is an important architectural decision.
Especially when designing ETL pipelines centered around Glue Jobs, the following two services are commonly compared:

AWS Step Functions
AWS Glue Workflow

Both services are workflow orchestration tools that manage job dependencies and execute processes sequentially.
However, since they are designed with different strengths and philosophies, choosing the right one based on your requirements is critical.

In this article, I’ll organize the characteristics, advantages, disadvantages, and use cases of these two services, and discuss how to decide which one to choose.

Personally, I really like Glue Workflow, but I’ve had fewer opportunities to use it recently, which is honestly a bit disappointing.
I also like Glue Python Shell, but compared to standard Glue Jobs, I rarely get to use it these days either...

Major Workflow Orchestration Tools

AWS provides multiple workflow orchestration services, but the two most commonly compared for Glue Job–based data pipelines are the following:

Service	Overview
AWS Step Functions	AWS managed workflow orchestration service. Can flexibly orchestrate a wide range of AWS services such as Lambda, Glue, ECS, and more
AWS Glue Workflow	Glue-native workflow feature. Defines pipelines using Glue Jobs, Crawlers, and Triggers

AWS also offers Amazon Managed Workflows for Apache Airflow (MWAA).
MWAA becomes a strong option when you need highly complex dependency management or cross-cloud orchestration. However, in this article, I’ll focus specifically on comparing Step Functions and Glue Workflow.

Recently, many teams have also adopted workflow tools such as Airflow, Dagster, and Prefect. As always, selecting the right tool depends heavily on your requirements and goals.

AWS Step Functions

Overview

(Figure above: Visual workflow definition using Workflow Studio)

AWS Step Functions is a workflow orchestration service that defines workflows using Amazon States Language (ASL), a JSON/YAML-based language.

It integrates with over 200 AWS services including Lambda, Glue, ECS, SNS, and DynamoDB, and supports flexible workflow controls such as conditional branching, parallel execution, and retry handling.

Features

Step Functions provides two workflow types:

Type	Characteristics
Standard Workflow	Long-running execution, audit logs, exactly-once execution guarantee
Express Workflow	High throughput, low cost, optimized for short-lived processing (at-least-once)

In the context of data pipelines, Standard Workflow is generally the more common choice.

Step Functions also provides AWS Step Functions Workflow Studio, a visual editor that allows workflows to be built through a GUI.

Advantages

Multi-service integration: Easily integrates Glue with other AWS services such as Lambda, ECS, SNS, and more
Flexible control flow: Supports conditional branching (Choice), parallel execution (Parallel/Map), and advanced error handling (Catch/Retry)
High observability: Execution history and per-step states can be visually inspected in the console
Event-driven integration: Easily triggered through EventBridge using S3 uploads or schedules
Parallel execution support (Map): Well suited for large-scale processing such as file-level or partition-level parallel execution. Distributed Map is especially useful for high-scale parallel workloads
Infrastructure as Code support: Can be fully managed through CloudFormation, CDK, or Terraform

Disadvantages

Learning curve: Requires understanding ASL, and complex workflows can become verbose
Cost: Standard Workflow pricing is based on state transitions, so costs can increase with larger workflows
Glue integration setup: IAM roles, parameter passing, and Glue Job integration must be configured manually

Use Cases

Step Functions is a strong fit for scenarios such as:

Hybrid pipelines combining Glue with Lambda, ECS, or other AWS services
Complex workflows requiring branching, parallel processing, or dynamic parameter passing
Pipelines that require SNS notifications or compensating actions on failures
Teams managing infrastructure strictly through IaC
Large-scale data platforms involving multiple teams

AWS Glue Workflow

Overview

(Figure above: Visual workflow definition using Glue Workflow)

AWS Glue Workflow is Glue’s native workflow orchestration feature.

It defines pipelines using three primary components:

Glue Jobs
Glue Crawlers
Glue Triggers

Pipelines can be configured directly from the Glue console GUI, making it easy to build Glue-centric ETL workflows quickly.

Features

The major components of Glue Workflow are as follows:

Component	Role
Glue Job	Actual ETL processing using Spark or Python Shell
Glue Crawler	Scans data sources such as S3 and registers table metadata into the Data Catalog
Glue Trigger	Defines execution conditions such as schedules, events, or conditional dependencies

Glue Triggers support three types:

SCHEDULED
ON_DEMAND
CONDITIONAL
EVENT

This allows chained execution patterns, such as triggering downstream jobs based on upstream job success or failure.

Advantages

Native Glue integration: Seamlessly integrates with Glue Jobs and Crawlers with minimal additional setup
Simple configuration: DAG-style workflows can be built intuitively through the console GUI
Low cost: No additional charge for the workflow itself beyond Glue Job execution costs
Easy crawler integration: Natural fit for workflows that update the Data Catalog after ETL execution
Glue Data Catalog integration: Job execution metadata and lineage can be managed centrally within Glue

Disadvantages

Glue-only orchestration: Cannot directly orchestrate Lambda, ECS, or other non-Glue services
Limited event-driven capabilities: Primarily Trigger- and schedule-based, making advanced event integration less flexible than Step Functions
Limited control flow: Weak support for advanced branching and dynamic parameter handling
Observability limitations: Detailed execution logs often require separate CloudWatch investigation
More difficult IaC management: CloudFormation and Terraform management can become cumbersome compared to Step Functions
Limited parallel execution control: Not ideal for fine-grained parallelization or Map-style orchestration
Weaker retry/re-execution control: Re-running only failed portions of a workflow is less flexible than in Step Functions

Use Cases

Glue Workflow is well suited for:

Simple ETL pipelines composed entirely of Glue Jobs and Crawlers
Periodic ingestion pipelines into S3-based data lakes with automatic catalog updates
Small teams or early-stage projects that need fast implementation
Organizations that prefer operating primarily through the Glue console

Which One Should You Choose?

Based on the characteristics of both services, the selection criteria can generally be summarized as follows.

When to Choose Glue Workflow

Your ETL pipeline consists only of Glue Jobs and Crawlers
Simple sequential execution and conditional triggers are sufficient
You want to build quickly (prototypes or small-scale projects)
Your operations are centered around the Glue console
You want to minimize costs

When to Choose Step Functions

You need integration with Lambda, ECS, or other AWS services
You require branching, parallel processing, or advanced error handling
You are adopting an EventBridge-centric event-driven architecture
You want strict Infrastructure as Code management
Multiple teams are involved in operating the data platform
Observability and audit logging are important

Summary of Decision Criteria

Perspective	Glue Workflow	Step Functions
Supported Services	Glue only	Broad AWS integration
Control Flow	Simple	Flexible and advanced
Observability	△	◎
Ease of Configuration	◎	△ (learning curve)
Cost	Low	Depends on state transitions
IaC Management	△	◎
Crawler Integration	◎	△ (manual setup required)

Glue Workflow is fundamentally Trigger-oriented and is not designed for advanced event orchestration like Step Functions.

Unless Glue Workflow specifically satisfies your requirements better, choosing Step Functions is generally the safer long-term option.
Personally, when designing architectures, I often start by considering Step Functions first.

That said, Glue Workflow remains a strong choice when the requirement is simply:

“I want to quickly build a Glue-centric ETL pipeline.”

Combining Both Services

Instead of choosing one or the other exclusively, it is also possible to invoke Glue Workflow from Step Functions.

For example, Step Functions can handle preprocessing and postprocessing with Lambda, while delegating the ETL core to Glue Workflow.

However, this introduces additional complexity because workflow state coordination between the two services must be managed carefully.
If simplicity is important, standardizing on one orchestration tool is generally easier operationally.

Caveats When Invoking Glue Workflow from Step Functions

1. Completion Detection Requires Polling

Step Functions provides a convenient .sync integration pattern for Glue: StartJobRun, which waits for job completion automatically.

However, Glue: StartWorkflowRun does not support the .sync integration pattern.

While you can invoke Glue Workflow through SDK integration, Step Functions will immediately proceed to the next state without waiting for completion.
As a result, you must implement custom polling logic to repeatedly check the WorkflowRun status.

2. Polling Logic Becomes Complex

You typically need to implement a loop like:

Wait
GetWorkflowRun
Choice (RUNNING / COMPLETED / FAILED)

This increases both the number of states and the verbosity of the ASL definition.

3. Error Handling Becomes More Complicated

Glue Workflow status is returned at the WorkflowRun level rather than the individual Job level.

As a result, identifying which specific Glue Job failed requires additional parsing logic against the GetWorkflowRun response.

Because of these complications, although combining both services is technically possible, I generally recommend avoiding Step Functions → Glue Workflow orchestration unless there is a compelling reason.

One possible use case is when you need to extend an existing Glue Workflow–based system incrementally.
Even then, rebuilding the orchestration directly in Step Functions using existing Glue Jobs often feels cleaner.

Migration Considerations

Some teams initially adopt Glue Workflow during the early stages of a project and later migrate to Step Functions as the data platform grows.

When migrating, the following considerations become important:

Workflow definitions must be rewritten: Glue Trigger definitions need to be converted into ASL
IAM roles must be redesigned: Step Functions requires permissions to invoke Glue Jobs
Extensive testing is necessary: Existing jobs must be validated carefully after migration

Migration is certainly possible, but it is not trivial.
This is why it is important to consider future extensibility and maintainability from the beginning when selecting your orchestration tool.

Conclusion

In this article, I organized the differences between AWS Step Functions and Glue Workflow for ETL orchestration.

To summarize:

Glue Workflow: Glue-native, simple, low-cost, and ideal for rapidly building straightforward ETL pipelines
Step Functions: Better suited for multi-service orchestration, advanced workflow control, observability, and large-scale pipelines

Neither service is universally “better.”
The right choice depends on your use cases, organizational structure, operational requirements, and future scalability needs.

For small-scale ETL pipelines, Glue Workflow is often sufficient. However, as data platforms evolve, requirements such as exception handling, notifications, conditional branching, and integrations with other services tend to grow over time. In many cases, architectures gradually move toward Step Functions as complexity increases.

A practical strategy can be to start simple with Glue Workflow during the early stages, and later migrate to Step Functions when requirements become more sophisticated.

That said, considering future migration costs, building on Step Functions from the beginning can also be a very reasonable approach.

I hope this article helps anyone currently evaluating workflow orchestration options on AWS.

Differences Between Snowflake Editions and Secure Connectivity with AWS

Aki — Fri, 15 May 2026 12:55:34 +0000

Original Japanese article: Snowflakeのエディションごとの違いとAWSとのセキュアな接続方法

Introduction

I'm Aki, an AWS Community Builder (@jitepengin).

More and more organizations are adopting Snowflake as their data platform.
However, once you actually start planning an implementation, there is often a surprisingly common question:

“Which Snowflake edition should we choose?”

In particular, many teams struggle to decide between Enterprise Edition and Business Critical Edition.

This becomes even more important when Snowflake is used together with AWS and there is a requirement for secure network connectivity.

Snowflake offers four editions, and higher editions provide stronger capabilities around security, compliance, and availability.
In addition, private connectivity using AWS PrivateLink requires Business Critical Edition or higher.

This means that if you initially start with Enterprise Edition and later realize that you need PrivateLink, migrating afterward can introduce additional operational and architectural effort.

In this article, I would like to briefly organize the differences between Snowflake editions and then explore how to securely connect Snowflake with AWS using Business Critical Edition, especially through AWS PrivateLink.

Differences Between Snowflake Editions (Quick Overview)

Snowflake provides the following four editions.
Each higher edition includes all functionality from the lower editions, while credit pricing increases accordingly.

Edition	Positioning	Major Additional Features
Standard	Entry-level	Core features, Time Travel (1 day), SSO, Network Policies
Enterprise	Large-scale / Production workloads	Multi-cluster warehouses, Materialized Views, Time Travel (up to 90 days), Column- and Row-level security
Business Critical	Regulated industries	Tri-Secret Secure, AWS PrivateLink, Failover, HIPAA / PCI DSS
Virtual Private Snowflake (VPS)	Highest isolation level	Fully isolated Snowflake environment with dedicated infrastructure

Standard Edition

This is the entry-level edition and provides the core Snowflake functionality.

It includes capabilities such as:

SQL processing
Semi-structured data support
Data sharing
1-day Time Travel

For organizations simply looking to use Snowflake as a data warehouse, Standard Edition is often sufficient.

It is generally a good fit for startups or smaller analytics teams that want to begin using Snowflake quickly.

Enterprise Edition

Enterprise Edition adds several features that are effectively essential for production workloads.

Key additions include:

Multi-cluster warehouses (for scaling concurrent users)
Materialized Views
Extended Time Travel (up to 90 days)
Column-level and row-level security
Dynamic Data Masking
Query Acceleration Service
Search Optimization Service

In practice, Enterprise Edition often feels like the real starting point for production Snowflake environments.

As data volume and concurrency increase, multi-cluster warehouses become especially important.

Business Critical Edition

Business Critical Edition is designed for organizations handling regulated or highly sensitive data.

In addition to all Enterprise features, it includes:

AWS PrivateLink / Azure Private Link / Google Private Service Connect
Tri-Secret Secure (dual encryption using customer-managed keys)
Account failover / failback
Client redirect
Compliance support for HIPAA, HITRUST CSF, PCI DSS, FedRAMP

If you are handling sensitive information such as:

PHI (medical data)
PCI-related cardholder data
Personally identifiable information

then Business Critical Edition becomes necessary.

It is also mandatory if your networking requirements specify that connectivity must avoid the public internet and use PrivateLink instead.

Data is one of the most important enterprise assets.
For that reason alone, I often see organizations adopting Business Critical Edition.

Virtual Private Snowflake (VPS)

This is Snowflake’s highest edition and provides a dedicated Snowflake environment.

Infrastructure is completely isolated and hardware resources are not shared with other customers.

It is intended for organizations with extremely strict security requirements, such as:

Financial institutions
Government agencies

I personally have not worked with VPS directly, and pricing/details require contacting Snowflake, so I will omit deeper discussion here.

Which Edition Should You Choose?

From a practical perspective, the decision criteria often look something like this:

Standard: Evaluation, PoC, small analytics teams
Enterprise: General production workloads (this is where many companies start)
Business Critical: Regulated industries, sensitive data, or mandatory PrivateLink requirements
VPS: Financial/government environments requiring complete isolation

One important point is that if you simply decide to “start with Enterprise,” you cannot use PrivateLink.

Because of this, it is safest to validate networking requirements at the beginning of the project.

Upgrading later is possible, but it may require revisiting architecture and pricing assumptions.

Considering Connectivity with AWS

When using Snowflake on AWS, network design—specifically how clients and applications connect—is tightly related to edition selection.

Here, I would like to organize connectivity approaches from both the Enterprise and Business Critical perspectives.

Connectivity with Enterprise Edition

Enterprise Edition does not support AWS PrivateLink, so connectivity to Snowflake is fundamentally internet-based.

Client / AWS Services
        ↓
    Internet
        ↓
    Snowflake

Hearing “internet-based connectivity” may sound concerning at first.
However, even with Enterprise Edition, it is possible to achieve a practical security level by combining multiple controls.

1. Restricting Source IPs with Network Policies

Snowflake network policies can restrict allowed source IP addresses.

By limiting access to known egress IPs such as:

Corporate network addresses
Elastic IPs attached to NAT Gateways

you can significantly reduce the risk of unauthorized access.

2. Key Pair Authentication + AWS Secrets Manager

For credential management, password authentication should generally be avoided in favor of key pair authentication.

As of 2026, Snowflake is actively moving away from single-factor password authentication.
For system integrations, Key Pair authentication or OAuth is now the recommended approach.

When connecting from services such as AWS Lambda, storing private keys in AWS Secrets Manager is a practical approach.

Event Source
    ↓
AWS Lambda
    ↓
Secrets Manager
    ↓
Retrieve Private Key
    ↓
Snowflake (TLS over Internet)

I covered this approach in a previous article as well:

Securely Implementing Snowflake AWS Lambda Integration with Key Pair Authentication + Secrets Manager

3. TLS Encryption

All communication with Snowflake is encrypted via TLS.

This means the communication channel itself remains confidential.

In other words, even with Enterprise Edition, combining:

IP restrictions
Key Pair authentication
TLS encryption

can provide a practical level of security.

However, traffic still traverses the public internet.

Limitations of Enterprise Edition

Enterprise Edition cannot satisfy requirements such as:

Audit/compliance mandates requiring “no internet-based connectivity”
Internal security policies mandating PrivateLink
Handling regulated data under HIPAA or PCI DSS
Encrypting data using customer-managed KMS keys (Tri-Secret Secure)

Once these requirements appear, Business Critical Edition or higher becomes necessary.

Connectivity with Business Critical Edition

Business Critical Edition supports AWS PrivateLink, enabling private connectivity between AWS VPCs and Snowflake.

In this architecture, traffic remains entirely within the AWS backbone network and never traverses the public internet.

    AWS VPC
        ↓
VPC Interface Endpoint
        ↓
 AWS PrivateLink
        ↓
   Snowflake VPC

High-Level Setup Procedure

Following the official documentation, the configuration can be summarized in several major steps.

1. Enable PrivateLink on the Snowflake Side

Using the ACCOUNTADMIN role, authorize AWS PrivateLink for the Snowflake account.

First, obtain a federation token using AWS STS.

aws sts get-federation-token --name your-user-name

Then execute authorization from the Snowflake side.

USE ROLE ACCOUNTADMIN;

SELECT SYSTEM$AUTHORIZE_PRIVATELINK(
  '<aws_account_id>',
  '<federated_token>'
);

You can verify authorization with the following function.

SELECT SYSTEM$GET_PRIVATELINK('<aws_account_id>', '<federated_token>');

If the response returns:

Account is authorized for PrivateLink.

then authorization succeeded.

2. Retrieve the VPC Endpoint ID

Retrieve the information required for AWS VPC Endpoint creation.

SELECT SYSTEM$GET_PRIVATELINK_CONFIG();

Take note of the privatelink-vpce-id value returned in the JSON response.

This ID becomes the “service name” when creating the VPC Endpoint on AWS.

3. Create the VPC Endpoint on AWS

Create an AWS VPC Interface Endpoint using the following configuration.

Service Name: the privatelink-vpce-id
VPC: the source VPC
Subnets: multi-AZ deployment is recommended
Security Groups: allow ports 443 and 80 from Lambda/EC2

Port 80 is required for OCSP (certificate revocation checking), so do not forget to allow it.

4. Configure DNS

When using PrivateLink, the Snowflake account URL changes to the following format:

<account_identifier>.privatelink.snowflakecomputing.com

You must create a CNAME record mapping the endpoint returned by SYSTEM$GET_PRIVATELINK_CONFIG() to the DNS name of the AWS VPC Endpoint.

Using a Route 53 Private Hosted Zone is the most common approach.

5. Verify Connectivity

Finally, verify connectivity from the client side.

For diagnostics, the SnowCD (Snowflake Connectivity Diagnostic Tool) is useful for validating PrivateLink connectivity.

snowcd <hostfile>

Configuring VPC Endpoints for S3 Access

This is an easy detail to overlook.

Snowflake drivers such as:

JDBC
ODBC
Python Connector

internally access Amazon S3 during data load/unload operations against stages.

Even if Snowflake connectivity itself is private via PrivateLink, S3 traffic may still traverse the public internet unless additional configuration is performed.

Available approaches include:

Creating AWS VPC Interface Endpoints for Snowflake internal stages (recommended)
Creating an S3 Gateway Endpoint to privatize S3 bucket access
Allowing internet-based S3 access (strongly discouraged)

If you want a fully private architecture, Snowflake officially recommends creating VPC Interface Endpoints for internal stages.

Blocking Public Access

After establishing PrivateLink connectivity, you can also block public access from the Snowflake side.

This allows only PrivateLink-based connectivity.

CREATE NETWORK POLICY privatelink_only
  ALLOWED_IP_LIST = ('10.0.0.0/8');

ALTER ACCOUNT SET NETWORK_POLICY = privatelink_only;

Snowflake also provides:

SYSTEM$ENFORCE_PRIVATELINK_ACCESS_ONLY
“Enforce privatelink-only access”

which are also valid approaches.

Combining VPN-based corporate IP ranges with PrivateLink-only access can create an even more secure architecture.

Leveraging Tri-Secret Secure

Business Critical Edition also supports Tri-Secret Secure using AWS KMS customer-managed keys (CMKs).

This mechanism requires both:

Snowflake-managed keys
Customer-managed keys

as an AND condition for decryption.

Even if Snowflake itself were compromised, data could not be decrypted without the customer-managed key.

Combining:

PrivateLink
Tri-Secret Secure

creates a very strong architecture for regulatory compliance.

I have not personally implemented this feature, so I will omit further details here.

Cross-Region Connectivity

AWS PrivateLink is fundamentally designed for same-region connectivity.

However, Business Critical Edition and above also support cross-region connectivity.

For example:

Snowflake account in US-EAST
AWS VPC in AP-NORTHEAST-1 (Tokyo)

can still communicate privately via PrivateLink.

That said, there are several caveats:

PaaS services such as S3 and KMS do not support cross-region PrivateLink
Government and China regions are not supported
“Enable Cross Region Endpoint” must be enabled in the VPC console

In practice, aligning the Snowflake region with the application region generally results in a simpler and easier-to-operate architecture.

Still, for globally distributed data platforms, these considerations become important.

Balancing Edition Selection and Cost

Business Critical Edition provides major security advantages, but the credit cost is roughly 1.3x higher than Enterprise Edition.

As rough on-demand reference pricing for US East in 2026:

Standard: approximately $2/credit
Enterprise: approximately $3/credit
Business Critical: approximately $4/credit

If you have a strict requirement that traffic must never traverse the public internet, Business Critical is effectively the only option.

However, from a practical standpoint, balancing data sensitivity and cost often leads to architectures such as:

Production data warehouse (including sensitive data): Business Critical
Development / testing environments: Enterprise
Dedicated data sharing accounts: Enterprise

Using multiple editions strategically within the same organization can also be a reasonable approach.

Conclusion

In this article, including some reflections from my own experience, I introduced the differences between Snowflake editions and explored secure AWS connectivity using Business Critical Edition.

The key points are:

Snowflake provides four editions (Standard / Enterprise / Business Critical / VPS), with higher editions adding stronger security and compliance capabilities
AWS PrivateLink requires Business Critical Edition or higher, so networking/security requirements should be validated early
Even Enterprise Edition can achieve reasonable security through network policies, Key Pair authentication, and TLS
Business Critical enables private connectivity between AWS VPCs and Snowflake through PrivateLink, fully isolating traffic from the public internet
S3 access must also be privatized, so VPC Endpoints for internal stages should be configured as well
Combining Tri-Secret Secure with PrivateLink enables architectures well suited for regulatory compliance

I think many teams struggle specifically with deciding between Enterprise Edition and Business Critical Edition.

Although edition upgrades are possible later, they can significantly impact both architecture and cost.
For that reason, it is best to organize these requirements carefully during the early stages of requirements definition and architecture design.

I hope this article helps anyone looking to use Snowflake securely on AWS.

What Is Apache Polaris? Why Open Data Catalogs Matter and How to Use Them with AWS

Aki — Sat, 02 May 2026 06:27:16 +0000

Original Japanese article: Apache Polarisとは何か？オープンなデータカタログが求められる理由とAWSとの組み合わせ方を整理する

Introduction

I'm Aki, an AWS Community Builder (@jitepengin).

In recent years, lakehouse architectures centered around Apache Iceberg have been rapidly expanding.

By placing Iceberg tables on object storage such as S3, it has become possible to query the same data from multiple engines such as Athena, Snowflake, Spark, Trino, and Dremio.
As a result, the discussion has shifted from “Where should data be placed, and which engine should be used for analysis?” to “Where should data ownership reside, and which catalog should be used to unify governance?”

Amid this trend, Apache Polaris has been attracting attention in recent years.
Apache Polaris is an open-source implementation of the Iceberg REST Catalog, led by Snowflake and donated to the Apache Software Foundation.

Multiple vendors—including Dremio, AWS, Google, Microsoft, and Confluent—are contributing to it, and it is positioned as an “open catalog” that enables cross-platform management of Iceberg tables while avoiding vendor lock-in.

In this article, I would like to think through the following:

What Apache Polaris is
Why open data catalogs are required
Differences from AWS Glue Data Catalog
Differences from Snowflake Horizon Catalog
How responsibilities should be divided when combining with AWS

In conclusion, Apache Polaris is not something that competes with AWS Glue Catalog or Snowflake Horizon Catalog; rather, they are catalogs that operate at different layers.

It may be easier to understand Apache Polaris as a component that enables an architecture such as:
“The data itself resides in AWS, the catalog is open, and analysis engines are selected based on use cases.”

What is Apache Polaris?

Apache Polaris is an open-source catalog implementation compliant with the Apache Iceberg REST Catalog specification.
It was announced by Snowflake in 2024 and later became an incubation project under the Apache Software Foundation.

The project has now graduated from incubation and has been promoted to a top-level Apache project.

Official site:
https://polaris.apache.org/

What Polaris aims to achieve is a common metadata and governance foundation in a lakehouse centered around Iceberg tables.

A major characteristic is that it is not tied to any specific query engine or cloud vendor, and anyone can access it using the same specification via REST APIs.

Key Features of Apache Polaris

Feature	Description
Implementation of Iceberg REST Catalog	Accessible via standardized REST APIs. Can be directly used from engines such as Spark, Trino, Flink, Snowflake, and Dremio
Multi-catalog architecture	Multiple catalogs can be defined within a single Polaris instance. Enables separation and management by team or business domain
RBAC (Role-Based Access Control)	Provides a permission model combining principals, principal roles, and catalog roles
External catalog integration	Can connect to other catalogs compliant with the Iceberg REST specification (e.g., Nessie, Gravitino)
OSS / Managed support	Can be self-hosted as OSS, or used as managed offerings such as Snowflake Open Catalog or Dremio Catalog

What Apache Polaris Solves

As Apache Iceberg has become more widely adopted, multiple Iceberg-compatible catalogs have emerged, including Hive Metastore, JDBC, Nessie, AWS Glue, and Snowflake.

Since each has its own client libraries and interfaces, the following challenges have arisen:

The need to implement catalog clients for each programming language
Inconsistent access control specifications across catalogs
Difficulty enforcing governance across multiple catalogs
As a result, the overall architecture becomes constrained by the chosen catalog

To solve these challenges, the Iceberg REST Catalog specification was introduced.
Apache Polaris is an open-source implementation of that specification, further enhanced with multi-catalog support and RBAC.

In other words, you can think of it as an open catalog for Apache Iceberg.

Polaris Security Model

The Polaris security model can be organized into the following three concepts:

Principal: An entity representing a user or service. Accesses Polaris via client ID/secret, etc.
Principal Role: A grouping of multiple catalog roles. Assigned to principals
Catalog Role: A set of permissions within a specific catalog. Includes permissions such as TABLE_READ_DATA, TABLE_CREATE, and NAMESPACE_LIST

For example, you can design it such that:

The data_engineer principal role is assigned both write access to prod_catalog and administrative access to dev_catalog
The data_analyst principal role is assigned only read access to prod_catalog

An important point is that RBAC is centralized on the catalog side, eliminating the need to implement access control separately for each engine.

Why Open Data Catalogs Are Required

Let us first consider why open data catalogs are required in the first place.

Separation of Data and Engines Has Become a Premise

The greatest value of open table formats such as Apache Iceberg is the ability to separate data storage from query engines.

It has become possible to freely choose engines such as Athena, Glue, Spark, Snowflake, Dremio, and DuckDB depending on the use case when querying Iceberg tables on S3.

As a result, the key question in data platforms has shifted from “Which product should we use?” to “Where should data ownership reside, and who should be responsible for governance at which layer?”

However, while engines can now be freely selected, the remaining challenge is the catalog.

What Happens When Catalogs Are Tied to Engines

When using catalogs tightly coupled with query engines, the following situations tend to occur:

The data itself is open (S3 + Iceberg), but the catalog is tied to a specific engine
You want to reference the same table from another engine, but the catalog does not support it
Access control is fragmented across engines, making governance difficult
Every time the catalog is changed, all engine-side configurations must be redone

In other words, even if storage and formats are open, a closed catalog significantly reduces the benefits of a lakehouse.

Especially in today’s environments where multi-cloud, multiple products, and multiple engines are commonly combined, how to unify catalogs becomes a key challenge.

Requirements for an Open Catalog

Based on this background, lakehouse catalogs are expected to meet the following requirements:

Requirement	Description
Compliance with standard APIs	Support vendor-neutral APIs such as the Iceberg REST Catalog specification
Multi-engine support	Usable across engines such as Spark, Trino, Flink, Snowflake, and Dremio
Centralized RBAC	Define permissions at the catalog level and apply consistent governance across all engines
Multi-cloud / hybrid	Not dependent on a specific cloud and capable of running on-premises when necessary
OSS sustainability	Not discontinued based on vendor decisions; continuously developed in a community-driven manner

Apache Polaris is a catalog designed to satisfy these requirements.

Differences from AWS Glue Data Catalog

When building on AWS, AWS Glue Data Catalog is often positioned as the central data catalog.
Here, we will organize the differences between AWS Glue Data Catalog and Apache Polaris.

Positioning of AWS Glue Data Catalog

AWS Glue Data Catalog is a core metadata management service in AWS.

It is natively integrated with AWS analytics services such as Athena, Glue, Redshift Spectrum, and EMR, and plays the role of managing data on S3 as a catalog.

As discussed in previous articles, Glue Data Catalog is an excellent technical catalog used by data platforms.

Is AWS Glue Data Catalog Sufficient as a Data Catalog? Organizing Its Design, Limitations, and Complementary Strategies

Functional Comparison

Aspect	AWS Glue Data Catalog	Apache Polaris
Offering	AWS-managed (closed)	OSS / Managed (Snowflake Open Catalog, Dremio Catalog, etc.)
API	AWS proprietary API (recently also provides Iceberg REST compatibility)	Iceberg REST Catalog specification (open)
Cloud support	AWS	Multi-cloud / on-prem
Engines	Athena, Glue, Redshift, EMR, Spark	Spark, Trino, Flink, Snowflake, Dremio, StarRocks, DuckDB
Multi-catalog	Account-level (logical separation via Lake Formation)	Native support for multiple catalogs within a single instance
Access control	IAM + Lake Formation	Built-in RBAC (Principal / Principal Role / Catalog Role)
External catalog integration	Limited	Can integrate with Iceberg REST-compliant catalogs (Nessie, Gravitino, etc.)
Non-Iceberg formats	Supports Hive, JSON, CSV, Parquet, etc.	Currently Iceberg-centric (Generic Table support on roadmap)

How to Interpret the Difference

Rather than being in a competitive relationship, it is easier to understand them as catalogs with different roles.

AWS Glue Data Catalog: Strong integration with AWS services, making it the primary choice for workloads completed within AWS. It supports a wide range of data lake formats beyond Iceberg and features such as S3 crawling.
Apache Polaris: A catalog that enables governance across multiple engines and clouds based on the industry-standard Iceberg REST API. It is effective when you want to enforce consistent RBAC across engines outside AWS (e.g., Snowflake, Dremio).

In summary:

If your use case is AWS-contained and includes formats beyond Iceberg, Glue Data Catalog is a practical choice
If you want common management of Iceberg across multiple engines and a vendor-neutral catalog layer, Polaris is suitable

Differences from Snowflake Horizon Catalog

This is often confused, so let’s clarify the difference between Snowflake Horizon Catalog and Apache Polaris.
Note that it is different from “Snowflake Open Catalog,” despite the similar name.

What is Snowflake Horizon Catalog?

Snowflake Horizon Catalog is a data governance and discovery suite provided by Snowflake.

For data managed within Snowflake (Snowflake-managed tables, stages, views, shared data, etc.), it provides:

Data discovery (search, tagging, descriptions)
Lineage
Data quality monitoring
Masking policies and row access policies
Automatic classification of sensitive data
Compliance management

In terms of positioning, it is similar to Amazon DataZone + Lake Formation + Glue Data Quality in AWS.

In other words, it is the layer responsible for cataloging and governance so that people can discover, understand, and trust data.

What is Snowflake Open Catalog (Relation to Polaris)

On the other hand, Snowflake Open Catalog is a managed offering of Apache Polaris.

Although the name is confusing, this is the lakehouse catalog that serves as an Iceberg REST Catalog.

In Snowflake’s model:

Snowflake Horizon Catalog: Business catalog and governance layer for Snowflake-managed data
Snowflake Open Catalog (= Apache Polaris): Lakehouse catalog layer for open table formats such as Iceberg

Functional Comparison

Aspect	Snowflake Horizon Catalog	Apache Polaris
Primary target	Data in Snowflake (internal tables, shared data, etc.)	Iceberg (Generic Table support for other formats is planned)
Layer	Business catalog / governance layer	Lakehouse catalog layer (technical catalog)
Offering	Built into Snowflake (closed)	OSS / Managed
API	Snowflake proprietary	Iceberg REST Catalog specification (open)
Data location	Snowflake internal storage or recognized external data	Iceberg tables on cloud storage
Scope	Within Snowflake organizations	Across multiple engines and clouds

How to Interpret the Difference

Again, these are not in opposition but complementary.

Snowflake Horizon Catalog: Upper layer that provides data to business users, handling discovery, quality, masking, etc.
Apache Polaris: Lower layer (metadata foundation) that exposes Iceberg tables to multiple engines

Conceptually, the structure looks like this:

┌──────────────────────────────────────────────┐
│  Business Catalog / Governance Layer         │ ← Snowflake Horizon Catalog
│  (Discovery / Lineage / Quality / Masking)   │   Amazon DataZone, etc.
└─────────────────────┬────────────────────────┘
                      │
┌─────────────────────┴────────────────────────┐
│  Lakehouse Catalog Layer                     │ ← Apache Polaris
│  (Iceberg REST Catalog / RBAC)               │   AWS Glue Data Catalog, etc.
└─────────────────────┬────────────────────────┘
                      │
┌─────────────────────┴────────────────────────┐
│  Data Lake (S3 / GCS / Azure Blob)           │
│  Iceberg / Parquet                           │
└──────────────────────────────────────────────┘

If you think of Snowflake Horizon Catalog and Apache Polaris as “choosing one or the other,” it feels unnatural, but when organized as different layers, the division of responsibilities becomes clear.

How to Combine with AWS

From here, we will consider cases where Apache Polaris is introduced into an AWS environment.
Since AWS already has a powerful catalog called Glue Data Catalog, it is important to clarify how Polaris should be positioned and who is responsible for what.

Expected Architecture

Representative configurations can be organized into the following three patterns.

Pattern 1: AWS-only (Glue Data Catalog-centered)

This is the simplest configuration.
It is a typical setup using S3 + Iceberg + Glue Data Catalog, along with Athena / Glue / Redshift Spectrum.

Catalog: AWS Glue Data Catalog
Governance: IAM + Lake Formation
Query engines: Athena, Redshift Spectrum, Glue ETL, EMR

If everything is completed within AWS and there is no strong need to share with external engines, this configuration remains the most practical.
There is no need to forcibly introduce Apache Polaris.

Pattern 2: AWS + Snowflake (Using Polaris as a shared catalog foundation)

This configuration is effective when you want to reference the same Iceberg tables from both AWS (e.g., Athena) and Snowflake.

Data storage: S3 + Iceberg
Catalog: Apache Polaris (OSS self-hosted or Snowflake Open Catalog)
AWS side: Reference Polaris as an Iceberg REST Catalog (via Spark or third-party tools)
Snowflake side: Connect to Polaris using External Volume and Catalog Integration (CATALOG_SOURCE = POLARIS)

From the Snowflake side, Polaris can be referenced directly as follows:

CREATE OR REPLACE CATALOG INTEGRATION polaris_catalog_int
  CATALOG_SOURCE = POLARIS
  TABLE_FORMAT = ICEBERG
  REST_CONFIG = (
    CATALOG_URI = 'https://<polaris-host>/api/catalog'
    CATALOG_NAME = '<your_polaris_catalog>'
  )
  REST_AUTHENTICATION = (
    TYPE = OAUTH
    OAUTH_CLIENT_ID = '<polaris_client_id>'
    OAUTH_CLIENT_SECRET = '<polaris_client_secret>'
    OAUTH_ALLOWED_SCOPES = ('PRINCIPAL_ROLE:ALL')
  )
  ENABLED = TRUE;

Pattern 3: Multi-engine / multi-cloud configuration

In addition to Snowflake, this configuration includes multiple engines such as Dremio, Databricks, Trino, and Flink.

In this case, all engines reference Polaris as a common Iceberg REST Catalog.

Data storage: S3 (and other cloud storage if needed)
Catalog: Apache Polaris (center of governance)
Query engines: Snowflake, Dremio, Spark, Trino, Flink, etc.
Governance: Polaris provides unified RBAC across all engines

How to Think About Responsibility Separation

This is the key point.
When combining Polaris, AWS, Snowflake, and others, it is important to clearly define who is responsible for which layer.

Layer	Primary Owner	Notes
Data storage (files)	AWS (S3)	Storage location of the data. Single Source of Truth
Storage access control	AWS (IAM)	Access permissions to S3 buckets/prefixes are defined on the AWS side
Table metadata	Apache Polaris	Source of Truth for Iceberg metadata such as schema, snapshots, partitions
Table-level RBAC	Apache Polaris	Applies consistent permission rules across engines
ETL / pipelines	AWS Glue / Lambda / EMR / Spark	Responsible for ingestion and transformation
Query execution	Athena / Snowflake / Dremio / Spark	Engines selected based on use case
Business catalog / discovery	Snowflake Horizon Catalog / Amazon DataZone	Higher-layer features for search, lineage, quality for users
Data quality	Glue Data Quality / Snowflake DMF	Implemented at engine or quality service layer

What is especially important is the three-layer separation:

Data resides in AWS, the catalog is Polaris, and usage is handled by each engine

By making this separation explicit:

AWS can focus on storage and IAM management
Polaris can focus on metadata and access control
Each query engine can focus on its strengths

Considerations When Adopting Polaris

Polaris is powerful, but there are also important considerations:

Operational cost when self-hosting OSS: Running on EKS or EC2 requires a metastore (e.g., PostgreSQL), authentication infrastructure, monitoring, and upgrade handling
Managed services are often more practical: Using Snowflake Open Catalog or Dremio Catalog significantly reduces operational burden
Less seamless integration with AWS services compared to Glue: For AWS-native services such as Athena, Redshift, and QuickSight, using Glue Data Catalog is far more straightforward
Need to avoid double governance: If IAM policies on S3 and RBAC in Polaris are inconsistent, troubleshooting becomes complex

In other words, when deciding whether to adopt Apache Polaris in an AWS environment, it is realistic to evaluate based on:

Whether multi-engine requirements exist
The organization’s stance on vendor lock-in
Whether operational cost is acceptable (or managed services can be used)

A Practical Approach

Personally, when considering Polaris in an AWS environment, the following phased approach is practical:

Build a lakehouse within AWS using Glue Data Catalog + Iceberg
When integration with other engines such as Snowflake becomes necessary, consider introducing an Iceberg REST layer
At that point, compare “Glue Iceberg REST endpoint,” “Apache Polaris OSS,” and “Snowflake Open Catalog” based on requirements
If multi-engine / multi-cloud requirements become clear, redesign with Polaris (especially managed) at the center

Rather than designing with Polaris from the beginning, it is often more practical to replace the catalog layer with an open one when requirements mature.

Conclusion

In this article, we organized the key points around Apache Polaris.

In the world of data platforms, while storage and formats have become open, a closed catalog reduces the benefits of a lakehouse by half.

Therefore, there is a need for an open catalog that complies with the Iceberg REST Catalog specification and enables unified governance across multiple engines and clouds.
Apache Polaris is designed to fulfill exactly that role.

However, it is important to think not in terms of “which one to choose” among Polaris, AWS Glue Data Catalog, and Snowflake Horizon Catalog, but rather which layer each is responsible for:

AWS Glue Data Catalog: Technical catalog within AWS (still the primary choice for AWS-only workloads)
Apache Polaris: Lakehouse catalog centered on Iceberg, shared across multiple engines
Snowflake Horizon Catalog: Business catalog and governance layer for Snowflake users

Even when combining with AWS, by consciously separating responsibilities as
“data in AWS, catalog in Polaris, analytics in engines, business catalog in another layer”,
you can design an architecture that leverages the strengths of each.

Going forward, lakehouse architectures are expected to increasingly adopt vendor-neutral designs.
Apache Polaris is likely to become an important component supporting that openness.

I hope this article will be helpful for those considering Apache Polaris or designing lakehouse architectures across multiple platforms such as AWS and Snowflake.

Lightweight ETL on AWS Lambda Using DuckDB and Snowflake Connector

Aki — Sat, 04 Apr 2026 13:54:10 +0000

Original Japanese article: AWS Lambda × DuckDB × Snowflake ConnectorによるETLの実装

Introduction

I'm Aki, an AWS Community Builder (@jitepengin).

In my previous article, I introduced how to connect to Snowflake from AWS Lambda using Key Pair authentication.

Securely Implementing Snowflake AWS Lambda Integration with Key Pair Authentication + Secrets Manager

This time, I would like to try the event-driven data ingestion approach that I introduced in the previous article.

In this article, I will implement an event-driven ETL pipeline that uses DuckDB on AWS Lambda to perform lightweight transformations on Parquet files stored in Amazon S3 and then load the processed data into Snowflake.

In addition, during the implementation process, I encountered an interesting limitation where write_pandas fails when writing to a Catalog-Linked Database. I will also summarize the root cause and the workaround.

Why Snowpipe Is Not Enough

Snowpipe is a very convenient feature for automatic data ingestion.

However, it has limitations when it comes to data transformation and complex filtering.

In other words, when you need preprocessing, filtering, or the integration of multiple events, you need to choose another approach.

In such cases, AWS Lambda becomes a strong option due to its high flexibility.

Architecture

A Parquet file is uploaded to S3
Lambda is triggered by the S3 event
DuckDB reads the data and performs the required transformations
snowflake.connector writes the data into Snowflake

The two key libraries used in this implementation are shown below.

DuckDB

DuckDB is an embedded database engine designed for OLAP (Online Analytical Processing).

Because DuckDB is extremely lightweight and supports in-memory processing, it can run efficiently even in a simple execution environment such as AWS Lambda.

It is said to provide particularly strong performance for batch workloads such as data analytics and ETL processing.

In addition, it enables SQL-based filtering and lightweight data transformations, allowing for intuitive implementations.

https://duckdb.org/

Snowflake Connector

Snowflake Connector for Python is a library that provides an interface for connecting to Snowflake and executing all standard operations.

By using this library, it becomes possible to operate Snowflake from runtime environments such as Lambda.

https://docs.snowflake.com/en/developer-guide/python-connector/python-connector

Sample Code

In the sample code below, WHERE VendorID = 1 is added as an ETL filter.

By performing filtering and data transformation inside Lambda, highly flexible preprocessing becomes possible.

import duckdb
import boto3
import json
import snowflake.connector
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.backends import default_backend
from snowflake.connector.pandas_tools import write_pandas
import time
import random

SECRET_ID = "snowflake-keypair"


def get_secret():
    client = boto3.client("secretsmanager")
    response = client.get_secret_value(SecretId=SECRET_ID)
    return json.loads(response["SecretString"])

def lambda_handler(event, context):
    conn = None
    duckdb_connection = None

    try:
        duckdb_connection = duckdb.connect(database=":memory:")
        duckdb_connection.execute("SET home_directory='/tmp'")
        duckdb_connection.execute("INSTALL httpfs")
        duckdb_connection.execute("LOAD httpfs")

        s3_bucket = event["Records"][0]["s3"]["bucket"]["name"]
        s3_object_key = event["Records"][0]["s3"]["object"]["key"]

        s3_input_path = f"s3://{s3_bucket}/{s3_object_key}"
        print(f"S3 input path: {s3_input_path}")


        query = f"""
            SELECT *
            FROM read_parquet('{s3_input_path}')
            WHERE VendorID = 1
        """

        result_arrow_table = duckdb_connection.execute(query).fetch_arrow_table()

        print(f"DuckDB filtered rows: {result_arrow_table.num_rows}")

        secret = get_secret()

        private_key_obj = serialization.load_pem_private_key(
            secret["privateKey"].encode("utf-8"),
            password=None,
            backend=default_backend()
        )

        conn = snowflake.connector.connect(
            user=secret["user"],
            account=secret["account"],
            private_key=private_key_obj,
            role=secret.get("role"),
            warehouse=secret.get("warehouse"),
            database=secret.get("database"),
            schema=secret.get("schema")
        )

        cur = conn.cursor()
        cur.execute(f"USE DATABASE {secret['database']}")
        cur.execute(f"USE SCHEMA {secret['schema']}")
        cur.close()

        import pandas as pd

        df = result_arrow_table.to_pandas()

        df["tpep_pickup_datetime"] = pd.to_datetime(
            df["tpep_pickup_datetime"],
            unit="us"
        ).dt.strftime("%Y-%m-%d %H:%M:%S")

        df["tpep_dropoff_datetime"] = pd.to_datetime(
            df["tpep_dropoff_datetime"],
            unit="us"
        ).dt.strftime("%Y-%m-%d %H:%M:%S")

        success, nchunks, nrows, _ = write_pandas(
            conn,
            df,
            table_name="YELLOW_TRIPDATA"
        )

        print(
            f"Snowflake write success={success}, "
            f"rows={nrows}, chunks={nchunks}"
        )

        return {
            "statusCode": 200,
            "body": (
                f"Processed {result_arrow_table.num_rows} rows "
                f"and wrote {nrows} rows to Snowflake."
            )
        }

    except Exception as e:
        print(f"An error occurred: {e}")
        import traceback
        traceback.print_exc()

        return {
            "statusCode": 500,
            "body": str(e)
        }

    finally:
        if conn:
            conn.close()

        if duckdb_connection:
            duckdb_connection.close()

Execution Result

As shown above, the data was successfully written.

Switching the Destination to a Catalog-Linked Database

As introduced in a previous article, what happens if we try writing to a table configured with a Catalog-Linked Database (Iceberg)?

Let’s test it.

AWS Snowflake Lakehouse: 2 Practical Apache Iceberg Integration Patterns

A Write Error Occurs

When attempting to write to the Catalog-Linked Database, the following error occurred:

{
  "statusCode": 500,
  "body": "093678 (0A000): SQL Compilation Error: This operation is not supported in a catalog-linked database."
}

Why Writing to a Catalog-Linked Database Fails

The reason this happens is due to the interaction between write_pandas in the Snowflake Connector and the constraints of a Catalog-Linked Database.

Internally, write_pandas creates a temporary stage.

Python Connector API

Writes a pandas DataFrame to a table in a Snowflake database.
To write the data to the table, the function saves the data to Parquet files, uses the PUT command to upload these files to a temporary stage, and uses the COPY INTO <table> command to copy the data from the files to the table.

You can use some of the function parameters to control how the PUT and COPY INTO <table> statements are executed.

However, stage creation is not supported in a Catalog-Linked Database.

Considerations for using a catalog-linked database for Iceberg tables

You can create schemas, externally managed Iceberg tables, or database roles in a catalog-linked database. Creating other Snowflake objects isn't currently supported.

This conflict causes write_pandas to fail with the error:

This operation is not supported in a catalog-linked database.

More specifically, the temporary stage created internally falls under the category of “other Snowflake objects,” so the error occurs at the point where CREATE TEMPORARY STAGE is executed.

That said, there is a workaround.

How to Write to a Catalog-Linked Database

A relatively simple approach is to use an INSERT statement directly.

Here is an example implementation:

        for i in range(0, len(df), 1000): 
            chunk = df.iloc[i:i+1000]
            columns = ", ".join(chunk.columns)
            placeholders = ", ".join(["%s"] * len(chunk.columns))
            sql = f"INSERT INTO {secret['schema']}.YELLOW_TRIPDATA ({columns}) VALUES ({placeholders})"
            cur.executemany(sql, chunk.values.tolist())

Alternatively, another good approach is to create a stage in a different database and execute the INSERT through that route.

Conclusion

In this article, I implemented a lightweight event-driven ETL pipeline triggered by S3 events using AWS Lambda, DuckDB, and the Snowflake Connector.

By using DuckDB inside Lambda, I was able to perform SQL-based filtering and lightweight transformations directly on Parquet files stored in S3, and successfully load the processed results into Snowflake.

In addition, I confirmed an important limitation: when using write_pandas against a Catalog-Linked Database (Iceberg), the write fails because the connector internally creates a temporary stage.

Although there are some constraints, combining DuckDB and the Snowflake Connector enables the construction of a low-cost and flexible data processing pipeline for Snowflake.

The key point is to clearly understand how Snowflake manages Iceberg tables.

It is important to determine whether the table is a Snowflake-managed Iceberg table or connected through mechanisms such as a Catalog-Linked Database, and to properly understand that structure.

In any case, the combination of Snowflake and Iceberg is an extremely powerful option for building a Lakehouse architecture.

I hope this article will be helpful for those considering lightweight data processing and real-time ETL pipelines with AWS and Snowflake when working with Iceberg tables.

Securely Implementing Snowflake AWS Lambda Integration with Key Pair Authentication + Secrets Manager

Aki — Thu, 02 Apr 2026 14:14:17 +0000

Original Japanese article: Snowflake × AWS Lambda連携をKey Pair認証 + Secrets Managerで安全に実装する

Introduction

I'm Aki, an AWS Community Builder (@jitepengin).

When building data pipelines and business systems on AWS, there are cases where you need to access Snowflake directly as part of your application processing.

For example, the following use cases are common:

Load data received via an API into Snowflake
Execute SQL on the Snowflake side after ETL / ELT processing is completed
Write results from external system integrations into Snowflake
Trigger Snowflake stored procedures as part of task execution

AWS Lambda works extremely well with serverless, event-driven processing, and by combining it with Snowflake, you can build a highly flexible data integration platform.

On the other hand, when connecting from Lambda to Snowflake, it is critically important to determine how authentication credentials should be managed securely.

As of 2026, for Snowflake system integrations, Key Pair authentication or OAuth has become the standard, and embedding passwords directly is no longer recommended.

In this article, I would like to explain an implementation pattern that uses AWS Secrets Manager to securely manage the private key and connects from Lambda to Snowflake using Key Pair authentication.

Reference: Deprecation plan for single-factor password sign-in
https://docs.snowflake.com/ja/user-guide/security-mfa-rollout

First of all, why access Snowflake from Lambda?

Before getting into the implementation, let’s first consider why it is necessary to access Snowflake from Lambda in the first place.

I’d like to explain this using three common use case patterns.

Case 1: Event-driven data ingestion

I think this is probably the most common pattern.

S3 / API Gateway / EventBridge
            ↓
         Lambda
            ↓
       Snowflake

For example:

order data received through an API
CSV / JSON files uploaded to S3
SaaS integration events

These can be received by Lambda and then directly loaded into Snowflake.

There are architectures that use Snowpipe, but when:

preprocessing is required
format conversion is required
multiple system integrations are involved

Lambda is often easier to work with.

Case 2: Snowflake integration after AWS ETL completion

This pattern executes SQL on Snowflake via Lambda after ETL processing has been completed on the AWS side.

This example assumes Apache Iceberg is being used as the data platform.

Glue / Lambda
      ↓
S3 Iceberg
      ↓
Lambda
      ↓
Execute SQL on Snowflake

For example:

ETL completed on AWS
REFRESH on Snowflake
execute MERGE SQL
data quality checks
update BI marts

This is a common pattern for downstream processing.

Case 3: Operations automation

This is another pattern that can often be seen in real-world implementations.

For example:

starting / stopping warehouses
executing tasks
automatic SQL execution

These operations can be automated from Lambda.

Architecture

The architecture used in this article is as follows.

Event Source
   ↓
AWS Lambda
   ↓
Secrets Manager
   ↓
Retrieve Private Key
   ↓
Snowflake

Lambda retrieves the private key from Secrets Manager and connects to Snowflake using Key Pair authentication.

Security best practices

1. Do not store the private key in environment variables

This is important not only for Snowflake, but for any sensitive credentials.

A common anti-pattern is storing it like this:

PRIVATE_KEY = os.environ["PRIVATE_KEY"]

This should absolutely be avoided.

In addition to security concerns, there are operational issues such as:

possibility of accidental log output
operational overhead during configuration changes
difficulty in rotation
weak access control

2. Centralized management with Secrets Manager

Sensitive information like this should ideally be centrally managed using Secrets Manager.

By using Secrets Manager, you gain the following benefits:

IAM-based access control
KMS encryption
audit logging with CloudTrail
credential rotation support

3. Principle of least privilege for IAM

Only the minimum required permissions should be granted to Lambda.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "arn:aws:secretsmanager:ap-northeast-1:123456789012:secret:snowflake-keypair-*"
    }
  ]
}

4. Use a dedicated Snowflake service user

Do not use personal users.

Instead, create a dedicated service user.

The service user should be created with:

TYPE = 'LEGACY_SERVICE'
or TYPE = 'SERVICE'

Registering authentication information

1. Create the private key

First, create the private key.

openssl genrsa 2048 | openssl pkcs8 -topk8 -inform PEM -out rsa_key.p8 -nocrypt
openssl rsa -in rsa_key.p8 -pubout -out rsa_key.pub

2. Configure Snowflake

ALTER USER lambda_service_user
SET RSA_PUBLIC_KEY='MIIBIjANBgkq...';

Use the contents of the rsa_key.pub file as the public key.

3. Register in Secrets Manager

Store the private key as follows.

{
  "account": "XXXXXXXXXX",
  "user": "lambda_service_user",
  "privateKey": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----",
  "passphrase": "",
  "role": "LAMBDA_ROLE",
  "warehouse": "COMPUTE_WH",
  "database": "ICEBERGDB",
  "schema": "PUBLIC"
}

account: Snowflake account identifier
user: Snowflake connection user
privateKey: private key created in step 1 (rsa_key.p8)

The <orgname>-<account_name> format is recommended for the account identifier.

Now the preparation is complete.

Lambda sample code

For verification purposes, this example simply executes:

SELECT CURRENT_VERSION()

The key point is using snowflake.connector, which makes connecting to Snowflake very straightforward.

Add snowflake.connector using a Lambda Layer, or build the Lambda function as a container image.

import json
import boto3
import snowflake.connector
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.backends import default_backend
import time
import random

SECRET_ID = "snowflake-keypair"

def get_secret():
    client = boto3.client("secretsmanager")
    response = client.get_secret_value(SecretId=SECRET_ID)
    return json.loads(response["SecretString"])

def execute_query_with_retry(conn, query, max_retries=3):
    retry_count = 0
    while retry_count < max_retries:
        try:
            cur = conn.cursor()
            cur.execute(query)
            result = cur.fetchone()
            cur.close()
            return result
        except snowflake.connector.Error as e:
            print(f"Query failed (attempt {retry_count+1}/{max_retries}): {e}")
            retry_count += 1
            sleep_time = min(2 ** retry_count, 30) + random.random()
            time.sleep(sleep_time)
    raise Exception("Query execution exceeded max retries.")

def lambda_handler(event, context):
    secret = get_secret()

    private_key_obj = serialization.load_pem_private_key(
        secret["privateKey"].encode("utf-8"),
        password=None,
        backend=default_backend()
    )

    conn = None
    try:
        conn = snowflake.connector.connect(
            user=secret["user"],
            account=secret["account"],
            private_key=private_key_obj,
            role=secret.get("role"),
            warehouse=secret.get("warehouse"),
            database=secret.get("database"),
            schema=secret.get("schema")
        )

        result = execute_query_with_retry(conn, "SELECT CURRENT_VERSION()")
        return {
            "statusCode": 200,
            "body": str(result)
        }

    except Exception as e:
        print(f"Lambda execution error: {e}")
        return {
            "statusCode": 500,
            "body": str(e)
        }
    finally:
        if conn:
            conn.close()

Depending on default service user settings can lead to unintended behavior during configuration changes.

Therefore, explicitly specifying:

role
warehouse
database
schema

is safer for production environments.

Execution result

{
  "statusCode": 200,
  "body": "('10.11.2',)"
}

Executed on Snowflake side:

SELECT CURRENT_VERSION()
>10.11.2

The results match, so the connection was successful.

Additional considerations

Strengthening network security

Using AWS PrivateLink enables communication within a private network and eliminates internet-based communication risks.

This can be implemented with the following steps:

enable private connectivity in Snowflake
create a VPC endpoint (com.amazonaws.vpce.snowflake)
pair it with Snowflake’s service principal
place Lambda in the same VPC
restrict access using security groups

Error handling

As implemented in the sample code, it is a good idea to add retry processing to tolerate temporary network failures and transient Snowflake issues.

Using exponential backoff improves resilience compared to fixed delays.

Monitoring

The following monitoring perspectives are recommended:

CloudWatch Logs: capture connection errors and query failures
CloudTrail: verify access logs to Secrets Manager
Snowflake audit logs: monitor user connection activity

Conclusion

In this article, I introduced the full flow from key management to connection setup for accessing Snowflake from AWS Lambda using Key Pair authentication.

There are more use cases for Lambda-to-Snowflake access than many people expect, especially for:

event-driven processing
downstream ETL workflows

And of course, secure credential management is critically important—not only for Snowflake, but for any system integration.

At this point in time, Key Pair authentication + Secrets Manager is likely the standard implementation pattern for connecting to Snowflake.

I hope this article helps as a reference when building application integrations between AWS and Snowflake.

AWS Snowflake Lakehouse: 2 Practical Apache Iceberg Integration Patterns

Aki — Tue, 31 Mar 2026 20:44:32 +0000

Original Japanese article: AWSのレイクハウス（Apache Iceberg）をSnowflakeと連携する2つのパターンを整理する

Introduction

I'm Aki, an AWS Community Builder (@jitepengin).

In recent years, it has become increasingly common to build lakehouse architectures centered around Apache Iceberg.

Before the rise of lakehouse architecture, it was common to design systems where data was consolidated into a specific platform, such as:

an architecture centered around Amazon Redshift on AWS
an architecture centered around internal tables in Snowflake

However, with the advent of Apache Iceberg, this assumption is rapidly changing.

Now that data on Amazon S3 can be directly accessed from multiple engines, what matters is no longer simply product selection.
Instead, the architecture design itself has become the central focus: where the data resides, who owns the write responsibility, and who holds governance authority.

In this article, focusing on the coexistence of AWS and Snowflake, I will organize the following:

two patterns based on S3 × Iceberg
connectivity with Power BI Service
future prospects including AI utilization

Why AWS × Snowflake Coexistence Is Necessary

The greatest value of Apache Iceberg lies in its ability to separate data from the query engine.

For example, while keeping the physical data stored in S3, the same data can be accessed from multiple tools such as Athena, AWS Glue / Spark, Redshift, and Snowflake.

In other words, while consolidating the physical data into a single location, it has become possible to choose the most suitable analytics platform depending on the use case.

As a result, the architectural discussion has shifted from “which product should we use?” to “where should data sovereignty reside, and where should analytical ownership be placed?”

The benefits of using Snowflake here include:

user-friendly UI/UX
powerful SQL analytics capabilities
integration with Cortex AI

In particular, I believe that an architecture where AWS serves as the data sovereignty layer while Snowflake is utilized as the analytics layer is highly compatible.

Two Patterns for Snowflake × S3 Integration

Here, I will organize the two commonly used patterns when integrating Snowflake with S3.

Pattern 1: Glue Catalog Integration

In this pattern, Iceberg tables stored on S3 are referenced from Snowflake through AWS Glue Data Catalog.

The advantages of this architecture are:

relatively simple configuration
S3 becomes the Single Source of Truth
because Snowflake cannot write, AWS retains sovereignty over data management (this can also be considered a disadvantage)
since user access is consolidated into Snowflake, access control can be centralized there

In other words, Snowflake focuses solely on the role of data analytics (query engine), while AWS retains authority over data management.

Setup Procedure

Step 1: Create an External Volume (S3 Access Configuration)

Run the following on the Snowflake side:

CREATE EXTERNAL VOLUME IF NOT EXISTS sample_iceberg_volume
  STORAGE_LOCATIONS = (
    (
      NAME = 'my-s3-location'
      STORAGE_PROVIDER = 'S3'
      STORAGE_BASE_URL = catalog S3 path
      STORAGE_AWS_ROLE_ARN = role ARN to use
      STORAGE_AWS_EXTERNAL_ID = 'my_external_id'
    )
  );

Step 2: Create Glue Catalog Integration

Run the following on the Snowflake side:

CREATE OR REPLACE CATALOG INTEGRATION glue_catalog_int -- arbitrary name
  CATALOG_SOURCE = GLUE
  CATALOG_NAMESPACE = Glue catalog namespace
  TABLE_FORMAT = ICEBERG
  GLUE_AWS_ROLE_ARN = role ARN to use
  GLUE_CATALOG_ID = Glue catalog ID to use
  GLUE_REGION = 'ap-northeast-1'
  ENABLED = TRUE;

Step 3: Retrieve Required Information for AWS Trust Policy

DESC EXTERNAL VOLUME sample_iceberg_volume;

→ Note down STORAGE_AWS_IAM_USER_ARN and STORAGE_AWS_EXTERNAL_ID

DESC CATALOG INTEGRATION glue_catalog_int;

→ Note down GLUE_AWS_IAM_USER_ARN and GLUE_AWS_EXTERNAL_ID

Since the two External IDs above will have different values, be sure to add both to the AWS IAM role Trust Policy.

Please configure the Trust Policy for the role used on the AWS side.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": STORAGE_AWS_IAM_USER_ARN
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringEquals": {
                    "sts:ExternalId": STORAGE_AWS_EXTERNAL_ID
                }
            }
        },
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": GLUE_AWS_IAM_USER_ARN
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringEquals": {
                    "sts:ExternalId": GLUE_AWS_EXTERNAL_ID
                }
            }
        }
    ]
}

Step 4: Create Database

Run the following on the Snowflake side:

CREATE DATABASE IF NOT EXISTS icebergdb;

Step 5: Create Table

Run the following on the Snowflake side:

CREATE OR REPLACE ICEBERG TABLE icebergdb.public.yellow_tripdata -- arbitrary name
  EXTERNAL_VOLUME = 'sample_iceberg_volume'
  CATALOG = 'glue_catalog_int'
  CATALOG_TABLE_NAME = 'yellow_tripdata'
  AUTO_REFRESH = TRUE;

Step 6: Verify Data

Run the following on the Snowflake side:

SELECT * FROM icebergdb.public.yellow_tripdata LIMIT 5;

We were able to read the data from S3!

Pattern 2: Catalog-Linked Database (Iceberg)

This pattern integrates Glue Catalog using the REST Catalog approach and manages Iceberg tables directly from the Snowflake side.

The advantages of this architecture are:

read and write operations from Snowflake are possible
the physical data remains stored in S3
Snowflake users can perform SQL-based updates and analytics
easier integration with Power BI and Cortex AI

In other words, the biggest feature is that analysis and updates can be performed from Snowflake while keeping the physical data in S3.

However, governance must be considered from both the AWS side and the Snowflake side.

Setup Procedure

Step 1: Create an External Volume (S3 Access Configuration)

CREATE EXTERNAL VOLUME IF NOT EXISTS sample_iceberg_volume
  STORAGE_LOCATIONS = (
    (
      NAME = 'my-s3-location'
      STORAGE_PROVIDER = 'S3'
      STORAGE_BASE_URL = catalog S3 path
      STORAGE_AWS_ROLE_ARN = role ARN to use
      STORAGE_AWS_EXTERNAL_ID = 'my_external_id'
    )
  );

Step 2: Create Glue Iceberg REST Catalog Integration

CREATE OR REPLACE CATALOG INTEGRATION glue_rest_catalog_int
  CATALOG_SOURCE = ICEBERG_REST
  TABLE_FORMAT = ICEBERG
  CATALOG_NAMESPACE = Glue catalog namespace
  REST_CONFIG = (
    CATALOG_URI = Glue catalog URI
    CATALOG_API_TYPE = AWS_GLUE
    CATALOG_NAME = AWS account ID
  )
  REST_AUTHENTICATION = (
    TYPE = SIGV4
    SIGV4_IAM_ROLE = role ARN to use
    SIGV4_SIGNING_REGION = 'ap-northeast-1'
  )
  ENABLED = TRUE;

Step 3: Retrieve Required Information for AWS Trust Policy

DESC EXTERNAL VOLUME sample_iceberg_volume;

→ Note down STORAGE_AWS_IAM_USER_ARN and STORAGE_AWS_EXTERNAL_ID

DESC CATALOG INTEGRATION glue_rest_catalog_int;

→ Note down GLUE_AWS_IAM_USER_ARN and GLUE_AWS_EXTERNAL_ID

Since the two External IDs above will have different values, be sure to add both to the AWS IAM role Trust Policy.

Please configure the Trust Policy for the role used on the AWS side.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": STORAGE_AWS_IAM_USER_ARN
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringEquals": {
                    "sts:ExternalId": STORAGE_AWS_EXTERNAL_ID
                }
            }
        },
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": GLUE_AWS_IAM_USER_ARN
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringEquals": {
                    "sts:ExternalId": GLUE_AWS_EXTERNAL_ID
                }
            }
        }
    ]
}

Step 4: Create Catalog Linked Database (Read/Write Enabled)

CREATE DATABASE my_iceberg_linked_db
  LINKED_CATALOG = (
    CATALOG = 'glue_rest_catalog_int',
    ALLOWED_WRITE_OPERATIONS = ALL
  )
  EXTERNAL_VOLUME = 'sample_iceberg_volume';

Step 5: Tables Are Automatically Discovered (Synced Every 30 Seconds)

SELECT * FROM my_iceberg_linked_db."icebergdb"."yellow_tripdata" LIMIT 5;

We were able to read the data from S3!

Step 6: Write Test

INSERT INTO my_iceberg_linked_db."icebergdb"."yellow_tripdata"
  (vendorid, tpep_pickup_datetime, tpep_dropoff_datetime, passenger_count,
   trip_distance, ratecodeid, store_and_fwd_flag, pulocationid, dolocationid,
   payment_type, fare_amount, extra, mta_tax, tip_amount, tolls_amount,
   improvement_surcharge, total_amount, congestion_surcharge, airport_fee)
VALUES
  (1, '2025-01-01 13:00:00', '2025-01-01 13:30:00', 2,
   3.5, 1, 'N', 100, 200,
   1, 15.0, 2.5, 0.5, 3.0, 0,
   1.0, 22.0, 2.5, 0);

Step 7: Verify Write

SELECT * FROM my_iceberg_linked_db."icebergdb"."yellow_tripdata"
WHERE tpep_pickup_datetime = '2025-01-01 13:00:00';

We were able to write data into S3!

Athena side

Before write

After write

Which Pattern Should You Choose?

For practical use, the following way of organizing it is easy to understand.

Use Case	Recommended Pattern
AWS-led ETL workloads, with Snowflake primarily used for read/query access	Pattern 1
BI / AI / SQL updates driven primarily from Snowflake	Pattern 2
Governance needs to be centralized on AWS	Pattern 1
Operations are mainly led by Snowflake users	Pattern 2

Integration with Power BI Service

Traditionally, when referencing an S3-based lakehouse from Power BI Service, many architectures used Redshift as an intermediary.

In such cases, securely connecting directly from Power BI Service often required provisioning an EC2 instance and configuring an on-premises data gateway.

This introduces additional operational costs such as EC2 management.

Now, how about Snowflake?

Power BI provides a native connector for Snowflake, allowing direct authentication and connection from Power BI Service.

This eliminates the need for relay servers or on-premises gateways that are often required in Redshift-based architectures.

In other words, it becomes possible to exclude costly operational components such as on-premises gateways.

In addition, since the semantically organized Gold layer on the Snowflake side can be directly connected to Power BI, this also improves usability for BI users.

Thinking in Terms of Medallion Architecture

Personally, I consider the following architecture to be highly practical for real-world use.

Bronze: S3 + Iceberg
Silver: S3 + Iceberg (with Snowflake integration as needed)
Gold: S3 + Iceberg → Snowflake integration

By especially aligning the Gold layer with Snowflake, it becomes easier to provide a semantic layer that is easy for BI users and business departments to consume.

Depending on the use case, the Silver layer can also be utilized for more detailed analysis.

In other words, this enables a separation of responsibilities between data management and data analytics.

Thinking in Terms of Separation of Ownership

This is the most important point in this article.

Domain	Ownership
Physical data	AWS
Governance	AWS / Snowflake
Analytics	Snowflake
AI interaction	Snowflake

What matters is not the product itself, but how ownership is separated.

By clearly defining this separation, it becomes easier to organize the scope of responsibilities across data engineering, BI, and AI utilization, which also provides benefits from an organizational management perspective.

The Change Brought by Snowflake Cortex AI

AI utilization will become even more important going forward.

As with other data platforms, AI adoption is progressing rapidly in Snowflake as well.

By leveraging Snowflake Cortex AI, it becomes possible to query Iceberg tables on S3 using natural language.

In other words, the data platform is evolving from “a platform for writing SQL” into “a platform for conversation.”

AI utilization is expected to continue evolving in many aspects.

One key point will be preparing data that is easier for AI to use—in other words, AI-ready data.

Conclusion

In this article, I organized two patterns for integrating an AWS lakehouse (Apache Iceberg) with Snowflake.

In recent data utilization scenarios, it is increasingly common not only to rely solely on AWS, but also to integrate with platforms such as Databricks and Snowflake as introduced here.

As mentioned earlier, what matters is not the product itself, but how ownership is separated.

Depending on which service takes responsibility for data, governance, and analytics, both the architecture and configuration will change.

In any case, what is truly important is not the product itself, but the perspective of how to design a data platform that users will continue to use over time.

Going forward, it will become even more important to design architectures not only from the perspective of where data is stored, but also from the viewpoint of who owns responsibility for each layer and how that responsibility connects to user value.

I hope this article will be helpful for those considering a combination of AWS and Snowflake.

Is AWS Glue Data Catalog Sufficient as a Data Catalog? Organizing Its Design, Limitations, and Complementary Strategies

Aki — Wed, 25 Mar 2026 14:41:14 +0000

Original Japanese article: AWS Glue Data Catalogはデータカタログとして十分か？設計・限界・補完戦略を整理する

Introduction

I'm Aki, an AWS Community Builder (@jitepengin).

As data utilization within organizations has advanced in recent years, the importance of data catalogs has continued to grow.

When building a data platform on AWS, the first thing that typically comes to mind as a data catalog is AWS Glue Data Catalog.
Especially in data lake architectures centered around Amazon S3, AWS Glue Data Catalog is almost a prerequisite service. By combining it with services like Athena, AWS Glue, and Redshift Spectrum, it is possible to quickly stand up a minimal data platform.

However, as data usage evolves, you may encounter challenges such as:

Not knowing which data to use
Multiple datasets that look similar
Being unable to determine whether data is trustworthy
Not being able to trace how data was generated

At first glance, these may appear to be separate issues, but in reality, they all stem from a single root cause: an insufficient data catalog.

In this article, starting from AWS Glue Data Catalog, we will explore:

The role of a data catalog
The strengths and limitations of AWS Glue Data Catalog
How to complement it within AWS
How to approach building a data catalog on AWS
A comparison with other data catalogs (OpenMetadata)

The conclusion is that AWS Glue Data Catalog is not a “data catalog” in the full sense.
Rather, it is a technical catalog used by query engines, and it is not sufficient as a catalog for humans to discover, understand, and trust data.

For this reason, a data catalog on AWS should not be designed as a single service, but as an architecture composed of multiple services.

What is a Data Catalog?

A data catalog is not simply about metadata management—it is a foundation that makes data usable.

Metadata can be understood as “data about data.”
Specifically, it includes information such as who created the data, what it means, how it is used, where it came from, how it flows, where it is stored, and what its quality is.

A data catalog centralizes this metadata and supports search and utilization.

Traditionally, data catalogs focused on table definitions and schema management. However, modern data catalogs are expected to include the following elements:

Category	Description
Metadata Management	Technical metadata (schemas, types, partitions), business metadata (meaning, usage, owner), operational metadata (job logs, processing metrics)
Data Discovery	Data discovery, filtering, classification
Data Lineage	Tracking data generation and transformation
Data Quality	Reliability indicators, anomaly detection
Data Governance	Access control and permission management

These elements span multiple domains defined in DMBOK, and it is rare for a single tool to cover all of them.

As will be explained later, AWS also requires combining multiple services to achieve this.
In the AWS context, it is more appropriate to think of a data catalog not as a single “service,” but as an “architecture.”

Role and Strengths of Glue Data Catalog

Glue Data Catalog is the core metadata management component in AWS and serves as a foundational element for operating a data platform.

Key Features of Glue Data Catalog

Feature	Description
Metadata Storage	Persistent storage of structured metadata
Schema Management	Definition and updates of table schemas
Partition Management	Management of partition information
Statistics	Column statistics such as min/max values and null counts
Tagging	Classification using key-value pairs
API/SDK	Programmatic access
Data Lineage	Basic lineage is available; advanced visualization requires additional tools
Operational Metadata	CloudWatch logs, Spark UI, job execution insights
Advanced Discovery	Console browsing, attribute filtering, unified search

Strengths of Glue Data Catalog

Seamless Integration with AWS Services

While this may seem obvious, it is an important point: Glue integrates natively with AWS services such as Athena, AWS Glue, and Redshift Spectrum.

Because these services reference the same catalog, it ensures consistency in how data is accessed across the platform.

Strong Affinity with Data Lakes

In modern lakehouse architectures, this is a significant advantage.

Glue Data Catalog allows data stored in S3 to be cataloged directly.
This makes it possible to build a lakehouse using formats like Iceberg and manage it through Glue Data Catalog.

(Note: Iceberg table metadata itself resides in S3, while Glue Data Catalog functions as the catalog endpoint.)

Is Glue Data Catalog Sufficient as a Data Catalog?

The conclusion is that Glue Data Catalog is a data platform–oriented catalog, not a user-oriented catalog.

It is highly effective as a technical metadata foundation referenced by analytics platforms.
However, its capabilities are limited when it comes to serving as a business catalog that enables users to discover, understand, and trust data.

In other words, Glue is extremely strong as the core (technical foundation) of a data catalog, but requires complementary services when used as a user-facing catalog that supports data utilization.

Category	Support by Glue Alone	Notes
Technical Metadata	○	Schemas, types, partitions, column statistics
Business Metadata	△	Descriptions, tags, classifications (advanced capabilities require Amazon DataZone)
Operational Metadata	△	Job execution history is stored; detailed metrics are managed in CloudWatch
Data Discovery	△	Console search and filtering (advanced capabilities require Amazon Q or Amazon DataZone)
Data Lineage	△	Basic lineage (input/output tables in Glue ETL jobs) is captured; no end-to-end lineage
Data Quality	△	Column statistics and auto statistics (advanced capabilities require Glue Data Quality or AWS Glue DataBrew)
Workflow Management	✕	Not handled by Glue Data Catalog
Data Governance	△	IAM integration, resource policies, encryption (advanced capabilities require Lake Formation)
Data Profiling	✕	Not supported

Supplement: How to Complement Areas Where Glue Alone Falls Short

Operational Metadata
Advanced metrics (e.g., processed record counts, error rates, memory usage) need to be managed using CloudWatch or AWS X-Ray.
Data Lineage
For end-to-end, advanced lineage visualization, you need Amazon DataZone, support for the OpenLineage specification, or the AWS Lineage API.
Data Quality
- Glue Data Quality: Rule-based validations (e.g., NULL checks, range checks)
- Glue DataBrew: Statistical profiling, distribution analysis, outlier detection (ML-based)
Workflow Management
Utilize AWS Step Functions, Amazon Managed Workflows for Apache Airflow (MWAA), or AWS Glue Workflows.
Data Profiling
Perform statistical profiling with Glue DataBrew, and detect sensitive data (PII classification) using Amazon Macie.

In summary, the following capabilities are not fully provided by Glue alone and must be complemented:

Business metadata management
Advanced data lineage
Data quality
Data discovery
Workflow management
Data governance
Data profiling

The question then becomes how to realize these capabilities, which leads to combining multiple AWS services.

Complementing Glue Data Catalog as a Data Catalog

As discussed earlier, Glue Data Catalog alone is not sufficient as a complete data catalog.
In AWS, this gap is addressed by combining multiple services to complement its capabilities.

Here, we organize which capabilities are complemented by which services.

Category	AWS Service	Description
Business Metadata	Amazon DataZone	Business glossary, data ownership definition, rich descriptions and context, data asset reviews and ratings
Data Lineage	Amazon DataZone	Lineage visualization, understanding data transformation flows, dependency management (end-to-end lineage requires OpenLineage or AWS Lineage APIs)
Data Quality	AWS Glue Data Quality / DataBrew	Data quality rule definition, scoring, anomaly detection, profiling (Glue Data Quality can auto-generate rules based on profiling results from DataBrew)
Data Discovery	Amazon DataZone / Amazon Q	Filtering, recommendations, related data suggestions, natural language search, AI-assisted analysis and insight generation
Workflow Management	AWS Step Functions / Amazon MWAA (Airflow) / Glue Workflows	Workflow orchestration
Data Governance	AWS Lake Formation	Column/row-level access control, tag-based access control, permissions management, data filtering
Data Profiling	AWS Glue DataBrew / Amazon Macie	Profiling, statistical analysis, sensitive data detection, PII classification

As shown above, AWS enables a data catalog by combining multiple services with Glue Data Catalog at the core.

Data Catalog Architecture

A data catalog on AWS, centered around Glue Data Catalog, can be organized into the following layered structure.

The key point is to view this not as individual services, but as an architecture composed of layers.

┌──────────────────────────────┐
│   Business Catalog Layer      │   ← Amazon DataZone / Amazon Q
│   (Discovery / Glossary)      │
└──────────────┬───────────────┘
               │
┌──────────────┼───────────────┐
│ Governance / Quality Layer   │   ← Lake Formation / Glue Data Quality
│ (Access Control / Quality)   │
└──────────────┬───────────────┘
               │
┌──────────────┼───────────────┐
│ Metadata Core Layer          │   ← Glue Data Catalog
│ (Technical Metadata)         │
└──────────────┬───────────────┘
               │
┌──────────────┼───────────────┐
│ Processing / Query Layer     │   ← Athena / Glue ETL / Redshift
│ (Query / ETL Processing)     │
└──────────────┬───────────────┘
               │
┌──────────────────────────────┐
│ Data Layer (S3)              │   ← Raw / Curated Data
└──────────────────────────────┘

Roles of Each Layer

Business Catalog Layer

Amazon DataZone: Entry point for business users to discover data, review it, and request access
Amazon Q: AI assistant that supports natural language search, data analysis, and insight generation (e.g., “Where is the sales data for 2023?”)

Governance / Quality Layer

Lake Formation: Column- and row-level access control, tag-based permission management
Glue Data Quality: Definition and validation of data quality rules (e.g., “Check that the age column does not contain negative values”)

Metadata Core Layer

Glue Data Catalog: Centralized management of technical metadata (schemas, statistics, partitions)
Integration with S3: Automatically catalogs file structures in the data lake

Processing / Query Layer

Athena / Redshift Spectrum: Query data directly on S3 using Glue Data Catalog
Glue ETL: Executes transformation jobs based on metadata from the catalog

Data Layer

S3: Stores raw data (CSV, Parquet, etc.) and processed (curated) data

Implementation Best Practices

1. Use Glue Data Catalog as the foundation

Place it at the center since it integrates natively with services like Athena, Glue, and Redshift

2. Add a business-facing layer

Introduce Amazon DataZone and build a business glossary
Define data ownership and utilize review/rating features

3. Strengthen data quality and governance

Define rules with Glue Data Quality (e.g., “Order amount must not be negative”)
Apply access control with Lake Formation (e.g., “Finance team can only view accounting data”) Note: DataZone also supports IAM integration and access control independently

4. Visualize data lineage

Use OpenLineage specifications to automatically capture input/output of Glue ETL jobs
Visualize lineage graphs in Amazon DataZone

5. Enable profiling and sensitive data detection

Use DataBrew for profiling (e.g., distribution analysis of columns)
Use Amazon Macie for detecting and classifying PII

6. Improve search experience

Integrate Amazon Q to enable natural language search (e.g., “Customer purchase history”)

Challenges of Adopting DataZone

Among the complementary services, one stands out as particularly important—but also challenging to adopt: Amazon DataZone.

In DataZone, data assets are managed as data products.
A data product represents a meaningful unit of business data (e.g., “Customer transaction data”) with clearly defined ownership and responsibility.

This structure clarifies who owns the data, forming the foundation for data quality and governance.
It also aligns well with Data Mesh principles, enabling domain-oriented data management.

DataZone provides what Glue lacks: a catalog for humans.

Data asset cataloging
Search and discovery
Lineage visualization
Data quality visibility
Governance management

In other words, it extends the technical catalog into a business catalog.

While Glue Data Catalog is a “catalog for systems,” DataZone is a “catalog for people.”

However, adopting DataZone requires meeting organizational, operational, and technical prerequisites.

1. Organizational Prerequisites

This is often the most difficult part.

Data Domain Design

Data domains—logical groupings of business data with clear ownership—must be defined.

Since DataZone manages data at the domain level, unclear boundaries make operations unsustainable.
In reality, many organizations have not formalized domain design, making this the first major challenge.

Data Ownership

Each data asset must have a clearly defined owner.

Data is treated as a “data product,” and each domain is responsible for managing its own data.
However, in many organizations, ownership is ambiguous or fragmented.

Responsibility Definition

Responsibilities for data quality, access control, and updates must be defined.
This forms the basis for governance and approval workflows.

In practice, aligning responsibilities across departments often becomes a bottleneck.

2. Operational Prerequisites

Approval Workflows

Processes for requesting and approving data access must be established.

Data Classification

Standardized classification rules based on sensitivity and usage are required.

Usage Policies

Guidelines and compliance rules for data usage must be clearly defined.

3. Technical Prerequisites

Lineage Collection

DataZone visualizes lineage, but only if lineage data exists.

This requires:

Integration with processing systems (Glue, Redshift, etc.)
Adoption of standards like OpenLineage
Designing metadata collection within pipelines

Metadata Integration

Metadata from various services must be integrated into DataZone:

Catalog integration (Glue / Redshift / S3)
Data quality metadata (Glue Data Quality)
Access control metadata (Lake Formation)

This integration enables a consistent data catalog experience.

In summary, DataZone does not automatically solve data governance problems.
It requires the following conditions:

A data-driven culture is emerging
Cross-functional collaboration exists
Awareness of data quality is high
Continuous improvement processes are in place

Without these, the catalog risks becoming a formality that is not actually used.

A Practical Approach to Adopting DataZone

Given the complexity, a phased approach is often effective.

Phase 1: Foundation (Data Platform Team)

Establish technical metadata with Glue Data Catalog
Basic data classification
Simple access control

Phase 2: Governance (Involving Governance Teams)

Implement fine-grained access control with Lake Formation
- Example: Apply classification tags at the column level and deny SELECT access to PII-tagged columns via IAM policies
Introduce data quality monitoring
Establish basic lineage

Phase 3: DataZone (Business-Led)

Introduce once organizational prerequisites are met
Manage business metadata
Enable self-service analytics

DataZone becomes effective only when the organization reaches a certain level of maturity.

It is not just a tool, but a mechanism for organizational transformation.
Technical readiness alone is not sufficient—cultural and process changes are required.

Considering OpenMetadata

OpenMetadata is an open-source data catalog that supports a wide range of platforms.
https://open-metadata.org/

It provides:

Connectors for various data sources
Data lineage
Data quality management
Search and UI capabilities

It can function as a comprehensive data catalog on its own.

This raises the question: why not use OpenMetadata from the start?

The answer depends on your context—specifically, which layer you want to implement the catalog in (infrastructure-oriented vs. business-oriented).

In AWS-centric environments, Glue Data Catalog integrates natively with services like Athena, Redshift, and Glue, providing consistency and operational simplicity.

Therefore, if your architecture is primarily within AWS, it is reasonable to center your design around Glue.

On the other hand, OpenMetadata becomes advantageous when:

Managing across multiple clouds (AWS / GCP / Azure)
Integrating diverse sources (SaaS, on-premises, etc.)
Requiring flexible and customizable metadata management
Designing a business-centric catalog from the beginning

However, adopting OpenMetadata requires:

Infrastructure setup (ECS/EKS or VMs, metadata storage such as PostgreSQL)
Operations (monitoring with Prometheus/Grafana, scaling, upgrades)
Security design (RBAC, authentication/authorization, encryption)

Compared to AWS managed services, it offers flexibility at the cost of increased operational overhead.

In summary:

Situation	Recommendation
AWS only / small scale	Glue Data Catalog centered
AWS + governance needs	+ Lake Formation
Advanced data usage	+ DataZone
Multi-cloud	Consider OpenMetadata

OpenMetadata can also integrate with Glue Data Catalog as a catalog provider.
Its Glue connector can ingest metadata from Glue and map it to OpenMetadata constructs such as glossaries and data products.

Conclusion

In this article, we explored data catalogs centered around AWS Glue Data Catalog.

A data catalog is essential to prevent a data lake from becoming a data swamp.

What matters is not the tool itself, but:

How metadata is managed
Who owns the data
How it continues to be used

This requires designing not only technology, but also organization and processes.

In other words, a data catalog is not just a tool—it is an organizational system.

Glue Data Catalog plays a central role, but it cannot form a complete data catalog on its own.

On AWS, a data catalog should be designed not as a single service, but as an architecture centered around Glue Data Catalog.

And most importantly, a data catalog should be viewed not merely as a technical foundation, but as a foundation for enabling organizations to operate with data.

I hope this article helps those considering data catalogs on AWS.

Boosting Lightweight ETL on AWS Lambda & Glue Python Shell with DuckDB and Apache Arrow Dataset

Aki — Fri, 06 Mar 2026 00:32:32 +0000

Original Japanese article: AWS Lambda/Glue Python Shell×DuckDBの軽量ETLをApache Arrow Datasetで高速化してみた

Introduction

I'm Aki, an AWS Community Builder (@jitepengin).

In my previous articles, I introduced lightweight ETL using AWS Lambda and Glue Python Shell.
In the process, I found that DuckDB's performance was not as high as expected:

Does Increasing AWS Lambda Memory to 10GB Really Make It Faster? (AWS Lambda chDB/DuckDB PyIceberg Benchmark)
AWS Lambda and AWS Glue Python Shell in the Context of Lightweight ETL

In this article, I will cover what became the bottleneck for DuckDB and how using Apache Arrow Dataset can improve performance, along with the trade-offs observed.

Recap of Previous Articles

Does Increasing AWS Lambda Memory to 10GB Really Make It Faster? (AWS Lambda chDB/DuckDB PyIceberg Benchmark)
AWS Lambda and AWS Glue Python Shell in the Context of Lightweight ETL

Using NYC taxi data, we compared performance on the same file:
data.page]https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page

Test files:

January 2024 Yellow Taxi Trip Records (2,964,624 records, 48MB)
Full-year 2024 Yellow Taxi Trip Records (41,169,720 records, 807MB)

Lambda measurements were taken with memory configurations of 1024MB, 2048MB, and the maximum 3008MB (without quota increase).
Glue Python Shell tests were performed with DPU settings of 1/16 and 1.

Since memory usage cannot be directly compared, we focus only on execution time.

48MB File (1 Month)

Execution Platform	Resource Setting	chDB Time (s)	DuckDB Time (s)
Glue Python Shell	1/16 DPU	46.000	40.000
Glue Python Shell	1 DPU	39.000	34.000
AWS Lambda	1024 MB	5.092	5.163
AWS Lambda	2048 MB	3.873	4.265
AWS Lambda	3008 MB	3.370	4.061

807MB File (1 Year)

Execution Platform	Resource Setting	chDB Time (s)	DuckDB Time (s)
Glue Python Shell	1/16 DPU	OutOfMemory	OutOfMemory
Glue Python Shell	1 DPU	51.0	212.0
AWS Lambda	1024 MB	OutOfMemory	OutOfMemory
AWS Lambda	2048 MB	OutOfMemory	OutOfMemory
AWS Lambda	3008 MB	27.171	187.332

What Caused the DuckDB Bottleneck?

When loading Parquet directly into DuckDB, the flow is typically:

S3
↓
DuckDB read_parquet
↓
Filter / Query

Key points:

S3 Scan: Reading the entire dataset involves heavy network I/O — this can take most of the time.
Parquet Decode: Decoding inside DuckDB adds CPU load.
Query Processing: For simple filters like WHERE VendorID = 1, query time is minimal.

Even if the query itself is light, S3 scanning becomes the bottleneck, lowering DuckDB’s standalone performance.
Measurements showed that in Glue Python Shell, of the total 210 seconds, 176 seconds (~83%) were spent in S3 Scan + Parquet Decode.

Solution: Use Apache Arrow Dataset to separate reading from querying and improve performance.

What is Apache Arrow Dataset?

https://arrow.apache.org/docs/python/dataset.html

Apache Arrow Dataset is a library for efficiently reading Parquet or CSV files using the columnar Arrow in-memory format.

Features:

Fast Parquet reading
Efficient decode operations
Parallelized S3 reads
Filter/Projection pushdown to reduce I/O

By leveraging these features, the S3 Scan + Parquet Decode bottleneck can be greatly reduced.

Architecture Overview

AWS Lambda

Same architecture as in previous articles:

Glue Python Shell

Also same as before:

Sample Code (AWS Lambda)

import duckdb
import pyarrow.dataset as ds
import pyarrow.fs as fs
import pyarrow as pa
import boto3
from pyiceberg.catalog.glue import GlueCatalog

def lambda_handler(event, context):
    try:

        # DuckDB setup in Lambda
        duckdb_connection = duckdb.connect(database=':memory:')

        # Retrieve S3 path from event
        s3_bucket = event['Records'][0]['s3']['bucket']['name']
        s3_object_key = event['Records'][0]['s3']['object']['key']
        s3_input_path = f"{s3_bucket}/{s3_object_key}"

        print(f"S3 input path: {s3_input_path}")

        # Read Parquet from S3 using Arrow Dataset
        # Use boto3 session to get temporary credentials
        session = boto3.Session()
        credentials = session.get_credentials().get_frozen_credentials()

        s3 = fs.S3FileSystem(
            region="ap-northeast-1",
            access_key=credentials.access_key,
            secret_key=credentials.secret_key,
            session_token=credentials.token
        )

        # Load dataset with Arrow Dataset
        dataset = ds.dataset(
            s3_input_path,
            filesystem=s3,
            format="parquet"
        )

        # Convert dataset to Arrow Table (in-memory)
        arrow_table = dataset.to_table()
        print(f"Number of rows retrieved: {arrow_table.num_rows}")
        print(f"Schema: {arrow_table.schema}")

        # DuckDB processing (SQL query)
        # Use DuckDB from_arrow to run SQL on Arrow 
        rel = duckdb_connection.from_arrow(arrow_table)
        result_arrow_table = duckdb_connection.execute(
            """
            SELECT * 
            FROM rel
            WHERE VendorID = 1
            """
        ).fetch_arrow_table()

        # Configure Glue Catalog (to access Iceberg table)
        catalog = GlueCatalog(region_name="ap-northeast-1", database="icebergdb", name="my_catalog")  # Adjust to your environment.

        # Load the table
        namespace = "icebergdb"  # Adjust to your environment.
        table_name = "yellow_tripdata"  # Adjust to your environment.
        iceberg_table = catalog.load_table(f"{namespace}.{table_name}")

        # Append data to the Iceberg table in bulk
        iceberg_table.append(result_arrow_table)

        print("Data has been appended to S3 in Iceberg format.")

    except Exception as e:
        print(f"An error occurred: {e}")

Note: This article focuses on the differences in operation, so version updates or conflict handling in Iceberg tables are omitted.

Key points:

Arrow Dataset separates S3 reading from DuckDB querying
Light filters can be applied in Arrow Dataset alone for speed

Sample Code (Glue Python Shell)

import boto3
import sys
import os
import pyarrow.dataset as ds
import pyarrow.fs as fs
from awsglue.utils import getResolvedOptions


def get_job_parameters():
    args = getResolvedOptions(sys.argv, ['s3_input'])
    s3_path = args['s3_input']
    print(f"object: {s3_path}")
    return s3_path


def setup_duckdb_environment():
    try:
        duckdb_dir = '/tmp/.duckdb'
        os.environ['HOME'] = '/tmp'
        os.makedirs(duckdb_dir, exist_ok=True)
        print(f"DuckDB environment setup completed: {duckdb_dir}")
        return True
    except Exception as e:
        print(f"DuckDB environment setup error: {e}")
        return False


def read_parquet_with_arrow_dataset(s3_input):

    print("Reading with Arrow Dataset...")

    session = boto3.Session()
    credentials = session.get_credentials().get_frozen_credentials()

    s3 = fs.S3FileSystem(
        region="ap-northeast-1",
        access_key=credentials.access_key,
        secret_key=credentials.secret_key,
        session_token=credentials.token
    )

    path = s3_input.replace("s3://", "")

    dataset = ds.dataset(
        path,
        filesystem=s3,
        format="parquet"
    )

    table = dataset.to_table()

    print(f"Arrow read rows: {table.num_rows}")

    return table


def process_with_duckdb(arrow_table):

    import duckdb

    con = duckdb.connect(":memory:")

    try:

        rel = duckdb.from_arrow(arrow_table)

        result = con.execute("""
            SELECT *
            FROM rel
            WHERE VendorID = 1
        """).arrow()

        print(f"DuckDB filtered rows: {result.num_rows}")

        return result

    finally:
        con.close()


def write_iceberg_table(arrow_table):

    try:

        print("Writing started...")

        from pyiceberg.catalog import load_catalog

        catalog_config = {
            "type": "glue",
            "warehouse": "s3://your-bucket/your-warehouse/",
            "region": "ap-northeast-1"
        }

        catalog = load_catalog("glue_catalog", **catalog_config)

        table_identifier = "icebergdb.yellow_tripdata"

        table = catalog.load_table(table_identifier)

        print(f"Target data to write: {arrow_table.num_rows:,} rows")

        table.append(arrow_table)

        return True

    except Exception as e:

        print(f"Writing error: {e}")

        import traceback
        traceback.print_exc()

        return False


def main():

    if not setup_duckdb_environment():
        print("Failed to set up DuckDB environment")
        return

    try:

        s3_input = get_job_parameters()

        # Arrow Dataset read
        arrow_tbl = read_parquet_with_arrow_dataset(s3_input)

        # DuckDB SQL filter
        result_tbl = process_with_duckdb(arrow_tbl)

        # Iceberg write
        if write_iceberg_table(result_tbl):
            print("Writing fully successful!")
        else:
            print("Writing failed")

    except Exception as e:

        print(f"Main error: {e}")

        import traceback
        traceback.print_exc()


if __name__ == "__main__":
    main()

Note: This article focuses on the differences in operation, so version updates or conflict handling in Iceberg tables are omitted.

Key points:

Glue Python Shell can execute ETL in the same Lambda-style configuration
Responsibilities separated: Arrow Dataset for reading, DuckDB for SQL query
Lightweight filters can be processed efficiently

Benchmarking

Using the same dataset and queries as in previous articles:

AWS Lambda

48MB File (1 Month) memory=3008MB

Method	Time (ms)	Memory (MB)
chDB	3,369.78	1115
DuckDB	4,061.33	524
DuckDB × Arrow Dataset	3,591.84	928

807MB File (1 Year) memory=10240MB

Method	Time (ms)	Memory (MB)
chDB	22,839.18	3490
DuckDB	189,678.02	2788
DuckDB × Arrow Dataset	15,220.6	8086

Glue Python Shell (1 DPU)

48MB File (1 Month)

Method	Time (s)
chDB	39
DuckDB	34
DuckDB × Arrow Dataset	32

807MB File (1 Year)

Method	Time (s)
chDB	51
DuckDB	212
DuckDB × Arrow Dataset	44

As a result, both AWS Lambda and Glue Python Shell were able to achieve significant performance improvements compared to chDB.
In other words, addressing the S3 scan and Parquet decoding bottlenecks seems to be the key to improving DuckDB processing.

However, in the case of Lambda, large file sizes can lead to high memory usage, potentially exceeding the memory limits.
This means that careful consideration of where and how to use this approach is necessary.

Memory Consideration with Arrow Dataset

dataset = ds.dataset(
    s3_input_path,
    filesystem=s3,
    format="parquet"
)

arrow_table = dataset.to_table()

In this process, dataset.to_table() materializes the entire dataset in memory as an Arrow Table.
Arrow Tables use a columnar in-memory format, which is very fast, but in this case, loading the entire file at once can result in high memory usage.

For example, reading an 807MB Parquet file in Lambda can cause the memory footprint of the Arrow Table to be much larger than the compressed Parquet file size.
While to_table() is convenient, it is important to be aware that it can significantly increase memory consumption depending on the processing.

Lambda can also use /tmp for disk-backed processing, but processing in memory is overwhelmingly faster.
However, due to memory limits, expanding a large file into an Arrow Table can quickly consume a large amount of memory.

For simple queries, one approach is to iterate over row groups instead of materializing the entire table in memory, processing small chunks at a time.
This method can potentially keep memory usage within a few hundred MBs.

Trade-Offs

Using Arrow Dataset significantly improves the speed of S3 reads and Parquet decoding.
However, expanding the dataset all at once with to_table() increases memory usage, which may hit Lambda’s memory limits.

Pros: Decoding and I/O are faster, resulting in improved performance.
Cons: Materializing the entire file consumes a lot of memory, and for large files, Lambda may run into OutOfMemory errors.

Therefore, it is important to design your ETL with a balance between performance and memory usage in mind.
For small files or when Lambda has sufficient memory, loading the full dataset at once is fine.
For larger files, consider chunked processing by row group or pushdown filters to keep memory usage under control.

Pushdown with Arrow Dataset

Using Arrow Dataset’s Filter/Projection Pushdown, you can load only the row groups you need from S3.

Here’s how you can apply it:

arrow_table = dataset.to_table(
    columns=["VendorID", "tpep_pickup_datetime"],
    filter=ds.field("VendorID") == 1
)

Only the necessary row groups are read (this is crucial for large datasets).
Reduces network I/O from S3.
Can further shorten processing time.
Arrow Dataset is optimal for lightweight reads and simple filters; complex queries should still be handled in DuckDB.

Experimenting in Lambda

You can integrate pushdown into your existing Lambda code. For example:

import duckdb
import pyarrow.dataset as ds
import pyarrow.fs as fs
import pyarrow as pa
import boto3
from pyiceberg.catalog.glue import GlueCatalog

def lambda_handler(event, context):
    try:

        # DuckDB setup in Lambda
        duckdb_connection = duckdb.connect(database=':memory:')

        # Retrieve S3 path from event
        s3_bucket = event['Records'][0]['s3']['bucket']['name']
        s3_object_key = event['Records'][0]['s3']['object']['key']
        s3_input_path = f"{s3_bucket}/{s3_object_key}"

        print(f"S3 input path: {s3_input_path}")

        # Read Parquet from S3 using Arrow Dataset
        # Use boto3 session to get temporary credentials
        session = boto3.Session()
        credentials = session.get_credentials().get_frozen_credentials()

        s3 = fs.S3FileSystem(
            region="ap-northeast-1",
            access_key=credentials.access_key,
            secret_key=credentials.secret_key,
            session_token=credentials.token
        )

        # Load dataset with Arrow Dataset
        dataset = ds.dataset(
            s3_input_path,
            filesystem=s3,
            format="parquet"
        )

        # Convert dataset to Arrow Table (in-memory)
        arrow_table = dataset.to_table(filter=ds.field("VendorID") == 1)
        print(f"Number of rows retrieved: {arrow_table.num_rows}")
        print(f"Schema: {arrow_table.schema}")

        # DuckDB processing (SQL query)
        # Use DuckDB from_arrow to run SQL on Arrow 
        rel = duckdb_connection.from_arrow(arrow_table)
        result_arrow_table = duckdb_connection.execute(
            """
            SELECT * 
            FROM rel
            """
        ).fetch_arrow_table()

        # Configure Glue Catalog (to access Iceberg table)
        catalog = GlueCatalog(region_name="ap-northeast-1", database="icebergdb", name="my_catalog")  # Adjust to your environment.

        # Load the table
        namespace = "icebergdb"  # Adjust to your environment.
        table_name = "yellow_tripdata"  # Adjust to your environment.
        iceberg_table = catalog.load_table(f"{namespace}.{table_name}")

        # Append data to the Iceberg table in bulk
        iceberg_table.append(result_arrow_table)

        print("Data has been appended to S3 in Iceberg format.")

    except Exception as e:
        print(f"An error occurred: {e}")

Using pushdown, memory usage was reduced significantly:

Method	Time (ms)	Memory (MB)
DuckDB × Arrow Dataset	15,220.6	8086
DuckDB × Arrow Dataset (Pushdown)	15,238.14	3458

The memory usage was significantly reduced with pushdown.
This two-step approach filtering unnecessary data with Arrow Dataset before passing it to DuckDB proves to be effective for large datasets.

Insights

DuckDB alone is slow due to S3 Scan + Parquet Decode
DuckDB shines with complex queries (JOIN, GROUP BY, WINDOW)
Pushdown is key in Arrow Dataset
Separating responsibilities (Arrow Dataset + DuckDB) enables efficient ETL in Lambda/Glue
chDB offers balanced memory and speed out-of-the-box

Responsibility-Separated Architecture

S3 Parquet (raw data)
      │
      ▼
Arrow Dataset → row group scan + simple filter
      │
      ▼
DuckDB → SQL query (JOIN, GROUP BY, Window functions)
      │
      ▼
PyIceberg → Iceberg table write

Conclusion

In this article, we explored performance improvements for a lightweight ETL built with AWS Lambda / Glue Python Shell × DuckDB × PyIceberg.

For lightweight ETL, especially on AWS Lambda, processing time is a critical factor. By using Apache Arrow Dataset, we were able to significantly improve performance by offloading S3 reading and Parquet decoding before running queries in DuckDB.

However, there are trade-offs. Expanding an entire dataset into memory with to_table() can lead to high memory usage, which may exceed Lambda’s limits for large files. Therefore, careful responsibility separation and chunked processing (e.g., row group iteration or pushdown filters) are important considerations.

With the architecture presented here, even large files can be processed quickly in a lightweight ETL on AWS. While complex queries may still face performance limitations, this approach provides a practical and efficient option for real-time or near-real-time ETL in a Lakehouse environment using Apache Iceberg.

We hope this article serves as a reference for those exploring lightweight data processing and ETL patterns on Iceberg tables.

Does Increasing AWS Lambda Memory to 10GB Really Make It Faster? (AWS Lambda chDB/DuckDB PyIceberg Benchmark)

Aki — Thu, 26 Feb 2026 13:33:33 +0000

Original Japanese article: AWS Lambdaを10GBにすると本当に速くなるのか？（AWS Lambda×chDB/DuckDB×PyIceberg検証）

Introduction

I'm Aki, an AWS Community Builder (@jitepengin).

In a previous article, I benchmarked Iceberg integration using AWS Lambda with DuckDB and chDB.

Lightweight ETL with AWS Lambda, chDB, and PyIceberg (Compared with DuckDB)

In that article, I tested two patterns on AWS Lambda:

chDB × PyIceberg
DuckDB × PyIceberg

Memory sizes were set to 1024 MB, 2048 MB, and 3008 MB (the maximum without quota increase at the time).

The results showed:

For small datasets, increasing memory generally improved performance.
For a large dataset (807 MB), 3008 MB was barely enough to complete processing.

This time, I extended the experiment:

What happens if we increase Lambda memory up to 10GB (10240 MB)?

Increasing the Lambda Memory Quota

To raise the Lambda memory limit beyond 3008 MB, you must request a quota increase.

Important:
You cannot increase Lambda memory from the Service Quotas console.

Steps

Go to the AWS Support Center and create a new case.
Clearly state:

The reason for the increase
The target region

Example request content:

We are building and validating a data processing platform using AWS Lambda.
The workload is memory-intensive, including large Parquet file loading, aggregation, and transformation.
The current 3008 MB limit is insufficient to complete processing.

We are performing analytical processing inside Lambda using columnar formats (Parquet), and the workload requires higher memory allocation.

Currently, we experience performance degradation and OutOfMemory errors.

We would like to request an increase of the Lambda memory limit in the Tokyo region to 10240 MB.

Although we considered migrating to other compute services, we determined that continuing with Lambda is the most appropriate option from both operational and architectural perspectives.

After submission, AWS responded in about 3 business days and applied the increase.

Architecture

The architecture is identical to the previous article.

Flow:

Load a Parquet file from S3 in Lambda
Process it using chDB or DuckDB
Write results into an Iceberg table

In short:

S3 → Lambda (chDB/DuckDB) → Iceberg (via Glue Catalog)

In this article, I focus on performance behavior differences.
Iceberg version conflicts and concurrency handling are omitted for simplicity.

Sample Code (chDB)

import chdb
import pyarrow as pa
from pyiceberg.catalog.glue import GlueCatalog


def _to_pyarrow_table(result):
    """
    Compatibility helper to extract a pyarrow.Table from a chDB query_result.
    """
    if hasattr(chdb, "to_arrowTable"):
        return chdb.to_arrowTable(result)

    if hasattr(result, "to_pyarrow"):
        return result.to_pyarrow()
    if hasattr(result, "to_arrow"):
        return result.to_arrow()

    raise RuntimeError(
        "Cannot convert chdb query_result to pyarrow.Table. "
        f"Available attributes: {sorted(dir(result))[:200]}"
    )


def normalize_arrow_for_iceberg(table: pa.Table) -> pa.Table:
    """
    Normalize Arrow types that Iceberg does not accept
    (mainly timezone-aware timestamps).
    """
    new_fields = []
    new_columns = []

    for field, column in zip(table.schema, table.columns):
        if pa.types.is_timestamp(field.type) and field.type.tz is not None:
            # Remove timezone information (values remain in UTC)
            new_type = pa.timestamp(field.type.unit)
            new_fields.append(pa.field(field.name, new_type, field.nullable))
            new_columns.append(column.cast(new_type))
        else:
            new_fields.append(field)
            new_columns.append(column)

    new_schema = pa.schema(new_fields)
    return pa.Table.from_arrays(new_columns, schema=new_schema)


def lambda_handler(event, context):
    try:
        # Extract S3 bucket and object key from the event
        s3_bucket = event['Records'][0]['s3']['bucket']['name']
        s3_object_key = event['Records'][0]['s3']['object']['key']

        # Build S3 HTTPS URL
        s3_url = (
            f"https://{s3_bucket}."
            f"s3.ap-northeast-1.amazonaws.com/"
            f"{s3_object_key}"
        )

        print(f"s3_url: {s3_url}")

        # Query Parquet data on S3 using chDB
        query = f"""
            SELECT *
            FROM s3('{s3_url}', 'Parquet')
            WHERE VendorID = 1
        """

        # Execute chDB query with Arrow output
        result = chdb.query(query, "Arrow")

        # Convert chDB result to pyarrow.Table
        arrow_table = _to_pyarrow_table(result)
        print(f"Original schema: {arrow_table.schema}")

        # Normalize schema for Iceberg compatibility
        arrow_table = normalize_arrow_for_iceberg(arrow_table)
        print(f"Normalized schema: {arrow_table.schema}")
        print(f"Rows: {arrow_table.num_rows}")

        # Initialize Iceberg Glue Catalog
        catalog = GlueCatalog(
            name="my_catalog",
            database="icebergdb",
            region_name="ap-northeast-1",
        )

        # Load Iceberg table
        iceberg_table = catalog.load_table("icebergdb.yellow_tripdata")

        # Append data to Iceberg table
        iceberg_table.append(arrow_table)

        print("Data appended to Iceberg table.")

    except Exception as e:
        print("Exception:", e)
        raise

Note: This article focuses on the differences in operation, so version updates or conflict handling in Iceberg tables are omitted.

Sample Code (DuckDB)

import duckdb
import pyarrow as pa
from pyiceberg.catalog.glue import GlueCatalog  

def lambda_handler(event, context):
    try:
        # Connect to DuckDB and set the home directory
        duckdb_connection = duckdb.connect(database=':memory:')
        duckdb_connection.execute("SET home_directory='/tmp'") 

        # Install and load the httpfs extension
        duckdb_connection.execute("INSTALL httpfs;")
        duckdb_connection.execute("LOAD httpfs;")

        # Load data from S3 using DuckDB
        s3_bucket = event['Records'][0]['s3']['bucket']['name']
        s3_object_key = event['Records'][0]['s3']['object']['key']

        s3_input_path = f"s3://{s3_bucket}/{s3_object_key}"

        print(f"s3_input_path: {s3_input_path}")

        query = f"""
            SELECT * FROM read_parquet('{s3_input_path}') WHERE VendorID = 1
        """
        # Execute SQL and retrieve results as a PyArrow Table
        result_arrow_table = duckdb_connection.execute(query).fetch_arrow_table()

        print(f"Number of rows retrieved: {result_arrow_table.num_rows}")
        print(f"Data schema: {result_arrow_table.schema}")

        # Configure Glue Catalog (to access Iceberg table)
        catalog = GlueCatalog(region_name="ap-northeast-1", database="icebergdb", name="my_catalog")  # Adjust to your environment.

        # Load the table
        namespace = "icebergdb"  # Adjust to your environment.
        table_name = "yellow_tripdata"  # Adjust to your environment.
        iceberg_table = catalog.load_table(f"{namespace}.{table_name}")

        # Append data to the Iceberg table in bulk
        iceberg_table.append(result_arrow_table) 

        print("Data has been appended to S3 in Iceberg format.")

    except Exception as e:
        print(f"An error occurred: {e}")

Note: Version updates and conflict handling are omitted here as well.

Test Conditions

Dataset:
NYC Taxi Trip Records
https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page

Test files:

January 2024 (Yellow Taxi)
- 2,964,624 records
- 48 MB
Full year 2024 (aggregated file)
- 41,169,720 records
- 807 MB

Memory configurations tested:

1024 MB
2048 MB
3008 MB (no quota increase maximum)
4096 MB
10240 MB

Each configuration was executed 5 times under warm conditions to eliminate cold start effects.

Results

48MB File (1 Month)

Memory (MB)	chDB Time (ms)	chDB Memory Used (MB)	DuckDB Time (ms)	DuckDB Memory Used (MB)
1024	5,092	1018	5,163	512
2048	3,872	1132	4,264	538
3008	3,369	1115	4,061	524
4096	3,197	1263	3,547	568
10240	3,087	1255	3,484	554

Performance improved as memory increased — but only marginally above 4096 MB.

Actual memory usage:

chDB ≈ 1.2 GB
DuckDB ≈ 550 MB

Allocated memory increased, but real usage did not scale proportionally.

807MB File (1 Year)

Memory (MB)	chDB Time (ms)	chDB Memory Used (MB)	DuckDB Time (ms)	DuckDB Memory Used (MB)
1024	OOM	-	OOM	-
2048	OOM	-	OOM	-
3008	27,170	3001	187,331	2732
4096	24,631	3322	188,880	2767
10240	22,839	3490	189,678	2788

OOM occurred because the memory allocation could not hold the temporary buffers required during the Parquet → Arrow → Iceberg transformation.

Did Performance Really Improve at 4096MB and 10GB?

What we really wanted to check in this experiment was whether performance would actually improve once memory goes beyond 3008 MB.

48MB (1-month data)

Increasing memory from 3008 → 4096 → 10240 MB resulted in:

chDB: 3,369 → 3,197 → 3,087 ms
DuckDB: 4,061 → 3,547 → 3,484 ms

Performance did improve, but the gains were limited.
In particular, the difference from 4096 → 10240 MB is almost negligible, within the margin of error.

Looking at Max Memory Used, chDB only used about 1.2 GB and DuckDB about 550 MB, meaning increasing allocated memory did not increase actual usage.

807MB (1-year data)

chDB:

3008 MB: 27,170 ms
4096 MB: 24,631 ms
10240 MB: 22,839 ms

Overall, going from 3008 → 10240 MB improved performance by roughly 16%,
but from 4096 → 10240 MB, the improvement was only about 7%.

Even though memory increased roughly 3.4×, performance only improved by ~16%, suggesting that performance is hitting a ceiling.

DuckDB:

3008 MB: 187,331 ms
4096 MB: 188,881 ms
10240 MB: 189,678 ms

Almost no improvement; in some cases, it was slightly slower.
Simply increasing memory does not affect execution time.

Analysis (Bottleneck Insights)

Memory behaves as a threshold parameter

For Lambda × DuckDB/chDB, memory seems to behave more like a threshold than a proportional scaling parameter.

1024 MB and 2048 MB → OOM
3008 MB → first point where processing succeeds

Beyond that, adding memory does not yield proportional performance gains.
This suggests the bottleneck is likely elsewhere, not just compute resources.

Does increasing vCPU help?

Lambda increases available CPU with memory.
However:

DuckDB barely scaled
chDB only slightly improved

Likely bottlenecks are I/O and serialization, such as:

Reading from S3

Iceberg metadata operations

Parquet → Arrow conversion

Engine-specific behavior

chDB: small improvements with more memory
DuckDB: almost no change

This difference may be due to internal implementations or parallelization strategies.
At least for this workload, simply going to 10 GB does not make DuckDB explode in speed.

Key Takeaways for Large Workloads on Lambda

From this experiment, a few points are clear:

Crossing the OOM threshold is the main goal.
Memory beyond that should be considered carefully, especially for cost.
Simply allocating 10 GB does not guarantee faster execution.

Looking at DuckDB results, it's clear that maxing out memory does not automatically make things faster.
From a cost perspective, finding the “just enough” memory is more practical.

For more complex or larger workloads, sticking to Lambda may not be optimal — Glue or EMR could be faster and more stable.

Conclusion

In this article, we walked through applying for a Lambda memory quota increase and measured the performance of lightweight ETL tasks with the expanded memory.

Both chDB and DuckDB are attractive open-source options, but they have significantly different characteristics. One clear takeaway is that crossing the OOM threshold should always be the first goal; beyond that, performance improvements will likely need to come from areas other than memory.

This experiment reinforced that designing for maximum memory by default is not necessarily the best approach. It's more important to understand your workload and identify the critical memory boundaries.

Also, keep in mind that Lambda quota increases cannot be requested from the Service Quotas screen, which can be useful knowledge in both personal projects and professional settings.

While both engines are still evolving, understanding their characteristics and using them appropriately allows you to build simple, yet highly extensible data processing workflows.

I hope this article serves as a helpful reference for anyone considering lightweight data processing or real-time ETL with Iceberg tables.

DEV Community: Aki

Rethinking Lakehouse Architecture Through Data Ownership: AWS vs. Snowflake

Introduction

Why Data Ownership Matters

Defining Ownership Across Three Layers

The Reality of Vendor Lock-In

Extensibility and Strategic Flexibility

What Changed After Iceberg?

Major Iceberg Catalog Options

Snowflake-Centric Architecture

Characteristics

Ownership Model

Benefits

Drawbacks

AWS-Centric Architecture

Characteristics

Ownership Model

Benefits

Drawbacks

Combining AWS and Snowflake

AWS Owns the Data, Snowflake Powers Analytics

1. Glue Catalog Integration (Read-Only)

2. Catalog-Linked Database (Read/Write)

Comparison: Three Architectural Patterns

Choosing the Right Pattern

What Happens When Ownership Is Unclear

"AWS or Snowflake?" Is a Secondary Question

Conclusion

Exploring Snowpark While Comparing It with Apache Spark

Introduction

What is Snowpark?

Differences Between Spark and Snowpark

Getting Started

Prerequisites

Obtaining a Session

Basic DataFrame Operations

Aggregations

Writing Results Back to a Table

How Does Lazy Evaluation Work?

Understanding Lazy Evaluation

Verifying Lazy Evaluation

Define the DataFrame

Execute collect()

Differences from Spark's Lazy Evaluation

Can We Use Caching?

Differences from Spark cache()

Example

Use Cases

ETL Pipelines

Custom Transformations Using UDFs

Data Quality Validation

Conclusion

Organizing How to Use AWS Glue Workflow

Introduction

What Is Glue Workflow?

Details of the Core Components

CONDITIONAL Trigger Usage Patterns

1. Simple Sequential Pattern

2. Waiting for Multiple Jobs to Complete (AND Condition)

3. Triggering When Any Job Completes (ANY Condition)

4. Failure Branching (Catching FAILED States)

Managing Parameters with default_run_properties

How It Works

How to Configure It

Configure from the Console

Configure via boto3

Configure via IaC (CloudFormation, etc.)

Static Parameters vs Dynamic Parameters

Passing Data Between Jobs

Dynamically Updating Run Properties

Retrieving Properties from a PySpark Job

Retrieving Properties from a Python Shell Job

Anti-Patterns for Data Passing

Integrating with EventBridge and Other Services

EventBridge-Based Startup (EVENT Trigger)

Batch Event Startup

Starting from S3 Events (The Parameter Passing Limitation)

Detecting Workflow Completion via EventBridge

Calling Glue Workflow from Step Functions

Operational Tips

Managing Parameters with `default_run_properties`

Resuming Failed Workflows (`ResumeWorkflowRun`)