Vikneshwara R B for AWS Community Builders

Posted on Oct 12

One Glue Job, Zero Orchestrators — My First Spark Declarative Pipeline on AWS Glue

#aws #beginners #python #tutorial

Disclaimer

This is an independent, personal write-up — not official AWS content, and not
endorsed by or affiliated with AWS. It documents a hobby project built while
learning AWS Glue 6.0's Spark Declarative Pipelines,
following the walkthrough in that official AWS Big Data blog post. All opinions
below are my own.

I've tried to explain every piece of jargon the first time it shows up, so if you're
new to data engineering or to AWS Glue, you should be able to follow this end to
end. No prior Glue or Spark experience assumed.

The problem, in plain terms

Imagine you need to process data in three steps: clean it up, validate it, then
summarize it for a report. Easy enough on its own. Now imagine step 2 must never
start before step 1 finishes, and if step 2 fails halfway through, you don't want to
redo all of step 1's work. And now imagine you have to add a step 4 next month.

That coordination problem — making sure things run in the right order, handling
retries, not redoing work that already succeeded — is called orchestration. It's
not the interesting part of a data pipeline. It's the part you have to get right
so the interesting part (the actual data transformations) works reliably.

The classic way to structure a pipeline like this is the medallion architecture:
raw data comes in as bronze (untouched, as-is — think raw ore), gets cleaned and
validated into silver (refined metal), and finally gets aggregated into gold
(the finished product, ready for a dashboard or report). It's a great mental model.
The part that quietly gets expensive is everything around it: three separate jobs,
an orchestrator (like AWS Step Functions or Apache Airflow) wiring them together in
order, retry logic at every stage, and bookkeeping so a partial failure doesn't force
you to reprocess everything from scratch. None of that is business logic. It's
plumbing that exists purely to make sure step 2 waits for step 1.

AWS's own blog post has a diagram of exactly this shape — three separate AWS Glue jobs wired together by an Amazon MWAA DAG:
"Three AWS Glue Jobs with orchestration running in Amazon MWAA",
from Build declarative ETL pipelines with AWS Glue 6.0 (AWS Big Data Blog). Here's my own version of that same idea:

Enter "declarative" pipelines

You've probably used something declarative before without calling it that. When you
write a SQL SELECT query, you don't tell the database how to fetch and join the
rows step by step — you describe what result you want, and the database figures
out the how. That's the declarative idea: say what you want, not the steps to get
there. (The opposite — a loop that manually fetches, filters, and combines rows
yourself — is called imperative: you spell out every step.)

AWS Glue 6.0 brings that same idea to entire pipelines, through a feature called
Spark Declarative Pipelines (SDP). Worth knowing: this is an Apache Spark 4.1
feature, not something AWS invented just for Glue — AWS Glue 6.0 just ships the
Spark version that includes it.

With SDP, you write code that says "silver_orders is bronze_orders, cleaned up
this way" and "gold_sales_summary is silver_orders, aggregated this way." You
never write "run bronze, then silver, then gold." The engine reads those table
references in your code, works out that silver depends on bronze (because your
silver code literally reads from the bronze table), and runs everything in the
correct order automatically — as one single Glue job, not three jobs stitched
together by an orchestrator.

And the "after" picture from the same post:
"AWS Glue 6.0 - Spark Declarative Pipeline (single job)" —
the same bronze/silver/gold flow, collapsed into one job, with the caption "SDP resolves dependency order automatically — no DAG wiring needed." That line is the whole pitch, honestly. My own version of that picture:

I built a small project around the walkthrough in AWS's own blog post to see what
this actually feels like end to end, including the infrastructure-as-code side
(using AWS CDK, a tool for defining cloud infrastructure in regular code instead of
clicking through the AWS console) that the original post doesn't cover. The repo is
here if you want to follow along or just skim the code:
github.com/vikneshwara-r-b/glue-sdp-demo.

What SDP actually does for you

A pipeline is just a small YAML file (spark-pipeline.yml, think of it as the
pipeline's table of contents) plus your SQL/Python transformation code. From those,
SDP:

figures out the dependency order from table references in your code (e.g. a line like spark.readStream.table("bronze_orders") tells it silver needs bronze first) — you never draw this graph by hand;
runs independent parts in parallel when it can;
keeps track of checkpoints (a bookmark of "which data have I already processed") for tables that update incrementally, so a rerun doesn't redo everything;
registers every output table in the Glue Data Catalog (AWS's metadata store — the thing that lets tools like Athena find and query your tables by name) as soon as it's created.

There are three kinds of tables you can define, and picking the right one matters
more than it sounds like it should:

Type	What it does	Good for
Streaming table	Only processes new data since last time	ingesting files as they arrive, change-tracking
Materialized view	Recomputes completely, every single run	summaries, aggregations, reports
Temporary view	Exists only during the run, never saved	a quick intermediate step, like a scratch variable

The pipeline I built: orders, bronze to gold

The demo reads order CSV files from S3 (object storage — think of it as a folder
that lives in the cloud) and produces three tables, all from one Glue job:

bronze_orders (streaming table) — the raw CSV, every single column kept as plain text, no cleanup at all. It's a streaming table on purpose: new order files should be picked up incrementally as they land, not reprocessed from scratch every time the job runs.
silver_orders (streaming table) — same data, but now typed properly (amounts become actual numbers, timestamps become actual timestamps), filtered down to only completed orders with a positive amount, and with a new amount_band column (small/medium/large) added.
gold_sales_summary (a SQL materialized view, not a streaming table) — order count, total sales, and average order value, grouped by region. This one's a materialized view on purpose: it's a pure aggregation, and since materialized views always fully recompute anyway, there's no incremental benefit being given up here — it's just the right tool for this specific job.

Here's the bit of code I'd flag if you're following along and writing your own
streaming table, because it's easy to get subtly wrong:

from pyspark import pipelines as dp
from pyspark.sql import DataFrame, SparkSession

dp.create_streaming_table("silver_orders", comment="...")

@dp.append_flow(target="silver_orders")
def ingest_silver_orders() -> DataFrame:
    spark = SparkSession.active()
    # readStream.table, not spark.table -- otherwise every run reprocesses
    # bronze's entire history into silver instead of just what's new.
    return spark.readStream.table("bronze_orders").select(...)

dp.create_streaming_table() paired with @dp.append_flow(target=...) is the
actual, documented Python API for a streaming table in AWS Glue's docs. You'll see
a simpler-looking @dp.table decorator mentioned informally in some write-ups
(including the original blog post's prose) — that's not the syntax Glue's own
developer guide specifies, so I stuck with the documented one.

The infrastructure side (not covered in the original post)

The original walkthrough is hands-on-the-CLI: you run aws glue create-job
yourself, zip and upload files by hand. My version wraps all of that in one AWS CDK
(Python) stack, so the whole thing is cdk deploy and done:

one private, encrypted S3 bucket holding the input CSV, the pipeline's checkpoint state, and the Data Catalog's storage location — one bucket, three folders inside it;
one IAM role (an AWS identity the Glue job "wears" to get permissions) scoped tightly to just that bucket, plus the one AWS-managed permission set Glue itself needs to operate;
a Glue database definition with an explicit storage location — SDP will register tables into this database, but it won't create the database itself, so it has to exist first;
one Glue job, version 6.0, with the two flags that turn SDP on and wire it up to the Data Catalog;
a small packaging script that fills in the bucket/database names into spark-pipeline.yml right before deployment, and zips everything up for upload.

Running it

Once deployed, there are two independent dials you turn via the job's arguments:
whether to actually write data or just check everything compiles, and how much to
reprocess.

# Dry run: checks the dependency graph and your SQL/Python compile correctly.
# Writes zero data. Good first check after a fresh deploy.
aws glue start-job-run --job-name "$JOB_NAME" \
  --arguments '{"--conf":"spark.glue.sdp.jobMode=VALIDATE"}' --region "$AWS_REGION"

# The real thing.
aws glue start-job-run --job-name "$JOB_NAME" \
  --arguments '{"--conf":"spark.glue.sdp.jobMode=RUN"}' --region "$AWS_REGION"

# Full refresh: throw away every table (including streaming checkpoints) and
# recompute everything from zero. Both settings have to share ONE --conf value,
# space-separated -- the underlying API only accepts one --conf entry at a time.
aws glue start-job-run --job-name "$JOB_NAME" \
  --arguments '{"--conf":"spark.glue.sdp.jobMode=RUN --conf spark.glue.sdp.runMode=--full-refresh-all"}' \
  --region "$AWS_REGION"

I wrapped these in a couple of small shell scripts
(scripts/run_job.sh --validate, --run --wait, --full-refresh-all, and
scripts/deploy.sh --upload for getting sample data into the bucket) — mostly so I
never have to hand-type a JSON string in the middle of a live demo.

Here's what a successful RUN actually looks like in the Glue console:

The job configuration/run screen after a successful RUN — Glue version 6.0, the
SDP job arguments, and a completed run.

And the three tables it registers, visible in the Glue Data Catalog:

bronze_orders, silver_orders, and gold_sales_summary — all three, registered
automatically, from one job run.

Querying the results

The whole point of getting data into bronze → silver → gold is being able to
actually query it. Once the tables are in the Data Catalog, Amazon Athena can see
them immediately — no extra wiring:

SELECT region, order_count, total_sales, average_order_value
FROM gold_sales_summary
ORDER BY total_sales DESC;

Real output, from the sample orders.csv in the repo — not a mockup.

What I took away from this

The core idea really does hold up. Three tables, zero hand-written orchestration code, dependency order worked out entirely from the table references already sitting in your transformation code.
Streaming table vs. materialized view is a real design decision, not just a style preference — pick the wrong one for a given table and you'll feel it in either unnecessary reprocessing or stale data, not in an obvious error up front.
Reading about a pattern and building it are genuinely different. Several small details only became obvious once I actually deployed and ran this against real AWS, not just from reading the walkthrough.

If you're newer to data engineering: don't let the jargon (DAG, checkpoint, orchestrator, materialized view) intimidate you out of trying this. Every one of those words describes something pretty ordinary once you've seen it in action once — and a three-table demo like this one is a genuinely good size to see it in action for the first time.

Original walkthrough this project is based on:
Build declarative ETL pipelines with AWS Glue 6.0
(AWS Big Data Blog). Companion repo, if you'd like to try it yourself:
github.com/vikneshwara-r-b/glue-sdp-demo.