AWS Glue Deep Dive: Crawlers, Job Bookmarks, DynamicFrames & Spark Tuning

AWS Glue is the service that most data engineers reach for the moment "we need serverless Spark ETL on AWS" enters a conversation — and it is also the service that quietly triples a bill or silently re-processes yesterday's data because four of its abstractions do not behave the way a plain Spark job would. A crawler infers a schema and writes it into the Glue Data Catalog; a Glue job runs Spark against that catalog; a job bookmark remembers which files it already read so the next run is incremental; and a DynamicFrame carries records whose schema was never fixed up front. Get any one of those four wrong and the pipeline still "works" — it just reads the entire S3 prefix every night, or drops a column the moment the source shape drifts, or emits ten thousand tiny Parquet files that make the next query crawl.

This guide is the deep dive you wished existed the first time an interviewer asked "how do Glue crawlers decide a table's partition keys?", or "how do job bookmarks know what's already been processed?", or "when would you drop from a DynamicFrame to a Spark DataFrame?", or "your Glue job scans all of S3 — how do you make it prune partitions?" It walks the four moving parts in order — crawler and catalog, bookmarks and incremental ETL job design, DynamicFrame vs DataFrame semantics, and Spark tuning with partitioning and predicate pushdown — and pairs each section with a Solution-Tail interview answer: code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the ETL practice library →, rehearse on the data transformation practice library →, and sharpen the tuning axis with the optimization practice library →.

---

On this page

• Why AWS Glue's serverless model changes how you build ETL
• Glue crawlers and the Data Catalog
• Job bookmarks and incremental ETL
• DynamicFrames vs Spark DataFrames
• Spark tuning, partitioning, and predicate pushdown
• Cheat sheet — AWS Glue recipes
• Frequently asked questions
• Practice on PipeCode

---

1. Why AWS Glue's serverless model changes how you build ETL

Four moving parts, one shared failure surface — the mental model that keeps a Glue job cheap and correct

The one-sentence invariant: AWS Glue is a managed Spark runtime wrapped around four cooperating abstractions — a crawler that infers schema into the Data Catalog, a Glue job that runs Spark against that catalog, a job bookmark that persists what has already been processed, and a DynamicFrame that tolerates schema drift — and almost every Glue incident traces back to misunderstanding one of those four, not to Spark itself. A plain EMR or Databricks Spark job asks you to hand it a schema and a file list; Glue instead offers to discover the schema for you (crawler), remember your progress for you (bookmark), and forgive your schema for you (DynamicFrame). Those conveniences are real, but each one hides state that you must reason about explicitly or the job drifts from correct to "technically running."

The four axes interviewers actually probe.

• Schema source of truth. Is the schema coming from a crawler-populated Data Catalog table, or is the job reading files directly and inferring on the fly? A crawler makes the catalog authoritative — great for Athena and downstream discovery, but it means schema changes require a re-crawl (or a schema-evolution policy) before the job sees them. Interviewers open here because the answer reveals whether you understand that the catalog is a cache of the schema, not the data.
• Incremental vs full reprocess. Does the job re-read the whole source every run, or only new data? Job bookmarks are the Glue-native answer, but they only work when you thread transformation_ctx through your reads and call job.commit(). A job without bookmarks silently re-processes everything — correct output, but O(all-data) cost every night.
• DynamicFrame vs DataFrame semantics. A DynamicFrame carries choice types (a column that is int in some records and string in others) and never fails a read on schema drift; a Spark DataFrame demands one type per column and is where joins, window functions, and tuned Spark SQL live. Knowing when to ResolveChoice and toDF() versus staying in DynamicFrame land is the tell of someone who has actually shipped Glue.
• Cost = DPU × runtime × bytes scanned. Glue bills per Data Processing Unit-hour. The three levers on the bill are worker count/size (DPU), how long the job runs (runtime), and — indirectly but dominantly — how much S3 you read (bytes scanned, controlled by partition pruning). A job that scans an unpruned prefix pays for data it throws away one line later.

The 2026 reality — Glue is the AWS-native default, but the footguns are unchanged.

• Glue 4.0 / 5.0 ship modern Spark (3.3+/3.5+) with autoscaling workers, and Glue is the path-of-least-resistance ETL for teams already on S3 + Athena + Redshift. If your stack is AWS-native, Glue is usually the first tool proposed.
• Crawlers remain the fastest way to get an Athena-queryable table over raw S3, but they are also the most common source of "the schema changed and half my columns became string" surprises — because a crawler merges schemas across files and has a configurable policy for what to do on conflict.
• Job bookmarks eliminate the nightly full-scan for append-only sources, but they are stateful and invisible: the single most common Glue bug in interviews is "my job re-processes everything" and the answer is always "you forgot transformation_ctx or job.commit()."
• DynamicFrames make Glue forgiving of messy, semi-structured, schema-drifting data — the exact data that breaks a naive DataFrame read — but they cost extra CPU to materialise choice types, so senior engineers drop to DataFrame the moment the schema is known and stable.

What interviewers listen for.

• Do you name all four abstractions — crawler, catalog, job, bookmark — without prompting? — senior signal.
• Do you say "the catalog is a schema cache, not the data" and explain re-crawl vs schema evolution? — required answer.
• Do you attribute a re-processing bug to missing transformation_ctx / job.commit() rather than "Spark being weird"? — senior signal.
• Do you connect cost to bytes scanned and reach for partition pruning / predicate pushdown first? — senior signal.
• Do you describe a DynamicFrame as "schema-flexible with choice types" rather than "like a DataFrame but Glue's"? — required answer.

Worked example — the four-part Glue component map

Detailed explanation. The most useful artifact for a Glue interview is a component map: for each moving part, name what state it owns, where that state lives, and the failure mode when you ignore it. Every Glue design discussion converges on this map. Walk through building it for a hypothetical pipeline that lands raw JSON clickstream in S3 and needs a partitioned Parquet table for Athena.

• Source. s3://raw/clickstream/dt=YYYY-MM-DD/*.json.gz — append-only, one prefix per day.
• Target. s3://curated/clickstream/ Parquet, partitioned by dt, registered in the catalog.
• Cadence. Hourly Glue job; only new files each run.
• Consumers. Athena ad-hoc, plus a downstream Redshift load.

Question. Build the four-component map and state which Glue feature owns each concern for this pipeline.

Input.

Component	State it owns	Where the state lives	Failure mode if ignored
Crawler	schema + partition keys	Data Catalog (metadata)	stale schema; new partitions invisible to Athena
Glue job	the Spark transform	job script + Spark cluster	logic bugs; wrong worker sizing
Job bookmark	files already processed	Glue service (per job/run)	re-processes all history every run
DynamicFrame	per-record schema tolerance	in-memory during the job	read fails or drops columns on drift

Code.

# Minimal Glue job skeleton showing all four parts in one script
import sys
from awsglue.transforms import ApplyMapping
from awsglue.utils import getResolvedOptions
from awsglue.context import GlueContext
from awsglue.job import Job
from pyspark.context import SparkContext

args = getResolvedOptions(sys.argv, ["JOB_NAME"])
sc = SparkContext()
glueContext = GlueContext(sc)
job = Job(glueContext)
job.init(args["JOB_NAME"], args)          # <-- bookmark state is loaded here

# Read FROM THE CATALOG (crawler-populated) as a DynamicFrame,
# with a transformation_ctx so the bookmark can track progress.
raw = glueContext.create_dynamic_frame.from_catalog(
    database="raw",
    table_name="clickstream",
    transformation_ctx="raw_clickstream",  # <-- bookmark key
)

# DynamicFrame transform: pin the schema we actually want.
mapped = ApplyMapping.apply(
    frame=raw,
    mappings=[
        ("event_id", "string", "event_id", "string"),
        ("ts", "string", "ts", "timestamp"),
        ("dt", "string", "dt", "string"),
    ],
    transformation_ctx="mapped_clickstream",
)

# Write partitioned Parquet.
glueContext.write_dynamic_frame.from_options(
    frame=mapped,
    connection_type="s3",
    connection_options={"path": "s3://curated/clickstream/", "partitionKeys": ["dt"]},
    format="parquet",
    transformation_ctx="write_clickstream",
)

job.commit()                               # <-- bookmark state is saved here

Step-by-step explanation.

1. job.init(args["JOB_NAME"], args) is where the bookmark for this named job is loaded. The job name is the bookmark's identity — rename the job and you start from a fresh bookmark, silently re-processing everything.
2. create_dynamic_frame.from_catalog(...) reads using the crawler-populated schema. The catalog is the source of truth for column names and types; if a crawler has not run since a schema change, the job sees the old shape. The transformation_ctx string is the stable key the bookmark uses to remember how far this specific read got.
3. ApplyMapping pins the schema we want downstream — it is where a DynamicFrame's flexibility gets converted into a committed shape. Any source column not listed here is dropped, which is a feature (explicit projection) and a footgun (silent drop) depending on whether you meant to.
4. The write declares partitionKeys=["dt"], so Glue lays the output out as dt=.../ folders — the layout a future crawler (or Athena's partition projection) expects.
5. job.commit() persists the bookmark. Skip it and the next run re-reads the same files, because from the bookmark's perspective the previous run never finished successfully.

Output.

Concern	Owned by	This pipeline's choice
Schema of clickstream	Crawler → Catalog	crawler runs before first job; schema-evolution policy set
Incremental read	Job bookmark	enabled; transformation_ctx on every read/write
Schema drift tolerance	DynamicFrame	read as DynamicFrame; ApplyMapping pins shape
Output layout	Glue job write	Parquet partitioned by dt

Rule of thumb. Before writing a line of transform logic, fill in the four-component map: which feature owns the schema, which owns incrementality, which owns drift tolerance, and where the output partitions live. Every Glue incident is one of those four cells left blank.

Worked example — when Glue wins and when it loses

Detailed explanation. Glue is not always the right tool, and a senior interview probes whether you know its edges. The decision comes down to how AWS-native the stack is, how spiky the workload is, and how much control over Spark internals you need. Walk through three scenarios and place each.

• Scenario A. A team on S3 + Athena + Redshift needs a nightly partitioned-Parquet build over raw logs.
• Scenario B. A team needs sub-minute streaming enrichment with custom state and fine-grained Spark tuning.
• Scenario C. A team runs one huge 6-hour daily transform that is CPU-bound and cost-sensitive.

Question. For each scenario, decide whether Glue is the right runtime and name the deciding axis.

Input.

Scenario	Workload shape	AWS-native?	Deciding axis
A — nightly Parquet build	batch, spiky	yes	serverless + catalog integration
B — streaming enrichment	continuous, stateful	yes	control / latency
C — long CPU-bound daily	batch, steady, long	maybe	cost at sustained load

Code.

Glue fit decision (say this out loud)
=====================================

A) Nightly Parquet build on S3 + Athena + Redshift
   → GLUE. Serverless (no cluster to babysit), crawler feeds Athena,
     bookmarks make it incremental, native Redshift connector.

B) Sub-minute streaming with custom state + deep Spark tuning
   → Usually NOT Glue's batch job. Consider Glue Streaming for
     simple cases, but heavy custom state / lowest latency →
     Kinesis Data Analytics / Flink / self-managed Spark.

C) One 6-hour CPU-bound transform, steady, cost-sensitive
   → LEAN AWAY from Glue at sustained load. Glue's per-DPU-hour
     price beats EMR for spiky/short jobs but loses for long,
     steady, predictable compute where a right-sized EMR /
     EC2 Spot cluster is cheaper per core-hour.

Step-by-step explanation.

1. Scenario A is Glue's sweet spot: spiky batch, AWS-native sinks, and a strong wish to not manage a cluster. The crawler + catalog + bookmark trio does exactly what this team needs with the least operational surface.
2. Scenario B stresses the control/latency axis. Glue's batch jobs are not for sub-minute streaming; Glue Streaming exists but heavy custom state or the lowest latencies push you toward Flink or a purpose-built streaming runtime.
3. Scenario C stresses the cost-at-sustained-load axis. Glue's convenience premium is worth it for short/spiky jobs; for a steady 6-hour daily grind, a right-sized EMR or EC2 Spot cluster usually wins on price per core-hour.
4. The general rule: Glue trades a per-DPU-hour premium for zero cluster management. That trade is great for spiky, AWS-native, discovery-heavy workloads and poor for long, steady, cost-optimised compute.
5. Naming the deciding axis (serverless convenience, latency/control, cost-at-load) rather than a blanket "Glue is good/bad" is the senior signal.

Output.

Scenario	Verdict	Why
A — nightly Parquet	Glue	serverless + catalog + bookmarks fit exactly
B — streaming/stateful	not batch Glue	latency/control axis favours Flink
C — long steady CPU	probably EMR/Spot	cost-at-load axis favours a managed cluster

Rule of thumb. Reach for Glue when the workload is spiky, AWS-native, and discovery-heavy; lean away when it is long, steady, and cost-critical, or when you need streaming state and sub-minute latency. State the deciding axis, not a verdict.

Worked example — reading the cost model before writing the job

Detailed explanation. Glue bills per DPU-hour, and the bill is dominated by a variable most beginners never mention: bytes scanned from S3. Two jobs with identical logic can differ 20× in cost purely from partition pruning. Walk through the cost model for the clickstream pipeline.

• Worker. G.1X = 1 DPU (4 vCPU, 16 GB); G.2X = 2 DPU (8 vCPU, 32 GB).
• Bill. DPUs × job runtime (hours) × price-per-DPU-hour, billed per second with a 1-minute minimum.
• Hidden driver. Runtime is a function of bytes scanned; scanning an unpruned prefix inflates runtime and the number of workers you feel you need.

Question. Estimate the cost delta between a job that scans one day's partition and one that scans a full year, all else equal.

Input.

Factor	Pruned (1 day)	Unpruned (365 days)
Bytes scanned	~5 GB	~1.8 TB
Runtime	~2 min	~90 min
Workers	4 × G.1X	20 × G.1X (added to cope)
DPU-hours	4 × (2/60) ≈ 0.13	20 × (90/60) = 30

Code.

# Back-of-envelope Glue cost model
PRICE_PER_DPU_HOUR = 0.44   # illustrative; check current AWS pricing

def dpu_hours(num_workers: int, dpu_per_worker: float, runtime_min: float) -> float:
    return num_workers * dpu_per_worker * (runtime_min / 60.0)

pruned   = dpu_hours(num_workers=4,  dpu_per_worker=1, runtime_min=2)
unpruned = dpu_hours(num_workers=20, dpu_per_worker=1, runtime_min=90)

print(f"pruned   : {pruned:6.2f} DPU-h  = ${pruned  * PRICE_PER_DPU_HOUR:6.2f}")
print(f"unpruned : {unpruned:6.2f} DPU-h  = ${unpruned * PRICE_PER_DPU_HOUR:6.2f}")
# pruned   :   0.13 DPU-h  = $  0.06
# unpruned :  30.00 DPU-h  = $ 13.20

Step-by-step explanation.

1. DPU-hours is the only thing Glue actually bills: workers × DPU-per-worker × runtime-in-hours. Everything else is a story about how those three numbers came to be.
2. The pruned job reads ~5 GB (one day), finishes in ~2 minutes on 4 small workers, and costs pennies. The unpruned job reads ~1.8 TB (a year), and the extra bytes inflate runtime to ~90 minutes.
3. The insidious part: because the unpruned job is slow, an engineer often "fixes" it by adding workers — turning a runtime problem into a runtime and worker-count problem, multiplying DPU-hours.
4. The 200× cost gap here is entirely from bytes scanned, driven by whether the read pruned to dt = today or scanned the whole prefix. No amount of worker tuning closes that gap; only partition pruning does.
5. This is why senior engineers reach for push_down_predicate and catalog partition filters before touching worker counts — the biggest lever on the bill is how much S3 you never read.

Output.

Job	DPU-hours	Cost	Cost driver
Pruned (1 day)	0.13	~$0.06	small read, short runtime
Unpruned (1 year)	30.0	~$13.20	full-prefix scan → long runtime → more workers

Rule of thumb. Before tuning worker counts, tune bytes scanned. In Glue, the dominant cost lever is partition pruning, not cluster size — a correctly pruned job on 4 small workers routinely beats an unpruned job on 20 large ones on both speed and price.

Senior interview question on the AWS Glue component model

A senior interviewer often opens with: "You inherit a Glue job that was cheap at launch but now costs 15× more and occasionally 'loses' recent data downstream. It reads a catalog table populated by a nightly crawler, transforms with DynamicFrames, and writes Parquet. Walk me through how you'd diagnose the cost blow-up and the data-loss, naming which of Glue's abstractions each symptom implicates."

Solution Using bookmark verification, partition-prune audit, and a schema-evolution check

# Diagnostic pass — instrument the existing job, don't rewrite it yet
import sys
from awsglue.utils import getResolvedOptions
from awsglue.context import GlueContext
from awsglue.job import Job
from pyspark.context import SparkContext

args = getResolvedOptions(sys.argv, ["JOB_NAME"])
glueContext = GlueContext(SparkContext())
job = Job(glueContext)
job.init(args["JOB_NAME"], args)

# 1. COST AUDIT — is the read pruning partitions?
#    Push a predicate and log the input partition count.
pruned = glueContext.create_dynamic_frame.from_catalog(
    database="raw",
    table_name="clickstream",
    push_down_predicate="dt >= date_format(current_date - 1, 'yyyy-MM-dd')",
    transformation_ctx="raw_clickstream",
)
print("input partitions after prune:", pruned.toDF().rdd.getNumPartitions())

# 2. DATA-LOSS AUDIT — is the bookmark committing?
#    If job.commit() was missing, the bookmark never advanced and a
#    failed run silently re-reads; a *rewound* bookmark skips new data.
print("bookmark-relevant ctx present:", "raw_clickstream")

-- 3. SCHEMA-EVOLUTION AUDIT — did the crawler change the table shape?
--    Compare current catalog columns against the job's ApplyMapping.
SELECT column_name, data_type
FROM   information_schema.columns          -- via Athena over the Glue catalog
WHERE  table_schema = 'raw'
  AND  table_name   = 'clickstream'
ORDER  BY ordinal_position;

Step-by-step trace.

Symptom	Suspected abstraction	Diagnostic	Likely root cause
Cost 15×	Job / read	log input partition count with/without predicate	read scans full prefix (no pushdown)
"Loses" recent data	Job bookmark	check job.commit() + transformation_ctx	commit missing or bookmark rewound
Column turned string	Crawler / Catalog	diff catalog schema vs ApplyMapping	crawler merged an incompatible file
New partitions unseen	Crawler	check last crawl time vs new dt= folders	crawler not run since new data landed
Job slow then failing	Job / Spark	check skew + small-file count	unpruned read + tiny-file explosion

After the diagnostic run, the input-partition log shows the read touching 365 partitions instead of 1 — the pushdown was never applied, confirming the cost blow-up is a partition-prune miss. The bookmark audit shows job.commit() present but a recent manual --job-bookmark-option job-bookmark-pause in the run config, which is why new data was skipped. The schema diff shows payload flipped from struct to string after the crawler merged a malformed file.

Output:

Finding	Abstraction	Fix
365-partition scan	read / pushdown	add push_down_predicate on dt
new data skipped	bookmark	remove pause; set job-bookmark-enable
payload → string	crawler policy	set schema policy; fix malformed file; re-crawl
slow shuffle	Spark tuning	coalesce small files with groupFiles

Why this works — concept by concept:

• Push-down predicate audit — logging the input partition count with and without a predicate on dt isolates the cost problem to the read layer. A read that reports 365 partitions when the job needs 1 is scanning ~365× the bytes it should, and bytes scanned is the dominant Glue cost driver.
• Bookmark commit and pause state — job bookmarks only advance when job.commit() runs and the bookmark is enabled. A paused bookmark (job-bookmark-pause) processes nothing new; a missing commit re-processes everything. Both are invisible in the job logic and only visible in the run configuration.
• Catalog-vs-mapping schema diff — because the catalog is a schema cache populated by the crawler, a column that "became a string" is almost always a crawler schema-merge decision, not a Spark bug. Diffing the live catalog against the job's ApplyMapping surfaces the drift.
• Crawler recency — new dt= partitions are invisible to catalog reads until a crawler (or MSCK-style partition add / partition projection) registers them, so "recent data missing" can also be a stale-crawler symptom independent of the bookmark.
• Cost — the diagnostic is O(1 day) of data because every read is predicate-pruned; the fixes turn an O(all-history) nightly scan into O(new-partitions). The dominant saving is bytes scanned (200×+ in the earlier cost model), not worker tuning.

ETL
Topic — etl
ETL problems on serverless pipeline design

Practice →

Design Topic — design Design problems on batch data platforms

Practice →

---

2. Glue crawlers and the Data Catalog

A crawler infers schema, detects partitions, and writes a queryable table — but it merges schemas, and that merge is where the surprises live

The mental model in one line: a Glue crawler walks an S3 prefix (or a JDBC source), runs classifiers to detect the format, infers a column schema by merging the shapes it sees across sampled files, detects dt=/key=value folder patterns as partition keys, and writes or updates a table in the Glue Data Catalog — so the catalog becomes an Athena-queryable, job-readable description of data the crawler never moved. The crawler is a metadata process, not a data process: it reads a sample, decides on a schema, and records partitions — and every "my column became a string" or "my table split into fifty tables" story is a consequence of how that merge and grouping behaved.

What a crawler actually does, step by ordered step.

• Classify. A chain of classifiers (built-in for JSON, CSV, Parquet, ORC, Avro, plus custom Grok/JSONPath) runs against sampled objects; the first to match with high confidence wins and determines the format and initial column set.
• Infer + merge. The crawler samples multiple files and unifies their schemas. If file A has age: int and file B has age: "N/A" (string), the merge must reconcile them — typically widening to string or recording a conflict, depending on data and settings.
• Detect partitions. Folder patterns like .../dt=2026-08-18/... become partition columns (dt) automatically. The crawler records each distinct partition value as a partition of the table.
• Register / update. The result is a Data Catalog table (columns + partition keys + serde + location). On subsequent runs the crawler updates that table subject to a schema-change policy.

The schema-change policy — the setting that decides your surprises.

• Update the table definition. New columns are added; changed types are updated in the catalog. Convenient, but a type change (int → string) propagates to every downstream reader.
• Add new columns only. Additive changes are applied; deletions/type-changes are ignored. Safer for stable pipelines.
• Log a warning, do not modify. The crawler detects drift but leaves the table as-is and logs it — the most conservative option; you evolve the schema deliberately.
• Partition changes. Separately, you choose whether the crawler updates all partitions to match the table schema or leaves per-partition schemas — critical when older partitions have a different shape than new ones.

Table grouping — one table or fifty?

• The default. If sibling folders under a prefix have compatible schemas, the crawler groups them into a single table with partition keys.
• The trap. If schemas differ enough (different columns, incompatible types), the crawler creates separate tables per folder — the dreaded "why do I have clickstream_2026, clickstream_2027, ..." explosion.
• The control. TableGroupingPolicy = CombineCompatibleSchemas and a well-designed prefix layout keep related data in one table. A TableLevelConfiguration (table level in the path) pins where the crawler draws the table boundary.

Common interview probes on crawlers.

• "Do you always need a crawler?" — no; you can define catalog tables by DDL or let a job write partitions and register them, but crawlers are the fastest discovery path.
• "How does a crawler decide partition keys?" — from key=value folder naming; ordered by depth.
• "What happens when two files disagree on a column type?" — schema merge widens or conflicts per policy; often ends up string.
• "How do you stop a table from re-splitting every crawl?" — grouping policy + consistent prefix layout + a schema-evolution policy that does not thrash.

Worked example — crawl a partitioned S3 JSON prefix into a catalog table

Detailed explanation. The canonical crawler setup: point a crawler at a partitioned JSON prefix, let it infer the schema and partition key, and produce one catalog table that Athena and a Glue job can both read. Walk through the layout, the crawler config, and the resulting table.

• Layout. s3://raw/clickstream/dt=2026-08-16/, .../dt=2026-08-17/, .../dt=2026-08-18/, each with gzip JSON.
• Crawler. One S3 target at s3://raw/clickstream/, JSON classifier, schema policy = add-new-columns, grouping = combine-compatible.
• Result. Table raw.clickstream with columns + a dt partition key.

Question. Configure the crawler (as Infrastructure-as-Code) and show the catalog table it produces.

Input.

Setting	Value
S3 target	s3://raw/clickstream/
Format	JSON (gzip)
Partition pattern	dt=YYYY-MM-DD
Schema change policy	add new columns only
Grouping	combine compatible schemas

Code.

# Create a crawler via boto3 (IaC-style)
import boto3

glue = boto3.client("glue")

glue.create_crawler(
    Name="clickstream-crawler",
    Role="arn:aws:iam::123456789012:role/GlueCrawlerRole",
    DatabaseName="raw",
    Targets={"S3Targets": [{"Path": "s3://raw/clickstream/"}]},
    # Schema-evolution policy: only add columns; never delete/retype in place
    SchemaChangePolicy={
        "UpdateBehavior": "LOG",          # log type changes, don't rewrite
        "DeleteBehavior": "LOG",          # log deletions, keep the table
    },
    RecrawlPolicy={"RecrawlBehavior": "CRAWL_NEW_FOLDERS_ONLY"},  # cheap re-crawls
    Configuration=(
        '{"Version":1.0,'
        '"CrawlerOutput":{"Partitions":{"AddOrUpdateBehavior":"InheritFromTable"}},'
        '"Grouping":{"TableGroupingPolicy":"CombineCompatibleSchemas"}}'
    ),
)

glue.start_crawler(Name="clickstream-crawler")

-- The catalog table the crawler produces (viewed via Athena SHOW CREATE TABLE)
CREATE EXTERNAL TABLE raw.clickstream (
    event_id  string,
    user_id   bigint,
    ts        string,
    payload   struct<page:string, referrer:string>
)
PARTITIONED BY (dt string)              -- inferred from dt=YYYY-MM-DD folders
ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
LOCATION 's3://raw/clickstream/';

Step-by-step explanation.

1. The crawler's single S3 target at the prefix root lets it discover all dt= folders under it. RecrawlBehavior = CRAWL_NEW_FOLDERS_ONLY means subsequent runs only look at folders added since the last crawl — dramatically cheaper than re-reading everything.
2. UpdateBehavior = LOG and DeleteBehavior = LOG implement the conservative schema policy: the crawler tells you about type changes and deletions but does not rewrite the table, so a single malformed file cannot silently retype a production column.
3. AddOrUpdateBehavior = InheritFromTable for partitions means every partition inherits the table-level schema rather than keeping its own — this is what prevents "old partitions have a different shape" query failures in Athena.
4. TableGroupingPolicy = CombineCompatibleSchemas keeps all the dt= folders under one clickstream table instead of splitting into a table per folder — the fix for the table-explosion trap.
5. The produced table lists data columns plus dt as a partition key. dt is a partition column (stored in the path, not the files), which is exactly what makes partition pruning possible later.

Output.

Aspect	Result
Table created	raw.clickstream (single table)
Data columns	event_id, user_id, ts, payload
Partition key	dt (string, from folder names)
Schema on drift	logged, not rewritten
Re-crawl cost	new folders only

Rule of thumb. Point one crawler at the prefix root, set the schema policy to LOG (or add-columns-only) for production tables, enable CRAWL_NEW_FOLDERS_ONLY, and turn on CombineCompatibleSchemas. Those four settings prevent the three classic crawler incidents: silent retype, table explosion, and expensive full re-crawls.

Worked example — the schema-merge type conflict

Detailed explanation. The most common crawler surprise: a numeric column becomes string because one file in the prefix wrote the value quoted (or wrote a sentinel like "N/A"). The crawler merges the two shapes and widens to the common type that can hold both — string. Walk through the diagnosis and the two fixes.

• Symptom. user_id was bigint; after a crawl it is string, and downstream SUM/joins on it break.
• Root cause. One new file wrote "user_id": "N/A"; the merge of bigint and string widens to string.
• Fixes. (a) conservative schema policy so the crawler logs instead of retyping; (b) fix the producer / cast in the job with ResolveChoice.

Question. Show how the merge produces the wrong type and how a schema policy plus a job-side cast recovers correctness.

Input.

File	user_id value	Inferred type
part-000.json	42	bigint
part-001.json	43	bigint
part-002.json	"N/A"	string
merged	mixed	string (widened)

Code.

# Job-side recovery: read as DynamicFrame, resolve the choice, cast cleanly
from awsglue.transforms import ResolveChoice

dyf = glueContext.create_dynamic_frame.from_catalog(
    database="raw", table_name="clickstream",
    transformation_ctx="raw_clickstream",
)

# If the catalog says user_id is a choice or string, coerce it to long,
# turning non-numeric sentinels into NULL rather than failing the whole read.
resolved = ResolveChoice.apply(
    frame=dyf,
    specs=[("user_id", "cast:long")],
    transformation_ctx="resolve_user_id",
)

-- Athena confirmation after a conservative-policy crawl:
-- the crawler LOGGED the conflict instead of retyping the column.
-- (CloudWatch crawler log)
--   INFO  Detected schema change for column 'user_id':
--         existing=bigint incoming=string  -> UpdateBehavior=LOG, table unchanged

Step-by-step explanation.

1. The crawler samples all three files and must produce one type for user_id. bigint and string have no common numeric supertype, so the only type that holds both is string — the merge widens, and the column silently becomes text.
2. Under the default "update in place" policy the catalog now reports user_id string, and every downstream SUM(user_id) or numeric join fails or silently mis-casts. This is the incident.
3. The first fix is preventive: a LOG schema policy makes the crawler report the conflict (as shown in the CloudWatch log) and leave the bigint table untouched, so production readers are not broken by one bad file.
4. The second fix is corrective in the job: read as a DynamicFrame and ResolveChoice with cast:long, which coerces valid numerics and turns "N/A" into NULL rather than failing the read. DynamicFrames exist precisely to survive this drift.
5. The durable fix is upstream: stop the producer from writing sentinels into a numeric field. Schema policy and ResolveChoice are guardrails, not a substitute for a clean contract.

Output.

Layer	Behaviour with fix
Crawler (LOG policy)	logs conflict; keeps bigint table
Job read	reads even with mixed types (DynamicFrame)
ResolveChoice cast:long	42→42, "N/A"→NULL
Downstream	numeric ops work; bad rows are NULL, not fatal

Rule of thumb. Treat "a column silently became string" as a crawler schema-merge signature, not a Spark bug. Set the schema policy to LOG for production, recover in the job with ResolveChoice cast:..., and fix the producing contract so the merge never widens in the first place.

Senior interview question on crawlers and the Data Catalog

A senior interviewer might ask: "You run a daily crawler over a large partitioned S3 prefix. Lately the crawler takes hours, occasionally splits your table into many tables, and once retyped a numeric column to string, breaking Athena dashboards. Redesign the crawler strategy — schema policy, grouping, re-crawl behaviour, and whether a crawler is even the right tool for new partitions."

Solution Using a conservative crawler plus partition projection for new data

# 1. Reconfigure the crawler: conservative, incremental, single-table
import boto3
glue = boto3.client("glue")

glue.update_crawler(
    Name="clickstream-crawler",
    Role="arn:aws:iam::123456789012:role/GlueCrawlerRole",
    DatabaseName="raw",
    Targets={"S3Targets": [{"Path": "s3://raw/clickstream/"}]},
    SchemaChangePolicy={"UpdateBehavior": "LOG", "DeleteBehavior": "LOG"},
    RecrawlPolicy={"RecrawlBehavior": "CRAWL_NEW_FOLDERS_ONLY"},
    Configuration=(
        '{"Version":1.0,'
        '"Grouping":{"TableGroupingPolicy":"CombineCompatibleSchemas"},'
        '"CrawlerOutput":{"Partitions":{"AddOrUpdateBehavior":"InheritFromTable"}}}'
    ),
)

-- 2. For a stable schema, stop crawling for partitions entirely:
--    use Athena partition projection so NEW dt= folders are query-visible
--    the instant they exist, with zero crawler runs.
ALTER TABLE raw.clickstream SET TBLPROPERTIES (
  'projection.enabled'      = 'true',
  'projection.dt.type'      = 'date',
  'projection.dt.range'     = '2024-01-01,NOW',
  'projection.dt.format'    = 'yyyy-MM-dd',
  'projection.dt.interval'  = '1',
  'projection.dt.interval.unit' = 'DAYS',
  'storage.location.template'   = 's3://raw/clickstream/dt=${dt}/'
);

Step-by-step trace.

Problem	Old behaviour	New behaviour
Slow crawls	full re-crawl each day	CRAWL_NEW_FOLDERS_ONLY → minutes
Table splitting	per-folder tables	CombineCompatibleSchemas → one table
Silent retype	update-in-place → string	LOG policy → conflict logged, table kept
New partitions	wait for next crawl	partition projection → instantly visible
Per-partition drift	mixed schemas	InheritFromTable → uniform

After the change, the crawler runs only to detect genuinely new columns (rare), while day-to-day new dt= partitions become queryable the moment they land via partition projection — no crawler run required. Table splitting stops because grouping is forced to combine, and the numeric-retype incident cannot recur because the schema policy logs conflicts instead of rewriting the table.

Output:

Metric	Before	After
Crawl runtime	~2 h	~3 min (new folders only)
Tables for clickstream	many	1
New-partition visibility	next crawl (hours)	immediate (projection)
Retype risk	high (update-in-place)	none (LOG policy)
Athena query correctness	breaks on drift	stable

Why this works — concept by concept:

• Crawl new folders only — RecrawlBehavior = CRAWL_NEW_FOLDERS_ONLY turns an O(all-partitions) daily scan into O(new-partitions), which is the single biggest crawler-cost lever for append-only prefixes.
• Combine compatible schemas — forcing the grouping policy keeps sibling dt= folders under one table instead of minting a table per folder, eliminating the table-explosion class of incident.
• LOG schema policy — logging drift instead of rewriting the table means one malformed file can never silently retype a production column; you evolve the schema deliberately after reading the log.
• Partition projection — moving partition discovery out of the crawler and into Athena TBLPROPERTIES makes new partitions visible with zero crawler runs, because Athena computes partition locations from the dt template rather than from catalog rows.
• Cost — the crawler now touches only new folders (near-zero), and partition projection is free at query time; compared to hours of daily full crawls, this is an O(new)-vs-O(all) improvement with strictly safer schema semantics.

ETL
Topic — etl
ETL problems on schema inference and catalogs

Practice →

Data processing Topic — data-processing Data processing problems on partitioned datasets

Practice →

---

3. Job bookmarks and incremental ETL

A bookmark is per-job persisted state of "what have I already read" — and it only works when you thread the context and commit

The mental model in one line: a job bookmark is a small piece of state that Glue persists per named job, recording how far each read source has been consumed — for S3 sources it tracks object paths and modification timestamps, for JDBC sources it tracks a monotonically increasing key column — so that the next run reads only new data, but it advances only if you pass a transformation_ctx on every read and write and call job.commit() at the end. Bookmarks are what turn a Glue job from "re-scan everything nightly" into a true incremental ETL job; they are also the single most common source of "it re-processes everything" and "it skipped yesterday's data" bugs, because the state is invisible in the script and lives in the Glue service.

How the bookmark tracks progress.

• S3 sources. The bookmark records the set of files already processed, keyed by object path and last-modified time. New files (or files with a newer mod-time) are in; already-seen files are out. This is why mutating an existing file can either be missed (mod-time unchanged) or cause reprocessing (mod-time bumped).
• JDBC sources. The bookmark tracks the maximum value of one or more jobBookmarkKeys columns (e.g. an auto-increment id or an updated_at). The next run reads WHERE key > last_seen. The key must be strictly increasing and unique-ish or rows are skipped or duplicated.
• Per-source identity. The bookmark is keyed by the job name plus the transformation_ctx of each read. Two reads in one job need two distinct transformation_ctx values or their state collides.

The four lines that make bookmarks work.

• job.init(job_name, args) — loads the bookmark for this job at the start. The job_name is the bookmark's identity.
• transformation_ctx="..." on every create_dynamic_frame and every write_dynamic_frame — the stable per-source key. Change or omit it and the bookmark loses the thread.
• job.commit() — persists the advanced bookmark. Without it, the run's progress is discarded and the next run re-reads.
• The run option --job-bookmark-option — job-bookmark-enable (use + advance), job-bookmark-pause (use but do not advance), job-bookmark-disable (ignore entirely; full reprocess).

Bookmark states and the reset button.

• Enable. Normal incremental operation.
• Pause. Reads from the current bookmark but does not advance it — useful for reprocessing the same delta repeatedly during debugging without moving the marker.
• Disable. Ignores the bookmark; every run is a full reprocess.
• Reset. aws glue reset-job-bookmark --job-name X clears the state so the next run starts from the beginning — the deliberate "reprocess all history" button.

Common interview probes on bookmarks.

• "Why is my Glue job reprocessing everything?" — missing transformation_ctx or job.commit(), or bookmark disabled.
• "Why did my job skip new rows?" — bookmark paused, or the JDBC key is not strictly increasing, or an S3 file's mod-time did not change.
• "How do bookmarks handle late-arriving files?" — an S3 file that appears with an older mod-time than the bookmark can be skipped; design for this.
• "How do you reprocess one bad day?" — reset the bookmark (full) or run a separate backfill job that reads a bounded partition range with the bookmark disabled.

Worked example — incremental S3 → Parquet with bookmarks

Detailed explanation. The canonical incremental job: read new clickstream files since the last run, transform, and append Parquet — with bookmarks doing the "only new files" work. Walk through the full script and the run configuration.

• Source. raw.clickstream catalog table over s3://raw/clickstream/.
• Bookmark. Enabled; transformation_ctx on read and write.
• Target. Append Parquet to s3://curated/clickstream/, partitioned by dt.

Question. Write the incremental job and the run arguments that make it process only new files.

Input.

Run	Files present	Files new since bookmark	Files read
1 (bootstrap)	100	100	100
2	112	12	12
3	112	0	0
4	130	18	18

Code.

import sys
from awsglue.transforms import ApplyMapping
from awsglue.utils import getResolvedOptions
from awsglue.context import GlueContext
from awsglue.job import Job
from pyspark.context import SparkContext

args = getResolvedOptions(sys.argv, ["JOB_NAME"])
glueContext = GlueContext(SparkContext())
job = Job(glueContext)
job.init(args["JOB_NAME"], args)          # load bookmark

# READ — transformation_ctx is the bookmark key for this source
src = glueContext.create_dynamic_frame.from_catalog(
    database="raw",
    table_name="clickstream",
    transformation_ctx="src_clickstream",
)

mapped = ApplyMapping.apply(
    frame=src,
    mappings=[
        ("event_id", "string", "event_id", "string"),
        ("user_id",  "bigint", "user_id",  "long"),
        ("ts",       "string", "ts",       "timestamp"),
        ("dt",       "string", "dt",       "string"),
    ],
    transformation_ctx="map_clickstream",
)

# WRITE — its own transformation_ctx; partitioned append
glueContext.write_dynamic_frame.from_options(
    frame=mapped,
    connection_type="s3",
    connection_options={"path": "s3://curated/clickstream/", "partitionKeys": ["dt"]},
    format="parquet",
    transformation_ctx="write_clickstream",
)

job.commit()                              # persist the advanced bookmark

# Run configuration (Glue job arguments)
--job-bookmark-option   job-bookmark-enable
--JOB_NAME              clickstream-incremental

Step-by-step explanation.

1. job.init("clickstream-incremental", args) loads the bookmark tied to that job name. Because the name is the identity, cloning this job under a new name would restart from empty and reprocess all 100 files.
2. The read passes transformation_ctx="src_clickstream". On run 1 the bookmark is empty, so all 100 files are read (the bootstrap). Glue records the processed file set.
3. On run 2, the bookmark knows the first 100 files; only the 12 newer files pass the filter. The transform runs on 12 files' worth of data — the incremental win.
4. Run 3 finds no new files; the read yields an empty frame and the write is a no-op. The job still calls job.commit(), keeping the bookmark valid.
5. job.commit() at the end persists the advance. If the job raised before commit (or you forgot the call), run 2 would re-read the original 100 files because the bookmark never moved past run 1.

Output.

Run	Files read	Bookmark after
1	100	knows 100 files
2	12	knows 112 files
3	0	unchanged (112)
4	18	knows 130 files

Rule of thumb. For any incremental S3 job: job.init at the top, a unique transformation_ctx on every read and every write, job.commit() at the bottom, and --job-bookmark-option job-bookmark-enable in the run config. Miss any one and you are back to full reprocessing.

Worked example — JDBC bookmark keys and the strictly-increasing requirement

Detailed explanation. Bookmarking a JDBC source is different: there are no file mod-times, so Glue tracks the maximum value of a jobBookmarkKeys column. The column must be strictly increasing (and ideally unique) or the incremental read skips or duplicates rows. Walk through a Postgres orders source keyed by id.

• Source. Postgres public.orders, primary key id BIGSERIAL.
• Bookmark key. id, ascending — new rows always have a larger id.
• Risk. If you key on updated_at (not strictly increasing under clock skew or bulk backfills), rows can be missed.

Question. Configure a JDBC incremental read with bookmark keys and explain the strictly-increasing requirement.

Input.

Option	Value
Source	Postgres public.orders
Bookmark keys	id
Sort order	ascending
Guarantee	id strictly increasing, unique

Code.

orders = glueContext.create_dynamic_frame.from_options(
    connection_type="postgresql",
    connection_options={
        "url": "jdbc:postgresql://db-primary:5432/production",
        "user": "cdc_reader",
        "password": args["DB_PASSWORD"],
        "dbtable": "public.orders",
        # Bookmark keys: strictly-increasing column(s) Glue tracks the MAX of
        "jobBookmarkKeys": ["id"],
        "jobBookmarkKeysSortOrder": "asc",
    },
    transformation_ctx="orders_jdbc",     # bookmark key for this JDBC source
)

# What Glue effectively runs each incremental pass:
#   run 1 (empty bookmark):  SELECT * FROM public.orders
#   run 2 (bookmark id=8123): SELECT * FROM public.orders WHERE id > 8123
#   run 3 (bookmark id=8140): SELECT * FROM public.orders WHERE id > 8140

Step-by-step explanation.

1. jobBookmarkKeys=["id"] tells Glue to remember the maximum id it has read. The next run reads only id > last_max, which is the JDBC analogue of the S3 file-set filter.
2. The asc sort order tells Glue the key increases; Glue reads in that order and records the top value. If the key actually decreased for some rows, those rows would fall below the watermark and be skipped forever.
3. This is why id (a BIGSERIAL) is the right key and updated_at is dangerous: a bulk backfill or clock skew can produce an updated_at smaller than the current watermark, so those updated rows are never re-read.
4. For sources that genuinely need change tracking on updated_at, bookmarks are the wrong tool — that is a change-data-capture problem (log-based CDC or a watermark table with a safety window), not a Glue bookmark.
5. The transformation_ctx="orders_jdbc" keeps this JDBC source's watermark separate from any other read in the same job. Two JDBC reads sharing a context would corrupt each other's watermark.

Output.

Run	Bookmark (max id)	Rows read
1	none	all rows
2	8123	id > 8123
3	8140	id > 8140
key = updated_at	risky	skips backfilled/skewed rows

Rule of thumb. For JDBC bookmarks, key on a strictly-increasing, unique column (a surrogate BIGSERIAL/identity), never on a wall-clock updated_at. If you need update tracking, that is a CDC problem — reach for log-based CDC or a watermark table with a safety window, not a Glue bookmark.

Senior interview question on job bookmarks

A senior interviewer might ask: "Your incremental Glue job over an S3 prefix started reprocessing the entire history after a refactor, tripling the bill. Later, after a bad deploy, it skipped two hours of data. Diagnose both failures in terms of bookmark mechanics, and design a job that is safe to re-run, safe to backfill, and observable."

Solution Using explicit context, commit discipline, and a bounded backfill path

import sys
from awsglue.transforms import ApplyMapping
from awsglue.utils import getResolvedOptions
from awsglue.context import GlueContext
from awsglue.job import Job
from pyspark.context import SparkContext

# Accept an optional backfill window; default to bookmark-driven incremental
args = getResolvedOptions(sys.argv, ["JOB_NAME"])
glueContext = GlueContext(SparkContext())
job = Job(glueContext)
job.init(args["JOB_NAME"], args)

# Stable transformation_ctx values — NEVER change these across deploys,
# because they ARE the bookmark identity for each source.
src = glueContext.create_dynamic_frame.from_catalog(
    database="raw",
    table_name="clickstream",
    transformation_ctx="src_clickstream_v1",   # frozen key
)

mapped = ApplyMapping.apply(
    frame=src,
    mappings=[
        ("event_id", "string", "event_id", "string"),
        ("user_id",  "bigint", "user_id",  "long"),
        ("ts",       "string", "ts",       "timestamp"),
        ("dt",       "string", "dt",       "string"),
    ],
    transformation_ctx="map_clickstream_v1",
)

# Observability: log input row count so a reprocess/skip is visible in logs
row_count = mapped.toDF().count()
print(f"BOOKMARK_AUDIT job={args['JOB_NAME']} input_rows={row_count}")

glueContext.write_dynamic_frame.from_options(
    frame=mapped,
    connection_type="s3",
    connection_options={"path": "s3://curated/clickstream/", "partitionKeys": ["dt"]},
    format="parquet",
    transformation_ctx="write_clickstream_v1",
)

job.commit()

# Backfill path (separate run config) — bounded, bookmark OFF
--job-bookmark-option   job-bookmark-disable
# plus a job param the read uses as a push_down_predicate, e.g.
--backfill_predicate    dt >= '2026-08-10' AND dt <= '2026-08-11'

Step-by-step trace.

Failure / need	Bookmark mechanic	Design response
Reprocessed all history	refactor changed/removed transformation_ctx	freeze _v1 ctx strings; treat as API
Skipped 2 hours	bookmark was left job-bookmark-pause	assert enabled; alert on unexpected pause
Safe re-run	commit discipline	single job.commit(); idempotent partitioned write
Backfill one range	bookmark bypass	separate config: bookmark-disable + bounded predicate
Observability	none by default	log input_rows every run for anomaly detection

After the redesign, the transformation_ctx strings are frozen and version-suffixed so a refactor can never silently reset them; the row-count log makes a reprocess (input spikes to full history) or a skip (input drops to zero unexpectedly) immediately visible in CloudWatch; and backfills go through a separate, bounded run that disables the bookmark and reads only the target dt range, so a one-day replay never disturbs the incremental marker.

Output:

Property	Result
Refactor-safe	frozen _v1 context keys
Skip-detectable	BOOKMARK_AUDIT input_rows= log
Re-run-safe	idempotent partitioned append + single commit
Backfill	bounded predicate, bookmark disabled
Blast radius of a bad deploy	bounded and observable

Why this works — concept by concept:

• Frozen transformation context — because the bookmark identity is job_name + transformation_ctx, treating those strings as a frozen public API (version-suffixed, never edited casually) removes the "refactor silently reset the bookmark" failure at its root.
• Single commit discipline — exactly one job.commit() on the success path means the bookmark advances if and only if the run truly finished; a raised exception leaves the marker where it was, so a retry re-reads the same delta safely.
• Row-count audit log — emitting input_rows every run turns invisible bookmark state into an observable signal: a spike means an accidental full reprocess, a surprise zero means a skip or a pause, both alertable.
• Bounded backfill with bookmark disabled — routing replays through a separate config that disables the bookmark and constrains the read to a dt range keeps history reprocessing off the incremental marker, so a backfill never rewinds or corrupts normal operation.
• Cost — steady-state reads stay O(new files); the only O(all-history) path is a deliberate reset or a bounded backfill. The audit log costs one count() action per run — a small, worthwhile price for making bookmark state observable.

ETL
Topic — etl
ETL problems on incremental and idempotent loads

Practice →

SQL Topic — sql SQL problems on watermarks and incremental keys

Practice →

---

4. DynamicFrames vs Spark DataFrames

A DynamicFrame tolerates schema drift with choice types; a DataFrame demands one type per column — knowing when to cross the bridge is the skill

The mental model in one line: a DynamicFrame is Glue's schema-flexible record collection where a single column can hold a choice of types across records (int in some, string in others) and a read never fails on drift, whereas a Spark DataFrame requires exactly one type per column and is where joins, window functions, and tuned Spark SQL live — so the Glue idiom is: read messy data as a DynamicFrame, resolve and clean it (ResolveChoice, Relationalize, ApplyMapping), then toDF() into a DataFrame for the heavy relational work. DynamicFrames exist because raw, semi-structured, drifting data breaks a naive DataFrame read; DataFrames exist because that is where Spark's optimiser and rich SQL actually operate.

What a DynamicFrame gives you that a DataFrame does not.

• Choice types. A column can be a choice<int, string> — the DynamicFrame carries both interpretations per record instead of failing the read or coercing blindly. You resolve the choice explicitly, later, on your terms.
• Self-describing records. Each record carries its own schema, so a file with an extra field does not break sibling files. This is the "schema-on-read that never throws" property.
• Glue-native transforms. ResolveChoice, Relationalize, ApplyMapping, DropNullFields, Unbox, SelectFields — transforms designed for messy ingestion, not available on plain DataFrames.
• No upfront schema. You can read genuinely unknown JSON without declaring a StructType first.

The transforms you will actually use.

• ResolveChoice. Collapse a choice column: cast:long (coerce), make_cols (split into col_int, col_string), make_struct (keep both in a struct), or project:type (keep one type, drop the rest).
• Relationalize. Flatten nested structures and unnest arrays into separate tables with generated join keys — the canonical way to turn nested JSON into relational tables.
• ApplyMapping. Rename, retype, and project columns in one step. Any source column not listed is dropped — powerful and a silent-data-loss risk.
• DropNullFields / SelectFields / Unbox. Prune all-null columns, keep a subset, or parse a string column into structured fields.

Crossing the bridge — toDF() and fromDF().

• dynamic_frame.toDF(). Materialise the DynamicFrame into a Spark DataFrame. Do this once the schema is known and stable, before joins/windows/Spark SQL. It has a cost — choice types must be resolved into concrete types first, or the conversion picks a representation for you.
• DynamicFrame.fromDF(df, glueContext, name). Convert back to a DynamicFrame — needed to use Glue-native writes (write_dynamic_frame) or bookmarks on the write side.
• The idiom. Read as DynamicFrame → resolve/clean → toDF() → relational work in DataFrame/Spark SQL → fromDF() → write_dynamic_frame (so partitioning + bookmarks still apply).

When to use which.

• DynamicFrame when: ingesting semi-structured or schema-drifting data, resolving type conflicts, flattening nested JSON, or you need bookmarks/partitioned Glue writes.
• DataFrame when: the schema is known and stable and you need joins, window functions, groupBy aggregations, UDFs, or you want the Catalyst optimiser and predicate pushdown into Parquet/ORC.

Common interview probes.

• "What is a choice type and how do you resolve it?" — ResolveChoice with cast/make_cols/make_struct/project.
• "How do you flatten nested JSON in Glue?" — Relationalize (produces a root frame + child frames for arrays).
• "Why not just always use DataFrames?" — they fail or mis-coerce on drift; DynamicFrames survive messy reads.
• "What is the risk of ApplyMapping?" — it silently drops unlisted columns; diff the schema before and after.

Worked example — resolve a choice-type column

Detailed explanation. A crawler flagged amount as a choice<long, string> because some records wrote it quoted. Reading it as a DataFrame would coerce or fail; as a DynamicFrame you resolve the choice deliberately. Walk through the three resolution strategies and pick one.

• Data. amount is long in most records, string ("12.50", "N/A") in some.
• Goal. A clean numeric amount in cents, with non-numeric values becoming NULL.
• Options. cast:long, make_cols (two columns), make_struct (keep both).

Question. Resolve the amount choice to a single numeric column and show what each strategy would produce.

Input.

Record	raw amount	choice member
1	1200	long
2	"12.50"	string
3	"N/A"	string
4	3400	long

Code.

from awsglue.transforms import ResolveChoice

dyf = glueContext.create_dynamic_frame.from_catalog(
    database="raw", table_name="payments",
    transformation_ctx="raw_payments",
)

# Strategy A — coerce everything to long; non-numeric -> NULL
resolved = ResolveChoice.apply(
    frame=dyf,
    specs=[("amount", "cast:long")],
    transformation_ctx="resolve_amount",
)

# Strategy B — split into amount_long and amount_string (inspect both)
# resolved = ResolveChoice.apply(dyf, specs=[("amount", "make_cols")], ...)

# Strategy C — keep both in a struct amount.{long, string}
# resolved = ResolveChoice.apply(dyf, specs=[("amount", "make_struct")], ...)

resolved.printSchema()

# printSchema() after Strategy A (cast:long)
root
|-- payment_id: string
|-- amount: long           # single clean numeric column
|-- dt: string

Step-by-step explanation.

1. The crawler recorded amount as choice<long, string> because it saw both shapes. The DynamicFrame preserves that ambiguity instead of failing — the whole reason to read messy data this way.
2. Strategy A (cast:long) coerces: 1200 → 1200, "12.50" → NULL (not an integer), "N/A" → NULL, 3400 → 3400. It produces one clean long column and quietly nulls anything non-integer — correct if you truly want integer cents and treat everything else as missing.
3. Strategy B (make_cols) produces amount_long and amount_string side by side — useful during investigation to see which records took which member before deciding a policy.
4. Strategy C (make_struct) nests both under amount.long and amount.string, preserving every value losslessly for a later, richer resolution.
5. The choice of strategy is a data-policy decision, not a syntax one: coerce when you trust the target type, split/struct when you need to audit the drift first.

Output.

Strategy	Result column(s)	Record 2 ("12.50")
cast:long	amount: long	NULL
make_cols	amount_long, amount_string	amount_string="12.50"
make_struct	amount.{long,string}	.string="12.50"

Rule of thumb. Resolve choice types explicitly with ResolveChoice. Use cast when you trust the target type and want non-conforming values nulled; use make_cols/make_struct when you must audit the drift before committing to a policy. Never let a silent DataFrame coercion make that decision for you.

Worked example — flatten nested JSON with Relationalize, then join in a DataFrame

Detailed explanation. Clickstream JSON has a nested payload struct and an items array. Relationalize flattens the struct and unnests the array into a child table with a generated join key; then you toDF() to run a window function the DynamicFrame API does not offer. Walk through the full crossing-the-bridge idiom.

• Shape. {event_id, payload:{page, referrer}, items:[{sku, qty}]}.
• Relationalize. Produces a root frame (flattened payload.page, payload.referrer, plus an items join key) and a child frame (one row per array element).
• Then. toDF() the root and rank events per page with a window function.

Question. Flatten the nested JSON and compute, per page, the running count of events ordered by time — using Relationalize then a DataFrame window.

Input.

event_id	payload.page	items	ts
e1	/home	[sku A]	09:00
e2	/home	[sku B, C]	09:05
e3	/cart	[sku A]	09:07

Code.

from awsglue.transforms import Relationalize
from pyspark.sql import functions as F, Window

dyf = glueContext.create_dynamic_frame.from_catalog(
    database="raw", table_name="events_nested",
    transformation_ctx="raw_events_nested",
)

# Relationalize returns a COLLECTION of frames: root + one per unnested array
dfc = Relationalize.apply(
    frame=dyf,
    staging_path="s3://tmp/relationalize/",
    name="root",
    transformation_ctx="relationalize_events",
)
root = dfc.select("root")            # flattened top-level, with an items join key
items = dfc.select("root_items")     # one row per array element (join key back to root)

# Cross the bridge: DataFrame for the window function
root_df = root.toDF()
w = Window.partitionBy("payload.page").orderBy("ts")
ranked = root_df.withColumn("events_so_far", F.count("*").over(w))

ranked.select("event_id", F.col("payload.page").alias("page"),
              "ts", "events_so_far").show()

# items child table (root_items) after Relationalize
+----+------------+--------+-----+
| id | index      | sku    | qty |   # id joins back to root's items key
+----+------------+--------+-----+
|  1 |  0         | A      |  1  |
|  2 |  0         | B      |  1  |
|  2 |  1         | C      |  2  |
+----+------------+--------+-----+

Step-by-step explanation.

1. Relationalize flattens payload.page/payload.referrer into top-level columns and replaces the items array with a foreign key, emitting a separate root_items frame with one row per array element — turning a document into relational tables.
2. staging_path is where Relationalize writes intermediate data; it needs a real S3 path. The output is a DynamicFrameCollection; select(name) pulls out the root and each unnested child.
3. root.toDF() crosses the bridge: window functions (COUNT(*) OVER (PARTITION BY page ORDER BY ts)) exist only in the DataFrame/Spark SQL API, not on DynamicFrames, so the relational work happens here.
4. The window computes the running event count per page — a textbook DataFrame operation that would be awkward or impossible in DynamicFrame land.
5. To write back with partitioning and bookmarks, you would DynamicFrame.fromDF(ranked, glueContext, "out") and write_dynamic_frame — completing the read-DynamicFrame → clean → toDF → work → fromDF → write idiom.

Output.

event_id	page	ts	events_so_far
e1	/home	09:00	1
e2	/home	09:05	2
e3	/cart	09:07	1

Rule of thumb. Use Relationalize to turn nested JSON into a root frame plus one child frame per array, then toDF() for anything relational — joins, windows, aggregations. Cross back with fromDF() only when you need Glue-native partitioned writes or bookmarks on the output.

Senior interview question on DynamicFrames vs DataFrames

A senior interviewer might ask: "You ingest a semi-structured JSON feed whose schema drifts weekly, then join it to a stable dimension table and compute per-user session windows before writing partitioned Parquet. Which parts use DynamicFrames, which use DataFrames, and why — and where would a naive all-DataFrame or all-DynamicFrame approach fail?"

Solution Using DynamicFrame ingestion, DataFrame relational work, and fromDF write-back

import sys
from awsglue.transforms import ResolveChoice, ApplyMapping, Relationalize
from awsglue.dynamicframe import DynamicFrame
from awsglue.utils import getResolvedOptions
from awsglue.context import GlueContext
from awsglue.job import Job
from pyspark.context import SparkContext
from pyspark.sql import functions as F, Window

args = getResolvedOptions(sys.argv, ["JOB_NAME"])
glueContext = GlueContext(SparkContext())
job = Job(glueContext)
job.init(args["JOB_NAME"], args)

# 1. INGEST drifting JSON as a DynamicFrame (survives schema drift)
raw = glueContext.create_dynamic_frame.from_catalog(
    database="raw", table_name="events",
    transformation_ctx="raw_events",
)

# 2. RESOLVE drift while still a DynamicFrame
clean = ResolveChoice.apply(raw, specs=[("amount", "cast:long")],
                            transformation_ctx="resolve_amount")
clean = ApplyMapping.apply(
    frame=clean,
    mappings=[
        ("user_id", "string", "user_id", "string"),
        ("amount",  "long",   "amount",  "long"),
        ("ts",      "string", "ts",      "timestamp"),
        ("dt",      "string", "dt",      "string"),
    ],
    transformation_ctx="map_events",
)

# 3. CROSS THE BRIDGE for relational work
events_df = clean.toDF()
dim_df = glueContext.create_dynamic_frame.from_catalog(
    database="dim", table_name="users", transformation_ctx="raw_users",
).toDF()

joined = events_df.join(F.broadcast(dim_df), on="user_id", how="left")
w = Window.partitionBy("user_id").orderBy("ts")
sessioned = joined.withColumn(
    "gap_min",
    (F.unix_timestamp("ts") - F.unix_timestamp(F.lag("ts").over(w))) / 60,
).withColumn("new_session", (F.col("gap_min") > 30).cast("int"))

# 4. CROSS BACK for a Glue-native partitioned, bookmarked write
out = DynamicFrame.fromDF(sessioned, glueContext, "out")
glueContext.write_dynamic_frame.from_options(
    frame=out,
    connection_type="s3",
    connection_options={"path": "s3://curated/sessions/", "partitionKeys": ["dt"]},
    format="parquet",
    transformation_ctx="write_sessions",
)

job.commit()

Step-by-step trace.

Stage	Frame type	Why this type
Read drifting JSON	DynamicFrame	survives weekly schema drift; choice types
Resolve amount	DynamicFrame	ResolveChoice only exists here
Project/retype	DynamicFrame	ApplyMapping normalises shape
Join + session window	DataFrame	joins, lag, windows, broadcast hints
Partitioned write	DynamicFrame (fromDF)	partitionKeys + bookmark on write

The ingestion and cleanup stay in DynamicFrame land because that is the only layer that survives the weekly drift and can resolve choice types. The join to the stable users dimension and the per-user session-gap window move to a DataFrame, where broadcast joins, lag, and window functions live and the Catalyst optimiser can push predicates into Parquet. The final write crosses back via fromDF so partitionKeys and the write-side bookmark still apply.

Output:

Approach	Failure mode
All-DataFrame	read fails / mis-coerces on the weekly JSON drift
All-DynamicFrame	no window functions; awkward joins; no Catalyst pushdown
Hybrid (this)	drift-safe ingest + optimised relational work + native write

Why this works — concept by concept:

• DynamicFrame ingest — reading the drifting feed as a DynamicFrame means a new or retyped field never fails the read; the schema flexibility absorbs the weekly change that would crash an all-DataFrame job.
• ResolveChoice before the bridge — resolving amount while still a DynamicFrame turns ambiguous choice types into a concrete long before toDF(), so the DataFrame conversion is not left guessing a representation.
• DataFrame for joins and windows — the broadcast join and the lag-based session window use Catalyst-optimised operators that DynamicFrames do not expose; this is where relational work belongs.
• fromDF for the write — converting back to a DynamicFrame for write_dynamic_frame preserves partitionKeys layout and lets the write-side bookmark advance, which a raw DataFrame write would not integrate with.
• Cost — the hybrid pays one toDF()/fromDF() materialisation but gains Catalyst pushdown and broadcast joins on the expensive relational step; net runtime is far below an all-DynamicFrame pipeline, and it does not crash on drift like an all-DataFrame one.

Data transformation
Topic — data-transformation
Data transformation problems on nested and messy data

Practice →

ETL Topic — etl ETL problems on schema-flexible ingestion

Practice →

---

5. Spark tuning, partitioning, and predicate pushdown

The three tuning levers are read less, size right, and write fewer files — and partition pruning dwarfs the other two

The one-sentence invariant: tuning a Glue Spark job is dominated by three levers applied in order — prune the read so you scan the fewest bytes (predicate pushdown on partition columns), size the cluster to the pruned work (worker type and count, autoscaling), and shape the output so you neither explode into tiny files nor skew into a few giant ones (groupFiles/groupSize on read, partitionKeys and repartitioning on write) — and of the three, pruning the read is worth more than the other two combined because bytes scanned drives both runtime and cost. A perfectly sized cluster on an unpruned read is still slow and expensive; a tiny cluster on a well-pruned read is fast and cheap.

Lever 1 — read less (predicate pushdown and partition pruning).

• push_down_predicate. A string filter on partition columns passed to create_dynamic_frame.from_catalog; Glue lists only matching partitions from S3, so unmatched partitions are never read. This is the biggest lever.
• catalogPartitionPredicate. Pushes the partition filter to the Glue Data Catalog server-side, so even the partition listing is filtered before any S3 access — essential when a table has hundreds of thousands of partitions.
• Column pruning + Parquet/ORC pushdown. Once in a DataFrame over columnar files, Catalyst pushes column projections and row-group filters into the file reader, skipping columns and row groups that cannot match.
• The failure mode. No partition predicate → Glue lists and reads every partition → full-prefix scan → the 200× cost blow-up from section 1.

Lever 2 — size right (workers, DPU, autoscaling).

• Worker types. G.1X (1 DPU, 4 vCPU, 16 GB) for light/IO-bound work; G.2X (2 DPU, 8 vCPU, 32 GB) for memory-heavy joins/shuffles; G.4X/G.8X for very heavy work.
• Count + autoscaling. --number-of-workers sets the ceiling; Glue autoscaling (Glue 3.0+) scales down when the DAG needs fewer executors, so you pay for what a stage actually uses.
• Right-sizing signal. Look at Spark stage metrics: if executors sit idle, you over-provisioned; if you spill to disk, you under-provisioned memory (go G.2X).
• The trap. Adding workers to fix a slow unpruned read — it multiplies DPU-hours without fixing the byte-scan problem.

Lever 3 — write fewer/right-sized files (small-file and skew control).

• groupFiles / groupSize. On read, {"groupFiles": "inPartition", "groupSize": "134217728"} coalesces many small S3 objects into ~128 MB input splits, so you spin up far fewer tasks — the fix for a small-file explosion on the read side.
• Output file count. write produces one file per output partition per Spark partition; repartition/coalesce before write controls how many files land per dt= folder. Aim for ~128–256 MB files.
• Skew. A single hot key (one dt, one user_id) makes one task run 10× longer than the rest; salt the key or repartition to spread it.

Common interview probes.

• "Your job scans all of S3 — fix it." — push_down_predicate (and catalogPartitionPredicate if partition listing is the bottleneck).
• "You have 200k tiny output files — why and fix?" — too many Spark partitions on write; coalesce/repartition; on read use groupFiles.
• "G.1X vs G.2X?" — IO-bound vs memory/shuffle-bound; watch for disk spill.
• "One task runs forever." — data skew on a hot partition key; salt or repartition.

Worked example — prune partitions with a pushdown predicate

Detailed explanation. The clickstream job only needs yesterday and today, but the catalog table has three years of dt= partitions. Without a predicate, Glue reads all of them. Walk through adding push_down_predicate and catalogPartitionPredicate and measuring the reduction.

• Table. raw.clickstream, ~1,095 daily partitions.
• Need. dt in the last 2 days.
• Fix. push_down_predicate on dt; for very wide tables, catalogPartitionPredicate too.

Question. Add partition pruning to the read and show partitions listed/read before and after.

Input.

Read config	Partitions listed	Partitions read	Bytes scanned
no predicate	1,095	1,095	~1.8 TB
push_down_predicate	1,095 (listed)	2	~5 GB
+ catalogPartitionPredicate	2 (listed)	2	~5 GB

Code.

# Partition pruning: filter on the dt partition column
dyf = glueContext.create_dynamic_frame.from_catalog(
    database="raw",
    table_name="clickstream",
    # Reads only matching partitions from S3 (skips the rest entirely)
    push_down_predicate="dt >= date_format(date_sub(current_date, 1), 'yyyy-MM-dd')",
    # Also filter the partition LISTING server-side in the catalog
    additional_options={
        "catalogPartitionPredicate": "dt >= (cast(current_date as date) - interval '1' day)"
    },
    transformation_ctx="raw_clickstream",
)

print("spark input partitions:", dyf.toDF().rdd.getNumPartitions())

# CloudWatch / Spark UI evidence
Before:  Opening 1,095 partitions, 1.8 TB, stage 1 = 90 min
After :  Opening 2 partitions,     5.1 GB, stage 1 = 2 min

Step-by-step explanation.

1. push_down_predicate is evaluated against the partition columns only (dt), so Glue reads objects solely from the two matching dt= folders and never opens the other 1,093 — that is the byte-scan collapse from 1.8 TB to ~5 GB.
2. catalogPartitionPredicate goes one step earlier: it filters the partition listing in the Glue Data Catalog server-side, so Glue does not even enumerate 1,095 partitions before pruning. On tables with hundreds of thousands of partitions, the listing itself is the bottleneck, and this option removes it.
3. The two predicates are complementary: catalogPartitionPredicate limits what gets listed; push_down_predicate limits what gets read. Using both is the belt-and-braces pattern for very wide tables.
4. The predicate must reference partition columns; putting a data column in push_down_predicate does nothing (data-column filters happen after the read, via DataFrame/SQL and Parquet row-group pushdown).
5. The measured effect — 90 min → 2 min, 1.8 TB → 5 GB — is the same 200×-class win from the cost model, achieved without touching worker counts.

Output.

Metric	No predicate	With pushdown
Partitions read	1,095	2
Bytes scanned	~1.8 TB	~5 GB
Stage 1 runtime	~90 min	~2 min
DPU-hours	~30	~0.13

Rule of thumb. Always pass a push_down_predicate on partition columns; add catalogPartitionPredicate when the table has tens of thousands of partitions so the listing is filtered server-side. Prune the read before you ever consider adding workers.

Worked example — coalesce small files and fix skew on write

Detailed explanation. After pruning, the job still runs slowly and emits 40,000 tiny Parquet files because the source is millions of ~10 KB objects and one dt is hot. Fix the read with groupFiles, and fix the write with a repartition on the partition key. Walk through both.

• Read problem. Millions of ~10 KB source files → millions of tiny tasks → scheduler overhead dominates.
• Write problem. Default write emits one file per Spark partition per dt → 40,000 files; one hot dt skews a single task.
• Fixes. groupFiles/groupSize on read; repartition("dt") (or salted) before write.

Question. Configure the read to coalesce small files and the write to emit ~128 MB files without a skewed straggler.

Input.

Symptom	Cause	Fix
millions of tiny tasks	~10 KB source files	groupFiles=inPartition, groupSize=128MB
40,000 output files	one file per Spark partition	repartition before write
one task runs 10× longer	hot dt key skew	salt the partition key

Code.

from pyspark.sql import functions as F

# READ — coalesce many small S3 objects into ~128 MB input splits
dyf = glueContext.create_dynamic_frame.from_catalog(
    database="raw", table_name="clickstream",
    push_down_predicate="dt >= '2026-08-17'",
    additional_options={"groupFiles": "inPartition", "groupSize": "134217728"},  # 128 MB
    transformation_ctx="raw_clickstream",
)

df = dyf.toDF()

# WRITE — control file count + spread a hot dt with a salt
salted = df.withColumn("_salt", (F.rand() * 8).cast("int"))
(salted
    .repartition("dt", "_salt")          # ~8 files per dt; hot dt no longer one task
    .drop("_salt")
    .write.mode("append")
    .partitionBy("dt")
    .parquet("s3://curated/clickstream/"))

# Before vs after
Read  : 3,000,000 tasks (10 KB each)  ->  14,000 tasks (128 MB splits)
Write : 40,000 files (~2 MB)          ->  ~8 files per dt (~180 MB each)
Skew  : dt=2026-08-17 task = 22 min   ->  spread across 8 tasks = ~3 min

Step-by-step explanation.

1. groupFiles=inPartition + groupSize=134217728 tells Glue to group many small S3 objects into ~128 MB input splits before creating Spark tasks, collapsing three million micro-tasks into ~14,000 right-sized ones and removing the scheduler-overhead bottleneck.
2. On the write, the default one-file-per-Spark-partition behaviour produced 40,000 tiny files; those files then make every downstream read slow (the small-file problem propagates). repartition controls how many files land per dt= folder.
3. The _salt column (0–7) added before repartition("dt", "_salt") spreads a single hot dt across 8 tasks instead of one, so the straggler that took 22 minutes now finishes in ~3. The salt is dropped before write so it does not pollute the output schema.
4. partitionBy("dt") on the write keeps the dt= folder layout intact, so partition pruning still works on the next job that reads this curated table.
5. The combined effect: fewer, larger files (good for downstream reads), no skewed straggler (good for this job's runtime), and pruning preserved (good for everyone downstream).

Output.

Metric	Before	After
Read tasks	~3,000,000	~14,000
Output files	~40,000 (~2 MB)	~8 per dt (~180 MB)
Hot-dt task time	~22 min	~3 min
Downstream read speed	slow (tiny files)	fast (right-sized)

Rule of thumb. Use groupFiles/groupSize to coalesce tiny input files, repartition before write to control output file count (target ~128–256 MB), and salt a hot partition key to kill stragglers. Small files are a tax you pay twice — once when you write them and again on every read that follows.

Senior interview question on Spark tuning in Glue

A senior interviewer might ask: "A Glue job over a 3-year, 500-partition-per-day table takes 3 hours and costs a fortune, emits hundreds of thousands of tiny files, and has one stage where a single task runs 10× longer than the rest. You have G.1X workers. Walk me through the tuning order — what you fix first, second, third — and justify why worker sizing is not first."

Solution Using pushdown-first tuning, right-sized workers, and coalesced skew-free writes

import sys
from awsglue.utils import getResolvedOptions
from awsglue.context import GlueContext
from awsglue.job import Job
from pyspark.context import SparkContext
from pyspark.sql import functions as F

args = getResolvedOptions(sys.argv, ["JOB_NAME"])
glueContext = GlueContext(SparkContext())
job = Job(glueContext)
job.init(args["JOB_NAME"], args)

# LEVER 1 (first) — read less: prune partitions server-side + on read
events = glueContext.create_dynamic_frame.from_catalog(
    database="raw", table_name="events",
    push_down_predicate="dt >= '2026-08-17'",
    additional_options={
        "catalogPartitionPredicate": "dt >= (cast(current_date as date) - interval '1' day)",
        "groupFiles": "inPartition",
        "groupSize": "134217728",
    },
    transformation_ctx="raw_events",
).toDF()

# LEVER 3 (third) — write shape: kill skew with a salt, size files with repartition
salted = events.withColumn("_salt", (F.rand() * 8).cast("int"))
(salted.repartition("dt", "_salt").drop("_salt")
    .write.mode("append").partitionBy("dt")
    .parquet("s3://curated/events/"))

job.commit()

# LEVER 2 (second) — right-size workers, AFTER pruning shrinks the work
# Run config:
--worker-type          G.2X          # was G.1X; join stage was spilling to disk
--number-of-workers    10            # autoscaling scales down when idle
--enable-auto-scaling  true

Step-by-step trace.

Order	Lever	Action	Effect
1st	read less	push_down_predicate + catalogPartitionPredicate	3 yrs → 2 days; 1.8 TB → 5 GB
1st	read less	groupFiles/groupSize	3M tiny tasks → 14k right-sized
2nd	size right	G.1X → G.2X + autoscaling	stop disk spill on the join stage
3rd	write shape	salt + repartition before write	kill the 10× straggler; ~180 MB files
3rd	write shape	partitionBy("dt")	preserve pruning for downstream

The order is deliberate. Pruning first shrinks the bytes scanned by ~200×, which alone takes the job from 3 hours toward minutes and makes the "we need more workers" instinct moot. Only after the read is pruned do worker-sizing decisions make sense — and the signal to move G.1X → G.2X is disk spill on the join stage, not raw slowness. The write fixes (salt + repartition + partitionBy) come last: they remove the skewed straggler and the tiny-file tax without which the downstream reads would inherit the problem.

Output:

Metric	Before	After
Bytes scanned	~1.8 TB	~5 GB
Runtime	~3 h	~6 min
Output files	~300,000	~8 per dt
Straggler task	~22 min	~3 min
Worker sizing	guessed	G.2X on spill evidence

Why this works — concept by concept:

• Pushdown before workers — pruning the read collapses bytes scanned ~200×, and since bytes scanned drives both runtime and the "add workers" instinct, fixing it first makes most of the perceived compute problem disappear before any sizing decision.
• Catalog partition predicate — on a 500-partition-per-day table (~500k partitions over 3 years), filtering the partition listing server-side stops Glue from enumerating half a million partitions just to discard them; without it the listing itself is a bottleneck.
• Group small files on read — groupFiles/groupSize turns millions of micro-objects into ~128 MB splits, replacing scheduler-dominated micro-tasks with right-sized ones — a runtime fix independent of cluster size.
• Salt then repartition — salting a hot dt spreads a single 22-minute straggler across eight parallel tasks, and repartition sizes output files to ~180 MB so downstream reads are not taxed by tiny files.
• Cost — DPU-hours drop from ~30 to well under 1 because runtime and bytes scanned both collapse; the G.1X → G.2X change is a targeted response to disk spill, not a blanket "throw hardware at it." Read-less is O(new partitions); the whole job is now bounded by the pruned working set, not the table size.

Optimization
Topic — optimization
Optimization problems on Spark and scan pruning

Practice →

Data processing
Topic — data-processing
Data processing problems on partitioning and skew

Practice →

---

Cheat sheet — AWS Glue recipes

• The four-component map. Crawler owns schema + partition keys (lives in the Data Catalog); the Glue job owns the Spark transform; the job bookmark owns what's already processed (lives in the Glue service, per job name + transformation_ctx); the DynamicFrame owns schema-drift tolerance (in-memory). Every incident is one of those four cells left blank. Cost = DPU × runtime × bytes scanned — and bytes scanned dominates.
• Crawler config recipe. One S3 target at the prefix root; SchemaChangePolicy = UpdateBehavior: LOG + DeleteBehavior: LOG for production (log drift, never silently retype); RecrawlBehavior: CRAWL_NEW_FOLDERS_ONLY for cheap incremental crawls; TableGroupingPolicy: CombineCompatibleSchemas to stop table explosion; partition AddOrUpdateBehavior: InheritFromTable for uniform partition schemas. For stable schemas, skip partition crawling entirely and use Athena partition projection.
• Bookmark boilerplate. job.init(args["JOB_NAME"], args) at the top; a unique, frozen transformation_ctx on every read and write; exactly one job.commit() on the success path; run with --job-bookmark-option job-bookmark-enable. Reprocessing everything = missing ctx/commit or bookmark disabled. Skipping new data = bookmark paused or non-increasing JDBC key. Reset with aws glue reset-job-bookmark --job-name X.
• JDBC bookmark keys. jobBookmarkKeys=["id"] with jobBookmarkKeysSortOrder="asc" on a strictly-increasing, unique surrogate key (BIGSERIAL/identity) — never a wall-clock updated_at (backfills and clock skew fall below the watermark and get skipped). Update tracking is a CDC problem, not a bookmark problem.
• ResolveChoice / Relationalize snippets. ResolveChoice.apply(frame, specs=[("col", "cast:long")]) to coerce (non-conforming → NULL); make_cols to split into col_int/col_string; make_struct to keep both losslessly. Relationalize.apply(frame, staging_path="s3://tmp/...", name="root") flattens nested structs and unnests arrays into child frames with generated join keys. ApplyMapping renames/retypes/projects — unlisted columns are dropped, so diff the schema.
• Cross-the-bridge idiom. Read messy data as a DynamicFrame → ResolveChoice/ApplyMapping/Relationalize to clean → dyf.toDF() for joins/windows/Spark SQL → DynamicFrame.fromDF(df, glueContext, "out") → write_dynamic_frame so partitionKeys and write-side bookmarks still apply. DynamicFrame for drift; DataFrame for relational work.
• Predicate pushdown templates. create_dynamic_frame.from_catalog(..., push_down_predicate="dt >= '2026-08-17'") reads only matching partitions; add additional_options={"catalogPartitionPredicate": "..."} to filter the partition listing server-side on wide tables. Predicates must reference partition columns; data-column filters happen post-read via Catalyst + Parquet row-group pushdown.
• Small-file recipe. On read: additional_options={"groupFiles": "inPartition", "groupSize": "134217728"} (128 MB) to coalesce tiny objects into right-sized splits. On write: repartition("dt") (or salted) before write.partitionBy("dt") to target ~128–256 MB output files. Small files are taxed twice — once on write, again on every downstream read.
• Worker sizing / DPU cost formula. DPU-hours = workers × DPU-per-worker × (runtime_min / 60). G.1X = 1 DPU (IO-bound); G.2X = 2 DPU (memory/shuffle-bound — move here on disk spill). Enable autoscaling so idle executors scale down. Never add workers to fix a slow unpruned read — prune first.
• Tuning order (memorise). 1) Read less — push_down_predicate + catalogPartitionPredicate + groupFiles. 2) Size right — G.1X→G.2X on spill evidence, autoscaling on. 3) Write shape — salt hot keys, repartition for file count, partitionBy to preserve downstream pruning. Pruning is worth more than the other two combined.
• Component decision matrix. Need Athena-visible schema over raw S3 → crawler (or partition projection). Need incremental reads → bookmarks. Need to survive schema drift / choice types → DynamicFrame. Need joins/windows/optimiser → DataFrame. Need cheap, fast, correct → prune the read, then everything else.
• Failure-signature quick map. "Column became string" = crawler schema merge. "Reprocesses everything" = missing transformation_ctx/commit or disabled bookmark. "Skipped recent data" = paused bookmark or bad JDBC key. "Scans all of S3" = no partition predicate. "200k tiny files" = too many Spark partitions on write. "One task runs forever" = hot-key skew.

Frequently asked questions

What is AWS Glue in one sentence?

AWS Glue is a serverless data-integration service that runs managed Apache Spark ETL jobs against the AWS ecosystem, wrapping Spark in four cooperating abstractions: a crawler that infers schema into the Glue Data Catalog, a Glue job that runs Spark against that catalog, a job bookmark that persists which data has already been processed so runs are incremental, and a DynamicFrame that tolerates schema drift with choice types. You pay per Data Processing Unit-hour rather than managing a cluster, which makes Glue the AWS-native default for spiky, discovery-heavy batch ETL over S3, Redshift, and JDBC sources. Almost every Glue incident traces back to one of those four abstractions rather than to Spark itself.

What is a Glue crawler and do I always need one?

A Glue crawler is a metadata process that walks an S3 prefix (or JDBC source), runs classifiers to detect the format, infers a column schema by merging the shapes it sees across sampled files, detects key=value folder patterns as partition keys, and registers or updates a table in the Data Catalog — it moves no data, only metadata. You do not always need one: you can define catalog tables by DDL, let a Glue job write partitioned output and register partitions itself, or use Athena partition projection so new partitions are query-visible without any crawl. Crawlers are the fastest way to get an Athena-queryable table over unknown raw data, but for a stable, known schema they are often replaced by partition projection to avoid slow re-crawls and schema-merge surprises. The classic crawler footguns — a numeric column silently becoming string, or one table splitting into many — come from the schema-merge and grouping behaviour, which you control with the schema-change and table-grouping policies.

How do job bookmarks track what's already been processed?

A job bookmark is state Glue persists per named job, keyed by the job name plus each read's transformation_ctx. For S3 sources it records the set of processed objects by path and last-modified time, so the next run reads only new or newer files; for JDBC sources it records the maximum value of the jobBookmarkKeys column(s) and reads WHERE key > last_seen. The bookmark only advances if you call job.init() at the start, thread a stable transformation_ctx through every read and write, and call job.commit() at the end — omit any of those and the job either reprocesses everything or discards its progress. Common failures are reprocessing all history (missing context or commit, or the bookmark disabled) and skipping recent data (bookmark paused, or a JDBC key that is not strictly increasing such as a wall-clock updated_at).

DynamicFrame vs DataFrame — when do I use each?

Use a DynamicFrame for ingestion and cleanup: it carries choice types (a column that is int in some records and string in others), never fails a read on schema drift, and exposes Glue-native transforms — ResolveChoice, Relationalize, ApplyMapping, DropNullFields — designed for messy, semi-structured, drifting data. Use a Spark DataFrame for relational work: joins, window functions, groupBy aggregations, UDFs, and anything that benefits from the Catalyst optimiser and columnar predicate pushdown. The idiomatic Glue job reads messy data as a DynamicFrame, resolves choice types and flattens nesting, calls toDF() to cross into DataFrame land for the heavy relational work, then DynamicFrame.fromDF(...) back so write_dynamic_frame keeps partitionKeys and write-side bookmarks. An all-DataFrame job fails or mis-coerces on drift; an all-DynamicFrame job has no window functions and misses Catalyst optimisation — the hybrid gets both.

How do I stop a Glue job from scanning all of S3?

Pass a push_down_predicate on the table's partition columns to create_dynamic_frame.from_catalog, e.g. push_down_predicate="dt >= '2026-08-17'" — Glue then lists and reads only the matching partitions, turning a full-prefix scan into a small pruned read (routinely a 100–200× reduction in bytes scanned). For tables with tens or hundreds of thousands of partitions, also pass catalogPartitionPredicate in additional_options so the partition listing is filtered server-side in the Data Catalog before any S3 access. Predicates must reference partition columns; filters on data columns happen after the read via Spark SQL and Parquet/ORC row-group pushdown. Because bytes scanned drives both runtime and the per-DPU-hour bill, partition pruning is the single biggest tuning lever — always prune the read before adding workers.

How is AWS Glue priced, and how do I control DPU cost?

Glue bills per Data Processing Unit-hour: DPU-hours = number of workers × DPU-per-worker × runtime-in-hours (billed per second with a one-minute minimum), where G.1X = 1 DPU and G.2X = 2 DPU. The dominant, often-overlooked cost driver is bytes scanned, because it inflates runtime and tempts you to add workers — two jobs with identical logic can differ 20–200× in cost purely on whether the read pruned partitions. Control the bill in this order: prune the read with push_down_predicate/catalogPartitionPredicate, coalesce tiny input files with groupFiles/groupSize, right-size workers (G.2X only when you see disk spill) with autoscaling on, and avoid emitting tiny output files that tax every downstream read. Adding workers to a slow unpruned read multiplies DPU-hours without fixing the actual problem.

Practice on PipeCode

• Drill the ETL practice library → for the incremental-load, bookmark, schema-inference, and serverless-pipeline problems Glue interviews lean on.
• Rehearse on the data transformation practice library → for the nested-JSON flattening, choice-type resolution, and mapping patterns DynamicFrames formalise.
• Sharpen the tuning axis with the optimization practice library → for partition pruning, predicate pushdown, small-file, and skew problems.
• Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the four-component Glue model against real graded inputs.

Lock in AWS Glue muscle memory

Docs explain the API. PipeCode drills explain the decision — when a crawler silently retypes a column, when a missing `transformation_ctx` reprocesses a year of data, when a DynamicFrame should become a DataFrame, and when a `push_down_predicate` turns a 3-hour job into a 6-minute one. Pipecode.ai is Leetcode for Data Engineering — pattern-first practice tuned for the production trade-offs data engineers actually face.

Practice ETL problems →
Practice optimization problems →