AWS Glue Job Bookmarks Explained: Build Incremental ETL Pipelines with PySpark

Imagine building an AWS Glue job that processes sales files from Amazon S3 every morning.

On Monday, the job processes 10 files. On Tuesday, five new files arrive.

You want the next job run to process only those five new files rather than reading all 15 again.

But how does AWS Glue know which files it has already processed?

That's where AWS Glue Job Bookmarks come in.

In this guide, we'll explore how AWS Glue Job Bookmarks work, how to enable them, and how to build a simple incremental ETL pipeline using PySpark.

What Are AWS Glue Job Bookmarks?

AWS Glue Job Bookmarks help supported ETL jobs remember which source data has already been processed.

When a job runs successfully and saves its bookmark state, subsequent runs can identify new or modified source data rather than repeatedly processing everything.

For supported Amazon S3 sources, AWS Glue uses object modification timestamps to determine which files need processing.

For example:

• Monday: Your job processes 10 CSV files.
• Tuesday: Five new CSV files arrive.
• Wednesday: Another three files are uploaded.

With correctly configured job bookmarks, each run can focus on files that are new or have been modified since the previous successful processing checkpoint.

This is called incremental data processing.

You can learn more in the official AWS Glue Job Bookmarks documentation.

Why Is Incremental ETL Important?

Imagine a company collecting thousands of transaction records daily.

If its data pipeline processes every historical file each time it runs, the workload can become increasingly expensive and time-consuming.

Incremental processing helps avoid unnecessary repeated work.

It is useful for:

• Daily sales data processing.
• Scheduled data lake updates.
• Customer transaction pipelines.
• Recurring reporting workloads.
• File-based data ingestion.
• Regular data warehouse updates.

However, bookmarks do not automatically prevent duplicate records in the final output. Your pipeline still needs suitable validation and output-handling logic.

Step 1: Prepare Your Data in Amazon S3

Start by creating an Amazon S3 bucket for your source data.

For example:

s3://your-input-bucket/sales/

Upload a few CSV files containing sample sales records.

A simple input file could contain:

order_id,customer_id,product,amount
1001,C101,Laptop,55000
1002,C102,Keyboard,1500
1003,C103,Mouse,800

Next, create an AWS Glue Crawler to discover the data structure and register a table in AWS Glue Data Catalog.

For this tutorial, we'll use the following example names:

Database: sales_db

Table: orders_raw

Replace these names with your actual AWS resources when testing.

Step 2: Enable Job Bookmarks in AWS Glue

Open the AWS Glue console and create or edit your Spark ETL job.

Find the Job bookmark setting in the job configuration.

Select Enable.

You can also configure the job using this argument:

--job-bookmark-option=job-bookmark-enable

AWS Glue supports three bookmark modes:

Enable: Tracks processed data and updates the bookmark state.

Disable: Does not use bookmarks, so the job processes the data according to its regular source-reading logic.

Pause: Uses existing bookmark information without updating the bookmark state.

For the latest configuration options, refer to the AWS Glue Spark Job Configuration documentation.

Step 3: Write a PySpark ETL Script

Now let's create a simple AWS Glue job that reads source data from the Glue Data Catalog and writes the output to Amazon S3 in Parquet format.

import sys

from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from awsglue.utils import getResolvedOptions

# Retrieve job parameters
args = getResolvedOptions(
    sys.argv,
    ["JOB_NAME"]
)

# Initialize Spark and Glue
sc = SparkContext.getOrCreate()
glue_context = GlueContext(sc)

# Initialize job bookmark state
job = Job(glue_context)
job.init(args["JOB_NAME"], args)

# Read data from Glue Data Catalog
source = glue_context.create_dynamic_frame.from_catalog(
    database="sales_db",
    table_name="orders_raw",
    transformation_ctx="orders_source"
)

# Write output to Amazon S3
glue_context.write_dynamic_frame.from_options(
    frame=source,
    connection_type="s3",
    connection_options={
        "path": "s3://your-output-bucket/sales/"
    },
    format="parquet",
    transformation_ctx="orders_sink"
)

# Save bookmark state
job.commit()

This is a minimal example that copies records from a supported catalog source into an S3 output location.

A production job would normally include transformations, validation checks, failure handling, and appropriate access controls.

Before running it, make sure your source table points to a supported input format and your Glue execution role has the required permissions.

Step 4: Understand the Important Parts of the Script

job.init()

The job.init() method initializes the AWS Glue job and retrieves its existing bookmark state.

This helps AWS Glue understand the previous processing checkpoint.

transformation_ctx

The transformation_ctx parameter gives the source a stable identifier.

AWS Glue uses this identifier to associate bookmark information with the relevant data source.

For example:

transformation_ctx="orders_source"

Avoid changing this identifier unnecessarily between job runs.

job.commit()

The job.commit() method saves the updated bookmark state.

It is important to call this method after the job finishes its intended processing steps successfully.

AWS documents the importance of both initialization and commit operations in its AWS Glue Job Bookmark Programming Guide.

Step 5: Test Your Incremental Data Pipeline

Once the job is configured, you can test whether incremental processing works as expected.

First Run

Upload three CSV files into your S3 source location.

Run the AWS Glue job.

The job should read the eligible source files and write the processed data to the output bucket.

Review the output to make sure the records were handled correctly.

Second Run

Upload another CSV file containing new records.

Run the same job again.

With bookmarks correctly configured, AWS Glue should process the new eligible input instead of rereading every unchanged source file.

Third Run

Run the job without uploading new data.

Check the logs and output.

For an unchanged supported S3 source, the job should not need to reprocess files already tracked by the bookmark.

This test helps confirm that your incremental processing configuration works as intended.

Common AWS Glue Job Bookmark Mistakes

Even a simple incremental pipeline can behave unexpectedly if its bookmark configuration is incorrect.

Here are some problems worth checking.

1. Job Bookmarks Are Disabled

Your script may contain bookmark-related code, but the job itself might still have bookmarks disabled.

Always check the job configuration.

2. Missing job.commit()

If the script does not commit the state, AWS Glue may not save the latest processing checkpoint.

This can cause unexpected reprocessing.

3. Changing the Transformation Context

AWS Glue uses transformation_ctx to identify bookmarked sources.

Changing the identifier or changing the underlying source while reusing old state can interfere with incremental processing.

4. Reading Data Through Unsupported Methods

Job bookmark tracking requires supported AWS Glue source-reading methods.

Reading a source directly using regular Spark DataFrame APIs does not automatically provide AWS Glue bookmark tracking.

5. Assuming Bookmarks Remove Duplicate Output

Bookmarks track input processing state, but they do not automatically clean previously written destination data.

If a job is retried or its bookmark state is reset, duplicate output can still occur.

A reliable pipeline should account for this possibility.

Can You Reset AWS Glue Job Bookmarks?

Yes.

AWS Glue allows you to reset or rewind bookmark state when reprocessing data is necessary.

Suppose your team discovers that an earlier transformation used the wrong date format.

You may need to process the original input again after correcting the transformation.

Resetting the bookmark allows the job to reconsider previously processed source data.

However, resetting a bookmark does not automatically delete the output created during earlier runs.

Before reprocessing, decide whether the previous output should be replaced, cleaned, or written to another destination.

For more information, see Tracking Processed Data Using AWS Glue Job Bookmarks.

When Should You Use AWS Glue Job Bookmarks?

Job bookmarks are useful when a recurring ETL job processes new data from supported sources.

For example, they can help when:

• New files arrive in S3 every day.
• A scheduled Glue job processes transaction files.
• A data lake receives regular updates.
• A recurring ETL job needs to avoid rereading unchanged inputs.

But they are not appropriate for every situation.

If your workflow requires full historical recalculation, complex change-data-capture logic, or record-level duplicate handling, you may need additional processing logic.

The important thing is to understand what bookmarks track and where their responsibilities end.

How to Build Stronger AWS Data Engineering Skills

AWS Glue Job Bookmarks are only one part of building reliable cloud data pipelines.

Data engineers also need to understand storage design, ingestion patterns, transformations, workflow orchestration, monitoring, security, and cost optimization.

For example, an AWS data engineering project might connect Amazon S3, AWS Glue, Amazon Athena, Amazon Redshift, and CloudWatch.

Learning how these services work together can help you understand both practical engineering tasks and the technical scenarios covered in the DEA-C01 certification exam.

For developers interested in structured training and additional hands-on exercises, Certification Mantra provides an AWS Data Engineer Certification Course in Hyderabad covering AWS data engineering concepts, ETL pipelines, practical projects, and preparation for AWS Certified Data Engineer – Associate.

Disclosure: This article is shared by Certification Mantra, the training provider referenced above.

Frequently Asked Questions

1. What is the purpose of AWS Glue Job Bookmarks?

AWS Glue Job Bookmarks help supported ETL jobs track previously processed source data so subsequent runs can focus on new or modified input.

They are useful for recurring incremental data pipelines.

2. Are AWS Glue Job Bookmarks enabled by default?

No. AWS Glue job bookmarks are disabled by default.

You need to enable the feature through the job configuration or appropriate job parameters.

3. Do AWS Glue Job Bookmarks work with Amazon S3?

Yes. Job bookmarks support certain Amazon S3 source formats.

For supported S3 sources, AWS Glue uses object modification timestamps to determine which files require processing.

4. Can AWS Glue Job Bookmarks prevent duplicate records?

Not completely.

Bookmarks can reduce unnecessary source reprocessing, but they do not automatically remove duplicate data or manage previous output files.

5. Can I reset an AWS Glue Job Bookmark?

Yes. AWS Glue allows bookmark state to be reset or rewound for supported ETL jobs.

Take care when reprocessing data because existing output is not automatically removed.

Final Thoughts

AWS Glue Job Bookmarks make incremental ETL processing easier to manage.

Instead of processing the same unchanged source files repeatedly, a properly configured job can track earlier work and process newly available data.

To use bookmarks correctly, remember these key points:

• Enable job bookmarks in your AWS Glue job.
• Read data through a supported Glue source.
• Use a stable transformation_ctx.
• Initialize the job with job.init().
• Save successful processing state with job.commit().
• Plan for retries, duplicate output, and data reprocessing.

Understanding these details will help you build more dependable AWS data pipelines.

If you're learning AWS data engineering, start with a small S3 dataset, create an incremental ETL job, and test the behavior across several runs.

That practical experience is a useful foundation for more advanced AWS projects.

Comments

[Karan Pulyala]

Good call on making clear that bookmarks track input state, not output. Two more gotchas worth adding to the mistakes list. First, the bookmark is only saved at job.commit(), so if the job writes its output and then fails or times out before the commit, the next run reprocesses the same files and appends them again. Writing each run to its own partition (for example a run or ingest date) and overwriting that partition on retry keeps reruns idempotent. Second, for JDBC sources the bookmark works off jobBookmarkKeys, which default to the primary key, and those keys need to increase monotonically. If rows can be updated in place or IDs aren't sequential, set jobBookmarkKeys to something like an updated_at column (with jobBookmarkKeysSortOrder) or the job will quietly skip changes. It's also safest to keep the job's max concurrency at 1, since overlapping runs share the same bookmark state.