Forem

Cover image for Azure Data Factory - Incrementally load data from Azure SQL to Azure Data Lake using Watermark
Balram Prasad
Balram Prasad

Posted on

3

Azure Data Factory - Incrementally load data from Azure SQL to Azure Data Lake using Watermark

Azure Data Factory - Incrementally load data from Azure SQL to Azure Data Lake using Watermark
In a data integration solution, incrementally (or delta) loading data after an initial full data load is a widely used scenario. The tutorials in this section show you different ways of loading data incrementally by using Azure Data Factory.

In this case, you define a watermark in your source database. A watermark is a column that has the last updated time stamp or an incrementing key. The delta loading solution loads the changed data between an old watermark and a new watermark.

In this Demo we will do following

Introduction -Agenda of Demo
Create Azure SQL Database
Create Azure Data Lake
Create an Azure Data Factory
Create required tables/SP in Azure SQL
Create Linked Services
Create ADF Pipeline
Run ADF Pipeline
Verify Azure Data Lake for File
Update Azure SQL Records
Run ADF Pipeline
Verify Azure Data Lake for Delta File

Image of Timescale

🚀 pgai Vectorizer: SQLAlchemy and LiteLLM Make Vector Search Simple

We built pgai Vectorizer to simplify embedding management for AI applications—without needing a separate database or complex infrastructure. Since launch, developers have created over 3,000 vectorizers on Timescale Cloud, with many more self-hosted.

Read more

Top comments (0)

Sentry image

See why 4M developers consider Sentry, “not bad.”

Fixing code doesn’t have to be the worst part of your day. Learn how Sentry can help.

Learn more

👋 Kindness is contagious

Discover a treasure trove of wisdom within this insightful piece, highly respected in the nurturing DEV Community enviroment. Developers, whether novice or expert, are encouraged to participate and add to our shared knowledge basin.

A simple "thank you" can illuminate someone's day. Express your appreciation in the comments section!

On DEV, sharing ideas smoothens our journey and strengthens our community ties. Learn something useful? Offering a quick thanks to the author is deeply appreciated.

Okay