Azure Data Factory Deep Dive: Mapping Data Flows, Triggers, Integration Runtimes & CI/CD

Azure Data Factory is the serverless orchestration service that most Azure data teams reach for first: you author pipelines that move and transform data, and the service provisions the compute for each run, bills you per activity execution, and never asks you to keep a server warm. It is not a database, not a Spark cluster you manage, and not a single connector — it is the control plane that wires connectors, compute, and schedules together. You describe what should happen (copy this table, run this transformation, then this one) and where the compute should live (a managed cloud runtime, a machine inside your network, a lifted SSIS cluster), and the service does the plumbing.

That is a genuinely different shape from the two things engineers reached for before it: a hand-scheduled box of Python and cron that silently rots when a credential rotates or a source adds a column, or a heavyweight ETL server you patch and capacity-plan yourself. This guide walks through the ideas an interviewer will actually probe — the pipeline / activity / dataset / linked-service object model and the Copy activity, Mapping Data Flows compiled to managed Spark, the three trigger types (schedule, tumbling window, storage event), the three integration runtimes (Azure, self-hosted, SSIS), and Git + ARM-based CI/CD — and pairs each with a Solution-Tail interview answer: code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the ETL practice library →, rehearse the transformation-shape decisions on the data-transformation practice set →, and sharpen your orchestration timing on the scheduling practice set →.

---

On this page

• Why Azure Data Factory is the cloud-native orchestrator
• Pipelines, activities & the Copy activity
• Mapping Data Flows on Spark
• Triggers — schedule, tumbling window, storage event
• Integration Runtimes & CI/CD
• Cheat sheet — Azure Data Factory recipes
• Frequently asked questions
• Practice on PipeCode

---

1. Why Azure Data Factory is the cloud-native orchestrator

ADF is a control plane you author, not a cluster you operate — that one fact decides where it fits

The one-sentence invariant: Azure Data Factory separates the control plane you author from the data plane it provisions, so orchestration becomes a declarative artifact rather than a fleet of servers you babysit. Everything that makes ADF attractive follows from that split. There is no scheduler VM to patch, no worker pool sized for peak that idles at night, no bespoke retry framework; a factory is a set of JSON definitions — pipelines, datasets, linked services, triggers — that the service executes on compute it spins up and tears down per run.

The control plane vs the data plane — the distinction interviewers open with.

• Control plane. ADF stores your pipeline, dataset, linked-service, and trigger definitions, evaluates expressions and control flow, schedules runs, and records run history and monitoring. This is metadata and orchestration — it is cheap and always on.
• Data plane. The actual byte-moving and row-crunching happens on an integration runtime: a managed Azure runtime, a self-hosted runtime inside your network, or a lifted SSIS cluster. ADF dispatches work to the runtime; the runtime touches the data.
• Why the split matters. You are billed mostly for activity runs and data-plane compute (DIUs, Data Flow vCore-hours), not for a server sitting idle. Scaling is the service's problem, not yours.

The four building blocks — the vocabulary every ADF answer needs.

• Pipeline. A logical grouping of activities with ordering, branching, and parameters. It is the unit you trigger and monitor.
• Activity. One step. Three families: data movement (the Copy activity), data transformation (Mapping Data Flow, Databricks notebook, Stored Procedure, HDInsight), and control flow (ForEach, If Condition, Until, Switch, Lookup, Get Metadata, Execute Pipeline, Set Variable, Wait, Web).
• Dataset. A named, typed view of data — "the CSVs in this container", "this SQL table". It points at data through a linked service and can be parameterised.
• Linked service. The connection — endpoint plus credentials — like a connection string. Datasets and activities reference linked services so secrets live in one governed place (ideally Key Vault).

ELT-first — what ADF does and deliberately does not do.

• Orchestrate and move. ADF's core job is to move data between 90+ connectors and to sequence work with control flow and dependencies.
• Push heavy transform down. For large transformations you either use Mapping Data Flows (ADF compiles them to Spark) or call out to Databricks / Synapse / SQL. ADF is the conductor; the orchestra is elastic compute.
• Not a warehouse, not a stream processor. ADF is batch/micro-batch orchestration. Sub-second streaming belongs in Event Hubs / Stream Analytics, not ADF.

What interviewers listen for.

• Do you say "ADF is a control plane; the integration runtime is the data plane" early? — senior signal.
• Do you place ADF as "orchestration + movement, with heavy transform pushed to Spark/SQL" unprompted? — required framing.
• Do you know which integration runtime a given source needs (on-prem → self-hosted) without being told? — the practical tell.
• Do you treat linked services as the credential boundary (Key Vault, managed identity) rather than inlining secrets? — the security signal.

Worked example — the smallest factory that does real work

Detailed explanation. The canonical ADF "hello world" is a one-activity pipeline that copies a table from a source to a sink. It looks trivial, and that is the point: the same four objects — linked service, dataset, pipeline with a Copy activity, trigger — scale unchanged from one table to a metadata-driven loop over five hundred, because ADF only ever composes the same primitives.

Question. Move a dbo.orders table from an Azure SQL source into a Parquet file in ADLS Gen2, and name the four objects involved.

Input.

object	what it names
linked service	the Azure SQL connection + credentials
dataset (source)	the dbo.orders table view
dataset (sink)	the target Parquet path
pipeline	one Copy activity binding source → sink

Code.

{
  "name": "CopyOrdersToLake",
  "properties": {
    "activities": [
      {
        "name": "CopyOrders",
        "type": "Copy",
        "inputs":  [ { "referenceName": "ds_sql_orders",   "type": "DatasetReference" } ],
        "outputs": [ { "referenceName": "ds_lake_orders",   "type": "DatasetReference" } ],
        "typeProperties": {
          "source": { "type": "AzureSqlSource" },
          "sink":   { "type": "ParquetSink" }
        }
      }
    ]
  }
}

Step-by-step explanation. The pipeline declares one Copy activity. Its inputs reference a dataset (ds_sql_orders) that in turn references a linked service holding the SQL connection; outputs reference the sink dataset for the lake path. When the pipeline runs, ADF resolves the datasets to concrete endpoints, picks an integration runtime to carry the bytes, and streams rows from source to sink. No compute is provisioned until the run starts, and none survives after it finishes.

Output.

ADF produced	value
pipeline run	CopyOrdersToLake — 1 activity, Succeeded
rows copied	dbo.orders → orders.parquet
compute	one Azure IR, billed as data-movement DIU-hours
monitoring	run id + duration + rows recorded in run history

Rule of thumb. If you can name the linked service, the two datasets, and the activity, you can describe any ADF job — everything else is control flow and parameters layered on top.

---

2. Pipelines, activities & the Copy activity

Pipelines sequence activities, and the Copy activity is the workhorse — learn the object graph and the rest is configuration

An interviewer who asks "walk me through an ADF pipeline" wants the object graph in order: a pipeline holds activities; activities read and write datasets; datasets resolve through linked services; the Copy activity moves bytes over an integration runtime. Get that chain crisp and the whole product snaps into focus.

The activity families.

• Data movement — Copy. One source, one sink, 90+ connectors. It handles format conversion (CSV ↔ Parquet ↔ JSON), schema mapping, compression, and fault tolerance in a single step.
• Data transformation. Mapping Data Flow (Spark), Databricks Notebook/Jar/Python, Stored Procedure, Synapse notebook, HDInsight. These hand heavy compute to an engine built for it.
• Control flow. ForEach (iterate, optionally in parallel), If Condition, Switch, Until, Lookup (read a value to branch on), Get Metadata (does the file exist? how big?), Execute Pipeline (call a child), Set Variable, Wait, Web/Webhook.

Datasets and linked services — the connection boundary.

• Linked service = connection + credential. It defines how to reach a store (server, database, auth). Point it at Key Vault or a managed identity so secrets never live in the pipeline JSON.
• Dataset = a typed view over a linked service. It defines what data — a table, a folder of files, a specific blob path — and its shape/format.
• Parameterise both. A dataset with a @dataset().fileName parameter becomes one reusable definition for a thousand files; a linked service parameter lets one connection definition target many databases.

The Copy activity, in detail (the one they drill).

• DIUs (Data Integration Units). A measure of the power (CPU, memory, network) ADF allocates to a copy. More DIUs = faster copy, higher cost; auto lets ADF size it.
• Parallel copy. ADF partitions the read (by table partition, by file, by a dynamic range) and copies partitions concurrently.
• Staged copy. When source and sink cannot talk directly (e.g. on-prem → Snowflake), ADF stages through blob storage as an interim hop.
• Fault tolerance & logging. Skip incompatible rows, log them, and continue instead of failing the whole load.

Parameters and expressions.

• Pipeline parameters (@pipeline().parameters.runDate) are set at trigger time; variables (@variables('counter')) mutate during a run with Set Variable.
• System variables give you @pipeline().RunId, @pipeline().TriggerTime, and @trigger().startTime for lineage and idempotent paths.
• The expression language (@concat, @formatDateTime, @item(), @activity('Lookup').output) is how one pipeline becomes metadata-driven instead of copy-pasted per table.

Worked example — a metadata-driven ForEach over many tables

Detailed explanation. Real ingestion rarely copies one table; it copies a list. The idiomatic ADF pattern is a Lookup that reads a control table of table names, feeding a ForEach that runs one parameterised Copy per item. One pipeline, any number of tables, driven by data instead of duplicated activities.

Question. Given a control table listing source tables, copy each one to the lake with a single pipeline, using a parameterised sink path.

Input. A LookupTables activity returns three rows: orders, customers, products.

Code.

{
  "name": "IngestAllTables",
  "properties": {
    "activities": [
      { "name": "LookupTables", "type": "Lookup",
        "typeProperties": { "source": { "type": "AzureSqlSource",
          "sqlReaderQuery": "SELECT table_name FROM etl.control" },
          "firstRowOnly": false } },
      { "name": "ForEachTable", "type": "ForEach",
        "dependsOn": [ { "activity": "LookupTables", "dependencyConditions": [ "Succeeded" ] } ],
        "typeProperties": {
          "items": "@activity('LookupTables').output.value",
          "isSequential": false, "batchCount": 8,
          "activities": [
            { "name": "CopyOne", "type": "Copy",
              "inputs":  [ { "referenceName": "ds_sql_generic",  "type": "DatasetReference",
                            "parameters": { "tableName": "@item().table_name" } } ],
              "outputs": [ { "referenceName": "ds_lake_generic", "type": "DatasetReference",
                            "parameters": { "path": "@concat('raw/', item().table_name)" } } ],
              "typeProperties": { "source": { "type": "AzureSqlSource" },
                                  "sink":   { "type": "ParquetSink" } } }
          ] } }
    ]
  }
}

Step-by-step explanation. LookupTables runs a query and returns the rows in output.value. ForEachTable iterates that array; because isSequential is false with batchCount: 8, up to eight copies run concurrently. Inside the loop, @item().table_name parameterises the generic source dataset and @concat('raw/', item().table_name) parameterises the sink path, so each iteration copies a different table to its own folder without a separate activity.

Output.

iteration	@item().table_name	sink path	result
1	orders	raw/orders	Succeeded
2	customers	raw/customers	Succeeded
3	products	raw/products	Succeeded

Rule of thumb. If you find yourself pasting a second Copy activity, stop — a Lookup + ForEach + parameterised datasets collapses N near-identical copies into one pipeline you can extend by inserting a control-table row.

Azure Data Factory interview question on the Copy activity

Question. You must copy a 400 GB partitioned SQL table to Parquet as fast as is reasonable, and the load must not abort if a handful of rows have unconvertible values. Which Copy settings do you set, and why?

Solution Using parallel copy with fault tolerance

Code.

{
  "name": "CopyBigTable", "type": "Copy",
  "inputs":  [ { "referenceName": "ds_sql_big",  "type": "DatasetReference" } ],
  "outputs": [ { "referenceName": "ds_lake_big", "type": "DatasetReference" } ],
  "typeProperties": {
    "source": {
      "type": "AzureSqlSource",
      "partitionOption": "PhysicalPartitionsOfTable"
    },
    "sink": { "type": "ParquetSink" },
    "parallelCopies": 16,
    "dataIntegrationUnits": 128,
    "enableSkipIncompatibleRow": true,
    "logSettings": {
      "enableCopyActivityLog": true,
      "copyActivityLogSettings": { "logLevel": "Warning" }
    }
  }
}

Step-by-step trace.

setting	value	effect on the run
partitionOption	PhysicalPartitionsOfTable	source read split along existing table partitions
parallelCopies	16	up to 16 partitions copied concurrently
dataIntegrationUnits	128	more CPU/network allocated to the copy
enableSkipIncompatibleRow	true	bad rows skipped, not fatal
logSettings	Warning	skipped rows written to a log for audit

1. ADF partitions the source read using the table's physical partitions, so 16 readers each pull a slice instead of one reader scanning 400 GB serially.
2. dataIntegrationUnits: 128 raises the compute ceiling so the 16 parallel streams are not starved; DIUs and parallelCopies are tuned together.
3. enableSkipIncompatibleRow makes a type-conversion failure skip that row rather than aborting the activity, and logSettings records each skip so the miss is auditable, not silent.
4. The result is a fast, partitioned copy that completes even with dirty rows, with a log you can reconcile against the source count.

Output:

metric	value
throughput	~16× a single-stream copy (partition-bound)
rows skipped	logged to the copy-activity log, run still Succeeded
tuning levers	parallelCopies × dataIntegrationUnits

Why this works — concept by concept:

• Parallel copy — partitioning the source read lets ADF move many slices at once; throughput scales with partitions up to the DIU ceiling, turning a serial scan into a fan-out.
• Data Integration Units — DIUs are the compute budget for the copy; raising them removes the starvation that would otherwise cap parallelCopies.
• Fault tolerance — enableSkipIncompatibleRow trades completeness for progress on known-dirty sources, and the log keeps the trade auditable instead of silent.
• Staged vs direct — for stores that cannot talk directly you would add enableStaging; here direct SQL → lake needs no interim hop.
• Cost — copy cost is O(data volume) in DIU-hours; parallelism cuts wall-clock time, not total DIU-seconds, so you pay for speed, not extra data.

ETL
Topic — etl
ETL extract-and-load pipeline problems

Practice →

Pipelines Topic — pipelines Pipeline-design and control-flow problems

Practice →

---

3. Mapping Data Flows on Spark

You draw the transformation graph, ADF compiles it to a Spark job — code-free transforms that still scale

The feature that turns ADF from a mover into a transformer is Mapping Data Flows: a visual, code-free transformation graph that ADF compiles to Apache Spark and runs on a cluster it manages for you. You never write Scala, never size a cluster by hand, never submit a job — you drag source, filter, join, aggregate, and alter-row nodes onto a canvas, and ADF generates and executes the Spark plan. This is the difference between "orchestrate and call Databricks" and "transform inside ADF."

How a Data Flow executes.

• Compiled to Spark. The visual graph becomes a Spark job. Execution happens on an Azure integration runtime configured with a compute type (General Purpose / Memory Optimized) and a core count — ADF spins the cluster up for the run and tears it down after.
• Debug session. While designing, you turn on a debug cluster so you can preview data at each transformation and see row counts live. The debug cluster has a time-to-live; a triggered run gets its own cluster.
• Data flow vs the pipeline. A Data Flow is invoked from a pipeline by an Execute Data Flow activity — the pipeline still orchestrates; the Data Flow is the compute step.

The transformation vocabulary (the nodes they ask about).

• Source / Sink. Read from and write to datasets or inline connections; a sink can insert, upsert, update, or delete.
• Row/column shaping. Select (rename/prune), Derived Column (compute new columns with the expression language), Filter, Sort, Aggregate (group-by with sum/avg/etc.), Pivot / Unpivot.
• Combining. Join (inner/left/right/full/cross), Lookup (enrich without dropping rows), Union, Exists (semi/anti join).
• Keys & change. Surrogate Key (generate incrementing keys), Window (ranking, running totals), and Alter Row — the transformation that tags each row as insert / update / upsert / delete so the sink applies the right operation.

When a Data Flow is the right tool.

• Yes — column-level transforms, joins, aggregations, SCD logic, and cleansing where you want no code and native Spark scale.
• No, use Copy — a straight move with at most format conversion; Copy is cheaper and simpler than spinning a Spark cluster.
• No, use Databricks — when you need custom libraries, ML, or fine-grained cluster control that the managed Data Flow runtime does not expose.

Worked example — aggregate orders into a daily revenue table

Detailed explanation. The everyday Data Flow is a source → derived column → aggregate → sink chain that turns raw rows into a summarised table, with the transform running on Spark under the hood. Here we roll raw orders up into daily_revenue.

Question. From a raw orders stream with order_ts and amount, produce one row per day with total revenue and order count, using a Mapping Data Flow.

Input.

order_id	order_ts	amount
1	2026-03-01 09:12	42.50
2	2026-03-01 18:40	17.00
3	2026-03-02 11:05	9.90

Code.

source(orders) ~> src
src derive(order_date = toDate(order_ts)) ~> withDate
withDate aggregate(
    groupBy(order_date),
    revenue = sum(amount),
    orders  = count()
) ~> daily
daily sink(dailyRevenue)

Step-by-step explanation. The source reads raw orders. derive adds order_date by truncating the timestamp with the Data Flow expression toDate(order_ts). aggregate groups by order_date and computes sum(amount) as revenue and count() as orders — this is the group-by that Spark executes in parallel across partitions. The sink writes one row per day. ADF compiles the four nodes into a single Spark job; you wrote no Spark.

Output.

order_date	revenue	orders
2026-03-01	59.50	2
2026-03-02	9.90	1

Rule of thumb. If the transform is a graph of joins, filters, and aggregations you would otherwise hand to Spark, do it in a Mapping Data Flow — you get the scale without the cluster management or the code.

Azure Data Factory interview question on Mapping Data Flows

Question. You must load a dim_customer dimension where existing customers get updated and new ones inserted, in one Data Flow, keyed by customer_id. Which transformation drives insert-vs-update, and how does the sink apply it?

Solution Using Alter Row with an upsert policy

Code.

source(stagedCustomers) ~> src
src lookup(
    dim_customer@customer_id == src@customer_id,
    broadcast: 'auto'
) ~> matched
matched alterRow(
    upsertIf(true())
) ~> tagged
tagged sink(dim_customer,
    keys: ['customer_id'],
    allowUpsert: true,
    skipDuplicateMapInputs: true
)

Step-by-step trace.

staged row (customer_id, email)	lookup found in dim?	alter-row tag	sink action
(5, b@x)	yes	upsert	UPDATE id 5 → b@x
(6, c@x)	no	upsert	INSERT id 6
(7, d@x)	yes	upsert	UPDATE id 7 → d@x

1. The lookup matches each staged row against the existing dim_customer on customer_id, so the graph knows which rows already exist.
2. alterRow(upsertIf(true())) tags every row with an upsert intent — the policy that tells the sink "update if the key exists, insert otherwise."
3. The sink is configured with keys: ['customer_id'] and allowUpsert: true, so it translates the upsert tag into an UPDATE on match and an INSERT on miss against the target.
4. Because the whole thing is one Spark job, the match, the tag, and the write happen in a single scaled-out pass instead of row-by-row SQL.

Output:

table	rows after run	invariant
dim_customer	one per customer_id	existing updated, new inserted

Why this works — concept by concept:

• Alter Row — the transformation that attaches an insert/update/upsert/delete policy to each row; it is where change logic lives, decoupled from the sink mechanics.
• Upsert on keys — declaring keys: ['customer_id'] with allowUpsert lets the sink resolve update-vs-insert without hand-written MERGE SQL.
• Lookup vs join — lookup enriches without dropping unmatched rows, so new customers survive to be inserted rather than filtered out by an inner join.
• Compiled to Spark — the match and write run as one distributed job, so a million-row dimension upserts with the same graph as a hundred-row one.
• Cost — Data Flow cost is O(cluster vCore-hours); it pays off when transform volume is high, and is overkill for a plain copy.

ETL
Topic — data-transformation
Transformation-graph and aggregation problems

Practice →

Spark Topic — spark-sql Spark SQL group-by and join problems

Practice →

---

4. Triggers — schedule, tumbling window, storage event

Three trigger types decide when a pipeline runs — schedule, tumbling window, and storage event are not interchangeable

A pipeline does nothing until something starts it, and ADF gives you three trigger types with genuinely different semantics. Say the distinction in one breath: schedule fires on the wall clock, a tumbling window is a stateful contiguous interval you can backfill, and a storage event reacts to a blob landing. Choosing wrong is a correctness bug — a schedule trigger cannot backfill, and a tumbling window is not many-to-many.

Schedule trigger.

• Wall-clock recurrence. Fires every N minutes/hours/days/weeks/months, optionally on specific weekdays or month-days, between a start and (optional) end time.
• Many-to-many. One schedule trigger can start several pipelines, and one pipeline can be started by several triggers.
• No memory of the past. It fires forward from "now"; if the factory was down, missed fires are gone. There is no backfill and no window identity.

Tumbling window trigger.

• Contiguous, non-overlapping windows. Time is chopped into fixed-size, back-to-back intervals; each window fires exactly once and carries WindowStart / WindowEnd system variables you pass into the pipeline to process that slice.
• Stateful and backfillable. Set the start time in the past and ADF replays every historical window in order — the native way to backfill. Each window tracks its own success/failure state.
• Dependencies & concurrency. A window can depend on the previous window (self-dependency) or on another trigger's window, and maxConcurrency bounds how many windows run at once. It also has a built-in retry policy.
• One-to-one. A tumbling window trigger drives exactly one pipeline.

Storage event trigger.

• Event-driven. Fires on blob created or blob deleted events in ADLS Gen2 / Blob Storage, delivered through Event Grid — so a file landing kicks the pipeline within seconds, no polling.
• Path filters. blobPathBeginsWith and blobPathEndsWith scope which blobs trigger it (e.g. begins with /raw/orders/, ends with .csv).
• Event payload. The trigger exposes @triggerBody().fileName and @triggerBody().folderPath, so the pipeline can process exactly the file that arrived.

Choosing a trigger.

• Fixed cadence, several pipelines, no backfill → schedule.
• Per-slice processing, historical replay, window dependencies → tumbling window.
• React to a file arriving, near-real-time → storage event.

Worked example — a tumbling window that processes one hour per run

Detailed explanation. The classic tumbling-window pattern passes WindowStart and WindowEnd into the pipeline so each run processes exactly its own hour of data. Point the start time at the past and ADF replays every hour since then, in order — backfill for free.

Question. Run an hourly pipeline where each run loads only the rows whose event_ts falls in that hour, and make a start date in the past backfill automatically.

Input. A tumbling window trigger with frequency: Hour, interval: 1, startTime: 2026-03-01T00:00:00Z.

Code.

{
  "name": "HourlyLoad",
  "properties": {
    "type": "TumblingWindowTrigger",
    "typeProperties": {
      "frequency": "Hour", "interval": 1,
      "startTime": "2026-03-01T00:00:00Z",
      "maxConcurrency": 4,
      "retryPolicy": { "count": 3, "intervalInSeconds": 300 }
    },
    "pipeline": {
      "pipelineReference": { "referenceName": "LoadHour", "type": "PipelineReference" },
      "parameters": {
        "windowStart": "@trigger().outputs.windowStartTime",
        "windowEnd":   "@trigger().outputs.windowEndTime"
      }
    }
  }
}

Step-by-step explanation. frequency: Hour / interval: 1 defines one-hour windows. Because startTime is in the past, ADF enumerates every hour from 2026-03-01T00:00 to now and schedules a run per window, up to maxConcurrency: 4 at a time. Each run receives windowStart / windowEnd, which the pipeline uses in a query like WHERE event_ts >= @{windowStart} AND event_ts < @{windowEnd} — so run boundaries are exact and non-overlapping. A failed window retries up to three times.

Output.

window	windowStart	windowEnd	rows processed
W1	2026-03-01 00:00	2026-03-01 01:00	that hour only
W2	2026-03-01 01:00	2026-03-01 02:00	that hour only
…	…	…	replayed in order

Rule of thumb. When each run should own a fixed slice of time and you may need to reprocess history, use a tumbling window — the WindowStart/WindowEnd pair makes every run deterministic and idempotent.

Azure Data Factory interview question on event-driven ingestion

Question. Files land in adls://raw/orders/ at unpredictable times and each must be processed the moment it arrives, passing the exact filename into the pipeline. Which trigger, which filters, and how does the pipeline learn the filename?

Solution Using a storage event trigger with path filters

Code.

{
  "name": "OnOrderFile",
  "properties": {
    "type": "BlobEventsTrigger",
    "typeProperties": {
      "scope": "/subscriptions/…/storageAccounts/lakestore",
      "events": [ "Microsoft.Storage.BlobCreated" ],
      "blobPathBeginsWith": "/raw/orders/",
      "blobPathEndsWith": ".csv"
    },
    "pipelines": [
      { "pipelineReference": { "referenceName": "ProcessOrderFile", "type": "PipelineReference" },
        "parameters": {
          "fileName":   "@triggerBody().fileName",
          "folderPath": "@triggerBody().folderPath"
        } }
    ]
  }
}

Step-by-step trace.

blob path written	begins /raw/orders/?	ends .csv?	trigger fires?	fileName passed
/raw/orders/2026-03-01.csv	yes	yes	yes	2026-03-01.csv
/raw/orders/2026-03-01.tmp	yes	no	no	—
/raw/invoices/x.csv	no	yes	no	—

1. The trigger subscribes to Microsoft.Storage.BlobCreated events on the storage account through Event Grid, so it reacts within seconds of a write with no polling.
2. blobPathBeginsWith and blobPathEndsWith filter events before a run is created, so only .csv files under /raw/orders/ start the pipeline — a .tmp staging file is ignored.
3. @triggerBody().fileName and @triggerBody().folderPath carry the arrived file's identity into the pipeline, so ProcessOrderFile reads exactly the blob that fired the event.
4. Because the filter and the payload are per-event, ten files landing at once create ten independent runs, each scoped to its own file.

Output:

aspect	value
latency	seconds after blob write (event-driven)
runs created	one per matching blob
pipeline input	the exact fileName that arrived

Why this works — concept by concept:

• Event Grid delivery — the trigger is push-based, not polling, so ingestion latency is seconds and you pay per event, not per poll.
• Path filters — beginsWith / endsWith evaluate before a run is created, keeping staging files and unrelated folders from starting work.
• Trigger body — @triggerBody().fileName binds the run to the specific blob, so each run is scoped and idempotent per file.
• One run per file — concurrent arrivals fan out into independent runs, which parallelises naturally without a ForEach.
• Cost — cost is O(events); an idle folder costs nothing, unlike a schedule that fires whether or not data arrived.

Scheduling
Topic — scheduling
Trigger, cadence and backfill problems

Practice →

Pipelines Topic — pipelines Event-driven pipeline-design problems

Practice →

---

5. Integration Runtimes & CI/CD

The integration runtime decides where compute runs, and Git + ARM decides how a factory ships — the two operational pillars

The two questions a senior ADF answer must settle are where does the work actually run and how does a change reach production. The first is the integration runtime (IR); the second is Git integration plus ARM templates. Get both right and a factory is a reproducible, promotable artifact instead of hand-edits in a portal.

The three integration runtimes.

• Azure IR — fully managed, cloud. Serverless compute ADF owns. It runs Copy between cloud stores, executes Mapping Data Flows (the Spark cluster), and dispatches transform activities. You pick a region (or auto-resolve); you never patch a VM.
• Self-hosted IR (SHIR) — hybrid / private network. Software you install on a VM or on-prem machine so ADF can reach data behind a firewall or in a private network. It runs Copy and dispatches transforms locally; you can add nodes for high availability and share one SHIR across factories.
• Azure-SSIS IR — lift-and-shift. A managed cluster of VMs that runs existing SSIS packages (from SSISDB) with little or no rewrite, so on-prem SSIS investments move to the cloud.

Choosing an IR.

• Source and sink are both in the cloud → Azure IR.
• A source lives on-prem or in a locked-down VNet → self-hosted IR (it is the only one that can reach it).
• You have legacy SSIS packages to run as-is → Azure-SSIS IR.

Git integration — the authoring workflow.

• Collaboration branch. ADF Studio connects to Azure DevOps Git or GitHub. You develop on feature branches and merge into the collaboration branch (usually main); the Studio saves each resource as JSON in the repo.
• Publish → adf_publish. Clicking Publish validates the collaboration branch and generates ARM templates (ARMTemplateForFactory.json + a parameters file) into the publish branch (adf_publish by default). ARM templates are the deployable artifact.

CI/CD — promoting across environments.

• One factory definition, many environments. Deploy the ARM template to dev, then test, then prod, overriding the ARM parameters per environment (connection strings, storage accounts, IR names) so the same pipelines point at the right endpoints.
• Global parameters & Key Vault. Environment-specific values live in ARM parameters or global parameters; secrets stay in Key Vault referenced by linked services, never in the template.
• Automated publish. The @microsoft/azure-data-factory-utilities npm package lets a CI pipeline validate the factory and export the ARM template automatically from the collaboration branch — removing the manual "Publish" button so deployment is fully scripted.

Worked example — one pipeline, three environments, per-env parameters

Detailed explanation. The point of ARM-based CI/CD is that the pipeline logic is identical across dev/test/prod and only the parameters differ. You deploy the same template three times with three parameter files, and each factory ends up pointing at its own storage and databases.

Question. Promote a factory from dev to prod without editing any pipeline, changing only the storage account and SQL server per environment.

Input. One ARM template plus a per-environment parameter override.

Code.

## CI step 1: validate + export ARM template from the collaboration branch
npm install
npm run build export ./ /subscriptions/…/factories/adf-dev "ArmTemplate"

## CD step 2: deploy the SAME template to prod with prod parameters
az deployment group create \
  --resource-group rg-data-prod \
  --template-file ArmTemplate/ARMTemplateForFactory.json \
  --parameters @params.prod.json

// params.prod.json — only environment values change
{
  "parameters": {
    "factoryName":            { "value": "adf-prod" },
    "ls_lake_accountName":    { "value": "lakeprod" },
    "ls_sql_serverName":      { "value": "sql-prod.database.windows.net" }
  }
}

Step-by-step explanation. The npm utilities export the collaboration branch to an ARM template (no manual Publish). The az deployment group create step deploys that same template to the prod resource group, but params.prod.json overrides the linked-service parameters — lakeprod instead of lakedev, sql-prod instead of sql-dev. Because only parameters change, the pipelines, datasets, and triggers are byte-identical across environments; prod is not a re-implementation, it is the same artifact with prod endpoints.

Output.

environment	template	storage account	SQL server
dev	ARMTemplateForFactory.json	lakedev	sql-dev
prod	ARMTemplateForFactory.json (same)	lakeprod	sql-prod

Rule of thumb. If promoting to prod means editing pipelines, your CI/CD is wrong — the template must be identical and only the ARM parameter file should change between environments.

Azure Data Factory interview question on integration-runtime placement

Question. A nightly job must copy from an on-prem SQL Server behind a corporate firewall into an Azure SQL warehouse. Which integration runtime carries the copy, where does it run, and why can the default Azure IR not do it?

Solution Using a self-hosted integration runtime

Code.

{
  "name": "OnPremToWarehouse", "type": "Copy",
  "inputs":  [ { "referenceName": "ds_onprem_sql", "type": "DatasetReference" } ],
  "outputs": [ { "referenceName": "ds_azure_sql",  "type": "DatasetReference" } ],
  "typeProperties": {
    "source": { "type": "SqlServerSource" },
    "sink":   { "type": "AzureSqlSink" }
  },
  "linkedServiceName": {
    "referenceName": "ls_onprem_sql", "type": "LinkedServiceReference"
  }
}

// ls_onprem_sql binds the source to the self-hosted IR
{
  "name": "ls_onprem_sql",
  "properties": {
    "type": "SqlServer",
    "typeProperties": { "connectionString": "…on-prem server…" },
    "connectVia": { "referenceName": "shir-corp", "type": "IntegrationRuntimeReference" }
  }
}

Step-by-step trace.

element	value	why it matters
source	on-prem SQL Server behind firewall	not reachable from the public cloud
connectVia	shir-corp (self-hosted IR)	runs inside the corporate network
SHIR node	installed on a domain VM	opens outbound-only to ADF, reads SQL locally
sink	Azure SQL warehouse	reachable from the SHIR's outbound path

1. The default Azure IR runs in Microsoft's cloud and cannot open an inbound connection through the corporate firewall, so it cannot read the on-prem server.
2. A self-hosted IR node is installed on a machine inside the network; it makes only outbound calls to ADF, so no inbound firewall rule is needed.
3. The source linked service sets connectVia: shir-corp, so ADF dispatches the read to the SHIR, which pulls rows locally and streams them out to the Azure SQL sink.
4. If you needed resilience, you would add a second SHIR node for high availability; the logical IR stays the same.

Output:

aspect	value
copy path	on-prem SQL → SHIR → Azure SQL
firewall change	none inbound (SHIR is outbound-only)
runtime	self-hosted IR (the only one that can reach the source)

Why this works — concept by concept:

• Self-hosted IR — the only runtime that lives inside your network, so it is the sole way to reach private / on-prem sources ADF cannot otherwise see.
• Outbound-only — the SHIR dials out to ADF, so hybrid connectivity needs no inbound firewall hole, which is what security teams require.
• connectVia — binding the linked service to a named IR is how ADF knows where to run the copy; change the binding, change the compute location.
• Azure vs self-hosted vs SSIS — Azure IR for cloud-to-cloud and Data Flows, SHIR for hybrid, Azure-SSIS for lifted packages; picking the right one is the placement decision.
• Cost — SHIR compute is your VM (fixed), while Azure IR is billed per DIU-hour; the trade is "own the box for reach" vs "pay per use for managed."

ETL
Topic — etl
Hybrid extract-and-load problems

Practice →

Reliability Topic — reliability Deployment and reliability problems

Practice →

---

Cheat sheet — Azure Data Factory recipes

Minimal Copy pipeline (activity core).

{ "name": "Copy1", "type": "Copy",
  "inputs":  [ { "referenceName": "ds_src", "type": "DatasetReference" } ],
  "outputs": [ { "referenceName": "ds_snk", "type": "DatasetReference" } ],
  "typeProperties": { "source": { "type": "AzureSqlSource" },
                      "sink":   { "type": "ParquetSink" } } }

Parameterised dataset (one definition, many files).

{ "name": "ds_lake_generic",
  "properties": { "type": "Parquet",
    "parameters": { "path": { "type": "string" } },
    "typeProperties": { "location": {
      "type": "AzureBlobFSLocation", "folderPath": "@dataset().path" } } } }

Tumbling window with WindowStart / WindowEnd.

{ "type": "TumblingWindowTrigger",
  "typeProperties": { "frequency": "Hour", "interval": 1,
    "startTime": "2026-03-01T00:00:00Z", "maxConcurrency": 4 },
  "pipeline": { "parameters": {
    "start": "@trigger().outputs.windowStartTime",
    "end":   "@trigger().outputs.windowEndTime" } } }

Storage event trigger filter.

{ "type": "BlobEventsTrigger",
  "typeProperties": { "events": [ "Microsoft.Storage.BlobCreated" ],
    "blobPathBeginsWith": "/raw/orders/", "blobPathEndsWith": ".csv" } }

Mapping Data Flow alter-row upsert (sink).

tagged alterRow(upsertIf(true())) ~> t
t sink(dim, keys: ['id'], allowUpsert: true)

Bind a linked service to a self-hosted IR.

{ "connectVia": { "referenceName": "shir-corp",
                  "type": "IntegrationRuntimeReference" } }

Integration-runtime picker.

Situation	Integration runtime
Cloud source and sink, or Data Flows	Azure IR
On-prem / private-network source	Self-hosted IR
Existing SSIS packages to run as-is	Azure-SSIS IR
Backfill historical time slices	(trigger) Tumbling window

---

Frequently asked questions

What is Azure Data Factory?

Azure Data Factory is a fully managed, serverless data-integration service for building ETL/ELT pipelines in the cloud. You author pipelines of activities — moving data with the Copy activity, transforming it with Mapping Data Flows or Databricks, and sequencing work with control flow — and ADF provisions the compute per run through an integration runtime. It is a control plane you author with JSON (or the Studio), not a server or cluster you operate.

What is the difference between the Copy activity and a Mapping Data Flow?

The Copy activity moves data from one source to one sink with optional format conversion and schema mapping; it is cheap and ideal for straight ingestion across 90+ connectors. A Mapping Data Flow is a visual transformation graph — joins, aggregates, derived columns, alter-row upserts — that ADF compiles to an Apache Spark job on a managed cluster. Use Copy to move bytes; use a Data Flow when you need real, scaled-out transformation without writing Spark code.

What is an integration runtime in ADF?

An integration runtime is the compute infrastructure ADF uses to actually run activities — the data plane. There are three: the Azure IR (fully managed, for cloud-to-cloud copy and Data Flow Spark execution), the self-hosted IR (installed inside your network to reach on-prem or private sources), and the Azure-SSIS IR (a managed cluster that runs lifted SSIS packages). You bind a linked service to an IR with connectVia to decide where the work runs.

How do ADF triggers work (schedule vs tumbling window vs event)?

A schedule trigger fires on a wall-clock recurrence, is many-to-many with pipelines, and cannot backfill. A tumbling window trigger chops time into contiguous, non-overlapping windows, is stateful, drives one pipeline, and can replay historical windows (backfill) while exposing WindowStart/WindowEnd. A storage event trigger fires through Event Grid when a blob is created or deleted, filtered by path, and passes the filename into the pipeline. Pick by cadence, backfill need, and time-vs-event.

How does CI/CD work in Azure Data Factory?

ADF Studio connects to Git (Azure DevOps or GitHub); you develop on feature branches and merge into a collaboration branch. Publishing generates ARM templates into the adf_publish branch, and you deploy that template across dev/test/prod, overriding only per-environment ARM parameters (storage accounts, servers, IR names). Secrets live in Key Vault, and the @microsoft/azure-data-factory-utilities npm package can validate and export the template automatically so deployment is fully scripted.

Does Azure Data Factory replace SSIS or Databricks?

Not exactly — it orchestrates alongside them. ADF can run existing SSIS packages via the Azure-SSIS integration runtime (a lift-and-shift path rather than a rewrite) and can call Databricks notebooks as transformation activities when you need custom code, libraries, or ML. ADF's own Mapping Data Flows cover many transformations code-free, but for specialised compute you keep Databricks and let ADF conduct.

Practice on PipeCode

Pipecode.ai is Leetcode for Data Engineering — every Azure Data Factory idea above, from the metadata-driven Copy loop to the tumbling-window backfill and the Mapping Data Flow upsert, maps to a hands-on practice room where you build the load against real graded inputs. PipeCode pairs each reading with 450+ DE-focused problems and a real-time scoring engine, so your answer to "how would you backfill this pipeline without double-loading?" holds up under a senior interviewer's depth probes.

Practice ETL problems now →
Scheduling & trigger drills →