Data Pipeline Orchestration with Apache Airflow

#dataengineering

Airflow is an open-source platform for programmatically authoring, scheduling, and monitoring workflows. It was developed at Airbnb in 2014 and has since become the standard for data pipeline orchestration.

Airflow does the following:

Schedules your tasks to run at the right time and in the right order.
Manages dependencies, i.e. if Task A fails, Task B never runs.
Gives you visibility into what succeeded, what failed, and how long it took.

Workflows are defined as codes (referred to as a DAG file) in Python.

Case scenario example:

You've been running the same three scripts for months:

extract.py pulls data from the API at 6 AM.
transform.py cleans it at 6:30AM.
load.py pushes it into the warehouse at 7AM.

One Tuesday, the API rate-limits you. extract.py fails silently. transform.py runs on yesterday's stale file, and load.py happily overwrites your production table with duplicate data. By the time the analytics team spots the error, it's noon. You spend the rest of the day cleaning up.

This is why orchestration exists. It handles dependencies, failures, retries, and visibility. Apache Airflow turns a fragile chain of scripts into a resilient, observable pipeline.

Airflow Components:

Scheduler - watches the clock, triggers tasks when conditions are met.
Executor - runs the task. The scheduler decides what to run; the executor does the running. Different executors handle different scales. SequentialExecutor for local development, CeleryExecutor or KubernetesExecutor for distributed production workloads.
Webserver - the dashboard at localhost:8080. Every DAG, every run, every task status, full logs are visible in one place.
Metadata database - Airflow's internal record. Every run is stored: what ran, when, the result, how long it took.

The DAG:

In Airflow, a pipeline becomes a DAG (Directed Acyclic Graph). Each step is a task; the relationships between tasks define the order.

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

with DAG(
    dag_id="gas_prices_pipeline",
    start_date=datetime(2026, 1, 1),
    schedule="0 6 * * *",
    catchup=False,
) as dag:

    extract_task = PythonOperator(
        task_id="extract",
        python_callable=extract_gas_prices,
    )

    transform_task = PythonOperator(
        task_id="transform",
        python_callable=transform_cities,
    )

    load_task = PythonOperator(
        task_id="load",
        python_callable=load_cities,
    )

    extract_task >> transform_task >> load_task

The >> operator defines dependency. Transform won't run until extract succeeds. Load won't run until transform succeeds. A failure at any step stops what's downstream and records exactly where and why — no manual inspection required.

XCom - How tasks pass data

Tasks in Airflow run independently. They don't share variables the way functions do in a script.
XCom (cross-communication) is Airflow's built-in mechanism for passing return values between tasks.

PythonOperator approach (explicit XCom pull):

def transform_cities(**context):
    data = context['ti'].xcom_pull(task_ids='extract')
    cities = data['result']['cities']
    cities_df = pd.DataFrame(cities)
    return cities_df.to_dict(orient='records')

TaskFlow approach ( automatic XCom via decorators):

@task
def transform_cities(data):
    cities = data['result']['cities']
    cities_df = pd.DataFrame(cities)
    return cities_df.to_dict(orient='records')

# wired by natural function calls:
data = extract_gas_prices()
records = transform_cities(data)
load_cities(records)

TaskFlow handles XCom automatically as the decorator handles all the repetitive setup code.

One constraint: XCom serializes data as JSON internally. DataFrames aren't JSON-serializable, so they convert to a list of dicts before passing through XCom, then reconstruct on the other side.