Apache Airflow Explained for Beginners

#dataengineering #airflow #python #programming

You have a data task. It runs every day. You run it manually.
That works. Until it doesn't.

What is Apache Airflow?

Airflow is an open-source platform to programmatically author, schedule, and monitor workflows.

In simple terms: you write your tasks in Python, tell Airflow when and how to run them — and it handles the rest.

Key concepts:
DAG — Directed Acyclic Graph. Just a fancy word for "a list of tasks with an order."
Task — one unit of work (run a script, move a file, query a database)
Scheduler — runs your DAG on time, every time

A simple DAG looks like this:

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

def extract():
    print("Extracting data...")

with DAG("my_first_dag", start_date=datetime(2024, 1, 1), schedule="@daily"):
    task = PythonOperator(task_id="extract", python_callable=extract)