Using 'tags' to Organize Airflow Dags

Airflow UI displays dags as a running list as shown below. While this serves the functional purpose, it becomes a bit busy for accessibility when running hundreds of tasks or in a shared deployment.

An engineer looking to see all dags related to 'Marketing' projects will have to go through a very long list of dags. Additionally, all dags are in one dags folder by default. Using the assigned dags folder and the tags keyword in dag definition, we can organize our projects in a clean way.

The server namespace

Consider the folder below called dags where we define our dags folder. We have additional folders for marketing, accounts and analytics departments.

dags/
│
├── marketing/
│   ├── peers.py
│   └── instances.py
│
├── accounts/
│   ├── prospects.py
│   └── perfs.py
│
└── analytics/
    ├── streams.py
    └── batch.py

We have organized our dags in a clean way by placing them inside namespace folders. This even allows us to enforce permissions and access to more sensitive dags on the server.

However, if we had 100 dags in each .py, Airflow UI will just show a running list of 600 dags. This spaghetti display is not very friendly especially when accessing specific needed dags. The server has our files well organized but Airflow just throws everything at us.

The 'tags' keyword

The 'tags' keyword argument helps achieve easy organization of the dags in the Airflow UI. They will have tags listed in the tags section and clicking on this allows us to view all related dags in that namespace.

#using `@dag` decorator
@dag(
    dag_id='my_dag',
    schedule=timedelta(minutes=20),
    catchup=False,
    tags=['marketing']
)

Blow is a screenshot showing how tags help organize dags in the UI. We see a namespace with odhiambo and sales that have all dags for the corresponding names.

We can even search with the dag tag names. Clicking on odhiambo gives a list of all dags in that namespace.