Raizan

Posted on May 27 • Originally published at chasebot.online

n8n vs Airflow for API workflow automation

#tutorial #devops #orchestration #selfhosted

What You'll Need

n8n Cloud or self-hosted n8n instance
Hetzner VPS or Contabo VPS for self-hosted deployments
DigitalOcean as an alternative hosting provider
Python 3.8+ (for Airflow)
Basic API knowledge and familiarity with REST endpoints

The Core Difference: When to Pick Each
n8n for Rapid API Workflows
Airflow for Complex DAGs
Hands-On: Building the Same Workflow in Both
Performance, Scaling, and Real Costs
Getting Started

The Core Difference: When to Pick Each

I've spent the last three years automating workflows with both n8n and Airflow, and I can tell you straight: they're built for different problems.

n8n is a no-code/low-code automation platform. You drag nodes, connect them visually, and deploy workflows in minutes. It's HTTP-first, has 400+ integrations baked in, and you don't need to write Python. Perfect for startups, small teams, and anyone who values speed.

Airflow is a data orchestration engine built by Airbnb. It's code-centric, Python-native, and designed for complex DAGs (Directed Acyclic Graphs). You write operators, define task dependencies, and gain granular control. It scales to thousands of jobs per second but requires engineering muscle.

The hard truth: if your workflow fits in 100 nodes and calls 5 APIs, n8n wins on velocity. If you're orchestrating 50 data pipelines with retry logic, monitoring, and multi-environment deployments, Airflow is built for that war.

Let me show you both in action.

n8n for Rapid API Workflows

I'll build a real example: a workflow that pulls GitHub repository stats, enriches them with API data, and saves results to a database. With n8n, this takes 15 minutes.

Step 1: Set Up n8n Cloud or Self-Hosted

If you're using n8n Cloud, sign up and skip to Step 2. If self-hosting, grab a Hetzner VPS with 2GB RAM ($4/month) or Contabo VPS ($3/month). SSH in and run:

curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
sudo curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
mkdir n8n && cd n8n

Create a docker-compose.yml:

version: '3.8'
services:
  n8n:
    image: n8nio/n8n
    container_name: n8n
    ports:
      - "5678:5678"
    environment:
      - N8N_HOST=your-domain.com
      - N8N_PROTOCOL=https
      - NODE_ENV=production
      - WEBHOOK_TUNNEL_URL=https://your-domain.com/
      - DB_TYPE=postgres
      - DB_POSTGRESDB_HOST=postgres
      - DB_POSTGRESDB_PORT=5432
      - DB_POSTGRESDB_DATABASE=n8n
      - DB_POSTGRESDB_USER=n8n
      - DB_POSTGRESDB_PASSWORD=securepassword123
    depends_on:
      - postgres
    volumes:
      - n8n_data:/home/node/.n8n
    networks:
      - n8n-network
    restart: unless-stopped

  postgres:
    image: postgres:15-alpine
    container_name: n8n-postgres
    environment:
      - POSTGRES_DB=n8n
      - POSTGRES_USER=n8n
      - POSTGRES_PASSWORD=securepassword123
    volumes:
      - postgres_data:/var/lib/postgresql/data
    networks:
      - n8n-network
    restart: unless-stopped

volumes:
  n8n_data:
  postgres_data:

networks:
  n8n-network:
    driver: bridge

Start it:

sudo docker-compose up -d

Access via http://localhost:5678 (or your domain after SSL setup with Nginx).

Step 2: Create the GitHub Stats Workflow

In n8n, create a new workflow:

Trigger: Click the "+" button, add a Webhook node set to POST
GitHub Node: Search "GitHub" in the node library, add it
- Authentication: Select OAuth or add your GitHub token
- Resource: "Repository"
- Operation: "Get"
- Owner: {{ $json.owner }}
- Repository: {{ $json.repo }}
HTTP Request Node: For additional API enrichment
- Method: GET
- URL: https://api.github.com/repos/{{ $json.owner }}/{{ $json.repo }}/traffic/popular
- Headers: Add Authorization: token {{ $env.GITHUB_TOKEN }}
Data Mapper Node: Transform the response
- Map output to name, stars, forks, language
PostgreSQL Node: Store results
- Operation: "Insert"
- Table: github_stats
- Columns: repo_name, stars, forks, language, fetched_at
Send Slack Notification: Add a Slack node (optional)
- Message: Fetched stats for {{ $json.name }} — {{ $json.stars }} stars

Test with a webhook POST:

curl -X POST http://localhost:5678/webhook/github-stats \
  -H "Content-Type: application/json" \
  -d '{"owner":"torvalds","repo":"linux"}'

Response will show 5000+ stars and you've built a real workflow in minutes. No Python, no DAG files.

💡 Fast-Track Your Project: Don't want to configure this yourself? I build custom n8n pipelines and bots. Message me with code SYS3-DEVTO.

Airflow for Complex DAGs

Now let's build the same workflow in Airflow. It's more verbose but gives you industrial-grade control.

Step 1: Install Airflow

On a fresh DigitalOcean droplet (Ubuntu 22.04, 2GB RAM, $6/month):

python3 -m venv airflow-env
source airflow-env/bin/activate
pip install --upgrade pip
pip install apache-airflow==2.8.0
pip install apache-airflow-providers-http
pip install apache-airflow-providers-postgres
pip install apache-airflow-providers-slack
pip install requests

Initialize the database:

export AIRFLOW_HOME=~/airflow
airflow db init
airflow users create --username admin --password admin --firstname Admin --lastname User --role Admin --email admin@example.com

Start the scheduler and webserver:

airflow scheduler &
airflow webserver --port 8080 &

Access at http://localhost:8080.

Step 2: Write the DAG

Create ~/airflow/dags/github_stats_dag.py:

from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.operators.http_operator import SimpleHttpOperator
from airflow.providers.postgres.operators.postgres import PostgresOperator
from airflow.providers.slack.operators.slack_webhook import SlackWebhookOperator
from airflow.models import Variable
import requests
import json

default_args = {
    'owner': 'data-team',
    'retries': 2,
    'retry_delay': timedelta(minutes=5),
    'email_on_failure': True,
    'email': ['ops@example.com'],
}

dag = DAG(
    'github_stats_pipeline',
    default_args=default_args,
    description='Fetch GitHub repo stats and store in Postgres',
    schedule_interval='0 */6 * * *',
    start_date=datetime(2024, 1, 1),
    catchup=False,
    tags=['github', 'api', 'analytics'],
)

def fetch_github_data(**context):
    owner = context['dag_run'].conf.get('owner', 'torvalds')
    repo = context['dag_run'].conf.get('repo', 'linux')
    token = Variable.get('GITHUB_TOKEN')

    headers = {'Authorization': f'token {token}'}
    url = f'https://api.github.com/repos/{owner}/{repo}'

    response = requests.get(url, headers=headers, timeout=10)
    response.raise_for_status()

    data = response.json()

    context['task_instance'].xcom_push(
        key='repo_data',
        value={
            'owner': owner,
            'name': data['name'],
            'stars': data['stargazers_count'],
            'forks': data['forks_count'],
            'language': data['language'],
            'description': data['description'],
        }
    )

def enrich_with_traffic(**context):
    task_instance = context['task_instance']
    repo_data = task_instance.xcom_pull(task_ids='fetch_github', key='repo_data')

    owner = repo_data['owner']
    repo = repo_data['name']
    token = Variable.get('GITHUB_TOKEN')

    headers = {'Authorization': f'token {token}'}
    url = f'https://api.github.com/repos/{owner}/{repo}/traffic/popular'

    try:
        response = requests.get(url, headers=headers, timeout=10)
        response.raise_for_status()
        traffic = response.json()
        repo_data['top_paths'] = len(traffic) if isinstance(traffic, list) else 0
    except:
        repo_data['top_paths'] = 0

    task_instance.xcom_push(key='enriched_data', value=repo_data)

def prepare_insert_statement(**context):
    task_instance = context['task_instance']
    enriched = task_instance.xcom_pull(task_ids='enrich_traffic', key='enriched_data')

    insert_sql = f"""
    INSERT INTO github_stats (repo_owner, repo_name, stars, forks, language, top_paths, fetched_at)
    VALUES ('{enriched['owner']}', '{enriched['name']}', {enriched['stars']}, {enriched['forks']}, '{enriched['language']}', {enriched['top_paths']}, NOW());
    """

    task_instance.xcom_push(key='insert_sql', value=insert_sql)

fetch_github = PythonOperator(
    task_id='fetch_github',
    python_callable=fetch_github_data,
    dag=dag,
    provide_context=True,
)

enrich_traffic = PythonOperator(
    task_id='enrich_traffic',
    python_callable=enrich_with_traffic,
    dag

Originally published on Automation Insider.

DEV Community