DEV Community

Cover image for Temporal vs Airflow for API-First Automation
Raizan
Raizan

Posted on • Originally published at chasebot.online

Temporal vs Airflow for API-First Automation

#devops #automation #orchestration #comparison

What You'll Need

  • n8n Cloud or self-hosted n8n for a lighter alternative
  • Hetzner VPS or Contabo VPS if self-hosting Temporal or Airflow
  • DigitalOcean as an alternative cloud hosting option
  • A REST API or webhook service to test against (we'll use examples)
  • Docker installed locally (for testing)
  • Basic Python knowledge for Airflow DAG writing

Table of Contents

Temporal vs Airflow: The API-First Showdown

I've spent the last few years automating APIs, and I've tested both Temporal and Apache Airflow in production environments. If you're building workflows that call external APIs—whether that's Slack notifications, Stripe payments, or data syncing—the choice between these two matters. A lot.

Here's the thing: Temporal is built for reliability first, APIs second. Airflow is built for scheduled jobs first, APIs second. That distinction changes everything about how you architect your automation.

Before we dive into code, I should mention that if you're evaluating orchestration platforms, our guide on Temporal vs n8n vs Make for Enterprise Automation covers how these fit into the broader automation landscape. And if you're comparing pricing structures across platforms that handle API automation, Make vs n8n vs Zapier API Automation Pricing breaks down cost-per-integration for different scales.

How Temporal Handles API Workflows

Temporal is a distributed workflow orchestration engine originally built by Uber. It's designed to handle long-running, fault-tolerant workflows where reliability is non-negotiable. When you call an API in Temporal, the framework handles retries, timeouts, and state management automatically.

The key advantage: Temporal persists workflow state to a database. If your API call fails, your entire workflow state is saved. You don't lose progress. You don't have to restart from the beginning. Temporal resumes exactly where it left off.

This matters for API calls because APIs fail. Rate limits happen. Servers go down. With Temporal, you define a retry policy once, and the platform enforces it globally across all workflows.

How Airflow Handles API Workflows

Apache Airflow is a task orchestration platform built by Airbnb. It's a Directed Acyclic Graph (DAG) engine, meaning you define tasks and dependencies, then Airflow schedules them.

For API workflows, you typically use Airflow's SimpleHttpOperator or write custom Python operators that call requests. Airflow is excellent at scheduling—running tasks on a cron schedule—but it requires you to handle API resilience yourself.

If an API call fails in Airflow, you define retries at the task level. Airflow will retry, but the entire DAG doesn't understand the business logic of your workflow. It just sees tasks and dependencies.

Building a Real API Automation: Temporal Edition

Let me show you a real example: automating a workflow that fetches data from a public API, transforms it, and sends it to Slack.

Here's the Temporal setup using TypeScript (the most common language for Temporal):

import {
  Activity,
  Client,
  Connection,
  WorkflowClient,
  Worker,
} from '@temporalio/client';
import { proxyActivities } from '@temporalio/workflow';
import * as activities from './activities';

// Define your workflow
export async function dataProcessingWorkflow(inputUrl: string): Promise<string> {
  const { fetchApiData, transformData, sendSlackMessage } = proxyActivities<
    typeof activities
  >({
    startToCloseTimeout: '1 minute',
    retry: {
      initialInterval: '1 second',
      maximumInterval: '1 minute',
      maximumAttempts: 3,
    },
  });

  const rawData = await fetchApiData(inputUrl);
  const transformedData = await transformData(rawData);
  const slackResponse = await sendSlackMessage(transformedData);

  return slackResponse;
}
Enter fullscreen mode Exit fullscreen mode

Now here's the activities file where the actual API calls happen:

import axios from 'axios';

export async function fetchApiData(url: string): Promise<any> {
  const response = await axios.get(url, {
    timeout: 30000,
  });
  return response.data;
}

export async function transformData(data: any): Promise<string> {
  const summary = `Processed ${Object.keys(data).length} fields at ${new Date().toISOString()}`;
  return summary;
}

export async function sendSlackMessage(message: string): Promise<string> {
  const slackWebhookUrl = process.env.SLACK_WEBHOOK_URL;
  const response = await axios.post(slackWebhookUrl, {
    text: `Automation complete: ${message}`,
  });
  return `Sent to Slack: ${response.status}`;
}
Enter fullscreen mode Exit fullscreen mode

To run this, you'd set up a Temporal server and worker:

async function main() {
  const connection = await Connection.connect({
    address: 'localhost:7233',
  });

  const client = new WorkflowClient({ connection });

  const worker = await Worker.create({
    connection,
    taskQueue: 'tutorial',
    workflowsPath: require.resolve('./workflows'),
    activitiesPath: require.resolve('./activities'),
  });

  await worker.start();
  console.log('Worker started, processing tasks...');
}

main().catch((err) => {
  console.error(err);
  process.exit(1);
});
Enter fullscreen mode Exit fullscreen mode

The magic here: if fetchApiData fails with a 429 rate limit, Temporal automatically waits 1 second, retries, waits 2 seconds, retries again. You don't write retry logic. The framework does it.

💡 Fast-Track Your Project: Don't want to configure this yourself? I build custom n8n pipelines and bots. Message me with code SYS3-DEVTO.

Building the Same Workflow in Airflow

Now let's build the same workflow in Apache Airflow using Python. You define a DAG:

from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.operators.http import SimpleHttpOperator
import requests
import json
import os

default_args = {
    'owner': 'airflow',
    'retries': 3,
    'retry_delay': timedelta(minutes=1),
    'start_date': datetime(2024, 1, 1),
}

dag = DAG(
    'api_processing_workflow',
    default_args=default_args,
    description='Fetch API data, transform, and notify Slack',
    schedule_interval='0 9 * * *',
    catchup=False,
)

def fetch_and_transform(**context):
    url = 'https://jsonplaceholder.typicode.com/posts/1'

    try:
        response = requests.get(url, timeout=30)
        response.raise_for_status()
        data = response.json()
    except requests.exceptions.RequestException as e:
        raise Exception(f'API fetch failed: {str(e)}')

    processed_data = {
        'original_id': data['id'],
        'title': data['title'],
        'processed_at': datetime.now().isoformat(),
    }

    context['task_instance'].xcom_push(
        key='processed_data',
        value=json.dumps(processed_data)
    )

    return 'Data fetched and transformed'

def send_slack_notification(**context):
    task_instance = context['task_instance']
    processed_data_json = task_instance.xcom_pull(
        task_ids='fetch_and_transform_task',
        key='processed_data'
    )
    processed_data = json.loads(processed_data_json)

    slack_webhook = os.getenv('SLACK_WEBHOOK_URL')
    message = {
        'text': f'Workflow complete: {json.dumps(processed_data)}'
    }

    try:
        response = requests.post(slack_webhook, json=message, timeout=10)
        response.raise_for_status()
    except requests.exceptions.RequestException as e:
        raise Exception(f'Slack notification failed: {str(e)}')

    return f'Slack notified with status {response.status_code}'

fetch_task = PythonOperator(
    task_id='fetch_and_transform_task',
    python_callable=fetch_and_transform,
    dag=dag,
    provide_context=True,
)

slack_task = PythonOperator(
    task_id='send_slack_task',
    python_callable=send_slack_notification,
    dag=dag,
    provide_context=True,
)

fetch_task >> slack_task
Enter fullscreen mode Exit fullscreen mode

Notice the differences:

  1. You handle retries manually in the default_args. Airflow will retry the entire task, but you manage exponential backoff yourself.
  2. XCom is how you pass data between tasks. This isn't built into the workflow definition; it's infrastructure you manage.
  3. Scheduling is built-in (schedule_interval='0 9 * * *'). Airflow excels here.

If you're evaluating whether Temporal or a platform like n8n works better for your enterprise needs, this is a key question—does your use case demand scheduled jobs or event-driven reliability?

Temporal vs Airflow: Direct Comparison

I've now built production workflows in both. Here's what I've learned:

Temporal Strengths for APIs

  • State persistence: The entire workflow state is stored. Network failure? Temporal resumes without losing context.
  • Activity retries with exponential backoff: Built-in, works globally. You don't rewrite retry logic for every API call.
  • Long-running workflows: Can run for days, weeks, or months. Perfect for monitoring or batch jobs.
  • No scheduler needed: Temporal workflows are triggered by external events or clients, making it event-driven by default.

Temporal weaknesses:

  • Steeper learning curve. You need to understand workflow state, activities, and child workflows.
  • Setup is more complex. You're running a distributed system.

Airflow Strengths for APIs

  • Cron scheduling is native: If you need tasks to run on a schedule, Airflow is built for this.
  • Visual DAG editor: You can see your workflow as a directed graph in the UI.
  • Mature ecosystem: Huge community, tons of operators, well-documented.
  • Easier initial setup for simple scheduled jobs.

Airflow weaknesses:

  • State isn't portable: If your worker crashes mid-task, you lose everything and restart from the beginning.
  • Task-level thinking: Airflow thinks in tasks, not business logic. Your workflows become implementation details scattered across task definitions.
  • Retry logic is per-task: You can't define workflow-level resilience policies.

For API-First Automation Specifically

Originally published on Automation Insider.

Top comments (0)