DEV Community

Cover image for 🚨 Fix Airflow 2.8.0 LocalExecutor Crashes on macOS: 3 Essential Solutions (2025)
Aineah Simiyu
Aineah Simiyu

Posted on

🚨 Fix Airflow 2.8.0 LocalExecutor Crashes on macOS: 3 Essential Solutions (2025)

#airflow #python #devops #postgres

From SIGILL crashes to zombie tasks—one checklist to rule them all

The Nightmare Scenario

You've just deployed your Airflow DAG on macOS. Everything looks perfect in the UI. You hit "Run", and then... nothing.

The symptoms:

  • Task starts → hangs for minutes → scheduler marks zombie/failed
  • Scheduler log shows: 
  Task exited with return code -4
  Detected zombie job ...
Enter fullscreen mode Exit fullscreen mode
  • Nothing ever prints past the first requests.get()

Sound familiar? You're not alone.

After weeks of debugging LocalExecutor issues on macOS, I discovered three silent killers that were sabotaging my workflows. Below is the exact three-step checklist that took my DAG from broken to blazing in under 30 seconds.

1. The macOS Proxy Phantom 👻

The Problem:
macOS silently routes all HTTP traffic through networkserviceproxy. If that proxy stalls (or becomes unreachable), requests.get() never returns—your tasks hang indefinitely.

The Symptoms:

  • Tasks hang on any HTTP request
  • No error messages, just eternal waiting
  • Works fine on Linux, breaks on macOS

The Fix:
Export once, forget forever:

# In the shell that launches airflow
export NO_PROXY="*"

# Make it permanent
echo 'export NO_PROXY="*"' >> ~/.zshrc
Enter fullscreen mode Exit fullscreen mode

Why this works: This bypasses macOS's network service proxy entirely, allowing direct connections.

2. The Objective-C Fork Bomb 💣

The Problem:
LocalExecutor (and Celery) fork workers to run tasks. On macOS 14 + Python 3.11, libdispatch crashes inside child processes with:

objc[...] BUG IN CLIENT OF LIBDISPATCH... SIGILL
Enter fullscreen mode Exit fullscreen mode

The Symptoms:

  • Tasks immediately crash with return code -4
  • SIGILL errors in system logs
  • Works in SequentialExecutor, breaks in LocalExecutor

The Fix:
Set one environment variable:

export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES
echo 'export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES' >> ~/.zshrc
Enter fullscreen mode Exit fullscreen mode

⚠️ Important: Restart Airflow completely so every worker inherits this setting.

Why this works: This disables Objective-C's fork safety checks that conflict with Python's multiprocessing on recent macOS versions.

3. Memory & Network Time-Bombs 🧨

The Problem:

  • Full-list API URLs (e.g., Binance /ticker/24hr without symbol filter)
  • No timeout configuration → hangs forever
  • pandas loading 2MB+ JSON → OOM → SIGKILL

The Symptoms:

  • Tasks hang on large API responses
  • Memory usage spikes then crashes
  • Inconsistent failures depending on data size

The Fix:
Make your requests tiny, scoped, and streamed:

import requests
import pandas as pd
from typing import Dict, Any

def fetch_data_safely(url: str, params: Dict[str, Any] = None) -> pd.DataFrame:
    """Fetch data with proper timeouts and memory management."""

    # Always set timeouts
    response = requests.get(
        url, 
        params=params,
        timeout=(10, 30),  # (connect_timeout, read_timeout)
        stream=True        # Don't load everything into memory
    )
    response.raise_for_status()

    # Process in chunks to avoid OOM
    data = response.json()

    # Filter early, filter often
    if isinstance(data, list) and len(data) > 1000:
        # Only take what you need
        data = data[:1000]

    return pd.DataFrame(data)

# Example: Instead of fetching all tickers
# BAD: requests.get("https://api.binance.com/api/v3/ticker/24hr")
# GOOD: 
df = fetch_data_safely(
    "https://api.binance.com/api/v3/ticker/24hr",
    params={"symbol": "BTCUSDT"}  # Be specific
)
Enter fullscreen mode Exit fullscreen mode

The Complete Fix Checklist ✅

Copy-paste this into your terminal:

# 1. Fix proxy issues
export NO_PROXY="*"
echo 'export NO_PROXY="*"' >> ~/.zshrc

# 2. Fix fork safety
export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES
echo 'export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES' >> ~/.zshrc

# 3. Restart your shell
source ~/.zshrc

# 4. Restart Airflow completely
pkill -f airflow
airflow scheduler &
airflow webserver &
Enter fullscreen mode Exit fullscreen mode

Why These Issues Are So Sneaky

  1. They're macOS-specific - Your code works fine on Linux production
  2. Silent failures - No obvious error messages
  3. Intermittent - Sometimes work, sometimes don't
  4. Environment-dependent - Different macOS/Python versions behave differently

Testing Your Fix

Create a simple test DAG:

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime
import requests

def test_http_request():
    """Test that HTTP requests work properly."""
    response = requests.get("https://httpbin.org/json", timeout=10)
    print(f"Success! Status: {response.status_code}")
    return response.json()

dag = DAG(
    'test_macos_fix',
    start_date=datetime(2024, 1, 1),
    schedule_interval=None,
    catchup=False
)

test_task = PythonOperator(
    task_id='test_http',
    python_callable=test_http_request,
    dag=dag
)
Enter fullscreen mode Exit fullscreen mode

If this runs successfully with LocalExecutor, you're golden! 🎉

🏆 Mission Accomplished: From Zero to Hero

These three nuclear-grade fixes completely transformed my Airflow development experience on macOS. No more:

  • 🚫 Mysterious hangs that steal your afternoon
  • 🚫 Zombie tasks haunting your scheduler
  • 🚫 3 AM debugging sessions fueled by caffeine and desperation
  • 🚫 "It works on my Linux server but not my Mac" embarrassment

🎯 The beauty of this approach: These are environment-level fixes—set them once, and every single DAG benefits automatically. It's like giving your entire Airflow installation a permanent immunity boost! 💉

📊 My before/after stats:

As you can see, the image below shows the consecutive 6 failures to the Consecutive 5 Success Executions

Before and After Dag Summary

As you can see image below from previous Loooong Executions, which failed to Short and Successful Executions

Before and After Deep Down

🤝 Join the Victory Squad!

Have you battled other macOS-specific Airflow demons that I missed? Found creative workarounds for edge cases? Drop them in the comments! 👇

Let's build the ultimate macOS Airflow troubleshooting arsenal together. Every shared fix saves countless developers from pulling their hair out!

🔥 Bonus points for:

  • Obscure M1/M2 chip issues 🍎
  • Obscure Intel Mac chip issues 🍎
  • Docker-related macOS quirks 🐳
  • VS Code + Airflow integration problems 💻
  • PostgreSQL connection hiccups 🐘

🎉 Spread the Victory!

Found this guide life-saving?

  • 💖 Smash that heart button
  • 🔄 Share with fellow developers drowning in Airflow issues
  • 💾 Bookmark for your future self (trust me, you'll need it)
  • 🗣️ Tag that colleague who's been struggling with macOS Airflow

Remember: Every developer you help with this guide is one less person cursing at their terminal at 2 AM! 😴➡️😊

May your DAGs run smoothly and your tasks never zombie! 🧟‍♂️❌

Top comments (0)