DEV Community

Cover image for Notes from a 2 AM On-Call: Why Our Android CI Containers Silently Froze (And the iptables Rule That Fixed It)
Hannah Schmidt
Hannah Schmidt

Posted on

Notes from a 2 AM On-Call: Why Our Android CI Containers Silently Froze (And the iptables Rule That Fixed It)

#docker #linux #devops #productivity

Notes from a 2 AM On-Call: Why Our Android CI Containers Silently Froze (And the iptables Rule That Fixed It)

The Failure Symptom 

2026-10-18 02:14:33 [ERROR] Emulator health check timeout (container: android-worker-3)
2026-10-18 02:14:48 [ERROR] ADB connection refused: 5555 (container: android-worker-7)
2026-10-18 02:15:12 [WARN] Docker exec into android-worker-3: process hangs, no response
Enter fullscreen mode Exit fullscreen mode

Our CI pipeline running budtmo/docker-android for parallel Appium test execution started dropping 40% of jobs at 2 AM UTC when concurrent load spiked from 8 to 24 containers. No crash logs. No OOM kills. Containers appeared healthy in docker ps but were zombies—ADB refused connections, noVNC showed frozen emulator screens, and docker exec commands hung indefinitely.

The Red Herring

Standard Android emulator docs and docker-android examples suggest exposing ADB ports with -p 5555:5555 for host access. Our compose stack did exactly that:

services:
  android-worker:
    image: budtmo/docker-android:emulator_11.0
    ports:
      - "5555-5578:5555"  # ADB port range for 24 workers
    privileged: true
Enter fullscreen mode Exit fullscreen mode

Under light load (< 10 concurrent emulators), this worked. At scale, containers froze without kernel panics or resource exhaustion. strace on the frozen emulator process revealed it was stuck in futex(FUTEX_WAIT) inside the Android emulator's internal lock—but why?

The Root Cause & Runnable Fix

The breakthrough came from a packet capture on the Docker bridge: external scanners were hitting our exposed ADB port range. When an external TCP SYN landed on an emulator's ADB port during emulator boot (a 15-second window), the emulator's ADB daemon entered a race condition in its internal socket accept loop, causing a deadlock in libc's mutex.

budtmo/docker-android containers don't need published ADB ports for internal orchestration—the MCP server and video recorder connect over the Docker bridge network. Publishing them created a 24-port attack surface.

The 3-line fix (added to docker-compose.yml):

services:
  android-worker:
    image: budtmo/docker-android:emulator_11.0
    networks:
      - android_internal
    # ports: []  # Removed: no published ports
    environment:
      - EMULATOR_ARGS="-no-snapshot -no-audio -no-window"

networks:
  android_internal:
    driver: bridge
    internal: false  # Allow outbound, block inbound
Enter fullscreen mode Exit fullscreen mode

Plus one iptables rule on the host to explicitly drop external traffic to the bridge subnet:

iptables -I DOCKER-USER -i eth0 ! -s 10.0.0.0/8 -j DROP
Enter fullscreen mode Exit fullscreen mode

After deploying, we ran 48-hour soak tests at 32 concurrent containers: zero freezes.

Invariant Rule

Never publish container ports unless the host needs direct access. Internal service-to-service communication over Docker bridge networks eliminates port conflicts, reduces attack surface, and avoids kernel-level port forwarding race conditions that manifest only under concurrency.

For production Android CI with budtmo/docker-android: use internal networks, enforce privileged: false where possible, and place a reverse proxy (Traefik, Nginx) in front of noVNC if browser access is required—never expose raw emulator ports to 0.0.0.0.

Top comments (0)