Notes from a 3 AM On-Call: Docker Daemon Socket Permission Denials That Survive Group Membership

The Failure Symptom

$ docker ps
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock:
Post "http://%2Fvar%2Frun%2Fdocker.sock/v1.47/containers/json": dial unix /var/run/docker.sock: connect: permission denied

Standard response: add user to docker group. Already done. Still fails. groups output shows docker. ls -l /var/run/docker.sock shows srw-rw---- 1 root docker. Everything looks correct.

Three deploys failed in a row because our ephemeral CI runners couldn't talk to the Docker daemon after provisioning. Build agents reported they were in the docker group, socket permissions were correct, daemon was running—yet every docker build call died with permission denied.

The Red Herring

Every tutorial says:

sudo usermod -aG docker $USER
newgrp docker  # or logout/login

We did that. Verified with id -nG. The docker group was listed. Should work.

Turns out the daemon socket gets recreated on every dockerd restart, and if your init system (systemd, in our case) runs the daemon before certain udev rules fire or if socket activation races with filesystem permissions, the socket can come up with the wrong group ownership.

Worse: if you're running Docker in a VM or inside another container (nested Docker, common in CI), the socket permissions are set at daemon start time based on whatever /etc/group snapshot the daemon sees—which may not reflect the group membership you just added.

The Root Cause & Runnable Fix

The issue has three layers:

1. Group membership isn't inherited by existing processes

After usermod -aG docker $USER, your current shell session still has the old group list. You must:

# Force the current shell to reload group membership
exec su -l $USER
# or just
newgrp docker

But in automated CI runners or systemd services, there's no interactive shell to reload.

2. The socket can be owned by root:root instead of root:docker

If dockerd starts before the docker group exists (race condition on boot or in ephemeral VMs), it falls back to root:root with 0600 permissions.

Fix: explicitly set socket group in /etc/docker/daemon.json:

{
  "group": "docker",
  "live-restore": true
}

Restart daemon:

sudo systemctl restart docker
ls -l /var/run/docker.sock
# Now: srw-rw---- 1 root docker

3. Nested Docker or rootless mode requires host socket forwarding

If you're running Docker inside a container (CI agent in Docker, docker-in-docker), mounting /var/run/docker.sock isn't enough. The container's user must have matching UID/GID or CAP_DAC_OVERRIDE.

Production-hardened docker-compose.yml for CI agents:

version: "3.9"
services:
  ci-agent:
    image: my-ci-agent:latest
    user: "1000:999"  # UID 1000, GID 999 (docker group on host)
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:rw
    environment:
      DOCKER_HOST: unix:///var/run/docker.sock
    group_add:
      - "999"  # Explicitly add docker GID

Key fix: group_add: ["999"] ensures the container process gets the docker group even if the image doesn't have it baked in.

The 3-Line Verification Script

We added this to our provisioning playbook:

#!/bin/bash
set -euo pipefail

# Verify daemon is reachable without sudo
if ! docker info >/dev/null 2>&1; then
  echo "FATAL: Docker daemon unreachable. Socket: $(ls -l /var/run/docker.sock)"
  echo "Current groups: $(id -nG)"
  exit 1
fi

echo "✓ Docker daemon accessible"

Runs immediately after provisioning. If it fails, we know the socket setup is broken before any builds start.

Invariant Rule

Never assume group membership propagates automatically. On every Docker daemon restart or VM reboot:

1. Verify /var/run/docker.sock ownership is root:docker with 0660.
2. If using containers that need Docker access, pass group_add: [<docker-gid>] explicitly in compose or --group-add in docker run.
3. In CI pipelines, run docker info as a health check before any build step.

We eliminated 90% of our "Docker permission denied" oncall pages by enforcing these three checks in our VM image baking process.

No more 3 AM socket archaeology.