DEV Community

Denis Morozov
Denis Morozov

Posted on

The Redis That Timed Out: A Docker Subnet Overlap Hiding in Plain Sight

#docker #redis #networking #devops

Postgres worked. Redis didn't. Same host, same Docker, same iptables, two containers started four minutes apart — and one of them refused to answer. This is the story of a TimeoutError that had nothing to do with Redis, and the one-line routing fact that explained everything.

The setup

My dev stack for a trading-desk API lives on an Alpine Linux VM under Windows 11 + Hyper-V. Inside Alpine, Docker runs two containers: Postgres and Redis, both port-published, both reachable from the host at 192.168.240.2. The Python app talks to Postgres happily. Then I wired up Redis for cache-aside and got:

redis.exceptions.TimeoutError: Timeout connecting to server
Enter fullscreen mode Exit fullscreen mode

Not connection refused. Timeout. That distinction is the whole plot.

  • Refused = the packet arrived and something said "no." Wrong port, dead service.
  • Timeout = the packet left and nothing ever came back. It's being black-holed.

So the SYN was going somewhere it shouldn't. But where? Redis was up (docker exec redis redis-cli ping → PONG), the DNAT rule looked identical to the Postgres one, and Postgres — same host, same IP — was fine.

The clue was in ifconfig

Two interfaces on the Docker host, side by side:

docker0   inet addr:172.17.0.1   Mask:255.255.0.0     -> 172.17.0.0/16   (Redis lives here)
eth0      inet addr:172.17.6.90   Mask:255.255.240.0   -> 172.17.0.0/20   (Hyper-V network)
Enter fullscreen mode Exit fullscreen mode

Read those two subnets again. docker0 claims 172.17.0.0/16. The Hyper-V-assigned eth0 sits on 172.17.0.0/20. They overlap. And Redis's container IP, 172.17.0.2, falls inside both.

When a client hit 192.168.240.2:6379, iptables DNAT'd the packet to 172.17.0.2 — correctly. Then the kernel had to route to 172.17.0.2, and it found two matching routes:

  • 172.17.0.0/20 via eth0
  • 172.17.0.0/16 via docker0

The kernel picks the longest prefix. /20 is more specific than /16, so eth0 wins. The packet was shoved out onto the Hyper-V LAN, hunting for a 172.17.0.2 that doesn't exist out there — instead of down docker0 to the container two hops away. Black hole. Timeout.

One command nailed it:

# ip route get 172.17.0.2
172.17.0.2 dev eth0  src 172.17.6.90
Enter fullscreen mode Exit fullscreen mode

dev eth0. Not docker0. There's your bug, printed in one line.

Why Postgres got away with it

Here's the part that makes it feel like magic until you see it: Postgres never did anything special. It just happened to be started with docker compose, and Redis with a bare docker run.

Compose silently creates a dedicated user-defined bridge network per project, and those are allocated from Docker's address pool starting at 172.18.0.0/16 — the 172.17 range is reserved for the built-in docker0. So Postgres landed on 172.18.x, which collides with nothing. Redis, via docker run, fell onto docker0/172.17 — straight into the Hyper-V overlap.

Nobody configured this. The outcome hinged entirely on docker compose vs docker run. It's a latent trap that only springs when your host network and Docker's defaults happen to share a prefix — and Hyper-V's Default Switch re-rolls its subnet on almost every reboot, drawing from exactly the ranges Docker likes: 172.16.0.0/12 and 192.168.0.0/16.

The fix: evict Docker from the danger zone

The robust move isn't to dodge one subnet — it's to relocate all of Docker into the one big private block Hyper-V never touches: 10.0.0.0/8.

1. Pin Docker's pools in /etc/docker/daemon.json:

{
  "bip": "10.200.0.1/24",
  "default-address-pools": [
    { "base": "10.201.0.0/16", "size": 24 }
  ]
}
Enter fullscreen mode Exit fullscreen mode

bip moves the default docker0 bridge; default-address-pools moves every Compose/user network. Nothing in 10.200/10.201 can ever share a prefix with a 172.x/192.168.x Hyper-V interface — even after the next reboot re-roll.

2. Make it deterministic by folding Redis into the same Compose file with a fixed subnet, so it stops being the odd docker run container that started this mess:

services:
  db:
    image: postgres:17
    networks: [opendesk]
    # ...
  redis:
    image: redis:7
    ports: ["6379:6379"]
    networks: [opendesk]

networks:
  opendesk:
    name: opendesk
    driver: bridge
    ipam:
      config:
        - subnet: 10.201.10.0/24
Enter fullscreen mode Exit fullscreen mode

3. Restart and rebuild (existing networks keep their old subnets until recreated):

rc-service docker restart
docker compose down && docker compose up -d
Enter fullscreen mode Exit fullscreen mode

4. Verify the route moved — the exact test that exposed the bug:

ip route get $(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' redis)
#  -> ... dev br-xxxxxxxx    (fixed)   instead of   dev eth0   (broken)
Enter fullscreen mode Exit fullscreen mode

And the client:

Connected successfully to Async Redis!
Enter fullscreen mode Exit fullscreen mode

The takeaway

Two things I'm keeping:

  1. Timeout vs refused is a routing hint, not a service hint. When a service is provably up but the client times out, stop poking the service and start asking where the packet actually goes. ip route get <ip> is the fastest question you can ask.
  2. Docker's defaults live inside Hyper-V's range. On any Windows/Hyper-V host, pin daemon.json to 10.x on day one. Don't wait for the overlap to bite — it only bites after a reboot re-rolls the subnet, which is the worst possible time to be surprised.

The docker exec ... PONG lied to me for a good ten minutes. It was answering over the container's own loopback, a network path the outside world never uses. The truth was one ip route get away the whole time.

Top comments (0)