How Docker networking broke checkout under load: a container ecommerce infrastructure case study

Checkout was timing out at 900 req/s, and it had nothing to do with CPU

A marketplace client came to us with a scary but familiar symptom: checkout p95 latency jumping from 280ms to over 2.1 seconds during traffic spikes. Their instinct was to throw more containers at it. That made things worse. Here's what was actually going on, and how we fixed it without touching a line of application code.

Link to the full writeup: Docker networking broke checkout under load

The setup

PHP monolith, containerized about a year prior, running on a single host via Docker Compose: app containers, Redis, a worker queue, behind a managed load balancer. 40k DAU, peak ~900 req/s. It ran fine for six months. Then flash sales started producing "the site is freezing" tickets.

The team's first move was scaling app containers, assuming CPU/memory pressure. Query times on Postgres stayed stable at 8-14ms the whole time, so the database wasn't the culprit either. The bottleneck was hiding in the network layer between containers, a place default tooling barely gives you visibility into.

What the audit found

Three compounding issues, none fatal alone:

1. Default bridge network overhead. Every inter-container hop (app to Redis, app to Postgres, app to search) was going through userland proxying on the default bridge. We measured ~1.8ms added latency per hop. At 900 req/s with 3-4 internal calls per request, that adds up fast.

2. Conntrack table exhaustion. nf_conntrack_max was still at the kernel default of 65,536. Short-lived Redis/Postgres connections churned through the table during spikes, filled it, and the kernel silently started dropping packets. Nobody noticed because syslog wasn't shipped anywhere useful.

3. DNS resolution overhead. Docker's embedded DNS (127.0.0.11) was resolving service names on every new connection instead of the app caching results. Under load, with connection churn, lookups started queuing behind each other.

None of these show up with 10 test users in staging. All three show up hard with 900 real ones under concurrent load.

What we didn't do

We ruled out "just move to Kubernetes." It wouldn't have fixed anything here; K8s has its own version of the same problems (CNI choice, kube-proxy mode, CoreDNS caching). Swapping orchestrators without fixing the root cause just relocates it.

We also ruled out rewriting the app to reduce internal calls. The call pattern was normal. The network layer needed to handle it efficiently, not the other way around.

The fix, in four layers

Kernel tuning for conntrack:

net.netfilter.nf_conntrack_max = 262144
net.netfilter.nf_conntrack_tcp_timeout_established = 600
net.ipv4.tcp_tw_reuse = 1
net.core.somaxconn = 4096

Applied via /etc/sysctl.d/99-docker-network.conf and sysctl --system. We also started monitoring conntrack table utilization going forward, since a full table fails silently.

Moved internal traffic off the default bridge:

docker network create \
  --driver bridge \
  --opt com.docker.network.bridge.enable_icc=true \
  --opt com.docker.network.driver.mtu=9000 \
  --subnet 172.28.0.0/16 \
  internal-services

App containers, Redis, and the search sidecar moved onto this network, keeping east-west traffic off Docker's default NAT path. The public-facing load balancer stayed on a separate network; no change to external attack surface.

Local DNS caching with dnsmasq as a sidecar:

# dnsmasq.conf
no-resolv
server=127.0.0.11
cache-size=1000
local-ttl=10
neg-ttl=5

A 10-second TTL absorbed connection churn during spikes without causing stale resolution issues when containers got replaced on deploy. We specifically tested that a container restart got picked up within one TTL window.

PgBouncer for connection pooling:

[databases]
marketplace = host=postgres-primary port=5432 dbname=marketplace

[pgbouncer]
pool_mode = transaction
max_client_conn = 2000
default_pool_size = 50

This was sequenced last on purpose. Fewer short-lived connections means fewer conntrack entries and fewer DNS lookups, so it made everything upstream easier once the network layer was already sound.

Takeaway

If your containerized app slows down only under load and CPU/memory graphs look fine, stop scaling containers and go look at conntrack, your bridge driver, and DNS resolution behavior. That's usually where it's actually hiding.

Originally published on binadit.com