DEV Community: kamal namdeo

nginx Event Loop — Complete Lifecycle Reference

kamal namdeo — Sat, 27 Jun 2026 15:21:43 +0000

nginx Event Loop — Complete Lifecycle Reference

A precise, bottom-up reference covering every buffer, syscall, interrupt, and data movement from the moment a TCP packet hits the NIC to the moment a response is sent back. Two concurrent users are used throughout as a concrete example.

Foundations — fd and Socket
Hardware Layer — NIC, DMA, Interrupts
Kernel Structures and All Buffers
epoll — How the Worker Waits Efficiently
nginx Startup Sequence
Complete Request Lifecycle — Two Concurrent Users
What Happens While Worker is Busy
All Buffers — Master Reference
All Syscalls — Master Reference
Failure Modes

1. Foundations

1.1 Everything is a File

Linux's core philosophy: every I/O resource — files on disk, network connections, pipes, terminals, devices — is represented as a file. This means one unified API (read, write, close) works on all of them. The kernel manages the actual resource. Your process holds a token.

1.2 File Descriptor (fd)

A file descriptor is just an integer. It is a per-process token that refers to a kernel-managed resource. The kernel maintains a table per process called the fd table — a simple array where the index is the fd and the value is a pointer into the kernel.

Process fd table:
┌─────┬───────────────────────────────┐
│ fd  │ points to                     │
├─────┼───────────────────────────────┤
│  0  │ stdin                         │
│  1  │ stdout                        │
│  2  │ stderr                        │
│  3  │ listen socket (nginx)         │
│  5  │ User A client connection      │
│  6  │ User B client connection      │
│ 12  │ backend connection for User A │
│ 13  │ backend connection for User B │
└─────┴───────────────────────────────┘

0, 1, 2 are always pre-assigned. Application fds start from 3 upward. The fd is meaningless on its own. It only means something when passed to a syscall — the kernel uses it to look up the real resource.

1.3 Socket

A socket is the kernel's internal data structure representing one end of a network connection. Created when your process calls socket(). Lives entirely in kernel RAM. Your process never touches it directly.

Socket struct (kernel RAM):
┌────────────────────────────────────┐
│ local_ip:    192.168.1.10          │
│ local_port:  80                    │
│ remote_ip:   203.0.113.5           │
│ remote_port: 54231                 │
│ state:       ESTABLISHED           │
│ recv_buffer: [ ...incoming bytes ] │
│ send_buffer: [ ...outgoing bytes ] │
│ tcp_state_machine: ...             │
│ timers, seq numbers, window size   │
└────────────────────────────────────┘

1.4 How fd and Socket Relate

PROCESS (user space)            KERNEL (kernel space)

fd table                        Open file table         Socket struct
┌──────┐                        ┌─────────────┐         ┌──────────────┐
│ fd=5 │ ─────────────────────► │ file entry  │ ──────► │ socket {     │
└──────┘                        │ (flags,     │         │  recv buffer │
                                │  offset)    │         │  send buffer │
                                └─────────────┘         │  state ...   │
                                                        └──────────────┘

This two-level indirection exists so that two processes (parent/child after fork) can share the same socket with different fd numbers.

2. Hardware Layer

2.1 NIC (Network Interface Card)

A physical chip on the motherboard (or PCIe slot). It receives electrical/optical signals from the network cable, decodes them into bytes, and writes them into RAM. The CPU has no involvement in receiving the bytes themselves — the NIC does it autonomously.

2.2 DMA (Direct Memory Access)

DMA is the mechanism that lets the NIC write bytes directly into RAM without involving the CPU. The kernel sets up a region of RAM called the NIC Ring Buffer at boot time and tells the NIC its address. The NIC writes incoming packets straight there. Zero CPU cycles spent moving bytes.

2.3 NIC Ring Buffer

A circular array of fixed-size slots in kernel RAM, allocated at NIC driver initialization.

NIC Ring Buffer (in kernel RAM):
┌──────────┬──────────┬──────────┬──────────┐
│ slot 0   │ slot 1   │ slot 2   │ slot 3   │  ← NIC writes here via DMA
│ pkt data │ pkt data │ (empty)  │ (empty)  │
└──────────┴──────────┴──────────┴──────────┘
      ▲                     ▲
   NIC writes            kernel reads (via interrupt)

Size: 256 to 4096 slots typically. Each slot holds one packet (up to 1500 bytes for standard Ethernet MTU).
Who writes: NIC hardware via DMA.
Who reads: Kernel interrupt handler (ISR).
What happens when full: Packets are dropped. Visible via ethtool -S eth0 | grep drop or ip -s link.

2.4 Hardware Interrupt (IRQ — Interrupt Request)

When the NIC finishes writing one or more packets into the Ring Buffer via DMA, it sends an electrical signal on a dedicated wire to the CPU. This signal is an IRQ (Interrupt Request).

The CPU responds by:

Finishing the current instruction (not mid-instruction).
Saving its register state (so it can resume later).
Looking up the interrupt number in the IDT (Interrupt Descriptor Table) — a kernel table mapping interrupt numbers to handler functions.
Jumping to the handler function.
Restoring state and resuming what it was doing before.

This entire sequence takes microseconds. The user process (nginx worker) does not know it happened.

2.5 ISR (Interrupt Service Routine)

The kernel function registered to handle NIC interrupts. When called:

Reads packet(s) from the NIC Ring Buffer.
Parses the Ethernet frame, IP header, TCP header.
Identifies which socket this packet belongs to (by matching src_ip:src_port:dst_ip:dst_port).
Copies payload bytes into that socket's recv buffer in kernel RAM.
Updates TCP sequence numbers, ACKs, window size.
Marks that socket's fd as ready (EPOLLIN) in the epoll ready list.
If the process was sleeping in epoll_wait, wakes it up.
Returns. CPU resumes what it was doing.

3. Kernel Structures and All Buffers

3.1 SYN Queue (Incomplete Connections Queue)

When a client sends a SYN packet to initiate a TCP connection:

Kernel puts this half-open connection into the SYN Queue.
Sends back SYN-ACK.
Waits for the client's final ACK to complete the handshake.

Tunable:  /proc/sys/net/ipv4/tcp_max_syn_backlog
Default:  128 (older kernels) / 1024 (newer)

Who writes: Kernel TCP stack on receiving SYN.
Who reads: Kernel TCP stack on receiving final ACK — moves entry to Accept Queue.
What happens when full: Kernel drops incoming SYNs silently. Client retries. Appears as connection timeout.

SYN flood attacks fill this queue deliberately. tcp_syncookies is the mitigation.

3.2 Accept Queue (Complete Connections Queue)

After the 3-way handshake completes, the connection moves from SYN Queue to the Accept Queue. These connections are fully established and waiting for your process to call accept().

Effective size = min(listen() backlog argument, net.core.somaxconn)

net.core.somaxconn default: 128 (kernel < 5.4) / 4096 (kernel >= 5.4)
nginx default backlog:      511

So on modern Linux: min(511, 4096) = 511 slots

Who writes: Kernel TCP stack when handshake completes.
Who reads: Your process via accept() syscall.
What happens when full: Kernel drops incoming SYNs. Client sees connection timeout.

3.3 Socket Receive Buffer

Per-connection buffer in kernel RAM. Incoming payload bytes (from ISR) land here after the TCP header is stripped.

Default size: ~208 KB  (/proc/sys/net/core/rmem_default = 212992)
Maximum size: 128 MB   (/proc/sys/net/core/rmem_max = 134217728)
Auto-tuning:  Linux adjusts size dynamically between rmem_default and rmem_max

Who writes: Kernel ISR (on hardware interrupt).
Who reads: Your process via read() or recv() syscall.
What happens when full: Kernel stops sending TCP ACKs to the sender. Sender's TCP stack sees its window shrink to zero and stops sending. This is TCP flow control. No data loss — sender just waits.

3.4 Socket Send Buffer

Per-connection buffer in kernel RAM. When your process calls write(), bytes go here. Kernel TCP stack drains this buffer by constructing TCP packets and sending them out via the NIC.

Default size: ~208 KB  (/proc/sys/net/core/wmem_default = 212992)
Maximum size: 128 MB   (/proc/sys/net/core/wmem_max = 134217728)

Who writes: Your process via write() or send() syscall.
Who reads: Kernel TCP stack (constructs packets and hands to NIC).
What happens when full: write() blocks (in blocking mode) or returns EAGAIN (in non-blocking mode). nginx uses non-blocking — it registers EPOLLOUT on the fd and retries when kernel signals the buffer has drained.

3.5 nginx Worker Memory (User Space)

When the worker calls read(fd, buf, size), the kernel copies bytes from the socket recv buffer (kernel RAM) into buf (process RAM). This is the boundary crossing — from kernel space to user space.

The worker parses HTTP in this memory and builds request_context objects:

request_context {
  client_fd:   5           ← incoming connection fd
  backend_fd:  12          ← outgoing connection fd
  state:       WAITING_BACKEND
  method:      GET
  path:        /api/users
  headers:     { host, auth, content-type ... }
  body:        (if POST)
  response:    (filled later)
}

This is the only place that links a client fd to its backend fd. The kernel has no concept of this pairing.

4. epoll

4.1 What is epoll

epoll is a Linux kernel subsystem that lets a single process monitor thousands of fds simultaneously without scanning all of them. It is the engine behind nginx's ability to handle 10,000+ concurrent connections in a single thread.

Three kernel structures make up epoll:

4.2 epoll Instance

Created via epoll_create1(). Returns an fd that represents the epoll instance itself. The instance contains:

Interest List — which fds to watch and what events.
Ready List — which fds currently have events pending.

4.3 Interest List

A red-black tree (O(log n) insertion/deletion) inside the kernel. Each node represents one watched fd and what event to watch for.

Interest list (red-black tree in kernel RAM):
┌─────────────────────────────────┐
│ fd=3,  events=EPOLLIN           │  ← new client connections
│ fd=5,  events=EPOLLIN           │  ← User A client data
│ fd=6,  events=EPOLLIN           │  ← User B client data
│ fd=12, events=EPOLLIN|EPOLLOUT  │  ← backend for User A
│ fd=13, events=EPOLLIN|EPOLLOUT  │  ← backend for User B
└─────────────────────────────────┘

Modified by: epoll_ctl(epfd, EPOLL_CTL_ADD/MOD/DEL, fd, event) — always called by worker.

4.4 Ready List

A doubly-linked list inside the kernel. When the ISR marks an fd as having data, it adds that fd to this list. epoll_wait() reads and clears this list.

Written by: Kernel ISR (on hardware interrupt) — completely independent of worker state.
Read by: Worker via epoll_wait().

This separation is critical: the ready list is updated even when the worker is busy. Nothing is lost.

4.5 Edge Triggered vs Level Triggered

nginx uses Edge Triggered (EPOLLET).

Mode	epoll_wait returns fd when	Risk if you don't drain fully
Level Triggered (default)	fd has data (keeps returning until empty)	None — will re-notify
Edge Triggered (EPOLLET)	fd transitions from no-data → has-data	Will NOT re-notify until new data arrives

Edge triggered means nginx must loop read() until it gets EAGAIN on every wakeup. If it reads only once and goes back to epoll_wait, remaining data in the buffer will never trigger a new event and that connection stalls.

Same applies to accept() — nginx must loop accept() until EAGAIN to drain the entire accept queue every time fd=3 fires.

4.6 epoll Syscalls

Syscall	What it does
`epoll_create1(flags)`	Creates epoll instance, returns epfd
`epoll_ctl(epfd, op, fd, event)`	ADD/MOD/DEL an fd in the interest list
`epoll_wait(epfd, events[], maxevents, timeout)`	Sleep until events ready; returns array of ready fds

timeout = -1 means sleep indefinitely. The process is removed from the CPU scheduler run queue. Zero CPU consumed while sleeping.

5. nginx Startup Sequence

Master Process — Before Workers Start

nginx has two process roles: one master process and one or more worker processes.

Master process:

Runs as root.
Reads nginx.conf. Determines worker_processes count — if set to auto, nginx reads the number of CPU cores on the machine (nproc) and spawns that many workers.
Calls socket(), bind(), listen() on port 80 once, before forking. This is deliberate — binding requires root privileges. Workers never need to bind themselves.
Calls fork() once per worker. Each forked child inherits fd=3 (the listen socket) automatically. This is how all workers share the same port without any of them needing root.
After forking, master drops into a supervision loop — watches workers, restarts any that crash, handles signals (nginx -s reload, nginx -s stop).
Master itself never handles a single HTTP request.

After fork, each worker independently:

Calls epoll_create1() — gets its own epoll instance.
Calls epoll_ctl(ADD, fd=3, EPOLLIN) — registers the inherited listen socket.
Calls epoll_wait() — sleeps, waiting for connections.

Master Process (root)
  socket() + bind() + listen() on fd=3
        │
        ├── fork() → Worker 1 (inherits fd=3) → own epoll → epoll_wait
        ├── fork() → Worker 2 (inherits fd=3) → own epoll → epoll_wait
        ├── fork() → Worker 3 (inherits fd=3) → own epoll → epoll_wait
        └── fork() → Worker 4 (inherits fd=3) → own epoll → epoll_wait

All workers call accept() on the same fd=3.
Kernel distributes incoming connections across workers.
Master just watches. Restarts crashed workers.

When multiple workers call accept() on the same fd=3 simultaneously, the kernel ensures only one worker gets each connection — no duplication. nginx also sets accept_mutex (older versions) or relies on EPOLLEXCLUSIVE flag (modern Linux) to avoid all workers waking up for every single connection — known as the thundering herd problem.

This runs once when nginx starts. The worker does not handle any requests until this is complete.

1.  socket(AF_INET, SOCK_STREAM, 0)
    → kernel creates socket struct
    → returns fd=3 (listen socket)

2.  setsockopt(fd=3, SO_REUSEADDR, 1)
    → allows reuse of port 80 after restart
    → avoids "Address already in use" error

3.  bind(fd=3, {ip=0.0.0.0, port=80})
    → registers this socket as owner of port 80 with kernel
    → kernel now routes port 80 packets to fd=3

4.  listen(fd=3, backlog=511)
    → kernel creates SYN Queue and Accept Queue for this socket
    → accept queue limit = min(511, somaxconn)
    → kernel begins accepting TCP handshakes even before accept() is called

5.  epoll_create1(EPOLL_CLOEXEC)
    → creates epoll instance
    → returns epfd=4

6.  epoll_ctl(epfd=4, EPOLL_CTL_ADD, fd=3, {EPOLLIN|EPOLLET})
    → registers listen socket in interest list
    → "wake me up when a new connection is ready to accept"

7.  epoll_wait(epfd=4, events[], 512, -1)
    → worker sleeps. 0 CPU consumed. Waiting for first knock.

6. Complete Request Lifecycle

Cast

Symbol	What
`fd=3`	nginx listen socket
`fd=5`	User A ↔ nginx TCP connection
`fd=6`	User B ↔ nginx TCP connection
`fd=12`	nginx ↔ backend TCP connection for User A
`fd=13`	nginx ↔ backend TCP connection for User B
`epfd=4`	epoll instance

Phase 1 — TCP Handshake (Kernel Only, Worker Sleeping)

Who acts: Kernel TCP stack + NIC. Worker is untouched.

User A browser                    Kernel (port 80)
      │── SYN ──────────────────► NIC receives packet
      │                           ISR runs: puts in SYN Queue
      │◄── SYN-ACK ─────────────  Kernel sends SYN-ACK via NIC
      │── ACK ──────────────────► ISR runs: moves to Accept Queue
      │                           Kernel assigns fd=5 internally
      │                           Marks fd=3 as EPOLLIN in ready list

User B browser (simultaneously)
      │── SYN ──────────────────► Same process
      │◄── SYN-ACK ─────────────
      │── ACK ──────────────────► fd=6 assigned internally
                                  fd=3 still marked EPOLLIN

Buffers touched:

SYN Queue: written (User A SYN), read (on ACK received), written again (User B SYN)
Accept Queue: written twice (one entry per completed handshake)
NIC Ring Buffer: written by NIC DMA, read by ISR

Interrupts: Hardware IRQ fires per packet. ISR runs. Worker untouched.

Phase 2 — Worker Wakes, Accepts Both Connections

Syscalls: epoll_wait, accept, epoll_ctl, fcntl

epoll_wait(epfd=4, events[], 512, -1)
→ returns: [{fd=3, EPOLLIN}]
→ worker wakes up

Worker enters accept loop (must drain fully because EPOLLET):

Iteration 1:
  accept(fd=3)
  → kernel dequeues User A's connection from Accept Queue
  → creates fd=5 for this connection
  → returns fd=5

  fcntl(fd=5, F_SETFL, O_NONBLOCK)
  → sets fd=5 to non-blocking mode

  epoll_ctl(epfd=4, EPOLL_CTL_ADD, fd=5, {EPOLLIN|EPOLLET})
  → fd=5 added to interest list

Iteration 2:
  accept(fd=3)
  → dequeues User B's connection
  → returns fd=6

  fcntl(fd=6, F_SETFL, O_NONBLOCK)
  epoll_ctl(epfd=4, EPOLL_CTL_ADD, fd=6, {EPOLLIN|EPOLLET})

Iteration 3:
  accept(fd=3)
  → returns -1, errno=EAGAIN
  → Accept Queue is empty. Stop looping.

Buffers touched:

Accept Queue: read (two entries dequeued)

Interest list now:

fd=3  EPOLLIN  ← always watching for new clients
fd=5  EPOLLIN  ← User A
fd=6  EPOLLIN  ← User B

Worker calls epoll_wait again. Sleeps.

Phase 3 — HTTP Request Bytes Arrive

Who acts: NIC + Kernel ISR. Worker sleeping.

User A's browser sends GET /api/users HTTP/1.1\r\nHost: ...\r\n\r\n

NIC receives TCP segment
→ DMA writes to NIC Ring Buffer
→ NIC fires hardware IRQ
→ CPU saves state, jumps to ISR

ISR:
  reads packet from NIC Ring Buffer
  parses IP + TCP headers
  matches src_ip:src_port:dst_ip:dst_port → identifies fd=5's socket
  copies HTTP payload into fd=5's recv buffer (kernel RAM)
  updates TCP: ACK sent back to User A's browser
  marks fd=5 as EPOLLIN in epoll ready list
  checks: is worker sleeping in epoll_wait? YES → wakes worker
  returns

User B's GET /api/orders HTTP/1.1 arrives simultaneously or milliseconds later. Same ISR sequence for fd=6.

Buffers touched:

NIC Ring Buffer: written by DMA, read by ISR
fd=5 recv buffer: written by ISR (~hundreds of bytes, the HTTP request)
fd=6 recv buffer: written by ISR

Interrupts: One IRQ per packet per user. ISR runs in microseconds each time.

Phase 4 — Worker Reads and Parses Both Requests

Syscalls: epoll_wait, read

epoll_wait(epfd=4, events[], 512, -1)
→ returns: [{fd=5, EPOLLIN}, {fd=6, EPOLLIN}]
→ both events in one batch

Processing fd=5 (User A):

Worker enters read loop (must drain, EPOLLET):

  Iteration 1:
    read(fd=5, buf, 4096)
    → kernel copies bytes from fd=5 recv buffer (kernel RAM)
      into buf (worker process RAM)
    → returns n=312 (bytes read, size of HTTP request)

  Iteration 2:
    read(fd=5, buf, 4096)
    → returns -1, errno=EAGAIN
    → recv buffer empty. Stop.

Worker parses buf:
  method  = GET
  path    = /api/users
  headers = { Host: ..., Authorization: Bearer xyz }

Worker creates:
  request_context_A {
    client_fd  = 5
    backend_fd = -1       ← not yet
    state      = PARSED
    request    = "GET /api/users..."
  }

Processing fd=6 (User B): Same sequence. request_context_B created with client_fd=6.

Buffers touched:

fd=5 recv buffer: read by worker (now empty)
fd=6 recv buffer: read by worker (now empty)
Worker heap memory: written (two request_context objects)

Kernel boundary crossing: Every read() call copies bytes from kernel RAM → process RAM.

Phase 5 — Worker Opens Backend Connections

Syscalls: socket, fcntl, connect, epoll_ctl

For User A:

socket(AF_INET, SOCK_STREAM, 0)
→ kernel creates new socket struct
→ returns fd=12

fcntl(fd=12, F_SETFL, O_NONBLOCK)
→ makes fd=12 non-blocking

connect(fd=12, {backend_ip, port=8080})
→ kernel initiates TCP handshake with backend (sends SYN)
→ returns immediately with errno=EINPROGRESS
   (handshake happening asynchronously in kernel)

epoll_ctl(epfd=4, EPOLL_CTL_ADD, fd=12, {EPOLLOUT|EPOLLET})
→ "tell me when fd=12 is writable" = when handshake completes

request_context_A.backend_fd = 12
request_context_A.state      = CONNECTING_BACKEND

For User B: Same. Creates fd=13, connects, registers EPOLLOUT.

Interest list now:

fd=3   EPOLLIN          ← new clients
fd=5   EPOLLIN          ← User A client
fd=6   EPOLLIN          ← User B client
fd=12  EPOLLOUT         ← backend for User A (connecting)
fd=13  EPOLLOUT         ← backend for User B (connecting)

Worker calls epoll_wait. Sleeps. 5 fds watched. 0 CPU consumed.

Phase 6 — Backend TCP Handshakes Complete

Who acts: Kernel. Worker sleeping.

Backend sends SYN-ACK for User A's connect (fd=12)
→ NIC receives, IRQ fires, ISR runs
→ Kernel completes handshake, fd=12 state = ESTABLISHED
→ fd=12 is now writable
→ ISR marks fd=12 EPOLLOUT in ready list
→ Worker is sleeping → kernel wakes it

Same for User B (fd=13)

Phase 7 — Worker Forwards Requests to Backend

Syscalls: epoll_wait, write, epoll_ctl

epoll_wait returns: [{fd=12, EPOLLOUT}, {fd=13, EPOLLOUT}]

Handling fd=12 (User A):

Look up context: fd=12 → request_context_A → request = "GET /api/users..."

write(fd=12, "GET /api/users HTTP/1.1\r\n...", len)
→ kernel copies bytes from worker RAM → fd=12 send buffer (kernel RAM)
→ kernel TCP stack will send these bytes to backend as TCP packets
→ returns immediately (non-blocking write)

epoll_ctl(epfd=4, EPOLL_CTL_MOD, fd=12, {EPOLLIN|EPOLLET})
→ switch from watching EPOLLOUT to EPOLLIN
→ "now tell me when backend sends a response back"

request_context_A.state = WAITING_BACKEND

Handling fd=13 (User B): Same. Forwards GET /api/orders.

Buffers touched:

fd=12 send buffer: written by worker (the forwarded HTTP request)
fd=13 send buffer: written by worker
Kernel TCP stack reads send buffers, constructs packets, hands to NIC

Interrupt on outgoing side: When NIC finishes sending packets from send buffer, it fires an IRQ to tell kernel "send buffer slots are free now." No worker involvement.

Worker calls epoll_wait. Sleeps.

Phase 8 — Backend Responds (Possibly Out of Order)

Who acts: Kernel ISR. Worker sleeping.

Backend processes /api/orders faster. Sends response for User B first.

Backend TCP segment for User B arrives at NIC
→ DMA → NIC Ring Buffer
→ IRQ fires → ISR runs
→ Payload copied into fd=13 recv buffer
→ fd=13 marked EPOLLIN in ready list

Later:
Backend TCP segment for User A arrives
→ Same path → fd=12 recv buffer filled
→ fd=12 marked EPOLLIN in ready list

Worker was sleeping → kernel wakes it

Buffers touched:

NIC Ring Buffer: written by DMA, read by ISR
fd=13 recv buffer: written by ISR (User B response body)
fd=12 recv buffer: written by ISR (User A response body)

Phase 9 — Worker Reads Backend Response, Writes to Client

Syscalls: epoll_wait, read, write

epoll_wait returns: [{fd=13, EPOLLIN}, {fd=12, EPOLLIN}]

Note: User B's response came first even though User A connected first. epoll returns what is ready, not what came first.

Handling fd=13 (User B's backend response):

read loop on fd=13:
  read(fd=13, buf, 65536)
  → kernel copies backend response from fd=13 recv buffer → worker RAM
  → returns n bytes

  read(fd=13, buf, 65536)
  → returns EAGAIN (buffer empty, done)

Look up context: fd=13 → request_context_B → client_fd = 6

write(fd=6, buf, n)
→ kernel copies bytes from worker RAM → fd=6 send buffer
→ kernel TCP stack sends packets to User B's browser

epoll_ctl(epfd=4, EPOLL_CTL_DEL, fd=13, NULL)
→ remove backend fd from interest list

close(fd=13)
→ kernel tears down backend connection, frees socket struct and buffers

request_context_B.state = DONE

Handling fd=12 (User A): Same. Reads fd=12 → writes to fd=5 → closes fd=12.

Buffers touched:

fd=13 recv buffer: read by worker (emptied)
fd=12 recv buffer: read by worker (emptied)
fd=6 send buffer: written by worker (User B response going to browser)
fd=5 send buffer: written by worker (User A response going to browser)

Kernel does the rest: TCP stack drains send buffers → NIC → wire → browsers.

Phase 10 — Connection Teardown

Depends on: HTTP version and Connection header.

HTTP/1.0 or Connection: close:

close(fd=5)   → kernel sends FIN to User A, tears down connection, frees socket
close(fd=6)   → same for User B
epoll_ctl(epfd=4, EPOLL_CTL_DEL, fd=5, NULL)
epoll_ctl(epfd=4, EPOLL_CTL_DEL, fd=6, NULL)

HTTP/1.1 keep-alive (default):

fd=5 and fd=6 remain open and registered in epoll interest list
Worker goes back to epoll_wait
If same client sends another request → fd=5 fires EPOLLIN → handled immediately
Kernel has idle timers: if no data for N seconds → close

Worker returns to epoll_wait. Sleeping again. Ready for next event.

7. What Happens While Worker is Busy

Scenario: Worker is processing User A's request. At the same moment, Users C, D, E connect and send requests.

Worker processing [User A]
        │
        │  ← IRQ fires (User C SYN arrives)
        │     ISR runs in microseconds
        │     Kernel completes C's handshake
        │     Adds C to Accept Queue
        │     Marks fd=3 EPOLLIN in ready list
        │     Worker is NOT sleeping → no wakeup sent. That's fine.
        │
        │  ← IRQ fires (User D SYN arrives)
        │     Same. D added to Accept Queue. fd=3 already marked ready.
        │
        │  ← IRQ fires (User E data arrives on existing connection)
        │     Bytes go into fd=E's recv buffer. fd=E marked EPOLLIN.
        │
Worker finishes [User A]
        │
epoll_wait()
        │
→ returns: [{fd=3, EPOLLIN}, {fd=E, EPOLLIN}]
  (fd=3 fires once for all queued connections — edge triggered fires on transition)

Worker drains Accept Queue:
  accept() → User C → fd=7 → register with epoll
  accept() → User D → fd=8 → register with epoll
  accept() → EAGAIN → done

Worker reads fd=E
Worker calls epoll_wait. Sleeps.

Key principle: The ready list is written by ISR independently of worker state. The worker sees everything accumulated while it was busy, on its very next epoll_wait call.

8. All Buffers — Master Reference

#	Buffer	Location	Default Size	Written By	Read By	Full Behaviour
1	NIC Ring Buffer	NIC / kernel RAM (DMA region)	256–4096 packet slots	NIC hardware via DMA	Kernel ISR	Packets dropped, visible via ethtool
2	SYN Queue	Kernel RAM	~128–1024 slots	Kernel TCP stack (on SYN)	Kernel TCP stack (on ACK)	SYN drops, connection timeout
3	Accept Queue	Kernel RAM	min(backlog, somaxconn) ≈ 511	Kernel TCP stack (post-handshake)	Process via `accept()`	SYN drops, connection timeout
4	Socket Recv Buffer (client fd)	Kernel RAM	208 KB default, 128 MB max	Kernel ISR (per interrupt)	Process via `read()`	TCP flow control, sender throttled
5	Socket Send Buffer (client fd)	Kernel RAM	208 KB default, 128 MB max	Process via `write()`	Kernel TCP stack → NIC	`write()` returns EAGAIN (non-blocking)
6	Worker Heap (request_context)	Process RAM	Unbounded (heap)	Process (after `read()` + parse)	Process (during response handling)	OOM if too many large requests
7	Socket Recv Buffer (backend fd)	Kernel RAM	208 KB default, 128 MB max	Kernel ISR (backend response)	Process via `read()`	TCP flow control
8	Socket Send Buffer (backend fd)	Kernel RAM	208 KB default, 128 MB max	Process via `write()`	Kernel TCP stack → NIC	`write()` returns EAGAIN

Data movement order for one request:

NIC Ring Buffer
  → (ISR) → Socket Recv Buffer [client fd]
  → (read syscall) → Worker Heap
  → (write syscall) → Socket Send Buffer [backend fd]
  → (kernel TCP) → NIC → wire → backend

Backend response:
NIC Ring Buffer
  → (ISR) → Socket Recv Buffer [backend fd]
  → (read syscall) → Worker Heap
  → (write syscall) → Socket Send Buffer [client fd]
  → (kernel TCP) → NIC → wire → client browser

9. All Syscalls — Master Reference

Syscall	Called By	When	What Kernel Does
`socket()`	Worker	Startup + per backend conn	Allocates socket struct, returns fd
`setsockopt()`	Worker	Startup	Sets socket options (SO_REUSEADDR etc.)
`bind()`	Worker	Startup	Claims port 80 for this process
`listen()`	Worker	Startup	Creates SYN + Accept queues
`accept()`	Worker	On fd=3 EPOLLIN	Dequeues one entry from Accept Queue, returns new fd
`fcntl(F_SETFL, O_NONBLOCK)`	Worker	After accept/socket	Sets fd to non-blocking mode
`epoll_create1()`	Worker	Startup	Creates epoll instance
`epoll_ctl(ADD)`	Worker	After accept/socket	Adds fd to interest list
`epoll_ctl(MOD)`	Worker	After connect/read	Changes watched events (EPOLLIN ↔ EPOLLOUT)
`epoll_ctl(DEL)`	Worker	Before close	Removes fd from interest list
`epoll_wait()`	Worker	After each batch	Sleeps until events; returns ready fd list
`connect()`	Worker	Per upstream conn	Initiates TCP handshake (non-blocking, returns EINPROGRESS)
`read()`	Worker	On EPOLLIN	Copies bytes from socket recv buffer → process RAM
`write()`	Worker	After parsing/response	Copies bytes from process RAM → socket send buffer
`close()`	Worker	After response sent	Sends FIN, frees socket struct and all buffers

10. Failure Modes

10.1 Accept Queue Full

Cause: Connections arriving faster than worker calls accept().
Symptom: Client sees Connection timed out (kernel drops SYN silently).
Fix: Increase net.core.somaxconn, increase nginx backlog, add more workers.

10.2 Event Loop Starvation

Cause: Worker doing heavy CPU computation or blocking syscall (sync file I/O, blocking DB call) for one connection.
Symptom: All other connections queue up. Latency spikes for everyone.
Fix: Never block the event loop. Offload CPU-heavy work to thread pool (aio threads in nginx). Use async database drivers.

10.3 Send Buffer Full (Slow Client)

Cause: Client reading response slowly. fd=5 send buffer fills up.
Symptom: write(fd=5) returns EAGAIN. Nginx registers EPOLLOUT on fd=5 and comes back to write more when buffer drains.
Fix: nginx handles this internally. Tune send_timeout to close connections from very slow clients.

10.4 Recv Buffer Full (Slow Worker)

Cause: Data arriving faster than worker reads it.
Symptom: TCP flow control kicks in. Sender throttled. No data loss.
Fix: Usually not a problem. If persistent, check for event loop blocking.

10.5 fd Exhaustion

Cause: Too many open connections. Each costs one fd.
Symptom: accept() returns EMFILE (too many open files).
Fix: ulimit -n 65536 in OS. worker_connections 10240 in nginx config.

Quick Mental Model

INTERNET
   │ raw bytes
   ▼
NIC Ring Buffer  ← DMA (no CPU)
   │ IRQ fires
   ▼
Kernel ISR  ← runs in microseconds, interrupts everything
   │ copies bytes
   ▼
Socket Recv Buffer  ← kernel RAM, per fd
   │ waits here until worker is ready
   ▼
epoll Ready List  ← kernel marks fd ready
   │ wakes worker if sleeping
   ▼
Worker (epoll_wait returns)
   │ read() → copies to process RAM
   │ parse HTTP
   │ write() to backend fd → send buffer
   │
   ▼
Socket Send Buffer  ← kernel RAM, per fd
   │ kernel drains via TCP
   ▼
NIC → wire → backend

Backend response takes same path in reverse.
Worker links client fd ↔ backend fd via request_context in its own memory.
Kernel has no concept of this pairing.
Everything the worker does is non-blocking.
When waiting, worker is off the CPU run queue. Zero cycles consumed.

Bulkhead Pattern - Go

kamal namdeo — Sun, 14 Jun 2026 15:22:59 +0000

Go Bulkhead Pattern — Complete Reference

What is the Bulkhead Pattern
Three Mechanisms
HTTP Transport Parameters — Deep Dive
Little's Law — Sizing Semaphores Correctly
Full Production Implementation
Wiring It Up
Correctly Sized Real-World Configs
Mental Model Summary

1. What is the Bulkhead Pattern

Named after watertight compartments in ship hulls. If one compartment floods, the others stay dry.

In software: isolate resources per dependency so a failure or slowdown in one cannot exhaust shared resources and take down unrelated parts of the system.

Without bulkheads:

Auth service slows down
  → goroutines pile up waiting
  → shared thread/connection pool exhausted
  → Payment service also fails (never touched auth)
  → Entire service down

With bulkheads:

Auth service slows down
  → only auth's semaphore/pool fills up
  → auth calls get fast 503s
  → Payment service unaffected
  → Ship stays afloat

2. Three Mechanisms

Mechanism 1 — Semaphore (Concurrency Bulkhead)

Caps how many concurrent in-flight calls exist to a dependency at any moment.

sem := semaphore.NewWeighted(20) // max 20 concurrent calls

func call(ctx context.Context) error {
    // TryAcquire = instant rejection if full (true bulkhead)
    // Acquire    = waits up to QueueTimeout for a slot
    acquireCtx, cancel := context.WithTimeout(ctx, 50*time.Millisecond)
    defer cancel()

    if err := sem.Acquire(acquireCtx, 1); err != nil {
        return fmt.Errorf("bulkhead full: %w", err) // fast 503
    }
    defer sem.Release(1)

    // make the actual call
}

TryAcquire vs Acquire:

Method	Behaviour	Use when
`TryAcquire(1)`	Instant reject if full	Strict load shedding
`Acquire(ctx, 1)`	Wait up to ctx deadline	Brief queuing acceptable

Each dependency gets its own independent semaphore — they do not share state.

Mechanism 2 — Connection Pool (TCP Bulkhead)

Caps how many TCP connections exist to each host. Without this, all services share http.DefaultTransport and one slow host monopolises all connections.

transport := &http.Transport{
    MaxConnsPerHost:     20, // hard ceiling on total connections
    MaxIdleConnsPerHost: 20, // warm connections kept alive
    MaxIdleConns:        20, // global idle pool
}

Mechanism 3 — Timeouts (Blast Radius Bulkhead)

Even if concurrency is high, tight timeouts ensure slots are released quickly when a dependency misbehaves. This is your blast radius control knob.

client := &http.Client{
    Transport: transport,
    Timeout:   600 * time.Millisecond, // absolute end-to-end deadline
}

3. HTTP Transport Parameters — Deep Dive

transport := &http.Transport{
    MaxIdleConnsPerHost:   N,
    MaxConnsPerHost:       N,
    MaxIdleConns:          N,
    DialContext:           ...,
    TLSHandshakeTimeout:   ...,
    ResponseHeaderTimeout: ...,
    IdleConnTimeout:       ...,
}
client := &http.Client{
    Transport: transport,
    Timeout:   ...,
}

`MaxConnsPerHost` — Hard Ceiling

What:    Maximum total TCP connections (active + idle) to a single host.
         New requests BLOCK when this is hit (until a connection is freed or context times out).
Effect:  This IS the TCP-level bulkhead. Beyond this number, no new connections are opened.

How to set it: Match your semaphore's MaxConcurrent. There is no point allowing 50 concurrent calls if you only have 10 TCP connections — requests would block waiting for a free connection anyway.

MaxConnsPerHost: cfg.MaxConcurrent // keep these equal

`MaxIdleConnsPerHost` — Warm Connection Cache

What:    How many idle (reusable) connections to keep open to a single host after a request completes.
Effect:  Avoids TCP+TLS handshake overhead on the next request. Higher = faster reuse, more FDs held open.

How to set it:

Set equal to or slightly below MaxConnsPerHost
For internal services (fast, high RPS): match MaxConnsPerHost exactly — you want all connections warm
For external APIs (slow, low RPS): can be lower since connections are held for longer and fewer are needed warm simultaneously

MaxIdleConnsPerHost = MaxConnsPerHost       // internal, high-throughput
MaxIdleConnsPerHost = MaxConnsPerHost / 2   // external, slow/bursty

`MaxIdleConns` — Global Idle Pool

What:    Total idle connections across ALL hosts combined.
Effect:  If this is too low, connections for one host get evicted to make room for another,
         causing unexpected TCP reconnects even though per-host limits haven't been hit.

How to set it: Sum of all your per-host idle limits across all bulkheads in the process.

// If you have 3 bulkheads: auth(20) + payment(10) + inventory(15)
MaxIdleConns: 20 + 10 + 15, // = 45, avoids cross-host eviction

Common mistake: Leaving this at Go's default of 100 when MaxIdleConnsPerHost is also 100. The global cap then silently limits what you think are independent pools.

`ResponseHeaderTimeout` — First Byte Timeout

What:    Time allowed between sending the request body and receiving the first byte of response headers.
Effect:  The most important timeout for catching slow/hung servers.
         Does NOT include time to read the response body.

How to set it: Slightly above your p99 latency for this dependency. This is the "server is stuck" detector.

External API p99 = 500ms → ResponseHeaderTimeout = 600ms
Internal svc p99 = 5ms   → ResponseHeaderTimeout = 20ms
Database p99     = 20ms  → ResponseHeaderTimeout = 50ms

`http.Client.Timeout` — Absolute End-to-End Deadline

What:    Time from request initiation to reading the last byte of response body.
         Includes: DNS + TCP dial + TLS + write request + wait for headers + read body.
Effect:  The outer hard deadline. Cancels the entire request if exceeded.

How to set it: ResponseHeaderTimeout + estimated body read time. Always set this — without it you can leak goroutines indefinitely.

ResponseHeaderTimeout = 600ms  (detect slow server)
body read estimate    = 200ms  (for your expected payload size)
Client.Timeout        = 800ms  (slightly above their sum)

`DialContext.Timeout` — TCP Handshake Limit

What:    Time allowed to establish the TCP connection itself.
Effect:  Catches unreachable hosts fast. Completely independent of request latency.

How to set it: 1–3 seconds universally. TCP handshake should be near-instant on healthy networks; longer means routing problem, not slowness.

DialContext: (&net.Dialer{
    Timeout:   2 * time.Second,  // TCP dial hard limit
    KeepAlive: 30 * time.Second, // TCP keepalive probes
}).DialContext,

`TLSHandshakeTimeout` — TLS Negotiation Limit

What:    Time allowed for TLS handshake after TCP is established.
Effect:  Catches TLS issues (expired certs, overloaded TLS terminator) separately from request latency.

How to set it: 3–5 seconds. TLS handshake involves a round trip plus crypto; 5s is generous but safe.

TLSHandshakeTimeout: 5 * time.Second,

`IdleConnTimeout` — Stale Connection Eviction

What:    How long an idle connection is kept in the pool before being closed.
Effect:  Prevents holding open TCP connections that the server has already closed (common after 60–90s).

How to set it: 60–90 seconds. Most servers close idle connections at ~90s. Setting this below their limit prevents "connection reset" errors.

IdleConnTimeout: 90 * time.Second, // safe default

`QueueTimeout` — Semaphore Wait Budget

What:    How long a request waits for a semaphore slot before being rejected.
Effect:  Controls the queue depth in time. Even if a slot frees in 10ms, you might not want to wait.

How to set it: Based on your own SLA minus downstream latency.

Your SLA:              500ms total
Downstream p99:        200ms
Budget for queuing:    500 - 200 - 50(overhead) = 250ms
QueueTimeout:          ~100ms (conservative, leaves room for retries)

For slow external APIs: lower QueueTimeout (fail fast, don't queue) For fast internal services: even lower (if pool is full, something is wrong)

All Parameters at a Glance

Parameter	Controls	Rule of Thumb
`MaxConnsPerHost`	TCP connection ceiling	= `MaxConcurrent`
`MaxIdleConnsPerHost`	Warm connection cache	= `MaxConnsPerHost` (internal), ÷2 (external)
`MaxIdleConns`	Global idle pool	= sum of all per-host idle limits
`ResponseHeaderTimeout`	"Server is slow" detector	p99 latency × 1.2
`Client.Timeout`	Absolute end-to-end limit	`ResponseHeaderTimeout` + body read estimate
`DialContext.Timeout`	TCP dial limit	1–2s universally
`TLSHandshakeTimeout`	TLS negotiation limit	3–5s universally
`IdleConnTimeout`	Stale connection eviction	60–90s (below server's idle timeout)
`QueueTimeout`	Semaphore wait budget	Your SLA − p99 − overhead

4. Little's Law — Sizing Semaphores Correctly

L = λ × W

L = concurrent requests in-flight (what your semaphore controls)
λ = throughput (requests per second you want to sustain)
W = average time per request (latency in seconds)

Key Insight

Slower dependency = MORE semaphore slots needed to sustain the same RPS.

Target: 100 rps

Auth svc  p99=100ms → L = 100 × 0.1 = 10 concurrent  (fast, slots free quickly)
Payment   p99=500ms → L = 100 × 0.5 = 50 concurrent  (slow, slots held longer)

This is counterintuitive but correct: a slow service holds each semaphore slot for longer, so you need more slots to keep throughput flowing.

What Happens When Semaphore Is Too Small

Payment svc p99=500ms, target 100rps, but MaxConcurrent=10:

Max throughput = 10 / 0.5 = 20 rps   ← you're throttling yourself to 20%

What Happens When Semaphore Is Too Large

Blast radius grows: more goroutines can pile up waiting on a misbehaving dep.
Blast radius is controlled by TIMEOUT, not concurrency.

The Correct Formula

min_concurrent = target_rps × p99_latency_seconds
semaphore_limit = min_concurrent × 1.3   // +30% headroom for bursts

NumCPU() is Wrong for IO-Bound Bulkheads

runtime.NumCPU() is the correct pool size only for CPU-bound work (hashing, compression, image processing) where goroutines burn CPU the entire time.

For IO-bound calls (HTTP, DB, Kafka): goroutines spend ~99% of their time parked waiting on the network. The Go scheduler puts them to sleep and runs other goroutines on the same OS thread. You can sustain thousands of concurrent IO goroutines on a 4-core machine.

IO call timeline:
[send ~50µs] [====== blocked waiting ~200ms ======] [read ~50µs]
                        ↑
              goroutine is PARKED here, OS thread runs other work

→ semaphore limit is a downstream capacity question, not a CPU question

5. Full Production Implementation

package bulkhead

import (
    "context"
    "fmt"
    "net"
    "net/http"
    "sync/atomic"
    "time"

    "golang.org/x/sync/semaphore"
)

// Config defines the bulkhead parameters for one downstream dependency.
type Config struct {
    Name           string
    MaxConcurrent  int64         // semaphore size — derived from Little's Law: target_rps × p99_latency
    MaxConnections int           // TCP connection ceiling — keep equal to MaxConcurrent
    RequestTimeout time.Duration // absolute end-to-end HTTP deadline (blast radius knob)
    QueueTimeout   time.Duration // how long to wait for a semaphore slot before rejecting
}

// Metrics holds live counters. All fields use atomic ops — safe to read from any goroutine.
type Metrics struct {
    Accepted atomic.Int64 // requests that acquired a semaphore slot
    Rejected atomic.Int64 // requests rejected because bulkhead was full
    Errors   atomic.Int64 // requests that acquired a slot but the HTTP call failed
}

// Bulkhead combines a per-dependency semaphore and an isolated HTTP client.
// Each instance is independent — a saturated bulkhead has zero effect on others.
type Bulkhead struct {
    cfg     Config
    sem     *semaphore.Weighted // Gate 1: concurrency cap
    client  *http.Client        // Gate 2: connection pool cap + timeouts
    Metrics Metrics
}

// New creates a Bulkhead. Call once per downstream dependency at startup.
func New(cfg Config) *Bulkhead {
    transport := &http.Transport{
        // --- Gate 2: TCP-level bulkhead ---
        MaxConnsPerHost:     cfg.MaxConnections,        // hard ceiling: new reqs block when hit
        MaxIdleConnsPerHost: cfg.MaxConnections,        // warm connections: avoids TCP+TLS overhead
        MaxIdleConns:        cfg.MaxConnections,        // global pool: set >= sum of all per-host

        // --- TCP dial ---
        DialContext: (&net.Dialer{
            Timeout:   2 * time.Second,  // TCP handshake hard limit (unreachable host detection)
            KeepAlive: 30 * time.Second, // TCP keepalive probe interval
        }).DialContext,

        // --- TLS ---
        TLSHandshakeTimeout: 5 * time.Second, // TLS negotiation limit

        // --- Request lifecycle ---
        // ResponseHeaderTimeout: time between sending request and receiving first response byte.
        // This is the "server is hung" detector. Set to p99 × 1.2.
        ResponseHeaderTimeout: cfg.RequestTimeout,

        // Evict idle connections before the server closes them (~90s on most servers).
        IdleConnTimeout: 90 * time.Second,
    }

    return &Bulkhead{
        cfg: cfg,
        sem: semaphore.NewWeighted(cfg.MaxConcurrent),
        client: &http.Client{
            Transport: transport,
            // Absolute deadline covering DNS+TCP+TLS+write+headers+body.
            // Always set — without this, goroutines can leak indefinitely.
            Timeout: cfg.RequestTimeout,
        },
    }
}

// Do executes an HTTP request through the bulkhead.
// Returns an error immediately if the semaphore is full (after QueueTimeout).
func (b *Bulkhead) Do(ctx context.Context, req *http.Request) (*http.Response, error) {
    // --- Gate 1: Semaphore ---
    // Use a child context with QueueTimeout so we don't wait forever for a slot.
    acquireCtx, cancel := context.WithTimeout(ctx, b.cfg.QueueTimeout)
    defer cancel()

    if err := b.sem.Acquire(acquireCtx, 1); err != nil {
        b.Metrics.Rejected.Add(1)
        return nil, fmt.Errorf("bulkhead [%s] rejected (semaphore full after %s): %w",
            b.cfg.Name, b.cfg.QueueTimeout, err)
    }
    defer b.sem.Release(1) // always release, even on HTTP error

    b.Metrics.Accepted.Add(1)

    // --- Gate 2: HTTP client (connection pool + timeouts) ---
    resp, err := b.client.Do(req.WithContext(ctx))
    if err != nil {
        b.Metrics.Errors.Add(1)
        return nil, fmt.Errorf("bulkhead [%s] call failed: %w", b.cfg.Name, err)
    }

    return resp, nil
}

// Stats returns a human-readable snapshot of metrics.
func (b *Bulkhead) Stats() string {
    return fmt.Sprintf("[%s] accepted=%d rejected=%d errors=%d",
        b.cfg.Name,
        b.Metrics.Accepted.Load(),
        b.Metrics.Rejected.Load(),
        b.Metrics.Errors.Load(),
    )
}

Why a Separate Semaphore per Dependency is Correct

authBulkhead.sem   ──► controls only auth calls
paymentBulkhead.sem ──► controls only payment calls

Auth saturated:
  authBulkhead.sem full → auth calls rejected
  paymentBulkhead.sem unaffected → payment calls proceed normally

A shared semaphore across all dependencies defeats the purpose entirely.

6. Wiring It Up

package main

import (
    "context"
    "fmt"
    "io"
    "net/http"
    "time"

    "yourmodule/bulkhead"
)

// Declare one bulkhead per downstream dependency at package level.
// These are long-lived, safe to use concurrently.
var (
    authBulkhead    = bulkhead.New(authConfig())
    paymentBulkhead = bulkhead.New(paymentConfig())
)

func HandleOrder(ctx context.Context) error {
    // Auth call — isolated behind its own semaphore + connection pool
    req, _ := http.NewRequest("GET", "https://auth.internal/verify", nil)
    resp, err := authBulkhead.Do(ctx, req)
    if err != nil {
        // Bulkhead full or auth down — fail this request fast.
        // Payment bulkhead is completely unaffected.
        return fmt.Errorf("auth unavailable: %w", err)
    }
    defer resp.Body.Close()
    io.Copy(io.Discard, resp.Body) // always drain body to return connection to pool

    // Payment call — completely isolated from auth
    req2, _ := http.NewRequest("POST", "https://payments.internal/charge", nil)
    resp2, err := paymentBulkhead.Do(ctx, req2)
    if err != nil {
        return fmt.Errorf("payment unavailable: %w", err)
    }
    defer resp2.Body.Close()
    io.Copy(io.Discard, resp2.Body)

    return nil
}

Important: Always io.Copy(io.Discard, resp.Body) before closing. If you close without draining, Go cannot reuse the TCP connection — it gets thrown away, causing unnecessary TCP handshakes on the next request.

7. Correctly Sized Real-World Configs

Sizing Formula

min_concurrent  = target_rps × p99_latency_seconds
semaphore_limit = min_concurrent × 1.3   // +30% burst headroom
MaxConnections  = semaphore_limit        // keep aligned
RequestTimeout  = p99_latency × 1.2     // slightly above p99, blast radius knob
QueueTimeout    = your_SLA - p99 - overhead_budget

External HTTP API (slow, p99=500ms, target 30 rps)

// Little's Law: 30 × 0.5 = 15, +30% = 20
// Blast radius: controlled by tight RequestTimeout (600ms), NOT by low concurrency
// Slower dep → more slots needed to sustain throughput
func externalAPIConfig() bulkhead.Config {
    return bulkhead.Config{
        Name:           "external-auth-api",
        MaxConcurrent:  20,                       // L = 30rps × 0.5s = 15, +30% = 20
        MaxConnections: 20,
        RequestTimeout: 600 * time.Millisecond,   // p99(500ms) × 1.2 — blast radius knob
        QueueTimeout:   100 * time.Millisecond,   // don't queue long for slow dep
    }
}

Internal Microservice (fast, p99=5ms, target 500 rps)

// Little's Law: 500 × 0.005 = 2.5 → floor at ~10 (connection overhead)
// Faster dep → naturally fewer slots needed, slots free in milliseconds
func internalServiceConfig() bulkhead.Config {
    return bulkhead.Config{
        Name:           "inventory-internal",
        MaxConcurrent:  10,                       // L = 500rps × 0.005s = 2.5, floor at 10
        MaxConnections: 10,
        RequestTimeout: 20 * time.Millisecond,    // tight: it's internal, same DC
        QueueTimeout:   5 * time.Millisecond,     // fail fast: if pool busy, something is wrong
    }
}

PostgreSQL (p99=20ms, target 200 rps, 5 app instances)

// Little's Law: 200 × 0.02 = 4 per instance
// Hard constraint: Postgres max_connections=100, shared across 5 instances → 20 per instance
// Use: max(Little's Law result × 1.3, safety floor) but never exceed hard cap
func postgresConfig() bulkhead.Config {
    return bulkhead.Config{
        Name:           "postgres",
        MaxConcurrent:  10,                       // L=4, ×1.3=5.2, safety floor=10, hard cap=20 ✓
        MaxConnections: 10,
        RequestTimeout: 100 * time.Millisecond,   // DB should be fast; slow DB = problem
        QueueTimeout:   20 * time.Millisecond,
    }
}

Kafka Producer (p99=2ms, target 2000 rps)

// Little's Law: 2000 × 0.002 = 4 concurrent
// Kafka batches internally — keep TCP connections low, semaphore has headroom for bursts
func kafkaConfig() bulkhead.Config {
    return bulkhead.Config{
        Name:           "kafka-producer",
        MaxConcurrent:  10,                       // L=4, headroom for burst
        MaxConnections: 3,                        // Kafka multiplexes internally; few TCP conns enough
        RequestTimeout: 50 * time.Millisecond,
        QueueTimeout:   5 * time.Millisecond,
    }
}

Summary Table

Dependency	p99	Target RPS	L=λ×W	+30%	MaxConcurrent	RequestTimeout	QueueTimeout
External API	500ms	30	15	20	20	600ms	100ms
Internal svc	5ms	500	2.5	4*	10*	20ms	5ms
Postgres	20ms	200	4	5**	10**	100ms	20ms
Kafka	2ms	2000	4	5	10	50ms	5ms

floor applied: connection establishment overhead makes sub-10 impractical ** floor applied, hard cap from Postgres max_connections / num_instances

8. Mental Model Summary

Two Laws, Two Knobs

Little's Law          → sets MaxConcurrent
  slow dep = MORE slots (slots held longer)
  fast dep = FEWER slots (slots released quickly)

Blast Radius Control  → sets RequestTimeout + QueueTimeout
  NOT concurrency — tight timeouts release slots fast
  even if MaxConcurrent is high, a 600ms timeout limits damage

The Failure Isolation Guarantee

Normal:                         Auth saturated:

User → Auth    ✓               User → Auth    ✗ (semaphore full → instant 503)
User → Payment ✓               User → Payment ✓ (own semaphore, unaffected)
User → S3      ✓               User → S3      ✓ (own semaphore, unaffected)

What Each Layer Protects Against

Layer                   | Protects against
────────────────────────┼──────────────────────────────────────────
Semaphore               | Goroutine pile-up from slow/hung deps
Connection pool         | TCP connection exhaustion from one host
ResponseHeaderTimeout   | Server that accepts connection but never responds
Client.Timeout          | Goroutine leak from infinite response body reads
QueueTimeout            | Cascading slowdowns from queue buildup

Bulkhead vs Circuit Breaker

These are complementary:

Pattern	Question it answers	Action
Bulkhead	"How much capacity do I allocate?"	Limits concurrent slots per dep
Circuit Breaker	"Should I even try this call?"	Opens when error rate exceeds threshold

Use both together: bulkhead limits blast radius, circuit breaker stops calling a dep that's known to be down.

Reference covers: golang.org/x/sync/semaphore, net/http.Transport, Little's Law (L=λW), Go scheduler IO parking, per-dependency isolation.

Note

So the minimal complete config for a single-host (not shared amont multiple deps) bulkhead transport is:

transport := &http.Transport{
    MaxConnsPerHost:     20,               // bulkhead ceiling
    MaxIdleConnsPerHost: 20,               // full connection reuse
    IdleConnTimeout:     90 * time.Second, // evict before server does

    DialContext: (&net.Dialer{
        Timeout:   2 * time.Second,
        KeepAlive: 30 * time.Second,
    }).DialContext,
    TLSHandshakeTimeout:   5 * time.Second,
    ResponseHeaderTimeout: cfg.RequestTimeout,
}

Everything else (MaxIdleConns, MaxConnsPerHost without PerHost suffix) is either irrelevant for single-host transports or has safe defaults for this use case.

False Sharing: The Silent Performance Killer in Concurrent Go

kamal namdeo — Sat, 11 Apr 2026 13:20:19 +0000

False Sharing: The Silent Performance Killer in Concurrent Go

Your goroutines never touch each other's data. Your atomics are correct. Your mutexes are in place. And yet your program slows down as you add more cores. Here is why — and how to fix it.

In this article, I would go through

The hardware reality: why memory access is never just one variable
What a cache line actually is
False sharing: correctness without performance
Three patterns that silently kill your throughput
The fix: padding and struct layout
Cache line sizes across architectures
How the Go standard library handles this
Rules to wire into your design instincts

THE HARDWARE REALITY MOST DEVELOPERS IGNORE

When you write a concurrent Go program, you think in variables: goroutine A writes to hits, goroutine B writes to misses. They are different variables. There is no data race. Your atomics are correct. Everything should be fine.

But your program gets slower as you add more cores. You profile it. There is no lock contention. No goroutine is blocked. The CPU utilisation looks high. And yet, throughput does not scale.

The bug is not in your code. It is in your mental model of how memory works.

The CPU does not fetch individual variables. It fetches 64-byte chunks of memory. Always.

This single fact — which most developers never internalise — is the root cause of an entire class of concurrency bugs called false sharing. Understanding it changes how you design every concurrent data structure you will ever write.

WHAT A CACHE LINE ACTUALLY IS

Modern CPUs have a memory hierarchy. Reading from main RAM costs roughly 100–300 clock cycles. Reading from L1 cache costs 3–5 cycles. This 60–100x difference is why caches exist — and why programs that use them well run dramatically faster than programs that do not.

But the cache does not work at the byte level. It works at the cache line level. A cache line is a 64-byte contiguous chunk of memory. Whenever the CPU needs any byte, it loads the entire 64-byte chunk that contains it into cache. You never get just the byte you asked for; you always get the whole 64-byte neighborhood.

RAM (simplified view):

Address:  0    8    16   24   32   40   48   56   64   72 ...
          |    |    |    |    |    |    |    |    |    |
          [────────────── cache line 1 ──────────────][── cache line 2 ──...
                         (64 bytes)                        (64 bytes)

Each core gets its own independent copy of any cache line it needs. If Core 1 and Core 2 both need data from the same 64-byte block, they both hold a copy. Now the hardware has a problem to solve: what happens when those copies diverge?

The coherence protocol

CPUs solve this with a cache coherence protocol (MESI is the most common variant). The rule is simple and absolute: when any core modifies a cache line, every other core's copy of that cache line is immediately marked invalid. Any core that subsequently tries to read or write that cache line must discard its invalidated copy and fetch a fresh one — from main memory, at full main-memory cost (100–300 cycles).

FALSE SHARING: WHEN INDEPENDENCE IS AN ILLUSION

False sharing occurs when two goroutines running on different cores are writing to different variables — logically unrelated variables — but those variables happen to sit within the same 64-byte cache line.

The defining characteristic of false sharing: your code is correct, your synchronisation is right, and you are still paying main-memory prices because of how your data happens to be laid out in RAM.

THREE PATTERNS THAT SILENTLY KILL YOUR THROUGHPUT

1. A slice of counters written by N goroutines

You have a slice of integers, one per goroutine, and each goroutine only ever writes to its own index.

counters := make([]int64, 8)

for i := 0; i < 8; i++ {
    go func(i int) {
        for {
            counters[i]++  // only touches index i
        }
    }(i)
}

In memory, eight int64 elements (8 bytes each) sit contiguously:

Address:  0        8        16       24       32       40       48       56
          [ c[0] ][ c[1] ][ c[2] ][ c[3] ][ c[4] ][ c[5] ][ c[6] ][ c[7] ]
          |────────────────────────────────────────────────────────────────|
                          one single cache line (64 bytes)

Every increment of c[0] by Core 0 invalidates the cache line for all seven other cores, because of the same copy of the cache line onto all eight cores, and changes by one core invalidate all other copies. Eight cores are now bouncing a single cache line through main memory like a hot potato.

2. A mutex and the data it protects on the same cache line

type SafeCounter struct {
    mu    sync.Mutex  // 8 bytes at offset 0
    value int64       // 8 bytes at offset 8
}

The act of contending for a lock invalidates the very cache line that holds the protected data. The goroutine that just won the lock pays a main memory penalty (100+ cycles) just to read the data it just unlocked.

Here is the step-by-step breakdown of the performance collapse:

Core 1 wants the lock: It executes an instruction to acquire mu. To do this, it pulls the cache line containing mu into its own L1 cache and marks it as Modified.
Core 2 tries to get the lock: It looks for mu. It sees that Core 1 owns the cache line. Core 2’s hardware must now wait for Core 1 to "give up" that line.
Core 1 updates the value: Since value is on the same cache line that Core 1 already "owns" for the mutex, this part is fast—initially.
Core 1 releases the lock: It writes to mu to unlock it.
Core 2 finally gets the line: The hardware intercepts.
Core 2’s request. Because Core 1 modified that line, the hardware invalidates Core 1's copy and forces the data to be synchronised and fetched from RAM.

This is the "Silent Killer" part: When Core 2 finally successfully acquires the mutex, it thinks it's ready to work. But because the entire 64-byte line was marked invalid during the "handover" from Core 1, the value variable is also gone from Core 2's cache.

Even though Core 2 now "owns" the lock, the very first time it tries to read SafeCounter.value, it hits a Cache Miss. It has to reach all the way out to the slow Main RAM to get the current value, wasting roughly 100–300 clock cycles.
The Result

You are paying a "RAM Tax" twice:

Once to fight for the Mutex.
Once to read the Data the mutex was supposed to protect.

3. Atomic counters declared side by side

var (
    hits   atomic.Int64   // offset 0
    misses atomic.Int64   // offset 8
)

atomic.Int64 guarantees correctness, but it says nothing about cache lines. The atomic instruction itself takes nanoseconds; the cache miss that precedes it takes 100+ cycles.

Let's understand with an example -

As shown in the snippet above, the vars hits and misses are on the same cache line.

Core 1 is responsible for incrementing hits.
Core 2 is responsible for incrementing misses.

Even though these two variables are logically unrelated, they are physically roommates. When Core 1 performs an atomic increment on hits:

It must lock the entire cache line to ensure atomicity.
It increments the value and marks the cache line as Modified (M).

Because the line is modified, Core 2’s cache copy of that entire 64-byte block is instantly marked Invalid (I). Now, Core 2 wants to increment misses.

Core 2 looks in its cache. It sees the line containing misses is Invalid.
The Penalty: Core 2 must wait for the hardware to fetch the updated line from Core 1 (or main memory). This takes ~100–300 cycles.
Core 2 performs its atomic increment, which in turn invalidates Core 1’s copy.

They are now "ping-ponging" the cache line back and forth across the CPU bus.

THE FIX: PADDING AND STRUCT LAYOUT

The solution is to force hot fields onto separate cache lines using padding.

Fix 1: Padded slice elements

type PaddedInt struct {
    Val int64
    _   [56]byte  // 8 + 56 = 64 bytes total = one full cache line
}

counters := make([]PaddedInt, 8)

Fix 2: Padding between mutex and protected data

type SafeCounter struct {
    mu    sync.Mutex
    _     [56]byte  // pad mu to its own cache line
    value int64
    _     [56]byte  // pad value to its own cache line
}

Field order is load-bearing

Go lays out struct fields in declaration order. Padding must go between the fields you want to isolate.

// CORRECT: padding acts as a wall between variables, hits and misses

type Good struct {
    hits     atomic.Int64
    _        [56]byte   // wall after hits
    misses   atomic.Int64
    _        [56]byte   // wall after misses
}

CACHE LINE SIZES ACROSS ARCHITECTURES

Architecture	Cache line size	Notes
x86-64 (Intel, AMD)	64 bytes	Standard server hardware
ARM64 (AWS Graviton)	64 bytes	Cloud ARM servers
Apple M1/M2/M3	64 / 128 bytes	L2 uses 128-byte chunks
IBM POWER9/10	128 bytes	128-byte lines throughout

Pro Tip: For portable code, pad to 128 bytes. It wastes a bit of RAM but ensures safety across all modern high-performance architectures.

HOW THE GO STANDARD LIBRARY HANDLES THIS

The Go team uses this sparingly but effectively.

runtime.p: The processor struct is explicitly padded to prevent adjacent p structs in the internal array from invalidating each other's run queues.
sync.Pool: The poolChain struct pads its head and tail pointers. Since producers write to the head and consumers read from the tail, padding prevents these two roles from fighting over the same cache line.

RULES TO WIRE INTO YOUR DESIGN INSTINCTS

Slices written by N goroutines: Use a padded wrapper struct.
Embedded Mutexes: Pad the mutex if the protected data is "hot."
Atomic Counters: Pad them apart if they are logically independent.
Scaling issues: If performance degrades as you add cores, suspect false sharing and profile cache miss rates.
Field Order: Remember that padding is a wall, not a footer. Place it physically between the fields you need to separate.

In distributed systems, you think about network round trips. In concurrent code, the equivalent cost is: will this memory access hit L1 cache, or main memory?

Getting this right is what separates code that scales from code that merely passes the race detector.

DEV Community: kamal namdeo

nginx Event Loop — Complete Lifecycle Reference

nginx Event Loop — Complete Lifecycle Reference

Table of Contents

1. Foundations

1.1 Everything is a File

1.2 File Descriptor (fd)

1.3 Socket

1.4 How fd and Socket Relate

2. Hardware Layer

2.1 NIC (Network Interface Card)

2.2 DMA (Direct Memory Access)

2.3 NIC Ring Buffer

2.4 Hardware Interrupt (IRQ — Interrupt Request)

2.5 ISR (Interrupt Service Routine)

3. Kernel Structures and All Buffers

3.1 SYN Queue (Incomplete Connections Queue)

3.2 Accept Queue (Complete Connections Queue)

3.3 Socket Receive Buffer

3.4 Socket Send Buffer

3.5 nginx Worker Memory (User Space)

4. epoll

4.1 What is epoll

4.2 epoll Instance

4.3 Interest List

4.4 Ready List

4.5 Edge Triggered vs Level Triggered

4.6 epoll Syscalls

5. nginx Startup Sequence

Master Process — Before Workers Start

6. Complete Request Lifecycle

Cast

Phase 1 — TCP Handshake (Kernel Only, Worker Sleeping)

Phase 2 — Worker Wakes, Accepts Both Connections

Phase 3 — HTTP Request Bytes Arrive

Phase 4 — Worker Reads and Parses Both Requests

Phase 5 — Worker Opens Backend Connections

Phase 6 — Backend TCP Handshakes Complete

Phase 7 — Worker Forwards Requests to Backend

Phase 8 — Backend Responds (Possibly Out of Order)

Phase 9 — Worker Reads Backend Response, Writes to Client

Phase 10 — Connection Teardown

7. What Happens While Worker is Busy

8. All Buffers — Master Reference

9. All Syscalls — Master Reference

10. Failure Modes

10.1 Accept Queue Full

10.2 Event Loop Starvation

10.3 Send Buffer Full (Slow Client)

10.4 Recv Buffer Full (Slow Worker)

10.5 fd Exhaustion

Quick Mental Model

Bulkhead Pattern - Go

Go Bulkhead Pattern — Complete Reference

Table of Contents

1. What is the Bulkhead Pattern

2. Three Mechanisms

Mechanism 1 — Semaphore (Concurrency Bulkhead)

Mechanism 2 — Connection Pool (TCP Bulkhead)

Mechanism 3 — Timeouts (Blast Radius Bulkhead)

3. HTTP Transport Parameters — Deep Dive

MaxConnsPerHost — Hard Ceiling

MaxIdleConnsPerHost — Warm Connection Cache

MaxIdleConns — Global Idle Pool

ResponseHeaderTimeout — First Byte Timeout

http.Client.Timeout — Absolute End-to-End Deadline

DialContext.Timeout — TCP Handshake Limit

TLSHandshakeTimeout — TLS Negotiation Limit

IdleConnTimeout — Stale Connection Eviction

QueueTimeout — Semaphore Wait Budget

All Parameters at a Glance

4. Little's Law — Sizing Semaphores Correctly

Key Insight

What Happens When Semaphore Is Too Small

What Happens When Semaphore Is Too Large

The Correct Formula

NumCPU() is Wrong for IO-Bound Bulkheads

5. Full Production Implementation

Why a Separate Semaphore per Dependency is Correct

6. Wiring It Up

`MaxConnsPerHost` — Hard Ceiling

`MaxIdleConnsPerHost` — Warm Connection Cache

`MaxIdleConns` — Global Idle Pool

`ResponseHeaderTimeout` — First Byte Timeout

`http.Client.Timeout` — Absolute End-to-End Deadline

`DialContext.Timeout` — TCP Handshake Limit

`TLSHandshakeTimeout` — TLS Negotiation Limit

`IdleConnTimeout` — Stale Connection Eviction

`QueueTimeout` — Semaphore Wait Budget