Implementing Graceful Shutdown in Go HTTP Servers

When you kill an HTTP server with SIGKILL, every in-flight request dies instantly. If your handler was halfway through writing to a database, you get partial writes. If it was streaming a large file, the client gets a broken download with no error message. This is not a theoretical edge case — it happens every time you deploy or restart without a proper shutdown sequence.

Why abrupt shutdown breaks production systems

kill -9 and docker stop without a timeout both send SIGKILL, which terminates the process unconditionally. The kernel reclaims file descriptors, open connections, and goroutines simultaneously — no cleanup hooks run.

The actual damage depends on what your handlers do:

• Partial database writes if you're not wrapping operations in transactions
• Broken HTTP responses causing clients to retry, fail silently, or see half-rendered pages
• Leaked resources: temp files on disk, advisory locks still held, in-progress S3 uploads that never complete
• Lost audit trail: if you log after writing to your store, you may get the store write without the log

SIGTERM is what you should be catching. Kubernetes (terminationGracePeriodSeconds), systemd (TimeoutStopSec), and docker stop all send SIGTERM first, then wait for the process to exit cleanly before escalating to SIGKILL. The default window is 30 seconds for Kubernetes and 90 for systemd — time you're leaving on the table if your server ignores SIGTERM.

The core pattern: os/signal + Server.Shutdown

Go's net/http package ships with Server.Shutdown(ctx) since Go 1.8. It stops accepting new connections and waits for active handlers to finish. The piece you wire yourself is the signal handler.

package main

import (
    "context"
    "log"
    "net/http"
    "os"
    "os/signal"
    "syscall"
    "time"
)

func main() {
    mux := http.NewServeMux()
    mux.HandleFunc("/health", func(w http.ResponseWriter, r *http.Request) {
        w.WriteHeader(http.StatusOK)
        w.Write([]byte("ok"))
    })
    mux.HandleFunc("/slow", func(w http.ResponseWriter, r *http.Request) {
        // Simulates a handler that takes a while (DB query, external call, etc.)
        time.Sleep(8 * time.Second)
        w.Write([]byte("done"))
    })

    srv := &http.Server{
        Addr:         ":8080",
        Handler:      mux,
        ReadTimeout:  15 * time.Second,
        WriteTimeout: 60 * time.Second,
    }

    // Start in a goroutine so we can block on signal below
    go func() {
        if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
            log.Fatalf("server error: %v", err)
        }
    }()
    log.Println("server started on :8080")

    // Block until SIGINT or SIGTERM
    quit := make(chan os.Signal, 1)
    signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
    sig := <-quit
    log.Printf("received signal: %s", sig)

    // Give active connections 30 seconds to finish
    ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()

    if err := srv.Shutdown(ctx); err != nil {
        log.Fatalf("forced shutdown: %v", err)
    }
    log.Println("server exited cleanly")
}

Two things to note. The buffered channel (make(chan os.Signal, 1)) prevents the signal from being dropped if your goroutine isn't ready to receive. And http.ErrServerClosed is the normal return value from ListenAndServe after a clean shutdown — treating it as fatal will mask that you actually shut down correctly.

Tracking background work spawned by handlers

Server.Shutdown() waits for HTTP connections to close, but it does not wait for goroutines your handlers spawn. If a handler fires async work and returns 202, the shutdown will drain the connection and then your goroutine keeps running — potentially after the database pool is closed.

The fix is a sync.WaitGroup that outlives the HTTP server:

package main

import (
    "context"
    "log"
    "net/http"
    "os"
    "os/signal"
    "sync"
    "syscall"
    "time"
)

func main() {
    var wg sync.WaitGroup

    mux := http.NewServeMux()
    mux.HandleFunc("/enqueue", func(w http.ResponseWriter, r *http.Request) {
        // wg.Add MUST happen before the goroutine starts — not inside it.
        // If you call Add inside, there's a race where Shutdown sees zero
        // and returns before the goroutine has incremented the counter.
        wg.Add(1)
        go func() {
            defer wg.Done()
            time.Sleep(5 * time.Second) // replace with real async work
            log.Println("background job completed")
        }()
        w.WriteHeader(http.StatusAccepted)
    })

    srv := &http.Server{Addr: ":8080", Handler: mux}

    go func() {
        if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
            log.Fatalf("server error: %v", err)
        }
    }()

    quit := make(chan os.Signal, 1)
    signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
    <-quit

    // Phase 1: stop accepting new requests
    httpCtx, httpCancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer httpCancel()
    if err := srv.Shutdown(httpCtx); err != nil {
        log.Printf("http shutdown error: %v", err)
    }

    // Phase 2: wait for in-flight background jobs
    done := make(chan struct{})
    go func() {
        wg.Wait()
        close(done)
    }()

    select {
    case <-done:
        log.Println("all background jobs finished")
    case <-time.After(30 * time.Second):
        log.Println("timeout: some background jobs may have been abandoned")
    }
}

The two-phase approach makes the sequence explicit: first refuse new traffic, then drain existing work. This mirrors how load balancers handle drain — and lets you set different timeouts for each phase.

Production edge cases worth handling

Health check conflicts. If your load balancer polls /health, it needs to see 503 before or right as shutdown begins — otherwise it keeps sending traffic to a server that's draining. A clean approach is to set an atomic flag when shutdown starts and check it in the health handler. Kubernetes's readinessProbe will remove the pod from the service endpoint set once it starts failing.

Multiple servers on different ports. If you expose a public API on :8080 and a metrics endpoint on :9090, shut down the public one first, then metrics. Share a single sync.WaitGroup and a parent context across both.

Timeout alignment. Your Server.Shutdown timeout must be less than Kubernetes's terminationGracePeriodSeconds. If they're equal, the pod gets SIGKILL before your shutdown context can return cleanly. A safe rule: set your app shutdown timeout to terminationGracePeriodSeconds - 5.

Testing the shutdown path. Start your server, issue a slow request (curl localhost:8080/slow &), then send SIGTERM (kill -TERM $(pgrep your-binary)). Verify that:

• New connections are immediately refused (connection refused error)
• The in-flight request completes normally
• The process exits with code 0

This smoke test catches the most common mistake: forgetting to start the server in a goroutine, which blocks main before the signal listener is reached.

For a broader security baseline — including timeouts, header hardening, and connection limits — the security hardening checklists we publish cover the settings most commonly skipped under deadline pressure.

The takeaway

Graceful shutdown in Go is under 50 lines of production-ready code, but the ordering matters. Catch SIGTERM with a buffered channel, call srv.Shutdown() with a context timeout, and track background work with a sync.WaitGroup. Miss the ordering on wg.Add() and you'll get a race that shows up only under load. Miss the timeout alignment with Kubernetes and your pods will get SIGKILLed anyway.

The pattern scales: whether you have one handler or fifty, the signal plumbing stays the same. What changes is which goroutines you register with the WaitGroup and how long you're willing to wait for them.

---

I run AYI NEDJIMI Consultants, a cybersecurity consulting firm. We publish free security hardening checklists — PDF and Excel.