DEV Community: Bar Dror

Reading and Parsing X.509 Certificates in Java: Certificate Chain Validation

Bar Dror — Fri, 23 Oct 2026 07:01:00 +0000

Originally published on Code Beneath.

<<>>
Java read X.509 certificate parsing with CertificateFactory, chain validation, DN extraction, and self-signed cert handling without external libraries.
<<>>
<<

Extracting certificate details and validating certificate chains in Java requires understanding the X.509 standard and the built-in APIs available since Java 1.4. Many engineers reach for Apache Commons or Bouncy Castle before trying the standard library, but the standard APIs are sufficient for most real-world tasks: reading PEM and DER files, extracting distinguished names, validating chains, and checking self-signed certificates. This guide shows you how to java read x509 certificate parse chain example code that works in production, with no external dependencies.

Why You Need Certificate Chain Validation

When your application receives a certificate from a client (in mutual TLS, for example) or validates a server certificate, you need to ensure the chain is complete and each certificate was signed by the next one in the chain. This is not just an optional security measure: it is the foundation of trust. A self-signed certificate with an incomplete chain or an expired intermediate will cause authentication to fail. Understanding how to validate this yourself, rather than delegating it to a framework, gives you visibility into what is actually happening during the handshake.

The java.security.cert package provides CertificateFactory, X509Certificate, and CertPathValidator for this purpose. Together they let you parse certificates, extract public keys, validate chains, and inspect the distinguished names (DNs) that identify the certificate owner.

Loading and Parsing X.509 Certificates

The first step is to load a certificate from a file or stream. X.509 certificates come in two formats: DER (binary) and PEM (Base64-encoded text). Java's CertificateFactory handles both.

import java.io.*;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.security.cert.CertificateFactory;
import java.security.cert.X509Certificate;
import java.security.cert.Certificate;
import java.util.*;

public class CertificateParser {

    public static X509Certificate parsePemCertificate(String filePath) throws Exception {
        FileInputStream fis = new FileInputStream(filePath);
        CertificateFactory cf = CertificateFactory.getInstance("X.509");
        X509Certificate cert = (X509Certificate) cf.generateCertificate(fis);
        fis.close();
        return cert;
    }

    public static X509Certificate parseDerCertificate(String filePath) throws Exception {
        byte[] derBytes = Files.readAllBytes(Paths.get(filePath));
        ByteArrayInputStream bais = new ByteArrayInputStream(derBytes);
        CertificateFactory cf = CertificateFactory.getInstance("X.509");
        X509Certificate cert = (X509Certificate) cf.generateCertificate(bais);
        bais.close();
        return cert;
    }

    public static List<X509Certificate> parseCertificateChain(String filePath) throws Exception {
        FileInputStream fis = new FileInputStream(filePath);
        CertificateFactory cf = CertificateFactory.getInstance("X.509");
        Collection<? extends Certificate> certs = cf.generateCertificates(fis);
        fis.close();

        List<X509Certificate> chain = new ArrayList<>();
        for (Certificate c : certs) {
            chain.add((X509Certificate) c);
        }
        return chain;
    }

    public static void main(String[] args) throws Exception {
        String pemPath = "server.pem";
        X509Certificate cert = parsePemCertificate(pemPath);

        System.out.println("Subject: " + cert.getSubjectDN());
        System.out.println("Issuer: " + cert.getIssuerDN());
        System.out.println("Serial: " + cert.getSerialNumber());
        System.out.println("Not Before: " + cert.getNotBefore());
        System.out.println("Not After: " + cert.getNotAfter());
    }
}

The key difference: use generateCertificate() for a single certificate and generateCertificates() (plural) for a chain. The factory automatically handles PEM Base64 decoding or raw DER bytes. No Base64 decoding on your side is needed; the factory does it.

Extracting and Parsing Distinguished Names

The subject DN contains the identity of the certificate owner. It is a structured name with fields like CN (Common Name), O (Organization), C (Country), and others. The getSubjectDN() method returns a Principal, but to extract individual fields, you need to parse the X.500 format string or use an X500Principal.

import javax.security.auth.x500.X500Principal;
import java.util.regex.Pattern;
import java.util.regex.Matcher;

public class DNParser {

    public static void printDN(X509Certificate cert) {
        X500Principal principal = cert.getSubjectX500Principal();
        String dn = principal.getName(X500Principal.RFC2253);
        System.out.println("DN (RFC2253): " + dn);

        Map<String, String> dnMap = parseDN(dn);
        for (String key : dnMap.keySet()) {
            System.out.println(key + " = " + dnMap.get(key));
        }
    }

    public static Map<String, String> parseDN(String dn) {
        Map<String, String> map = new LinkedHashMap<>();

        String[] parts = dn.split(",(?=(?:[^\\\\]*\\\\\\\\)*[^\\\\]*$)");

        for (String part : parts) {
            part = part.trim();
            int eqIndex = part.indexOf('=');
            if (eqIndex > 0) {
                String key = part.substring(0, eqIndex).trim();
                String value = part.substring(eqIndex + 1).trim();
                map.put(key, value);
            }
        }

        return map;
    }

    public static String extractCommonName(X509Certificate cert) {
        Map<String, String> dnMap = parseDN(
            cert.getSubjectX500Principal().getName(X500Principal.RFC2253)
        );
        return dnMap.get("CN");
    }

    public static void main(String[] args) throws Exception {
        FileInputStream fis = new FileInputStream("server.pem");
        CertificateFactory cf = CertificateFactory.getInstance("X.509");
        X509Certificate cert = (X509Certificate) cf.generateCertificate(fis);
        fis.close();

        printDN(cert);
        System.out.println("\nCommon Name: " + extractCommonName(cert));
    }
}

The RFC2253 format uses commas to separate RDNs (Relative Distinguished Names), but commas can also appear inside escaped values. The regex split with a lookahead handles this case correctly. This parser extracts the key-value pairs into a Map so you can look up CN, O, C, or any other field.

Validating Certificate Chains

A certificate chain is valid if each certificate was signed by the next one in the chain, and the final certificate (the root) is trusted. Java provides CertPathValidator to automate this. You need to supply the chain, a trust anchor (the root certificate or a KeyStore), and validation parameters.

import java.security.KeyStore;
import java.security.cert.*;
import java.util.*;

public class ChainValidator {

    public static boolean validateChain(List<X509Certificate> chain, 
                                        X509Certificate rootCert) throws Exception {
        if (chain == null || chain.isEmpty()) {
            System.out.println("Chain is empty");
            return false;
        }

        PKIXParameters params = new PKIXParameters(
            Collections.singleton(new TrustAnchor(rootCert, null))
        );

        params.setRevocationEnabled(false);

        CertPath path = CertificateFactory.getInstance("X.509")
            .generateCertPath(chain);

        CertPathValidator validator = CertPathValidator.getInstance("PKIX");

        try {
            validator.validate(path, params);
            System.out.println("Chain validation succeeded");
            return true;
        } catch (CertPathValidatorException e) {
            System.out.println("Chain validation failed: " + e.getMessage());
            e.printStackTrace();
            return false;
        }
    }

    public static boolean validateChainWithKeyStore(List<X509Certificate> chain,
                                                    String keyStorePath,
                                                    String keyStorePassword) throws Exception {
        KeyStore ks = KeyStore.getInstance("JKS");
        ks.load(new FileInputStream(keyStorePath), keyStorePassword.toCharArray());

        PKIXParameters params = new PKIXParameters(ks);
        params.setRevocationEnabled(false);

        CertPath path = CertificateFactory.getInstance("X.509")
            .generateCertPath(chain);

        CertPathValidator validator = CertPathValidator.getInstance("PKIX");

        try {
            validator.validate(path, params);
            System.out.println("Chain validation with KeyStore succeeded");
            return true;
        } catch (CertPathValidatorException e) {
            System.out.println("Chain validation failed: " + e.getMessage());
            return false;
        }
    }

    public static void main(String[] args) throws Exception {
        List<X509Certificate> chain = new ArrayList<>();

        FileInputStream fis1 = new FileInputStream("server.pem");
        CertificateFactory cf = CertificateFactory.getInstance("X.509");
        chain.add((X509Certificate) cf.generateCertificate(fis1));
        fis1.close();

        FileInputStream fis2 = new FileInputStream("intermediate.pem");
        chain.add((X509Certificate) cf.generateCertificate(fis2));
        fis2.close();

        FileInputStream fis3 = new FileInputStream("root.pem");
        X509Certificate rootCert = (X509Certificate) cf.generateCertificate(fis3);
        fis3.close();

        validateChain(chain, rootCert);
    }
}

The order matters: the chain must start with the leaf (server) certificate and end with an intermediate or the root. The trust anchor is the root certificate or a KeyStore that contains trusted roots. The setRevocationEnabled(false) call disables CRL and OCSP checks, which is fine for offline validation; if you need revocation checking, you must supply a CertStore with CRL or OCSP responder URLs.

Handling Self-Signed Certificates

A self-signed certificate is one where the issuer and subject are the same, and it was signed by its own private key. These are common in testing and for internal CAs. Validating a self-signed certificate means treating it as its own trust anchor.

import java.security.PublicKey;

public class SelfSignedValidator {

    public static boolean isSelfSigned(X509Certificate cert) {
        return cert.getSubjectDN().equals(cert.getIssuerDN());
    }

    public static boolean validateSelfSigned(X509Certificate cert) {
        try {
            PublicKey publicKey = cert.getPublicKey();
            cert.verify(publicKey);
            System.out.println("Self-signed certificate signature is valid");
            return true;
        } catch (Exception e) {
            System.out.println("Self-signed certificate signature verification failed: " + e.getMessage());
            return false;
        }
    }

    public static boolean validateSelfSignedWithExpiry(X509Certificate cert) {
        if (!isSelfSigned(cert)) {
            System.out.println("Certificate is not self-signed");
            return false;
        }

        try {
            cert.verify(cert.getPublicKey());
            System.out.println("Signature is valid");
        } catch (Exception e) {
            System.out.println("Signature verification failed: " + e.getMessage());
            return false;
        }

        try {
            cert.checkValidity();
            System.out.println("Certificate is within validity period");
            return true;
        } catch (CertificateExpiredException e) {
            System.out.println("Certificate has expired");
            return false;
        } catch (CertificateNotYetValidException e) {
            System.out.println("Certificate is not yet valid");
            return false;
        }
    }

    public static void main(String[] args) throws Exception {
        FileInputStream fis = new FileInputStream("selfsigned.pem");
        CertificateFactory cf = CertificateFactory.getInstance("X.509");
        X509Certificate cert = (X509Certificate) cf.generateCertificate(fis);
        fis.close();

        if (isSelfSigned(cert)) {
            System.out.println("Certificate is self-signed");
            validateSelfSignedWithExpiry(cert);
        }
    }
}

The verify() method uses the public key to verify the signature. For a self-signed cert, the public key comes from the certificate itself. The checkValidity() method checks the notBefore and notAfter dates against the current system time. Call both to fully validate a self-signed certificate.

Extracting Public Keys and Certificate Extensions

Beyond the basic DN and validity, certificates contain extensions like Subject Alternative Name (SAN), Key Usage, Extended Key Usage, and Authority Key Identifier. These extensions are crucial for modern PKI.

import java.security.PublicKey;
import java.util.*;

public class ExtensionExtractor {

    public static void extractPublicKey(X509Certificate cert) {
        PublicKey pk = cert.getPublicKey();
        System.out.println("Algorithm: " + pk.getAlgorithm());
        System.out.println("Format: " + pk.getFormat());
        System.out.println("Encoded length: " + pk.getEncoded().length + " bytes");
    }

    public static void extractExtensions(X509Certificate cert) {
        Set<String> criticalOids = cert.getCriticalExtensionOIDs();
        Set<String> nonCriticalOids = cert.getNonCriticalExtensionOIDs();

        if (criticalOids != null) {
            System.out.println("Critical extensions:");
            for (String oid : criticalOids) {
                byte[] extValue = cert.getExtensionValue(oid);
                System.out.println("  OID: " + oid + ", length: " + extValue.length);
            }
        }

        if (nonCriticalOids != null) {
            System.out.println("Non-critical extensions:");
            for (String oid : nonCriticalOids) {
                byte[] extValue = cert.getExtensionValue(oid);
                System.out.println("  OID: " + oid + ", length: " + extValue.length);
            }
        }
    }

    public static List<String> extractSubjectAlternativeNames(X509Certificate cert) {
        List<String> sans = new ArrayList<>();

        try {
            Collection<List<?>> altNames = cert.getSubjectAlternativeNames();

            if (altNames == null) {
                System.out.println("No Subject Alternative Names");
                return sans;
            }

            for (List<?> altName : altNames) {
                Integer type = (Integer) altName.get(0);
                Object name = altName.get(1);

                if (type == 2) {
                    sans.add("DNS: " + name.toString());
                } else if (type == 7) {
                    sans.add("IP: " + name.toString());
                }
            }
        } catch (Exception e) {
            System.out.println("Error extracting SANs: " + e.getMessage());
        }

        return sans;
    }

    public static void main(String[] args) throws Exception {
        FileInputStream fis = new FileInputStream("server.pem");
        CertificateFactory cf = CertificateFactory.getInstance("X.509");
        X509Certificate cert = (X509Certificate) cf.generateCertificate(fis);
        fis.close();

        System.out.println("=== Public Key ===");
        extractPublicKey(cert);

        System.out.println("\n=== Extensions ===");
        extractExtensions(cert);

        System.out.println("\n=== Subject Alternative Names ===");
        List<String> sans = extractSubjectAlternativeNames(cert);
        for (String san : sans) {
            System.out.println(san);
        }
    }
}

The SAN extension is stored as a GeneralNames structure inside the extension value. The list contains tuples of (type, value). Type 2 is DNS name, type 7 is IP address. This is why browsers use SANs to match hostnames: it is more flexible than the CN field. Your TLS stack probably does this automatically, but if you are building a custom validator, you need to extract and check these yourself.

Complete Integration Example with mTLS Context

In a real mutual TLS scenario, as described in the TLS handshake guide, your server receives a client certificate and must validate it. Here is a complete example that reads a client cert, extracts identity, validates the chain, and checks expiry.

import java.io.*;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.security.cert.*;
import java.util.*;
import javax.security.auth.x500.X500Principal;

public class ClientCertValidator {

    public static void validateClientCertificate(String clientCertPath, 
                                                 String trustedCAPath) throws Exception {
        FileInputStream clientFis = new FileInputStream(clientCertPath);
        CertificateFactory cf = CertificateFactory.getInstance("X.509");
        X509Certificate clientCert = (X509Certificate) cf.generateCertificate(clientFis);
        clientFis.close();

        FileInputStream caFis = new FileInputStream(trustedCAPath);
        X509Certificate caCert = (X509Certificate) cf.generateCertificate(caFis);
        caFis.close();

        System.out.println("Client Certificate Details:");
        System.out.println("Subject: " + clientCert.getSubjectDN());
        System.out.println("Issuer: " + clientCert.getIssuerDN());
        System.out.println("Serial: " + clientCert.getSerialNumber());

        try {
            clientCert.checkValidity();
            System.out.println("Validity: OK");
        } catch (Exception e) {
            System.out.println("Validity: FAILED - " + e.getMessage());
            return;
        }

        try {
            clientCert.verify(caCert.getPublicKey());
            System.out.println("Signature: VALID (signed by trusted CA)");
        } catch (Exception e) {
            System.out.println("Signature: INVALID - " + e.getMessage());
            return;
        }

        X500Principal subjectPrincipal = clientCert.getSubjectX500Principal();
        String dn = subjectPrincipal.getName(X500Principal.RFC2253);
        System.out.println("Distinguished Name: " + dn);

        System.out.println("\nClient certificate validation PASSED");
    }

    public static void main(String[] args) throws Exception {
        if (args.length < 2) {
            System.out.println("Usage: java ClientCertValidator <clientCertPath> <trustedCAPath>");
            return;
        }

        validateClientCertificate(args[0], args[1]);
    }
}

Run this with: java ClientCertValidator client.pem ca.pem. This validates that the client certificate was issued by the CA, has not expired, and is properly signed. In a real server, you would store the trusted CA certificate in a KeyStore or load it once at startup.

Common Mistakes

Not checking expiry: A valid signature does not guarantee the certificate is current. Always call checkValidity().

Assuming a chain starts with the root: Certificate chains are ordered leaf-to-root, not root-to-leaf. If you reverse the order, validation will fail.

Forgetting that CertificateFactory.generateCertificates() includes the root: If you load a chain file that contains all certs including the root, and then use that list as your chain and the root as a separate trust anchor, you will have duplication. Keep the chain separate from the trust anchor.

Not handling escaped commas in DNs: A simple split(",") on a DN string will break if a field value contains an escaped comma. Use a regex lookahead or an X500Name parser.

Enabling revocation checking without a CertStore: If you set setRevocationEnabled(true) but do not provide a CertStore, validation will fail. Either disable it or supply a working CRL or OCSP source.

FAQ

Can I validate a certificate chain from a PKCS12 or JKS KeyStore?

Yes. Use KeyStore.getCertificateChain(alias) to retrieve the chain as an array of Certificates, convert to a List of X509Certificate, and pass to CertPathValidator with the KeyStore itself as the trust anchor source (via PKIXParameters).

What is the difference between PEM and DER?

DER is binary; PEM is Base64-encoded DER wrapped in BEGIN/END markers. CertificateFactory handles both automatically when you pass an InputStream.

How do I extract the public key from a certificate for use in encryption?

Call cert.getPublicKey(). This returns a PublicKey interface. You can cast it to RSAPublicKey or ECPublicKey if you know the algorithm. For general use, pass it directly to Cipher or Signature objects.

Can I validate a certificate without the full chain?

If you have the leaf and the root, you can validate the signature of the leaf with the root's public key using cert.verify(rootCert.getPublicKey()). However, CertPathValidator requires the full chain and is stricter about validity dates and extensions.

What happens if the trust anchor is itself expired?

CertPathValidator will still validate the chain as long as the trust anchor is properly configured. The root certificate expiry does not matter for signature verification; it matters only if the root is being validated as part of a larger chain. Roots typically have very long validity periods (20+ years).

How do I get the fingerprint of a certificate?

Use cert.getEncoded() to get the DER bytes, then compute a hash (SHA-256 is common): MessageDigest.getInstance("SHA-256").digest(cert.getEncoded()). Convert to hex for display.

Go Defer Order and Stack Unwinding: Execution Gotchas Explained

Bar Dror — Wed, 21 Oct 2026 07:00:50 +0000

Originally published on Code Beneath.

Go's defer statement is powerful for cleanup, but golang defer execution order multiple defer statements follow a Last-In-First-Out (LIFO) stack that catches many engineers off guard. Unlike finally blocks in languages such as Java, deferred functions execute in reverse order of declaration, and that order becomes critical when managing file handles, database connections, or locks. Understanding how the defer stack unwinds, especially when panics occur, is the difference between a resource leak and bulletproof cleanup code.

The Defer Stack: LIFO Execution Model

When you write multiple defer statements in Go, each one pushes a function onto a stack. At function exit (whether normal return, panic, or error), Go pops and executes defers from the top of the stack downward. This LIFO behavior means the last defer you write runs first.

Here is a minimal example to demonstrate:

package main

import "fmt"

func main() {
    defer fmt.Println("First defer (declared first, runs THIRD)")
    defer fmt.Println("Second defer (declared second, runs SECOND)")
    defer fmt.Println("Third defer (declared third, runs FIRST)")
    fmt.Println("Main body executes")
}

Output:

Main body executes
Third defer (declared third, runs FIRST)
Second defer (declared second, runs SECOND)
First defer (declared first, runs THIRD)

This LIFO order is not accidental. It mirrors how a call stack unwinds. If you acquire resources in order A, B, C, you must release them in reverse: C, B, A. Go's defer ordering enforces that discipline automatically.

Resource Cleanup: A Real-World Pattern

The most common use case is opening multiple resources and ensuring they close in the correct order. Consider opening a file, then a gzip reader, then scanning lines:

package main

import (
    "bufio"
    "compress/gzip"
    "fmt"
    "os"
)

func processGzipFile(path string) error {
    // Open file
    file, err := os.Open(path)
    if err != nil {
        return err
    }
    defer func() {
        fmt.Println("Closing file")
        file.Close()
    }()

    // Wrap in gzip reader
    reader, err := gzip.NewReader(file)
    if err != nil {
        return err
    }
    defer func() {
        fmt.Println("Closing gzip reader")
        reader.Close()
    }()

    // Create scanner
    scanner := bufio.NewScanner(reader)
    defer func() {
        fmt.Println("Closing scanner")
    }()

    // Process lines
    for scanner.Scan() {
        fmt.Println(scanner.Text())
    }

    return scanner.Err()
}

func main() {
    // Create a test gzip file for demonstration
    tmpfile, _ := os.Create("test.gz")
    defer os.Remove("test.gz")

    // Write some gzipped content
    w := gzip.NewWriter(tmpfile)
    w.Write([]byte("Hello\nWorld\n"))
    w.Close()
    tmpfile.Close()

    // Now process it
    if err := processGzipFile("test.gz"); err != nil {
        fmt.Println("Error:", err)
    }
}

Output:

Hello
World
Closing scanner
Closing gzip reader
Closing file

Notice the order: scanner closes first (most recently acquired), gzip reader next, file last. This ordering ensures that the gzip reader's Close can flush buffered data to the file before the file itself closes. Reverse the order, and you risk data loss or dangling file descriptors.

Defer and Panic: Stack Unwinding Behavior

When a panic occurs, Go immediately unwinds the stack, executing defers along the way. If a defer itself calls recover(), the panic stops propagating. This is the mechanism for exception handling in Go.

package main

import "fmt"

func main() {
    defer func() {
        if r := recover(); r != nil {
            fmt.Println("Recovered from panic:", r)
        }
    }()

    defer fmt.Println("Defer 1: Runs even during panic")
    defer fmt.Println("Defer 2: Runs even during panic")

    fmt.Println("About to panic")
    panic("something went wrong")
    fmt.Println("This line never executes")
}

Output:

About to panic
Defer 2: Runs even during panic
Defer 1: Runs even during panic
Recovered from panic: something went wrong

Even during a panic, defers execute in LIFO order. The recover() function, when called inside a deferred closure at the top of the stack, catches the panic and prevents the program from crashing.

Deferred Values vs. Deferred Closures

A subtle gotcha: when you defer a function call with arguments, Go evaluates those arguments immediately, not when the defer executes. This matters when the arguments depend on mutable state.

package main

import "fmt"

func main() {
    x := 10
    defer fmt.Println("Deferred with value:", x)  // x is evaluated NOW as 10
    x = 20
    defer fmt.Println("Deferred with closure:", x) // x is captured, evaluated LATER as 20

    fmt.Println("Current x:", x)
}

Output:

Current x: 20
Deferred with closure: 20
Deferred with value: 10

The first defer uses a function call with an argument, so x is printed as 10 (its value when defer was declared). The second defer is a closure, so it captures x by reference and prints 20 (its value at execution time). This distinction is critical when deferring cleanup logic that depends on state.

Loop Defers: A Common Mistake

Declaring defer inside a loop creates one stack frame per iteration, but all defers still execute after the loop completes. If you are opening files in a loop and deferring close(), every close happens at the end, potentially keeping all files open simultaneously.

package main

import (
    "fmt"
    "os"
)

func main() {
    // WRONG: All files stay open until the loop exits
    for i := 1; i <= 3; i++ {
        filename := fmt.Sprintf("file%d.txt", i)
        file, err := os.Create(filename)
        if err != nil {
            fmt.Println("Error:", err)
            continue
        }
        defer func() {
            fmt.Printf("Closing %s\n", filename)
            file.Close()
            os.Remove(filename)
        }()
        fmt.Printf("Opened %s\n", filename)
    }
    fmt.Println("Loop done, now closing all files")
}

Output:

Opened file1.txt
Opened file2.txt
Opened file3.txt
Loop done, now closing all files
Closing file3.txt
Closing file2.txt
Closing file1.txt

All three files remain open for the duration of the loop. If you are processing thousands of files, this approach exhausts file descriptor limits. The correct pattern is to wrap the resource allocation in a separate function call:

package main

import (
    "fmt"
    "os"
)

func processFile(filename string) error {
    file, err := os.Create(filename)
    if err != nil {
        return err
    }
    defer func() {
        fmt.Printf("Closing %s\n", filename)
        file.Close()
        os.Remove(filename)
    }()

    fmt.Printf("Opened %s\n", filename)
    return nil
}

func main() {
    // CORRECT: Each processFile call defers and then exits, closing the file
    for i := 1; i <= 3; i++ {
        filename := fmt.Sprintf("file%d.txt", i)
        processFile(filename)
    }
    fmt.Println("All files closed as each processFile returned")
}

Output:

Opened file1.txt
Closing file1.txt
Opened file2.txt
Closing file2.txt
Opened file3.txt
Closing file3.txt
All files closed as each processFile returned

By extracting the file processing into its own function, the defer fires at each function return, ensuring files close immediately after use.

Defer with Named Return Values

Named return values can be modified by deferred functions, creating elegant error recovery patterns. A deferred function can inspect and even modify the return value:

package main

import (
    "fmt"
)

func divideWithCleanup(a, b int) (result int, err error) {
    defer func() {
        if err != nil {
            fmt.Println("Cleanup running due to error:", err)
            result = -1 // Modify the return value
        }
    }()

    if b == 0 {
        err = fmt.Errorf("division by zero")
        return
    }

    result = a / b
    return
}

func main() {
    val, err := divideWithCleanup(10, 2)
    fmt.Printf("Result: %d, Error: %v\n", val, err)

    val, err = divideWithCleanup(10, 0)
    fmt.Printf("Result: %d, Error: %v\n", val, err)
}

Output:

Result: 5, Error: <nil>
Cleanup running due to error: division by zero
Result: -1, Error: division by zero

The deferred function sees the named return values, err and result, and can modify them before the function exits. This is more precise than returning a new value, because the defer logic runs after the explicit return statement.

Comparison: Defer vs. Finally in Java

Go's defer is often compared to Java's finally block, but they differ in subtle ways:

Feature	Go Defer	Java Finally
Execution order with multiple blocks	LIFO (reverse declaration order)	FIFO (top to bottom)
Argument evaluation	Immediate (at defer declaration)	At finally block execution
Stack unwinding on panic	Executes all defers in LIFO order	Executes finally block
Can modify return value	Yes (via named returns)	Yes (if return type is not primitive)
Common use case	Resource cleanup, locks	Resource cleanup, try-catch-finally pattern

Java's finally is scoped to a try block, whereas Go's defer is scoped to the enclosing function. This means Go's defer is more flexible for cleanup distributed across a function, but also more error-prone if not used carefully. For more details on Go fundamentals, see our Go language guide.

Common Mistakes

1. Forgetting LIFO order: Declaring defer in the wrong order and then wondering why resources close prematurely or in the wrong sequence.

2. Deferring in loops without function boundaries: All defers from a loop execute after the loop, not after each iteration. Extract resource allocation into a helper function to close resources immediately.

3. Ignoring argument evaluation: Writing defer fmt.Println(someVariable) and expecting someVariable to be evaluated at defer execution, not at declaration. Use a closure if you need late binding.

4. Panicking inside a defer without recovering: If a deferred function panics, it will cause the entire program to crash if no higher-level recover() catches it. Always guard panicky operations.

5. Deferring heavy operations: Heavy I/O or computation inside a defer can block the function exit. Keep defers lightweight and focused on cleanup.

FAQ

What happens if a deferred function panics?

The panic propagates up to the next defer in the stack. If that defer calls recover(), the panic stops. Otherwise, the program crashes. This allows you to build layered error handling.

Can I use defer to modify the return value of a function?

Yes, but only if the return values are named. A deferred function can access and modify named return variables before the function truly exits. Unnamed returns cannot be modified by a defer.

Is there a limit to how many defers I can create?

Technically, no hard limit in the Go runtime, but each defer consumes a small amount of memory. Creating millions of defers in a loop is wasteful. Use function boundaries to limit defer stack depth.

How do I ensure cleanup happens even if the function returns early?

That is the whole point of defer. Place the defer immediately after acquiring the resource, and it executes at any function exit point: return, panic, or implicit return at the end of the function.

Can I cancel a defer once it is declared?

No, there is no built-in way to undefer a function. However, you can make the deferred function a no-op by setting a flag or checking state. For example, defer could check a flag before performing cleanup.

What is the performance cost of defer?

Modern Go (1.13+) has near-zero overhead for defer in the common case. The compiler optimizes tail-call defers efficiently. The cost is in allocating the defer frame on the stack, not in the execution itself. Avoid the micro-optimization of removing defer for performance unless profiling shows it is a real bottleneck.

HTTP Chunk Encoding and Transfer-Encoding: chunked in Practice

Bar Dror — Mon, 19 Oct 2026 07:00:54 +0000

Originally published on Code Beneath.

When you need to send an HTTP response but do not know the total size in advance, the chunked transfer encoding is your tool. Instead of buffering everything or setting Content-Length, you declare Transfer-Encoding: chunked and send data in discrete chunks. This is how live streams, real-time logs, and server-sent events work beneath the framework abstractions. An HTTP transfer encoding chunked example implementation shows the real wire format: chunk size in hex, the data, a CRLF pair, repeat, then a final zero-sized chunk with optional trailers. This article walks through working code that does exactly this, why size selection matters, and the gotchas that trap most engineers the first time.

Why Chunked Transfer Encoding

In HTTP/1.1, a response must tell the client how many bytes are coming. You either set Content-Length (which means you must know the size before you start sending headers) or you close the connection (HTTP/1.0 style, wasteful). Chunked encoding lets you send headers immediately, start streaming data, and signal the end later. The server never waits to buffer the entire body.

This is critical for:

Long-running computations that produce output incrementally
Large files served from disk without precomputing size
Real-time data feeds (stock tickers, log tail, metrics)
Server-sent events (SSE), which relies on chunked encoding by default
Proxies that do not know downstream content length

The HTTP/1.1 spec (RFC 7230, Section 4.1) defines the wire format precisely. You send each chunk as: a line with the chunk size in hexadecimal, a CRLF pair (carriage return, line feed: "\r\n"), the chunk data itself, another CRLF pair, and repeat. The stream ends with a chunk of size 0 followed by optional trailer headers and a final CRLF.

The Wire Format in Detail

Let me show you exactly what travels over the socket:

5\r\n
Hello\r\n
6\r\n
 World\r\n
0\r\n
\r\n

The "5" is hex for 5 bytes. Then 5 bytes of data ("Hello"). Then "6" for 6 bytes ("space World"). Then "0" signals the end, followed by a blank line (the final CRLF). The client reads the hex size, reads exactly that many bytes, discards the trailing CRLF, and repeats until it sees a zero chunk. Simple, but easy to get wrong when writing raw sockets.

Go Implementation: Manual Chunk Writing

Go's http.ResponseWriter detects when you do not set Content-Length and automatically uses chunked encoding. But let us do it manually first to show what really happens:

package main

import (
    "bufio"
    "fmt"
    "net"
    "time"
)

func main() {
    listener, err := net.Listen("tcp", ":8080")
    if err != nil {
        panic(err)
    }
    defer listener.Close()

    fmt.Println("Listening on :8080")

    for {
        conn, err := listener.Accept()
        if err != nil {
            panic(err)
        }
        go handleConnection(conn)
    }
}

func handleConnection(conn net.Conn) {
    defer conn.Close()

    reader := bufio.NewReader(conn)
    // Read the HTTP request line
    line, err := reader.ReadString('\n')
    if err != nil {
        return
    }

    // Consume the rest of the headers
    for {
        hline, err := reader.ReadString('\n')
        if err != nil {
            return
        }
        if hline == "\r\n" {
            break
        }
    }

    // Send HTTP response headers with chunked encoding
    response := "HTTP/1.1 200 OK\r\n"
    response += "Transfer-Encoding: chunked\r\n"
    response += "Content-Type: text/plain\r\n"
    response += "\r\n"

    conn.Write([]byte(response))

    writer := bufio.NewWriter(conn)

    // Send chunks manually
    chunks := []string{"Hello", " ", "World", "!", "\n", "Streaming ", "data."}

    for _, data := range chunks {
        // Write chunk size in hex
        chunkSize := fmt.Sprintf("%x\r\n", len(data))
        writer.WriteString(chunkSize)

        // Write chunk data
        writer.WriteString(data)

        // Write CRLF after data
        writer.WriteString("\r\n")

        writer.Flush()

        time.Sleep(500 * time.Millisecond)
    }

    // Send final zero-sized chunk (terminator)
    writer.WriteString("0\r\n")
    writer.WriteString("\r\n")
    writer.Flush()
}

Run this with `go run main.go` and connect with `curl -v http://localhost:8080`. You will see each chunk arrive as a separate write over the socket. curl will buffer and reassemble them transparently. The key lines:

`fmt.Sprintf("%x\r\n", len(data))` converts the byte count to hex and adds a CRLF
Every chunk has data and a trailing CRLF
The terminator "0\r\n\r\n" signals the end (chunk size 0, then blank line for optional trailers)
`writer.Flush()` forces the data onto the socket immediately, simulating streaming

Python Implementation with Real-Time Output

Python's http.server.BaseHTTPRequestHandler hides chunking by default. Here is a low-level socket version:

#!/usr/bin/env python3

import socket
import time

def handle_client(client_socket, addr):
    # Read request
    request = client_socket.recv(4096).decode('utf-8')
    if not request:
        client_socket.close()
        return

    # Send HTTP headers
    response_headers = (
        "HTTP/1.1 200 OK\r\n"
        "Transfer-Encoding: chunked\r\n"
        "Content-Type: text/plain\r\n"
        "\r\n"
    )
    client_socket.sendall(response_headers.encode('utf-8'))

    # Generator that yields chunks
    def stream_chunks():
        yield "Line 1: Processing started\n"
        time.sleep(0.5)
        yield "Line 2: Still working\n"
        time.sleep(0.5)
        yield "Line 3: Almost done\n"
        time.sleep(0.5)
        yield "Line 4: Complete\n"

    # Send each chunk
    for data in stream_chunks():
        # Chunk size in hex
        chunk_size = hex(len(data))[2:]  # Remove '0x' prefix

        # Format: size\r\ndata\r\n
        chunk_line = f"{chunk_size}\r\n"
        client_socket.sendall(chunk_line.encode('utf-8'))
        client_socket.sendall(data.encode('utf-8'))
        client_socket.sendall(b"\r\n")

    # Send terminating chunk
    client_socket.sendall(b"0\r\n\r\n")
    client_socket.close()

def main():
    server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    server_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    server_socket.bind(('127.0.0.1', 8081))
    server_socket.listen(5)

    print("Server listening on 127.0.0.1:8081")

    try:
        while True:
            client_socket, addr = server_socket.accept()
            print(f"Connection from {addr}")
            handle_client(client_socket, addr)
    except KeyboardInterrupt:
        print("Shutting down")
    finally:
        server_socket.close()

if __name__ == "__main__":
    main()

Run this with `python3 main.py` and curl it:

curl -v http://127.0.0.1:8081

You will see the response arrive in real time, 500ms apart, with each chunk size printed by curl. The hex conversion (`hex(len(data))[2:]`) is the Python equivalent of Go's `fmt.Sprintf("%x", ...)`.

Chunk Size Selection

How big should each chunk be? There is no magic number. Common strategies:

Strategy	Size	Use Case
Line-based	Variable (each line)	Logs, line-delimited JSON, text streams
Fixed size	4KB, 8KB, 64KB	File streaming, general-purpose
Semantic unit	1 JSON object, 1 frame	SSE, protocol buffers, video frames
Flush on timeout	Buffered + 100ms	Interactive streams (avoid latency)

Smaller chunks mean lower latency (client sees data sooner) but more overhead (more CRLF pairs, more writes). Larger chunks mean better throughput but higher latency. For interactive streams, prefer smaller chunks or flush on timeout. For bulk file transfer, 64KB chunks are standard.

Common Mistakes

Forgetting the trailing CRLF after data

Every chunk must have a CRLF after the data itself, not just after the size line. Missing this breaks the format and hangs the client.

Not flushing to the socket

Buffering all your chunks and sending them at once defeats the purpose of chunked encoding. Call `flush()` (Python) or `Flush()` (Go bufio) after each chunk if you want streaming behavior.

Mixing chunked with Content-Length

Set Transfer-Encoding: chunked OR Content-Length, never both (HTTP/1.1 forbids it). If both are present, Transfer-Encoding takes precedence and Content-Length is ignored, leading to confusion.

Forgetting the terminating zero chunk

The client waits for "0\r\n\r\n" to know the response is complete. Without it, the client hangs. Always send it, even if you send nothing else.

Sending non-hex chunk sizes

The size must be in hexadecimal, lowercase or uppercase (RFC says lowercase is preferred). Sending decimal sizes (e.g., "10" for 10 bytes instead of "a") creates a protocol violation.

Trailers: Sending Headers After Data

Chunked encoding allows optional trailer headers after the final chunk. This is useful for checksums, signatures, or metadata computed during streaming:

// In Go, after the last chunk:
writer.WriteString("0\r\n")
writer.WriteString("X-Checksum: abc123\r\n")
writer.WriteString("X-Rows-Processed: 1000\r\n")
writer.WriteString("\r\n")
writer.Flush()

The format is: zero chunk size, then trailer headers in the same format as response headers, then a final CRLF. Not all clients support trailers (trailers were poorly supported in HTTP/1.1 and are better in HTTP/2), so test thoroughly. For dynamic HTTP response types, trailers can signal computed metadata.

Browser and Client Behavior

Browsers handle chunked responses transparently. curl, wget, and httpx all reassemble chunks into the full body before the application sees it. This is why developers often forget chunking exists: the abstraction hides it. The real test is watching with tcpdump or Wireshark to confirm chunks arrive separately on the wire.

Some differences:

Browsers do not display partial content incrementally by default (though they do for SSE with text/event-stream)
CLI tools like curl buffer everything before printing
JavaScript fetch() receives events as chunks arrive if you use a ReadableStream
HTTP/2 and HTTP/3 do not use chunked encoding at all (they have native streaming), so Transfer-Encoding: chunked is HTTP/1.1 specific

See HTTP basics for more on protocol versions.

Testing and Debugging

To see the raw bytes on the wire:

curl --raw http://localhost:8080 | od -c

Or with netcat:

echo -e "GET / HTTP/1.1\r\nHost: localhost\r\n\r\n" | nc localhost 8080 | od -c

You will see the hex chunk sizes, the data, and the CRLF sequences clearly. Wireshark is also invaluable: capture port 8080, filter by "http.transfer_encoding == chunked", and inspect the frames.

FAQ

Do I have to use chunked encoding if I don't know Content-Length?

No. You can close the connection after the body (Connection: close), and HTTP/1.0 clients will work. But chunked encoding is cleaner for HTTP/1.1 because it allows Connection: keep-alive and pipelining.

What happens if I send a chunk size larger than the actual data?

The client reads that many bytes and hangs waiting for more. You have created a protocol violation. The client times out, confused. Always send the exact number of bytes you declare in the chunk size.

Can I send chunks in different orders or out of sequence?

No. The HTTP/1.1 specification requires chunks to arrive in order and to be processed sequentially. Reordering them breaks the protocol.

Does HTTP/2 support chunked encoding?

No. HTTP/2 uses frames instead, with native streaming support. The Transfer-Encoding header is not sent over HTTP/2 connections. The proxy or server translates chunked to frames transparently.

What is the maximum chunk size?

Theoretically, the chunk size hex can be as large as your system's integer can represent. Practically, most servers and clients limit it to 2GB or less. Stick to reasonable sizes (4KB to 1MB) unless you have a specific reason not to.

Should I compress chunked responses?

Yes. You can send Transfer-Encoding: chunked and Content-Encoding: gzip together. Many servers (nginx, Apache) do this automatically. Write your data to a gzip compressor, then chunk the compressed output.

TCP Socket Graceful Shutdown in Go: Closing Connections Without Data Loss

Bar Dror — Fri, 16 Oct 2026 07:00:58 +0000

Originally published on Code Beneath.

When you shut down a network service, you cannot simply close the listening socket and exit. Clients might have in-flight requests, half-written buffers, or active reads waiting for your response. A golang TCP socket graceful shutdown example requires coordinating context cancellation, draining pending connections, and enforcing timeouts so no data disappears. This is not theoretical: lose proper shutdown and you will drop customer transactions, corrupt state, and make debugging hell.

The Problem: Why Abrupt Shutdown Fails

A naive shutdown looks like this: close the listener, exit the main function, the OS kills all goroutines, sockets close hard. Any client mid-read gets an EOF. Any server trying to write gets a broken pipe error. Uncommitted state, incomplete transactions, nothing graceful about it.

The correct pattern is: stop accepting new connections, give existing connections time to finish, close those connections cleanly, only then exit. TCP and UDP sockets have different shutdown semantics, and TCP requires a coordinated close where both sides send FIN. Go's context package and sync.WaitGroup are the tools to choreograph this.

Architecture: Context, WaitGroup, and Timeouts

The pattern uses three layers. First, a context tied to the service lifecycle. Second, a WaitGroup that tracks active connection handlers. Third, a shutdown timeout that forces closure if handlers don't finish fast enough. When shutdown is triggered, you close the listener (which unblocks Accept calls), drain remaining handlers by waiting on the WaitGroup, then close sockets with a deadline.

Here is a complete server and client demonstrating proper golang TCP socket graceful shutdown:

package main

import (
    "bufio"
    "context"
    "fmt"
    "io"
    "log"
    "net"
    "os"
    "os/signal"
    "sync"
    "syscall"
    "time"
)

// Server holds the listener and connection tracking.
type Server struct {
    listener net.Listener
    wg       sync.WaitGroup
    ctx      context.Context
    cancel   context.CancelFunc
}

// NewServer creates a server bound to addr.
func NewServer(addr string) (*Server, error) {
    listener, err := net.Listen("tcp", addr)
    if err != nil {
        return nil, err
    }
    ctx, cancel := context.WithCancel(context.Background())
    return &Server{
        listener: listener,
        ctx:      ctx,
        cancel:   cancel,
    }, nil
}

// Start accepts connections in a goroutine.
func (s *Server) Start() {
    s.wg.Add(1)
    go func() {
        defer s.wg.Done()
        for {
            conn, err := s.listener.Accept()
            if err != nil {
                // Listener closed or accept failed.
                select {
                case <-s.ctx.Done():
                    // Shutdown in progress, exit cleanly.
                    return
                default:
                    log.Printf("accept error: %v\n", err)
                    return
                }
            }
            // Handle each connection in its own goroutine.
            s.wg.Add(1)
            go s.handleConnection(conn)
        }
    }()
}

// handleConnection reads a message, processes it, and sends a response.
func (s *Server) handleConnection(conn net.Conn) {
    defer s.wg.Done()
    defer conn.Close()

    // Wrap the connection with a reader and writer.
    reader := bufio.NewReader(conn)
    writer := bufio.NewWriter(conn)

    for {
        // Set read deadline using context or a fixed timeout.
        // This prevents reads from blocking forever during shutdown.
        deadline, ok := s.ctx.Deadline()
        if !ok {
            // No deadline on context, use a reasonable read timeout.
            deadline = time.Now().Add(10 * time.Second)
        }
        conn.SetReadDeadline(deadline)

        // Read a line from the client.
        line, err := reader.ReadString('\n')
        if err != nil {
            if err == io.EOF {
                // Client closed cleanly.
                return
            }
            // Check if this is a deadline exceeded error during shutdown.
            if netErr, ok := err.(net.Error); ok && netErr.Timeout() {
                select {
                case <-s.ctx.Done():
                    // Context canceled, shutdown in progress.
                    return
                default:
                    log.Printf("read timeout: %v\n", err)
                    return
                }
            }
            log.Printf("read error: %v\n", err)
            return
        }

        // Check if shutdown was initiated.
        select {
        case <-s.ctx.Done():
            // Write a goodbye message and close.
            fmt.Fprintf(writer, "Server shutting down\n")
            writer.Flush()
            return
        default:
        }

        // Echo the message back with a timestamp.
        response := fmt.Sprintf("Echo: %s received at %s\n", line[:len(line)-1], time.Now().Format(time.RFC3339))
        if _, err := fmt.Fprint(writer, response); err != nil {
            log.Printf("write error: %v\n", err)
            return
        }
        if err := writer.Flush(); err != nil {
            log.Printf("flush error: %v\n", err)
            return
        }
    }
}

// Shutdown gracefully closes the server.
// It stops accepting connections, waits for existing handlers to finish,
// and enforces a timeout to prevent indefinite hangs.
func (s *Server) Shutdown(timeout time.Duration) error {
    log.Println("initiating graceful shutdown...")

    // Close the listener. This unblocks Accept() calls.
    if err := s.listener.Close(); err != nil {
        log.Printf("listener close error: %v\n", err)
    }

    // Signal all handlers that shutdown is in progress.
    s.cancel()

    // Wait for all handlers to finish, with a timeout.
    done := make(chan struct{})
    go func() {
        s.wg.Wait()
        close(done)
    }()

    select {
    case <-done:
        log.Println("all connections closed")
        return nil
    case <-time.After(timeout):
        log.Printf("shutdown timeout after %v\n", timeout)
        return fmt.Errorf("shutdown timeout")
    }
}

// Client connects to the server and sends messages.
func runClient(addr string, messages []string) error {
    conn, err := net.Dial("tcp", addr)
    if err != nil {
        return err
    }
    defer conn.Close()

    writer := bufio.NewWriter(conn)
    reader := bufio.NewReader(conn)

    for _, msg := range messages {
        // Send message.
        if _, err := fmt.Fprintf(writer, "%s\n", msg); err != nil {
            return err
        }
        if err := writer.Flush(); err != nil {
            return err
        }

        // Read response with a timeout.
        conn.SetReadDeadline(time.Now().Add(5 * time.Second))
        response, err := reader.ReadString('\n')
        if err != nil {
            return err
        }
        fmt.Print(response)

        time.Sleep(500 * time.Millisecond)
    }
    return nil
}

func main() {
    // Start server.
    server, err := NewServer("127.0.0.1:9999")
    if err != nil {
        log.Fatalf("failed to create server: %v\n", err)
    }
    server.Start()
    log.Println("server listening on 127.0.0.1:9999")

    // If an argument is passed, run as client. Otherwise run as server.
    if len(os.Args) > 1 && os.Args[1] == "client" {
        time.Sleep(1 * time.Second) // Give server time to start.
        err := runClient("127.0.0.1:9999", []string{"hello", "world", "test"})
        if err != nil {
            log.Fatalf("client error: %v\n", err)
        }
        return
    }

    // Server: wait for interrupt signal.
    sigChan := make(chan os.Signal, 1)
    signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)
    <-sigChan

    // Shutdown with a 10-second timeout.
    if err := server.Shutdown(10 * time.Second); err != nil {
        log.Printf("shutdown error: %v\n", err)
    }
}

To run this code:

go run main.go &
sleep 0.5
go run main.go client
kill %1

In another terminal, press Ctrl+C to trigger the graceful shutdown. The server closes the listener, prevents new accepts, waits for the current connection handler to finish reading and writing, then exits cleanly.

Why This Works: The Three Phases

Phase 1: Stop accepting. Close the listener. The Accept loop receives an error and exits if the context is already canceled, or it checks the context on the next iteration.

Phase 2: Signal handlers. Call cancel() on the context. Every goroutine listening on the context exits cleanly. Handlers that are mid-read will eventually see the context done, then they break the read loop and defer close the connection.

Phase 3: Wait with timeout. Use a WaitGroup to block until all handlers are done. If they are not done after the timeout, the shutdown returns an error and the process exits anyway (because main exits), but at least we gave handlers a chance.

Key Details and Edge Cases

Use SetReadDeadline and SetWriteDeadline on the connection. If you leave reads and writes undeadlined, a handler can hang forever waiting for a client, and your shutdown will block forever waiting for that handler. The deadline forces the read to unblock, you check the context, and you exit if shutdown is in progress.

Check the context immediately after returning from a blocking I/O call. A read might timeout naturally (not due to shutdown). You need to distinguish: did the timeout happen because shutdown was signaled, or did the client just go silent? If the context is done, exit. If not, it might be a transient issue; log and return.

Use sync.WaitGroup carefully. Add to the group before starting a goroutine, not after. Use defer Done() so the count decreases even if a goroutine panics. Forget to Add or Done and you will have goroutine leaks or a panic in Wait.

Buffered I/O (bufio.Writer) requires explicit Flush before the connection closes. A writer might hold data in a 4096-byte buffer. If you close the connection without flushing, the last chunk of the response is lost. Always defer Flush or call it after the final Write in critical sections.

Go's goroutine model makes this pattern clean because goroutines are cheap and the select statement is built for multiplexing context. In languages without goroutines (Java threads are heavier, C requires manual event loops), this requires more infrastructure.

Common Mistakes

Mistake 1: Closing the listener but not canceling the context. The Accept loop exits because Accept fails, but handlers still run. You wait on a WaitGroup that never decrements because no handler checks for shutdown. Result: deadlock on Shutdown.

Mistake 2: Not setting read/write deadlines. A handler blocked on a read never sees the context. The WaitGroup.Wait() call hangs forever. You hit the shutdown timeout and kill the process, dropping the in-flight data from that handler.

Mistake 3: Forgetting to flush buffered writers before closing. The response is never sent. The client sees EOF mid-response and logs an error. The data is lost even though the handler thought it succeeded.

Mistake 4: Too short a shutdown timeout. If you set timeout to 1 millisecond, a handler doing any non-trivial work (database query, file I/O) cannot finish. You force-close and lose data. Start with 10-30 seconds and tune based on your handler latency.

Comparing Approaches

Approach	Pros	Cons
Abrupt close (no shutdown)	Simple, no coordination needed	Data loss, client errors, no cleanup
Close listener, immediate exit	Faster than waiting, some handlers finish naturally	Pending handlers still run but may be interrupted
Context cancel with WaitGroup and timeout (recommended)	Clean signal, handlers have time to finish, timeout prevents hangs	More code, requires discipline on deadlines
HTTP server with Server.Shutdown()	Built-in, handles connection draining, request context automatically canceled	Limited to HTTP, abstracts the mechanics

For raw TCP services, you implement the pattern yourself. For HTTP, use the standard library's Server.Shutdown() method, which does all this under the hood.

Testing Graceful Shutdown

Write a test that connects to the server, starts a slow operation (sleep for a second), then sends a shutdown signal. Verify the slow operation completes (no data loss). Then test with a very short timeout: verify the shutdown function returns a timeout error but the process exits.

func TestGracefulShutdown(t *testing.T) {
    server, err := NewServer("127.0.0.1:0") // Port 0 = OS chooses
    if err != nil {
        t.Fatalf("failed to create server: %v", err)
    }
    server.Start()
    addr := server.listener.Addr().String()

    // Connect and send a message.
    go func() {
        conn, _ := net.Dial("tcp", addr)
        defer conn.Close()
        fmt.Fprintf(conn, "test\n")
        io.ReadAll(conn) // Read until EOF
    }()

    time.Sleep(100 * time.Millisecond)

    // Shutdown should complete without timeout.
    err = server.Shutdown(5 * time.Second)
    if err != nil {
        t.Errorf("shutdown failed: %v", err)
    }
}

FAQ

What if a handler is stuck in an infinite loop and never checks the context?

The shutdown timeout expires and you force-exit. This is a bug in your handler code, not the shutdown mechanism. Every handler must have checkpoints where it checks the context or has a deadline. Handlers doing long-running compute (no I/O) should explicitly check the context periodically using select <-ctx.Done().

Can I use context.WithTimeout instead of context.WithCancel?

Yes, if you know exactly how long the service should run. But WithCancel is cleaner for shutdown because the timeout is not about the service lifetime, it is about the shutdown grace period. Use WithCancel for the service, then pass a separate timeout to the Shutdown function.

Do I need a WaitGroup if I am not spawning many goroutines?

No, but do not skip it. Even if you think your service only handles a few connections, adding WaitGroup costs nothing and prevents bugs when load increases. It is defensive programming.

What happens if a client reconnects after I close the listener?

If the listener is closed, new connection attempts will fail with "connection refused". Existing connections are not affected until you close them. This is correct behavior: you stop accepting new work but let current work finish.

Should I set deadlines on the server side or let the client timeout?

Set them on both. Server-side deadlines prevent handlers from hanging indefinitely, especially during shutdown. Client-side timeouts are for client resilience. If either one fires, the connection is closed and resources are freed.

Does this pattern work for UDP?

UDP is connectionless, so there is no connection to gracefully close. You close the socket and any pending receives return an error immediately. The pattern is simpler: just cancel the context and exit the receive loop.

How do I know if my shutdown was successful?

Check the error return from Shutdown. If it is nil, all handlers finished before the timeout. If it is a timeout error, handlers were force-closed. Log this and monitor it in production. A persistent timeout error might indicate handlers are not checking the context properly.

Unix Domain Sockets vs TCP Loopback: Faster Local IPC

Bar Dror — Wed, 14 Oct 2026 07:00:58 +0000

Originally published on Code Beneath.

When you need two processes on the same machine to communicate, you have a choice: bind to 127.0.0.1 on a TCP port, or use a Unix domain socket. The question of unix domain socket vs tcp localhost matters because one is faster, simpler to clean up, and skips the entire TCP stack. Yet many engineers default to TCP loopback out of habit or framework convenience, never measuring the actual cost. This article shows you exactly what happens beneath each approach and gives you runnable code to benchmark both.

What is a Unix Domain Socket?

A Unix domain socket (UDS) is a bidirectional communication channel between processes on the same host. It is a file-like object, not a network socket. The kernel passes data through shared memory and queues, not through the TCP/IP stack. You create a socket file, usually in /tmp, and both client and server open it by path.

In contrast, TCP loopback (127.0.0.1) is a real network socket. Even though traffic never leaves the machine, it still travels through the network stack: network layer (IP), transport layer (TCP), checksum computation, buffer management, and all the overhead designed for remote peers.

The key difference: UDS works at the filesystem and IPC layer; TCP loopback works at the network layer. For local processes only, UDS is a lighter tool.

Architecture and Kernel Behavior

When you send data over a Unix domain socket, the kernel writes it directly into a shared buffer associated with that socket. The receiving process reads from the same buffer. No IP headers, no TCP state machine, no retransmission logic.

TCP loopback still runs the full TCP state machine. The sender's kernel constructs a TCP packet, the receiver's kernel parses it, manages window size, handles ACKs, and manages the socket buffer. For every send and receive on a loopback TCP socket, you pay this cost.

Unix domain sockets also have simpler connection semantics. There is no SYN-ACK handshake, no TIME-WAIT state, no lingering file descriptor cleanup. When you close a Unix socket file, it is gone.

A third advantage: credentials. Unix domain sockets let you query the peer process's UID and GID directly via SO_PEERCRED (Linux) or LOCAL_PEERCRED (BSD). This is a built-in security feature that TCP loopback cannot match without an application-level protocol.

Performance: Runnable Benchmark

Here is a complete, runnable Go program that creates both a Unix domain socket server and a TCP loopback server, runs concurrent clients against each, and measures throughput. This is the real data you need to make a decision.

package main

import (
    "fmt"
    "io"
    "net"
    "os"
    "sync"
    "sync/atomic"
    "time"
)

// Message is a fixed-size payload for predictable latency testing
type Message struct {
    payload [256]byte
}

// startUnixServer starts a Unix domain socket server
func startUnixServer(socketPath string, done chan struct{}) error {
    // Remove stale socket file
    os.Remove(socketPath)

    listener, err := net.Listen("unix", socketPath)
    if err != nil {
        return err
    }
    defer os.Remove(socketPath)

    go func() {
        for {
            select {
            case <-done:
                listener.Close()
                return
            default:
            }
            listener.SetDeadline(time.Now().Add(100 * time.Millisecond))
            conn, err := listener.Accept()
            if err != nil {
                continue
            }
            go handleConn(conn)
        }
    }()

    return nil
}

// startTCPServer starts a TCP loopback server
func startTCPServer(addr string, done chan struct{}) error {
    listener, err := net.Listen("tcp", addr)
    if err != nil {
        return err
    }

    go func() {
        for {
            select {
            case <-done:
                listener.Close()
                return
            default:
            }
            listener.SetDeadline(time.Now().Add(100 * time.Millisecond))
            conn, err := listener.Accept()
            if err != nil {
                continue
            }
            go handleConn(conn)
        }
    }()

    return nil
}

// handleConn echoes back everything it receives
func handleConn(conn net.Conn) {
    defer conn.Close()
    buf := make([]byte, 256)
    for {
        n, err := conn.Read(buf)
        if err != nil {
            return
        }
        conn.Write(buf[:n])
    }
}

// benchmarkClient runs a fixed number of requests and counts round-trip time
func benchmarkClient(network, addr string, numRequests int) time.Duration {
    start := time.Now()
    conn, err := net.Dial(network, addr)
    if err != nil {
        panic(err)
    }
    defer conn.Close()

    msg := [256]byte{1, 2, 3, 4}
    reply := make([]byte, 256)

    for i := 0; i < numRequests; i++ {
        conn.Write(msg[:])
        conn.Read(reply)
    }

    return time.Since(start)
}

// benchmarkConcurrent runs multiple concurrent clients
func benchmarkConcurrent(network, addr string, numClients, requestsPerClient int) (time.Duration, int64) {
    start := time.Now()
    var wg sync.WaitGroup
    var successCount int64

    for i := 0; i < numClients; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            conn, err := net.Dial(network, addr)
            if err != nil {
                return
            }
            defer conn.Close()

            msg := [256]byte{1, 2, 3, 4}
            reply := make([]byte, 256)

            for j := 0; j < requestsPerClient; j++ {
                if _, err := conn.Write(msg[:]); err != nil {
                    return
                }
                if _, err := conn.Read(reply); err != nil {
                    return
                }
                atomic.AddInt64(&successCount, 1)
            }
        }()
    }

    wg.Wait()
    return time.Since(start), atomic.LoadInt64(&successCount)
}

func main() {
    socketPath := "/tmp/test.sock"
    tcpAddr := "127.0.0.1:9876"

    done := make(chan struct{})

    // Start servers
    if err := startUnixServer(socketPath, done); err != nil {
        panic(err)
    }
    if err := startTCPServer(tcpAddr, done); err != nil {
        panic(err)
    }

    // Let servers start
    time.Sleep(100 * time.Millisecond)

    fmt.Println("=== Single Client Benchmark (1000 requests) ===")
    unixTime := benchmarkClient("unix", socketPath, 1000)
    tcpTime := benchmarkClient("tcp", tcpAddr, 1000)

    fmt.Printf("Unix Domain Socket: %v\n", unixTime)
    fmt.Printf("TCP Loopback:       %v\n", tcpTime)
    fmt.Printf("Ratio (TCP/Unix):   %.2fx\n", float64(tcpTime)/float64(unixTime))

    fmt.Println()
    fmt.Println("=== Concurrent Benchmark (10 clients, 1000 requests each) ===")
    unixDur, unixCount := benchmarkConcurrent("unix", socketPath, 10, 1000)
    tcpDur, tcpCount := benchmarkConcurrent("tcp", tcpAddr, 10, 1000)

    fmt.Printf("Unix Domain Socket: %v (%d requests/sec: %.0f)\n",
        unixDur, unixCount, float64(unixCount)/unixDur.Seconds())
    fmt.Printf("TCP Loopback:       %v (%d requests/sec: %.0f)\n",
        tcpDur, tcpCount, float64(tcpCount)/tcpDur.Seconds())
    fmt.Printf("Ratio (TCP/Unix):   %.2fx\n", float64(tcpDur)/float64(unixDur))

    close(done)
}

Run this on your machine:

go run benchmark.go

Expected output on a typical Linux system (results vary by hardware and load):

=== Single Client Benchmark (1000 requests) ===
Unix Domain Socket: 45ms
TCP Loopback:       78ms
Ratio (TCP/Unix):   1.73x

=== Concurrent Benchmark (10 clients, 1000 requests each) ===
Unix Domain Socket: 198ms (10000 requests/sec: 50505.05)
TCP Loopback:       312ms (10000 requests/sec: 32051.28)
Ratio (TCP/Unix):   1.58x

On modern hardware with low load, Unix domain sockets are consistently 1.5x to 2x faster for local IPC. The gap widens under system load or with smaller message sizes, because the TCP overhead becomes a larger fraction of the total time.

When TCP Loopback Makes Sense

Unix domain sockets are not universally better. TCP loopback has real advantages in specific scenarios:

Process isolation: If you run a process in a container or a separate mount namespace, it may not have access to the Unix socket file. TCP loopback works across namespace boundaries in some configurations.

Network debuggability: You can use netstat, ss, lsof, and packet capture tools to inspect TCP loopback traffic. Unix sockets require different tools (lsof still works, but packet capture does not).

Language and platform compatibility: Every language supports TCP sockets. Not every language has first-class Unix socket support (though most do now).

Firewall and network policies: If your organization has strict socket-level policies that only allow TCP, you may have no choice. Unix sockets are not network sockets, so policies written for the network layer do not apply.

Portability to non-Unix systems: Windows has Unix domain sockets in recent versions (Windows 10+), but if you need to support older Windows, use TCP. See our TCP and UDP socket article for comprehensive coverage of network sockets across platforms.

File Descriptor Management

A subtle difference: Unix domain socket files persist on disk until explicitly deleted. If your process crashes or hangs, the socket file remains, and the next invocation may fail because the file already exists. You must clean it up in code or with a systemd service configuration.

TCP sockets are cleaned up automatically by the kernel after TIME-WAIT expires (typically 60 seconds). If you restart a TCP loopback server on the same port within a few seconds, you risk "Address already in use" errors, but this is temporary and usually predictable.

The benchmark code above includes os.Remove(socketPath) in the server to handle this. In production, you should wrap this in error handling or use Go's runtime/debug.SetTraceback to help diagnose cleanup issues.

Comparison Table

Aspect	Unix Domain Socket	TCP Loopback
Speed (local)	1.5x-2x faster	Baseline
Overhead	Minimal (IPC layer only)	Full TCP/IP stack
File cleanup	Manual (os.Remove required)	Automatic (kernel manages)
Credentials access	Yes (SO_PEERCRED)	No (requires app protocol)
Cross-namespace	No (filesystem-based)	Yes (loopback interface)
Debuggability	lsof only	lsof, netstat, tcpdump
Language support	Most (not 100%)	Universal

Common Mistakes

Forgetting to clean up socket files: If your code creates a Unix socket and the process crashes, the file remains. The next run will fail with "Address already in use" because listen() will try to bind to an existing file. Always defer os.Remove or handle startup cleanup.

Assuming Unix sockets are always faster: They usually are for simple echo-like workloads, but large message sizes and high-bandwidth scenarios may saturate the socket buffer, and the advantage shrinks. Benchmark your actual workload.

Neglecting permissions: Unix socket files have filesystem permissions. If your client and server run under different UIDs, you may get permission denied errors. Explicit chmod or running under the same user avoids this.

Ignoring namespace boundaries: Containerized services in separate namespaces cannot share Unix socket files by default. TCP loopback inside a container works fine, but you need explicit network configuration for Unix sockets (volume mounts or shared namespaces).

Not setting read/write deadlines: Both Unix and TCP sockets can hang if the peer crashes or stops reading. Use conn.SetReadDeadline() and conn.SetWriteDeadline() to avoid blocking indefinitely.

Integration with Existing Code

If you have an existing TCP loopback service, switching to Unix sockets is straightforward in Go, Java, Python, and most languages. Change only the Listen and Dial calls:

// Before (TCP)
listener, _ := net.Listen("tcp", "127.0.0.1:8080")
conn, _ := net.Dial("tcp", "127.0.0.1:8080")

// After (Unix)
listener, _ := net.Listen("unix", "/tmp/myservice.sock")
conn, _ := net.Dial("unix", "/tmp/myservice.sock")

The rest of your I/O code stays identical because both satisfy net.Listener and net.Conn in Go. In other languages (Java, Python), the socket abstraction is similarly neutral.

FAQ

Do I always need to use Unix sockets for local IPC?

No. Use TCP loopback if you need cross-namespace communication, broader tool support, or language compatibility you are uncertain about. Unix sockets win for same-machine performance and security (credentials). If the service will eventually be remote, TCP is the better long-term choice.

Can I use Unix domain sockets on Windows?

Yes, as of Windows 10 (build 17063) and Windows Server 2019. Older Windows versions do not support them. Most production Windows services still use TCP loopback.

How do I set socket permissions safely?

After creating a Unix socket file, use os.Chmod in Go (or chmod in shell) to set restrictive permissions: 0600 for private, 0660 for a shared group. Query peer credentials with SO_PEERCRED to verify identity if needed.

What happens if two processes try to bind the same Unix socket path?

The second listen() call fails with "Address already in use" (technically EADDRINUSE). Unlike TCP, where TIME-WAIT allows eventual reuse, the Unix socket file must be deleted explicitly before rebinding.

Can I use Unix sockets for inter-container communication?

Only if the containers share a volume or mount namespace. Docker's bridge network does not extend to Unix socket files. For inter-container IPC, TCP loopback on a user-defined network is cleaner. For intra-container (separate processes in one container), Unix sockets are ideal.

Do abstract Unix sockets avoid the file cleanup problem?

Yes. On Linux, you can use a leading null byte in the socket path to create an abstract Unix socket: "\0myservice" (null-terminated string in the socket path). These live only in kernel memory and are cleaned up automatically. Not portable to other Unix variants, but good for containerized services.

How does performance scale with message size?

Unix sockets maintain their advantage even with larger messages (up to the socket buffer size, often 208 KB). The TCP overhead is mostly in the protocol machine and buffer management, not message copying. Benchmark your actual message size to be sure.

Can I multiplex multiple connections on one Unix socket?

Yes. A single Unix domain socket listener accepts multiple connections, just like TCP. The server then multiplexes them (with goroutines, select, epoll, etc.) the same way it would for TCP connections. See our general socket architecture article for multiplexing patterns.

HTTP Keep-Alive and Connection Pooling, Measured

Bar Dror — Mon, 12 Oct 2026 07:00:45 +0000

Originally published on Code Beneath.

When you make an HTTP request without keep-alive, the client opens a fresh TCP connection, performs a three-way handshake, sends the request, receives the response, and closes the connection. If you're making dozens of requests per second to the same host, you're burning CPU and network round-trips on handshakes alone. HTTP keep-alive and connection pooling explained: keep-alive lets you reuse a TCP connection across multiple HTTP requests, while connection pooling lets you maintain a pool of warm connections ready to go. The real cost difference is measurable, and this article shows you exactly what happens inside the socket layer when you enable or disable keep-alive, and how to measure it in your own services.

Why Keep-Alive Matters

A TCP handshake costs at least one round-trip time (RTT). In a data center with 100 microsecond RTT, that's already a 100 microsecond tax on every request. Over the network, a typical RTT is 50-200 milliseconds. If your service makes 100 requests per second to a backend, you're spending 5-20 seconds per second just on handshakes. That's not hyperbole. Add TLS negotiation (one more RTT for TLS 1.3, two for TLS 1.2), and a "simple" request to a remote HTTPS endpoint can take 100-300 milliseconds just for the ceremony before the first byte of the request even sends.

Keep-alive solves this by keeping the TCP connection open after the response. The next request reuses the same socket, skipping the handshake. Connection pooling takes this further: instead of sharing a single connection (which becomes a bottleneck under load), you maintain a pool of N warm connections, so multiple concurrent requests can proceed in parallel without waiting for each other.

How Keep-Alive Works

HTTP/1.1 made keep-alive the default. Each request includes a Connection header; the client sends Connection: close to signal "close after this response," or the server does the same. If neither side sends close, the connection stays open. The server may also send Keep-Alive: timeout=60, max=100 to tell the client "I'll keep this connection alive for up to 60 seconds and 100 requests."

At the socket level, nothing special happens. The kernel buffers the data, TCP stays established, and the client can send a new HTTP request immediately. The server reads the new request from the same socket, processes it, and sends a response. From a system calls perspective, keep-alive eliminates the socket(), connect(), and close() calls, and for TLS, it eliminates the TLS 1.3 handshake.

Measuring the Real Difference

Let's build a tiny benchmark that makes HTTP requests with and without keep-alive, measuring requests per second. This Go program starts a local server and a client that fires requests at it. The server responds with a fixed 1 KB payload. We'll measure two scenarios: one connection per request (no keep-alive), and one pool of connections with keep-alive.

package main

import (
    "fmt"
    "io"
    "net"
    "net/http"
    "sync"
    "sync/atomic"
    "time"
)

func startServer() *http.Server {
    mux := http.NewServeMux()
    mux.HandleFunc("/test", func(w http.ResponseWriter, r *http.Request) {
        payload := make([]byte, 1024)
        w.Write(payload)
    })

    server := &http.Server{
        Addr:           ":8765",
        Handler:        mux,
        ReadTimeout:    10 * time.Second,
        WriteTimeout:   10 * time.Second,
        MaxHeaderBytes: 1 << 20,
    }

    go func() {
        server.ListenAndServe()
    }()

    time.Sleep(100 * time.Millisecond)
    return server
}

func benchmarkNoKeepAlive(duration time.Duration, numGoroutines int) int64 {
    var count int64
    var wg sync.WaitGroup

    start := time.Now()
    for i := 0; i < numGoroutines; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            for time.Since(start) < duration {
                req, _ := http.NewRequest("GET", "http://localhost:8765/test", nil)
                req.Header.Set("Connection", "close")

                client := &http.Client{
                    Transport: &http.Transport{
                        DisableKeepAlives: true,
                    },
                }
                resp, err := client.Do(req)
                if err != nil {
                    continue
                }
                io.ReadAll(resp.Body)
                resp.Body.Close()
                atomic.AddInt64(&count, 1)
            }
        }()
    }

    wg.Wait()
    return count
}

func benchmarkWithKeepAlive(duration time.Duration, numGoroutines int) int64 {
    var count int64
    var wg sync.WaitGroup

    client := &http.Client{
        Transport: &http.Transport{
            MaxIdleConns:        100,
            MaxIdleConnsPerHost: 10,
            IdleConnTimeout:     90 * time.Second,
        },
    }

    start := time.Now()
    for i := 0; i < numGoroutines; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            for time.Since(start) < duration {
                req, _ := http.NewRequest("GET", "http://localhost:8765/test", nil)
                resp, err := client.Do(req)
                if err != nil {
                    continue
                }
                io.ReadAll(resp.Body)
                resp.Body.Close()
                atomic.AddInt64(&count, 1)
            }
        }()
    }

    wg.Wait()
    return count
}

func main() {
    server := startServer()
    defer server.Close()

    testDuration := 5 * time.Second
    numGoroutines := 4

    fmt.Println("Benchmarking HTTP keep-alive and connection pooling...")
    fmt.Println()

    fmt.Printf("Test duration: %v, Goroutines: %d\n", testDuration, numGoroutines)
    fmt.Println()

    fmt.Print("Running without keep-alive (Connection: close)...")
    noKeepAliveCount := benchmarkNoKeepAlive(testDuration, numGoroutines)
    fmt.Printf(" Done\n")
    fmt.Printf("Requests completed: %d\n", noKeepAliveCount)
    fmt.Printf("Requests per second: %.0f\n", float64(noKeepAliveCount)/testDuration.Seconds())
    fmt.Println()

    fmt.Print("Running with keep-alive and connection pooling...")
    withKeepAliveCount := benchmarkWithKeepAlive(testDuration, numGoroutines)
    fmt.Printf(" Done\n")
    fmt.Printf("Requests completed: %d\n", withKeepAliveCount)
    fmt.Printf("Requests per second: %.0f\n", float64(withKeepAliveCount)/testDuration.Seconds())
    fmt.Println()

    improvement := float64(withKeepAliveCount) / float64(noKeepAliveCount)
    fmt.Printf("Improvement factor: %.2fx\n", improvement)
}

Compile and run this:

go run main.go

On a modern laptop with a local server, you should see something like:

Benchmarking HTTP keep-alive and connection pooling...

Test duration: 5s, Goroutines: 4

Running without keep-alive (Connection: close)... Done
Requests completed: 892
Requests per second: 178

Running with keep-alive and connection pooling... Done
Requests completed: 8342
Requests per second: 1668

Improvement factor: 9.35x

That 9x difference is real. Without keep-alive, the client spends most of its time in socket setup and teardown. With keep-alive and a small pool, requests pipeline through warm connections.

Connection Pooling Details

Connection pooling is not magic. It's a queue of established TCP connections. When a goroutine needs to make a request, it checks if a connection is available in the pool. If yes, it reuses it. If no, it opens a new one (up to a limit) or waits for one to be returned. Most HTTP client libraries handle this automatically.

In Go's net/http.Transport, key settings are:

MaxIdleConns: Total idle connections across all hosts. Default is 100.
MaxIdleConnsPerHost: Idle connections per host. Default is 2. If you're hammering one backend, increase this.
MaxConnsPerHost: Absolute limit of connections per host (idle + active). Default is unlimited.
IdleConnTimeout: How long an idle connection sits before the client closes it. Default is 90 seconds.

Under the hood, the Transport maintains a map of "http://hostname:port" to a queue of connections. When you call client.Do(), the Transport looks up that host, grabs an idle connection if available, or opens a new one. After the response is read, the connection is returned to the pool (unless you set DisableKeepAlives: true or the server sent Connection: close).

TLS and Keep-Alive

For HTTPS, keep-alive saves even more work. A TLS 1.3 handshake is one additional round-trip. TLS 1.2 is two. Without keep-alive, every request eats that cost. With keep-alive, the TLS session is cached inside the TCP connection, and subsequent requests on the same connection use the same session key. No re-negotiation.

If you're integrating a third-party authentication service or making calls to token-based security systems, the latency difference between connection pooling and no pooling can be the difference between 50 ms per request and 500 ms. That adds up quickly.

Idle Connections and Resource Limits

Idle connections consume resources: kernel socket buffers, file descriptors, and TCP control structures. On high-concurrency services, you need to balance pool size against system limits. Linux has /proc/sys/fs/file-max and per-process limits (ulimit -n). If your pool maintains 10,000 idle connections, that's 10,000 file descriptors.

The IdleConnTimeout is your safeguard. Set it to match the server's keep-alive timeout (often 60-90 seconds). If the server closes after 60 seconds and your client keeps the connection open for 120 seconds, you'll get surprise connection resets when trying to reuse a dead connection. Most robust clients detect this and transparently retry, but it's latency you could avoid.

When to Disable Keep-Alive

Keep-alive is usually right. But there are exceptions:

Proxies or gateways: If you're a stateless proxy passing through requests, connection pooling can mask load distribution. Some setups prefer connection-per-request to spread load across backend instances.
Long-lived connections that timeout: If the server aggressively closes idle connections (e.g., every 10 seconds), keeping a pool of idle connections wastes resources and causes timeouts.
Old HTTP/1.0 servers: Some legacy services don't support keep-alive. Set DisableKeepAlives: true in the Transport.
Memory-constrained environments: Embedded systems or containers with tight memory budgets may not afford pool overhead.

HTTP/2 and Beyond

HTTP/2 multiplexes multiple requests over a single connection by default. No connection pooling needed; you get stream-level concurrency on one connection. This is one reason HTTP/2 is faster for many workloads. However, if the connection is interrupted or the server closes it, you still pay for a fresh handshake.

WebSockets, by contrast, establish a persistent bidirectional connection and do not benefit from HTTP's connection pooling model; see our WebSocket article for that pattern.

Common Mistakes

Creating a new http.Client per request: Clients include a Transport with its own pool. If you do client := &http.Client{} for every request, each one starts with an empty pool. Reuse a single client across your application.

Not reading response bodies: If you ignore resp.Body, the Transport doesn't return the connection to the pool; it closes it instead. Always call io.ReadAll(resp.Body) and resp.Body.Close() or use io.Copy to drain the body.

Setting MaxIdleConnsPerHost too low: If you have 10 goroutines all hammering one endpoint and MaxIdleConnsPerHost is 2, you're thrashing the pool. Increase it to match your concurrency.

Mismatched timeouts: If your IdleConnTimeout is 90 seconds but the server closes after 60, you'll get silent connection resets. Check the server's Keep-Alive header.

FAQ

Does HTTP/2 make connection pooling irrelevant?

No. HTTP/2 multiplexes streams on a single connection, which is more efficient than pooling multiple connections. However, you still want to maintain a pool of HTTP/2 connections (even if small) to handle reconnection and geographic distribution across multiple servers.

Can I manually close an idle connection to force refresh?

Yes. Call client.CloseIdleConnections(). Useful when testing or when the backend rotates servers. But in production, let the IdleConnTimeout do the work.

What if the server doesn't send a Keep-Alive header?

In HTTP/1.1, connections are keep-alive by default unless Connection: close is sent. The absence of a Keep-Alive header does not mean the connection closes. However, some servers do close after a fixed duration anyway (often 30-60 seconds). Monitor your logs for reset errors and adjust IdleConnTimeout if needed.

Is connection pooling safe with multiple goroutines?

Yes. http.Client and Transport are goroutine-safe. Multiple goroutines can call client.Do() concurrently, and the Transport serializes access to the pool internally with mutexes.

How do I know what pool size is optimal?

Measure. Start with MaxIdleConnsPerHost equal to your expected concurrency (e.g., 20 goroutines, set it to 20). Monitor TCP connection count and request latency. If latency is stable and connection count grows to your pool size during load, you're sized right. If you're hitting MaxConnsPerHost limits and getting delays, increase the limit.

How Container Networking Works: From veth to the Bridge

Bar Dror — Fri, 09 Oct 2026 07:01:04 +0000

Originally published on Code Beneath.

When you run a Docker container, you assume the network just works. A container gets an IP, you can reach it from the host, and you don't think about the mechanics. But understanding how docker container networking works is essential if you want to debug network issues, build custom networks, or understand why your container can't reach a service. The magic is not magic at all: it is virtual ethernet devices, kernel bridges, network namespaces, and some iptables rules. This article strips away the Docker abstraction and builds a bridge and veth pair by hand using only the ip command, so you see exactly what Docker does for you.

The Problem: Why Containers Need Special Networking

A container is not a separate physical machine. It is a Linux process with its own network namespace: its own set of network interfaces, routing table, and firewall rules. The kernel uses namespaces to isolate processes. A container's process thinks it owns eth0, but that interface does not exist on the host. The host has eth0 or wlan0, not the container.

So how does a packet from inside a container reach the outside world? How does the host reach the container? Docker solves this with virtual ethernet devices and a bridge. The bridge acts like a network switch: it connects the host to one side and the container to the other, and it forwards packets between them. Each container gets a virtual ethernet pair, one half in the container namespace and one half connected to the bridge on the host.

Virtual Ethernet Pairs (veth): The Tunnel Between Namespaces

A veth pair is a Linux kernel mechanism that creates two virtual network interfaces permanently connected to each other. Send a packet to one side, it emerges from the other side. Think of it as a pipe between two network namespaces.

Let us create a veth pair by hand:

#!/bin/bash
set -e

# Create a veth pair
# One side: veth-host, one side: veth-container
ip link add veth-host type veth peer name veth-container

# Check that both exist
ip link show veth-host
ip link show veth-container

# Bring both up
ip link set veth-host up
ip link set veth-container up

# Verify status
ip link show | grep veth

Run this script and you will see two new network interfaces. They are not connected to anything yet, just hanging there. If you send a packet to veth-host, the kernel forwards it to veth-container and vice versa. But since there is no namespace setup yet, both are on the host namespace.

Network Namespaces: Isolating the Container

A container's network isolation comes from a network namespace. The kernel separates each namespace so a process in a namespace sees only its own network interfaces. Create a namespace, move veth-container into it, and now the container process sees only veth-container, not the host's real interfaces.

#!/bin/bash
set -e

# Create a network namespace called "container1"
ip netns add container1

# Move veth-container into the container1 namespace
ip link set veth-container netns container1

# Verify: veth-container is no longer visible on the host
ip link show veth-container 2>&1 || echo "veth-container not on host (expected)"

# But it is visible from inside the namespace
ip netns exec container1 ip link show

# We can also run a shell inside the namespace
# ip netns exec container1 bash

Once veth-container is in the container1 namespace, the host kernel still manages it, but the container process cannot see veth-host or eth0. This is the isolation. The container thinks it has a brand new veth-container interface that is its only link to the outside.

The Bridge: The Network Switch

Now we need a way for the host and container to communicate. Enter the bridge: a virtual network switch that lives on the host. A bridge interface on Linux can connect multiple physical or virtual interfaces and forward frames between them, just like a hardware switch.

#!/bin/bash
set -e

# Create a bridge
ip link add br0 type bridge

# Bring the bridge up
ip link set br0 up

# Connect veth-host to the bridge
ip link set veth-host master br0

# Give the bridge an IP address (the gateway for containers)
ip addr add 172.17.0.1/16 dev br0

# Verify
ip link show br0
ip addr show br0

The bridge is now up and has an IP address on the 172.17.0.0/16 subnet. veth-host is connected to it. When the host sends to 172.17.0.x, it goes to the bridge. The bridge, seeing veth-host, forwards it into veth-container. The container receives it on its veth-container interface.

Assigning IPs to the Container and Host Side

The bridge has 172.17.0.1. Now we assign the container side an IP on the same subnet. Inside the container namespace, we also need a default route pointing back to the bridge (the gateway).

#!/bin/bash
set -e

# Inside container1 namespace: assign IP to veth-container
ip netns exec container1 ip addr add 172.17.0.2/16 dev veth-container

# Bring up the interface inside the namespace
ip netns exec container1 ip link set veth-container up

# Set default route inside the container to the bridge (host's veth-host IP is not visible, but gateway on bridge is)
ip netns exec container1 ip route add default via 172.17.0.1 dev veth-container

# Verify from inside the container
ip netns exec container1 ip addr show
ip netns exec container1 ip route show

Now the container has 172.17.0.2 on its veth-container interface, and its default route points to 172.17.0.1 (the bridge). The host has 172.17.0.1 on br0. They are on the same subnet.

Testing Connectivity

Let us verify that packets flow between the host and container. The simplest test is a ping from the container to the gateway.

#!/bin/bash
set -e

# Ping from container to the bridge (gateway)
echo "Pinging gateway from container:"
ip netns exec container1 ping -c 2 172.17.0.1

# Ping from host to the container
echo "Pinging container from host:"
ping -c 2 172.17.0.2

# Verify packet flow with tcpdump (optional, requires another terminal)
# sudo tcpdump -i veth-host -n icmp

If both pings work, packets are flowing through the veth pair and the bridge. The host sees 172.17.0.2 on the bridge. The container sees 172.17.0.1 as its gateway. This is the core of container networking.

Reaching External Networks

So far, the container can reach the host on the bridge. But can the container reach the outside world (for example, 8.8.8.8)? No, not yet. The host knows about the 172.17.0.0/16 subnet (it has a route to it via br0), but external networks do not know how to reach 172.17.0.2. The container's packets leave the bridge but hit a dead end.

Docker solves this with NAT (network address translation). The host adds an iptables rule to masquerade outgoing packets from the container, rewriting the source IP from 172.17.0.2 to the host's real IP. Replies come back to the host, and iptables rewrites them back to 172.17.0.2. The container never knows.

#!/bin/bash
set -e

# Enable IP forwarding on the host
echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward

# Add a masquerade rule for the bridge subnet
sudo iptables -t nat -A POSTROUTING -s 172.17.0.0/16 ! -o br0 -j MASQUERADE

# Now test external connectivity from the container
ip netns exec container1 ping -c 2 8.8.8.8 || echo "External ping may fail if no default route outside"

# Check the iptables rule
sudo iptables -t nat -L -n

With masquerade, outgoing packets from 172.17.0.2 are rewritten to the host's IP. The outside world replies to the host, and the host rewrites the response back to 172.17.0.2. The container receives it as if it was talking directly to 8.8.8.8.

Inbound Traffic: Exposing Container Ports

A container runs a web server on port 8080. You want to access it from outside. The container listens on 172.17.0.2:8080, but the outside world cannot reach 172.17.0.2. Docker adds a port mapping: the host listens on 0.0.0.0:8080 and forwards incoming connections to 172.17.0.2:8080.

This is also done with iptables rules in the PREROUTING and POSTROUTING chains:

#!/bin/bash
set -e

# Map host port 8080 to container port 8080
sudo iptables -t nat -A PREROUTING -d 172.17.0.1 -p tcp --dport 8080 -j DNAT --to-destination 172.17.0.2:8080
sudo iptables -t nat -A POSTROUTING -d 172.17.0.2 -p tcp --dport 8080 -j MASQUERADE

# Or use docker-proxy, which Docker does internally
# docker-proxy -container-ip 172.17.0.2 -container-port 8080 -host-port 0.0.0.0:8080

# To test, run a simple HTTP server inside the container
# ip netns exec container1 python3 -m http.server 8080 &
# curl http://172.17.0.1:8080 (from host)

The first rule intercepts incoming traffic to the bridge IP on port 8080 and rewrites the destination to the container IP. The second rule masquerades the reply so the container thinks it is talking to the outside.

What Docker Automates

Docker does all of the above automatically when you run a container. Here is what it does under the hood:

Creates a veth pair
Moves one side into the container's network namespace
Creates or reuses a bridge (docker0, or a user-defined bridge network)
Connects the host side of the veth to the bridge
Assigns IPs from the bridge subnet to both the host side and container side
Sets up the default route inside the container
Adds iptables rules for masquerading and port mapping
Ensures IP forwarding is enabled on the host

The primitives are simple: veth pairs, bridges, network namespaces, and iptables. Everything else is orchestration.

Comparison: Host, Bridge, and Overlay Networks

Mode	Isolation	Performance	Use Case
Host	None; container shares host namespace	Minimal overhead; direct access to host NICs	High-performance services, monitoring agents
Bridge (default)	Full; container in separate namespace	One veth pair per container, iptables overhead	Typical single-host containerized apps
Overlay	Full; encrypted tunnel between hosts	Additional encapsulation, slight latency	Multi-host Docker Swarm or Kubernetes

How to Inspect the Real Thing in Docker

Start a real Docker container and inspect the networking:

#!/bin/bash
set -e

# Start a container in the background
docker run -d --name test-container alpine sleep 1000

# Get the container's PID
PID=$(docker inspect -f '{{.State.Pid}}' test-container)
echo "Container PID: $PID"

# Inspect the veth pair from the container side
# The container's interface is usually eth0
docker exec test-container ip link show

# Get the container IP
docker exec test-container ip addr show

# Find the host-side veth by looking at the bridge
docker network inspect bridge

# Look at iptables rules Docker created
sudo iptables -t nat -L -n

# Ping the container from the host
docker exec test-container ip addr show eth0 | grep "inet " | awk '{print $2}' | cut -d/ -f1 | xargs ping -c 2

# Clean up
docker stop test-container
docker rm test-container

Socket and TCP/UDP Fundamentals

Understanding veth pairs and bridges is the foundation, but it sits on top of sockets and the TCP/UDP layer. If you are debugging network issues at the container level, you also need to understand how TCP and UDP sockets work and how the kernel routes packets through the network stack. A container's process makes a socket call, the kernel forwards it through the veth pair, the bridge, and eventually out the host's real NIC. If a packet does not arrive, the problem could be at any layer: the veth pair, the bridge, the routing table, or the firewall rules.

Debugging Tips

When container networking breaks:

Check that veth pairs exist: ip link show | grep veth
Verify the bridge exists and the veth is attached: ip link show br0 and brctl show
Confirm IPs are assigned: ip addr show br0, docker inspect <container>
Test connectivity with ping from the host to the container IP
Check iptables rules: sudo iptables -t nat -L -n
Enable IP forwarding if it is disabled: echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward
Use tcpdump to see packet flow: sudo tcpdump -i br0 -n

Common Mistakes

IP forwarding disabled. By default, Linux does not forward packets between interfaces. If you set up a bridge and try to ping from the container to an external host, you will see packets arrive at the bridge but no response. Enable forwarding with echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward.

Forgot masquerade rules. Without iptables masquerading, outgoing packets from the container carry the source IP 172.17.0.2, which the outside world does not know how to route back to. The host receives no reply. Add the masquerade rule.

Assigning the same IP to multiple containers. If two containers get the same IP on the same bridge, the kernel will not know which veth pair to use, and packets will be dropped or routed incorrectly. Each container needs a unique IP on the bridge subnet.

Confusing the bridge IP with the host IP. The bridge is a separate interface with its own IP (172.17.0.1). It is not the host's primary NIC. The host might have eth0 with 192.168.1.100, and br0 with 172.17.0.1. They are independent. The container talks to the bridge, not directly to eth0.

Tools and Resources

If you want to explore this further, use tools like strace, tcpdump, and iptables to watch the kernel do its work. strace shows every system call a process makes, including socket creation and bind. tcpdump shows every frame on every interface. iptables lets you see and modify the rules the kernel uses to route traffic.

FAQ

Why does Docker use bridges and not just connect containers directly?

A bridge scales. One bridge can connect dozens of containers. The bridge acts as a central hub, so you do not need a unique veth pair for every pair of containers. Also, the bridge sits on the host, so it can apply global rules like masquerading and port mapping in one place.

Can I put a container on the host network?

Yes. Docker supports --network host, which runs the container in the host's network namespace. The container sees the host's eth0, listens on the host's ports, and has no isolation. It is fast but not isolated, so it is only safe for trusted workloads.

What is the difference between a bridge and overlay network?

A bridge is single-host only. All containers on a bridge must be on the same physical machine. An overlay network spans multiple machines: it uses a tunnel (usually VXLAN) to connect containers on different hosts. Docker Swarm and Kubernetes use overlay networks for multi-host deployments.

Does every container get its own veth pair?

Yes, when using bridge networking. Each container has one veth pair, with the container side in the container's namespace and the host side connected to the bridge. If a container joins multiple networks, it gets multiple veth pairs.

What happens if I delete the bridge while containers are running?

The containers lose connectivity. If you try to delete a bridge that has veth pairs attached, the kernel will refuse. You must disconnect the veth pairs first, then delete the bridge. Docker handles this cleanup automatically.

Can I manually create a Docker network?

Yes. docker network create my-net creates a user-defined bridge network. Docker assigns it a subnet and creates a new bridge interface. Containers on this network can communicate with each other by name (if they use the embedded DNS). This is better than the default bridge because it offers service discovery.

Pointers and Memory in C and C++, Without the Fear

Bar Dror — Wed, 07 Oct 2026 07:01:13 +0000

Originally published on Code Beneath.

If you come from Java or Go, pointers and memory management in C++ feel like someone handed you a loaded gun without a safety. The language will not stop you from dereferencing null, reading freed memory, or leaking gigabytes. But pointers are not evil. They are the mechanism underneath every network socket, every memory pool, every high-performance system. Once you see what actually happens in memory, the fear dissolves into understanding. This article shows you pointers and memory management C++ example code that prints real addresses, demonstrates the stack, the heap, and the specific mistakes that burn production services.

The Stack and Heap, Visible

In Java, you think in references. In Go, the language decides whether a value lives on the stack or heap, and you rarely care. In C++, you decide, and you see the consequences immediately. Let us start with a program that prints actual memory addresses so you can watch the stack and heap in action.

#include <iostream>
#include <string>

using namespace std;

int main() {
    // Stack allocation
    int stack_int = 42;
    int* ptr_to_stack = &stack_int;

    cout << "Stack int address: " << ptr_to_stack << endl;
    cout << "Stack int value: " << *ptr_to_stack << endl;

    // Heap allocation
    int* heap_int = new int(100);
    cout << "Heap int address: " << heap_int << endl;
    cout << "Heap int value: " << *heap_int << endl;

    // Another stack variable
    int another_stack = 999;
    cout << "Another stack int address: " << &another_stack << endl;

    // Typical output on a 64-bit system:
    // Stack int address: 0x7ffc6b7ff65c
    // Stack int value: 42
    // Heap int address: 0x55a7c1a19260
    // Heap int value: 100
    // Another stack int address: 0x7ffc6b7ff658

    delete heap_int;
    return 0;
}

Compile and run this: the stack addresses are close together and large (typically 0x7fff...). The heap address is different and typically smaller. That is not magic. The operating system maps these regions. When your program starts, it gets a stack that grows downward (on most systems) and a heap that grows upward. Variables declared directly in a function live on the stack. Memory allocated with new lives on the heap. The pointer itself (the address value) lives wherever you declare it. The thing it points to lives on the heap or stack depending on how it was created.

Stack variables are fast and automatic. They vanish when the function returns. Heap memory persists until you explicitly deallocate it with delete. That freedom is also a trap.

Ownership and the Use-After-Free Bug

Here is the most common nightmare in C++: you free memory, then accidentally read or write it anyway.

#include <iostream>

using namespace std;

int* create_number() {
    int* num = new int(42);
    return num;
}

int main() {
    int* my_number = create_number();

    cout << "Value: " << *my_number << endl;  // OK

    delete my_number;

    cout << "After delete: " << *my_number << endl;  // UNDEFINED BEHAVIOR

    return 0;
}

Compile and run this. It might print garbage, crash, or appear to work (do not trust that). The memory is gone, but the pointer is still valid in the sense that it holds an address. You can still dereference it. The operating system did not magic away the bits. It just marked that memory as "available to reuse." When you read or write to freed memory, you are corrupting whatever now lives there. In a server, this is a data corruption bug that shows up as silent, random failures weeks later.

Who owns the memory? In the code above, create_number allocates but does not deallocate. The caller gets the pointer and must remember to delete it. If you forget, memory leaks. If you delete twice, crash. If you delete in the wrong place, the other place crashes. This is why everyone dreads C++ in large codebases.

Smart Pointers: Automatic Ownership

In C++11 and later, smart pointers solve ownership by using a concept borrowed from Rust: the memory is owned by an object that cleans itself up. The two main types are unique_ptr (exclusive ownership) and shared_ptr (reference counted).

#include <iostream>
#include <memory>

using namespace std;

int* create_number_raw() {
    return new int(42);
}

unique_ptr<int> create_number_smart() {
    return make_unique<int>(42);
}

int main() {
    // The old way: you manage
    int* raw_num = create_number_raw();
    cout << "Raw: " << *raw_num << endl;
    delete raw_num;  // Must not forget

    // The smart way: the pointer manages itself
    unique_ptr<int> smart_num = create_number_smart();
    cout << "Smart: " << *smart_num << endl;
    // Automatically deleted when smart_num goes out of scope

    return 0;
}

A unique_ptr owns its memory exclusively. When the unique_ptr object is destroyed (at the end of the function, or when a variable goes out of scope), it automatically calls delete. You cannot copy it. You can only move it (transfer ownership). This makes ownership explicit in the code: if a function returns a unique_ptr<int>, you know who owns the memory.

#include <iostream>
#include <memory>
#include <string>

using namespace std;

class Config {
public:
    Config(const string& name) : name(name) {
        cout << "Config created: " << name << endl;
    }
    ~Config() {
        cout << "Config destroyed: " << name << endl;
    }

    string name;
};

unique_ptr<Config> load_config() {
    return make_unique<Config>("production");
}

int main() {
    cout << "Before scope" << endl;
    {
        auto config = load_config();
        cout << "Inside scope, config name: " << config->name << endl;
    }
    cout << "After scope" << endl;

    return 0;
}

Run this and watch the output. You will see "Config created", then "Inside scope", then "Config destroyed", then "After scope". The destructor runs automatically when the unique_ptr goes out of scope. No manual cleanup, no leak, no use-after-free possible.

shared_ptr is similar but counts references. If multiple parts of your code own a piece of data, use shared_ptr. When the last reference is released, the memory is freed.

#include <iostream>
#include <memory>

using namespace std;

class Resource {
public:
    Resource(int id) : id(id) {
        cout << "Resource " << id << " allocated" << endl;
    }
    ~Resource() {
        cout << "Resource " << id << " freed" << endl;
    }
    int id;
};

int main() {
    shared_ptr<Resource> r1 = make_shared<Resource>(1);
    cout << "r1 use_count: " << r1.use_count() << endl;  // 1

    {
        shared_ptr<Resource> r2 = r1;  // Copy, increment reference count
        cout << "r1 use_count after r2 = r1: " << r1.use_count() << endl;  // 2
    }

    cout << "r1 use_count after r2 destroyed: " << r1.use_count() << endl;  // 1
    return 0;
}

This prints: "Resource 1 allocated", "r1 use_count: 1", "r1 use_count after r2 = r1: 2", "r1 use_count after r2 destroyed: 1". When you exit main, r1 is destroyed, use_count drops to 0, and "Resource 1 freed" prints. You never write a destructor or delete. The shared_ptr handles it. This is closest to Go's garbage collection, but deterministic and transparent.

Stack vs. Heap Performance and Semantics

When should you allocate on the stack versus the heap? The rule is simple: stack when the size and lifetime are known at compile time, heap for large objects, long-lived data, or polymorphism. Here is a comparison.

Aspect	Stack	Heap
Speed	Very fast, one instruction	Slower, involves allocator
Lifetime	Automatic, function scope	Manual (you decide when to free)
Size limits	Limited (usually 1-8 MB)	Large (limited by RAM)
Fragmentation	None	Possible
Suitable for	Small, local values	Large structures, containers

In practice, on a native language like C++, you allocate small local structures on the stack and wrap heap allocations in smart pointers. Containers like vector and map manage their own heap memory internally, so you do not have to think about it. When you need polymorphism (virtual functions), you use pointers to base classes, and again, wrap them in smart pointers.

Common Mistakes

Mistake 1: Returning a pointer to a stack variable.

int* bad_function() {
    int x = 42;
    return &x;  // WRONG: x is destroyed when function returns
}

The caller gets a pointer to memory that no longer belongs to the program. This is a classic dangling pointer. Always return by value or return a unique_ptr.

Mistake 2: Deleting the same pointer twice.

int* p = new int(42);
delete p;
delete p;  // CRASH: undefined behavior

If you use smart pointers, this is impossible. A unique_ptr can only be deleted once because it cannot be copied.

Mistake 3: Mixing new and delete with stack allocation.

int x = 42;
delete &x;  // WRONG: x is on the stack, not heap

Only delete memory you allocated with new. Stack variables clean themselves up.

Mistake 4: Creating a shared_ptr from a raw pointer twice.

int* raw = new int(42);
shared_ptr<int> sp1(raw);
shared_ptr<int> sp2(raw);  // WRONG: both think they own it
// When sp1 is destroyed, it deletes raw
// When sp2 is destroyed, it tries to delete it again: crash

If you have a raw pointer, create one smart pointer and use that. If you need multiple references, copy the smart pointer.

Pointers to Pointers and Indirection

Sometimes you need a pointer to a pointer. This is rare but essential for callbacks and certain data structures. Here is what is happening at the memory level.

#include <iostream>

using namespace std;

int main() {
    int value = 42;
    int* ptr = &value;              // Pointer to int
    int** ptr_ptr = &ptr;           // Pointer to pointer

    cout << "value: " << value << endl;
    cout << "ptr (address of value): " << ptr << endl;
    cout << "ptr_ptr (address of ptr): " << ptr_ptr << endl;
    cout << "*ptr (dereference once): " << *ptr << endl;
    cout << "**ptr_ptr (dereference twice): " << **ptr_ptr << endl;

    return 0;
}

ptr_ptr holds the address of ptr, which holds the address of value. Dereference once to get the pointer, twice to get the value. You will see this in C APIs (like sorting callbacks) and when working with arrays of pointers. In modern C++, avoid it when possible. Use references or containers instead.

References vs. Pointers

C++ has references, which are safer than pointers because they cannot be null and cannot be reseated (changed after initialization). A reference is a compiled-away alias to the original variable.

#include <iostream>

using namespace std;

void modify_by_pointer(int* p) {
    *p = 100;
}

void modify_by_reference(int& r) {
    r = 100;
}

int main() {
    int x = 42;

    modify_by_pointer(&x);
    cout << "After pointer call: " << x << endl;

    x = 42;
    modify_by_reference(x);
    cout << "After reference call: " << x << endl;

    return 0;
}

Both produce the same machine code. Use pointers when you need to store an address, pass a pointer to another function, or use nullptr as a sentinel. Use references for function parameters and return values. This makes the calling code cleaner and the intent clearer.

Arrays and Pointer Decay

In C++, an array name decays to a pointer to its first element. This is where many off-by-one bugs and buffer overruns live.

#include <iostream>

using namespace std;

int main() {
    int arr[5] = {10, 20, 30, 40, 50};

    int* ptr = arr;  // arr decays to pointer to first element

    cout << "arr[2]: " << arr[2] << endl;
    cout << "ptr[2]: " << ptr[2] << endl;  // Same
    cout << "*(ptr + 2): " << *(ptr + 2) << endl;  // Same

    cout << "Size of arr: " << sizeof(arr) << endl;  // 20 (5 ints)
    cout << "Size of ptr: " << sizeof(ptr) << endl;  // 8 (one address)

    return 0;
}

When arr is assigned to ptr, you lose the size information. ptr is now just a pointer. In a function, if you pass an array as a parameter, it also decays. This is why C APIs that take arrays also take a size parameter. In C++, use std::vector or std::array to avoid this pitfall. They maintain bounds and size.

Function Pointers and Callbacks

Function pointers are addresses of executable code. They enable callbacks and late binding without virtual functions.

#include <iostream>

using namespace std;

int add(int a, int b) {
    return a + b;
}

int multiply(int a, int b) {
    return a * b;
}

int operate(int x, int y, int (*func)(int, int)) {
    return func(x, y);
}

int main() {
    int (*func_ptr)(int, int) = add;

    cout << "Using add: " << func_ptr(5, 3) << endl;

    func_ptr = multiply;
    cout << "Using multiply: " << func_ptr(5, 3) << endl;

    cout << "Using operate: " << operate(5, 3, add) << endl;

    return 0;
}

Function pointers are rare in modern C++. Use std::function or lambdas instead. They are cleaner and type-safer.

The const Pointer

Const pointers confuse everyone. Remember: read from right to left. const int* p means "pointer to const int" (you cannot change the int). int* const p means "const pointer to int" (you cannot change the pointer, but can change the int).

#include <iostream>

using namespace std;

int main() {
    int x = 42;
    int y = 100;

    // Pointer to const int: cannot modify x through this pointer
    const int* p1 = &x;
    // *p1 = 50;  // COMPILE ERROR
    p1 = &y;      // OK, can change where p1 points

    // Const pointer to int: cannot change where p2 points
    int* const p2 = &x;
    *p2 = 50;      // OK, can modify x
    // p2 = &y;  // COMPILE ERROR

    // Const pointer to const int: cannot do either
    const int* const p3 = &x;
    // *p3 = 50;  // COMPILE ERROR
    // p3 = &y;  // COMPILE ERROR

    return 0;
}

This matters for function signatures. If a function takes const int*, it promises not to modify the data, and you can safely pass a pointer to const data. This is a key tool for communicating intent and preventing bugs.

Memory Alignment and Padding

Pointers and memory addresses must respect alignment. On a 64-bit system, a 64-bit value must be at an address divisible by 8. Smaller types have looser constraints. The compiler adds padding to structures to maintain alignment.

#include <iostream>

using namespace std;

struct Bad {
    char a;    // 1 byte
    long b;    // 8 bytes
    char c;    // 1 byte
};

struct Good {
    long b;    // 8 bytes
    char a;    // 1 byte
    char c;    // 1 byte
};

int main() {
    cout << "sizeof(Bad): " << sizeof(Bad) << endl;   // 24 (not 10)
    cout << "sizeof(Good): " << sizeof(Good) << endl;  // 16 (not 10)

    Bad b;
    cout << "Address of b.a: " << (void*)&b.a << endl;
    cout << "Address of b.b: " << (void*)&b.b << endl;
    cout << "Address of b.c: " << (void*)&b.c << endl;

    return 0;
}

The compiler reorders and pads to keep b aligned to an 8-byte boundary. You do not control this directly (without compiler attributes), but you can optimize by grouping similar-sized fields. This is not usually critical for correctness, but it matters for tight memory constraints or cache-heavy workloads.

Common Debugging with Valgrind

When you suspect a memory bug, use Valgrind (on Linux). It detects use-after-free, leaks, and invalid writes.

g++ -g -o program program.cpp
valgrind --leak-check=full ./program

The -g flag includes debug symbols so Valgrind can report line numbers. Valgrind runs your program in an emulator and tracks every memory operation. On a native language project, this is your safety net until you can refactor to smart pointers.

FAQ

What is the difference between a pointer and a reference?

A reference is an alias that cannot be null or reseated. A pointer holds an address and can be reassigned or null. Use references in function signatures and return types for clarity. Use pointers when you need to store addresses, check for null, or interface with C code.

When should I use unique_ptr vs. shared_ptr?

Use unique_ptr by default. It has zero overhead and clear ownership. Use shared_ptr only when multiple objects need to own the same resource. Reference counting has a small performance cost.

Can I return a raw pointer from a function?

Only if the caller is not responsible for deleting it. For example, returning a pointer to a member of an object that owns it. If the function allocates and the caller must delete, return a unique_ptr instead. It makes ownership explicit.

What causes a segmentation fault?

Reading or writing memory you do not own. Common causes: dereferencing null, using a dangling pointer, buffer overflow, or corrupted heap. Use Valgrind to find the exact line. Use smart pointers to prevent most causes.

Is C++ memory management faster than garbage collection?

Yes, but only if you write correct code. Smart pointers have negligible overhead compared to garbage collection, and you control exactly when cleanup happens. However, manual memory management is error-prone. For most code, choose clarity and correctness over micro-optimized manual cleanup.

How do I know if a pointer is valid?

You do not, without additional bookkeeping. This is why smart pointers are essential. A unique_ptr guarantees the memory is valid or null. For raw pointers, you must document ownership and lifetime in code comments or use static analysis tools.

What about null pointers?

A null pointer (address 0) is not valid to dereference. Always check before dereferencing a raw pointer: if (ptr != nullptr) { ... }. Smart pointers and references eliminate most null checks by design.

Should I worry about memory alignment when writing network code?

Yes, for parsing binary protocols. Misaligned casts can cause crashes or undefined behavior. Use memcpy for unaligned reads and writes, or use #pragma pack carefully. See native language guides for serialization best practices.

How do I learn more about low-level memory in Go or languages like it?

Go abstracts memory, but reading about Go runtime and escape analysis is enlightening. Go shows you that memory decisions are still being made, just by the compiler, not you. C++ teaches you to make those decisions explicitly and understand the cost.

Linux Processes and System Calls, Traced With strace

Bar Dror — Mon, 05 Oct 2026 07:01:04 +0000

Originally published on Code Beneath.

Every program on Linux is a series of system calls. When you read a file, write to a socket, or spawn a child process, your code is asking the kernel to do the real work. The operating system intercepts these requests, checks permissions, manages resources, and returns results. A linux strace tutorial example shows you exactly where your program pauses, which syscalls it makes, and how long each one takes. This is not theoretical: strace is a production tool that lets you see the gap between what you think your code does and what it actually tells the kernel to do. Most developers skip this step and wonder why their program hangs, leaks file descriptors, or wastes syscalls on redundant operations.

This article walks through a real C program, runs strace on it, and maps each kernel request to what happens inside the kernel. You will see fork, execve, open, read, write, and close in action. You will understand why a single Python print statement can trigger dozens of syscalls. And you will learn to read strace output fast enough to debug a production issue in minutes, not hours.

What is strace and Why You Need It

strace is a system call tracer. It intercepts all syscalls made by a process and prints them to stderr. Each line shows the syscall name, arguments, and return value. strace works by ptrace, a Linux syscall that lets one process inspect and control another. The kernel pauses the traced process before and after each syscall, and strace reads the register state to extract argument and return values.

Why is this useful? Because it shows you the truth. Your code might think it is reading from a file, but strace shows you the actual sequence: open, fstat, mmap, read, close. If your program is slow, strace reveals whether it is spending time in user code or blocked in the kernel. If a program crashes mysteriously, strace shows the last few syscalls before death. If you are integrating low-level code, as we discuss in native language integration, strace is often the fastest way to see what went wrong.

A Minimal C Program to Trace

Let us start with a tiny program that does several things: open a file, read from it, write to stdout, fork a child, and execute a new program. This will show you the full life cycle of syscalls.

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <fcntl.h>
#include <sys/wait.h>
#include <string.h>

int main() {
    // Open and read a file
    int fd = open("test.txt", O_RDONLY | O_CREAT, 0644);
    if (fd == -1) {
        perror("open");
        return 1;
    }

    // Write some content first
    int wfd = open("test.txt", O_WRONLY | O_TRUNC);
    write(wfd, "hello from test file\n", 21);
    close(wfd);

    // Read it back
    char buf[256];
    ssize_t n = read(fd, buf, sizeof(buf));
    if (n > 0) {
        write(STDOUT_FILENO, buf, n);
    }
    close(fd);

    // Fork and exec
    pid_t pid = fork();
    if (pid == 0) {
        // Child process
        execlp("echo", "echo", "child process output", NULL);
        perror("execlp");
        exit(1);
    } else if (pid > 0) {
        // Parent waits
        wait(NULL);
    } else {
        perror("fork");
        return 1;
    }

    return 0;
}

Compile this with gcc -o tracetest tracetest.c. Run it normally and you see some output. Now run it under strace and you see the machinery.

Running strace and Reading the Output

The simplest invocation is: strace ./tracetest

The output will be hundreds of lines. Here is what you get (truncated and annotated):

execve("./tracetest", ["./tracetest"], 0x7ffee55ad6e0 /* 54 vars */) = 0
brk(NULL)                               = 0x55a4c6a4d000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1234560000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
mmap(NULL, 3449960, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f1234100000
mmap(0x7f12342a0000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x190000) = 0x7f12342a0000
mmap(0x7f12342a6000, 14440, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f12342a6000
mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1234556000
munmap(0x7f1234560000, 4096)            = 0
set_tid_address(0x7f12342a68c0)         = 24601
set_robust_list(0x7f12342a68d0, 24)    = 0
rt_sigaction(SIGRTMIN, {sa_handler=0x7f12342a1f10, sa_mask=[], sa_flags=SA_RESTORER|SA_ONSTACK}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 0) = 0
prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8388608, rlim_max=RLIMIT_INFINITY}) = 0
open("test.txt", O_RDONLY|O_CREAT, 0644) = 3
open("test.txt", O_WRONLY|O_TRUNC, 0)  = 4
write(4, "hello from test file\n", 21) = 21
close(4)                                = 0
lseek(3, 0, SEEK_SET)                   = 0
read(3, "hello from test file\n", 256)  = 21
write(1, "hello from test file\n", 21)  = 21
close(3)                                = 0
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_USEREGS|CLONE_VM, child_tidptr=0x7f12342a68c0) = 24606
wait4(24606, NULL, 0, NULL)             = 24606
exit_group(0)                           = ?
+++ exited with code 0 +++

The first many lines are the loader and libc initialization. They happen before your main() runs. The interesting part is where your code actually works:

1. open("test.txt", O_RDONLY|O_CREAT, 0644) = 3: Opens the file for reading. The kernel returns file descriptor 3.

2. open("test.txt", O_WRONLY|O_TRUNC, 0) = 4: Opens the same file for writing and truncates it. fd 4.

3. write(4, "hello from test file\n", 21) = 21: Writes 21 bytes to fd 4. The kernel returns 21 (success).

4. close(4) = 0: Closes fd 4.

5. lseek(3, 0, SEEK_SET) = 0: Seeks to the beginning of fd 3. Read position is now 0.

6. read(3, "hello from test file\n", 256) = 21: Reads 21 bytes from fd 3 into the buffer.

7. write(1, "hello from test file\n", 21) = 21: Writes to fd 1 (stdout).

8. close(3) = 0: Closes fd 3.

9. clone(...) = 24606: This is fork. It creates a new process with PID 24606.

10. wait4(24606, NULL, 0, NULL) = 24606: Parent waits for child to exit.

11. exit_group(0): Parent exits with code 0.

Filtering strace Output: Focus on What Matters

Full strace output is noisy. Use filters to cut the signal from the noise. The -e option filters by syscall type.

strace -e openat,open,read,write,close,fork,execve ./tracetest 2>&1

Note: Modern Linux uses openat instead of open. Both do the same thing, but openat includes a directory fd for relative paths. Here is the filtered output:

open("test.txt", O_RDONLY|O_CREAT, 0644) = 3
open("test.txt", O_WRONLY|O_TRUNC, 0)  = 4
write(4, "hello from test file\n", 21) = 21
close(4)                                = 0
read(3, "hello from test file\n", 256)  = 21
write(1, "hello from test file\n", 21)  = 21
close(3)                                = 0
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_USEREGS|CLONE_VM, child_tidptr=0x7f12342a68c0) = 24618
+++ exited with code 0 +++

Much cleaner. You can also filter by syscall group. For example, -e trace=file shows all syscalls that touch the filesystem. -e trace=process shows fork, execve, exit. Combine multiple filters with commas.

Timing and Performance: -c and -T Options

To see how much time each syscall takes, use -c:

strace -c ./tracetest

Output:

% time     seconds  usecs/call     calls  syscall
------ ----------- ----------- --------- ----------------
 15.23    0.000456      114.00         4 mmap
 12.45    0.000373       62.17         6 mprotect
  9.87    0.000296       49.33         6 open
  8.34    0.000250       41.67         6 close
  7.56    0.000227       56.75         4 read
  6.12    0.000183       30.50         6 write
  5.34    0.000160       40.00         4 lseek
  4.89    0.000147       24.50         6 brk
  3.21    0.000096       16.00         6 rt_sigaction
------ ----------- ----------- --------- ----------------
100.00    0.003000              total

This tells you the percentage of time spent in each syscall, how many times each was called, and the average duration. If your program spends 50% of its time in read, you know where to optimize.

To see per-call timing, use -T:

strace -T ./tracetest 2>&1 | grep read
read(3, "hello from test file\n", 256)  = 21 <0.000012>

The <0.000012> is the time in seconds for that one syscall.

Tracing Child Processes: -f Option

When you fork, the child process is a separate process. Without -f, strace only traces the parent. With -f, it traces all children too.

strace -f -e trace=process ./tracetest

Output:

fork()                                  = 24625
[pid 24625] execve("/bin/echo", ["echo", "child process output"], [/* 54 vars */]) = 0
[pid 24625] write(1, "child process output\n", 22) = 22
[pid 24625] exit_group(0)               = ?
[pid 24624] wait4(24625, NULL, 0, NULL) = 24625
+++ exited with code 0 +++

Notice [pid 24625] in brackets. That is the child process. The parent (24624) calls wait4 to wait for the child. This is the fork/exec pattern every process on Unix follows.

File Descriptor Tracking: Why It Matters

One hidden danger in system programming is file descriptor leaks. A fd is an integer, so the kernel can run out if you open and never close. Debugging this by reading code is hard. strace shows it immediately.

Here is a program that leaks fds:

#include <stdio.h>
#include <fcntl.h>
#include <unistd.h>

int main() {
    for (int i = 0; i < 10; i++) {
        int fd = open("test.txt", O_RDONLY);
        // Oops, forgot to close(fd);
    }
    return 0;
}

Compile and run with strace -e open,close:

strace -e open,close ./leak

You see 10 open calls and 0 close calls. The fds are leaked. A production program that does this will eventually hit the fd limit (usually 1024 per process) and start failing with "too many open files".

Digging Deeper: System Call Arguments and Return Codes

The O_RDONLY, O_WRONLY, O_CREAT flags are integers. strace decodes them for you. If you want to see the raw numbers, use -r or -v (verbose).

strace -v -e open ./tracetest

Also, strace can decode pointers and buffer contents. If you pass a struct to a syscall, strace will try to unpack it.

For example, if you use mmap:

mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1234560000

The arguments are decoded. The return value is the address where the kernel mapped memory.

Comparing: strace vs ltrace vs perf

Tool	What It Traces	Overhead	Best For
strace	System calls (kernel boundary)	High (tens of percent)	Debugging syscall behavior, fd leaks, permission issues
ltrace	Library function calls (user space)	Medium	Debugging library misuse, tracking malloc/free
perf	CPU cycles, cache misses, events	Low	Finding CPU hotspots, optimization

For low-level debugging, strace is the tool. It shows what your code asked the kernel to do. If you want to know what the code itself is doing (which functions it calls, how much time in each), use ltrace or perf.

Real World Example: Password File Reading

Suppose you are reading user data from the password file, as covered in password management. You write code to open /etc/passwd and parse it. strace shows you exactly what the system is doing:

#include <stdio.h>
#include <stdlib.h>

int main() {
    FILE *f = fopen("/etc/passwd", "r");
    if (!f) {
        perror("fopen");
        return 1;
    }

    char line[256];
    int count = 0;
    while (fgets(line, sizeof(line), f)) {
        count++;
    }

    fclose(f);
    printf("Read %d lines\n", count);
    return 0;
}

Compile and run with strace -e openat,read,close:

strace -e openat,read,close ./passwd_reader

Output (simplified):

openat(AT_FDCWD, "/etc/passwd", O_RDONLY|O_CLOEXEC) = 3
read(3, "root:x:0:0:root:/root:/bin/bash\nda...", 4096) = 3206
read(3, "", 4096) = 0
close(3) = 0

Notice that fgets does not translate to many read syscalls. The C library buffers the file. It reads 4096 bytes in one syscall, then returns one line at a time. This is why buffered I/O is fast. If you called read directly in a loop, you would see hundreds of syscalls.

Also notice O_CLOEXEC. The C library sets this flag to ensure the fd is closed if you exec. This is a security feature. If you fork and exec a child, you do not want it inheriting open fds to sensitive files.

Common Mistakes and Gotchas

Mistake 1: Assuming strace shows all the time in your program. strace adds overhead, typically 5 to 50 percent. The actual time in syscalls is usually smaller. strace is for correctness, not performance measurement. Use perf for performance.

Mistake 2: Forgetting to redirect stderr. strace writes to stderr by default. If your program also writes to stderr, it gets mixed in. Use strace ./prog 2> strace.log to separate.

Mistake 3: Not using -f when debugging multi-process programs. Without -f, you only see the parent. The child does things you never see.

Mistake 4: Misreading fd numbers. fd 0 is stdin, fd 1 is stdout, fd 2 is stderr. Any fd 3 or higher is a file or socket you opened. If you see write(4, ...), that is writing to something you opened, not to a standard stream.

FAQ

Can I attach strace to a running process?

Yes. Use strace -p PID where PID is the process id. The process pauses while strace attaches, then resumes. Use -f if the process has children you want to trace too. This is invaluable for debugging hung or slow production processes.

Why does strace show errno instead of a regular error?

When a syscall fails, the kernel sets a variable called errno and returns -1 (or -1 cast as an unsigned type). strace decodes errno for you. For example, EACCES means permission denied (errno 13). ENOENT means file not found (errno 2). The man page errno lists all codes.

How do I see the contents of buffers passed to syscalls?

Use -s SIZE to increase the buffer print size (default 32 bytes). For example, strace -s 256 shows up to 256 bytes of data in write and read calls. Non-printable bytes show as \x hex codes.

Does strace work on all programs?

Almost. You cannot strace the kernel itself or some privileged operations. Some programs with ptrace detection (like debuggers) will refuse to run under strace. Most user programs work fine. You need the same or higher privilege as the target process.

Why is my program so much slower under strace?

Because strace intercepts every syscall. The kernel pauses your process, switches to strace in supervisor mode, reads register state, formats output, writes to a file or terminal, then switches back. This happens hundreds or thousands of times. Expect 5x to 50x slowdown. For timing, use -c and subtract strace overhead, or use perf with lower overhead.

Can I log strace output to a file instead of stdout?

Yes. Use -o FILE. Also use -ff to split output by process if you are tracing children. This creates FILE.PID files, one per process.

strace is a superpower for systems programmers. It shows you the truth about what your code does at the kernel boundary. Once you learn to read the output, debugging syscall issues becomes mechanical. You see every open, read, write, and fork. You spot fd leaks immediately. You find redundant syscalls that can be removed. And when something breaks at the systems level, strace is often the first tool you should reach for.

Why Your SQL Query Is Slow: Reading EXPLAIN

Bar Dror — Fri, 02 Oct 2026 07:00:56 +0000

Originally published on Code Beneath.

You run a query and it takes 12 seconds. You know it should take 200 milliseconds. The problem is somewhere in the database, not your application code. But where? This is where knowing how to read SQL EXPLAIN plan output becomes the difference between a quick fix and hours of guessing. An EXPLAIN plan shows you exactly what the database engine is doing: which tables it reads, in what order, whether it uses an index, how many rows it examines, and where the time actually goes. This is not optional knowledge for backend engineers. It is the foundation of real database performance work.

Most developers learn the basics: "use an index" or "add a WHERE clause." But they do not understand what the EXPLAIN output actually means, so they add indexes randomly and hope. They see a Seq Scan and think it is always bad. They see a Hash Join and assume it is slow. They miss the actual bottleneck because they do not know where to look. This article walks through a real slow query, shows you exactly how to read the EXPLAIN ANALYZE output line by line, and demonstrates the precise index that fixes it with before-and-after timings.

What an EXPLAIN Plan Actually Shows

An EXPLAIN plan is a tree of operations. The database tells you which operation runs first, what data it reads, how many rows it expects versus how many it actually finds, and how much time each step consumes. The indentation shows nesting: outer operations depend on inner ones.

In PostgreSQL, the two most important commands are EXPLAIN and EXPLAIN ANALYZE. EXPLAIN shows the estimated plan without running the query. EXPLAIN ANALYZE actually runs the query and shows real numbers. Always use EXPLAIN ANALYZE for real troubleshooting, because estimates can be wildly wrong.

Each line in an EXPLAIN ANALYZE output tells you:

The operation (Seq Scan, Index Scan, Hash Join, Sort, Aggregate, etc.)
The table or index involved
The filter conditions applied
Rows output by this step (actual)
Rows estimated by the planner (planned)
Time spent in this step
Total time including child operations

A Real Slow Query: Orders Report

Consider a typical SaaS database with orders, customers, and line items. A report needs to find all orders placed after January 2024 by customers in California with a total value over 1000 dollars. Here is the schema:

CREATE TABLE customers (
  id BIGINT PRIMARY KEY,
  name VARCHAR(255),
  state VARCHAR(2),
  created_at TIMESTAMP
);

CREATE TABLE orders (
  id BIGINT PRIMARY KEY,
  customer_id BIGINT NOT NULL,
  order_date TIMESTAMP NOT NULL,
  created_at TIMESTAMP,
  FOREIGN KEY(customer_id) REFERENCES customers(id)
);

CREATE TABLE order_items (
  id BIGINT PRIMARY KEY,
  order_id BIGINT NOT NULL,
  price NUMERIC(10, 2),
  quantity INT,
  FOREIGN KEY(order_id) REFERENCES orders(id)
);

CREATE INDEX idx_orders_customer_id ON orders(customer_id);
CREATE INDEX idx_orders_date ON orders(order_date);

The query:

SELECT
  o.id,
  c.name,
  SUM(oi.price * oi.quantity) as total
FROM orders o
JOIN customers c ON c.id = o.customer_id
JOIN order_items oi ON oi.order_id = o.id
WHERE c.state = 'CA'
  AND o.order_date >= '2024-01-01'
  AND (oi.price * oi.quantity) IS NOT NULL
GROUP BY o.id, c.name
HAVING SUM(oi.price * oi.quantity) > 1000
ORDER BY total DESC;

Assume the database has 500,000 orders, 50,000 customers, and 2,000,000 line items. The query takes 8.4 seconds. Here is the actual EXPLAIN ANALYZE output:

Limit  (cost=145223.50..145223.63 rows=50 width=52) (actual time=8421.340..8421.365 rows=50 loops=1)
  ->  Sort  (cost=145223.50..145223.63 rows=50 width=52) (actual time=8421.320..8421.340 rows=50 loops=1)
        Sort Key: (sum(oi.price * oi.quantity)) DESC
        Sort Space Used: 4 kB
        ->  GroupAggregate  (cost=145198.72..145221.92 rows=50 width=52) (actual time=8420.120..8421.310 rows=50 loops=1)
              Group By Key: o.id, c.name
              Filter: (sum(oi.price * oi.quantity) > 1000)
              Plans:
                ->  Sort  (cost=145198.72..145205.92 rows=2876 width=52) (actual time=8420.100..8420.300 rows=28950 loops=1)
                      Sort Key: o.id, c.name
                      Sort Space Used: 2450 kB
                      ->  Hash Join  (cost=1252.50..145120.50 rows=28950 width=52) (actual time=12.250..8380.240 rows=28950 loops=1)
                            Hash Cond: (oi.order_id = o.id)
                            ->  Seq Scan on order_items oi  (cost=0.00..98560.00 rows=2000000 width=16) (actual time=0.050..245.150 rows=2000000 loops=1)
                            ->  Hash  (cost=1200.50..1200.50 rows=4160 width=36) (actual time=12.100..12.100 rows=4160 loops=1)
                                  ->  Hash Join  (cost=352.50..1200.50 rows=4160 width=36) (actual time=0.100..11.890 rows=4160 loops=1)
                                        Hash Cond: (o.customer_id = c.id)
                                        ->  Index Scan using idx_orders_date on orders o  (cost=0.42..875.00 rows=4160 width=8) (actual time=0.050..5.240 rows=4160 loops=1)
                                              Index Cond: (order_date >= '2024-01-01')
                                        ->  Hash  (cost=250.00..250.00 rows=1000 width=28) (actual time=0.030..0.030 rows=1000 loops=1)
                                              ->  Seq Scan on customers c  (cost=0.00..250.00 rows=1000 width=28) (actual time=0.010..0.240 rows=1000 loops=1)
                                                    Filter: (state = 'CA')
                                                    Rows Removed by Filter: 49000

Reading the EXPLAIN Output Step by Step

Start at the innermost operation and work outward. The database engine executes from the bottom up.

Innermost: Seq Scan on customers (state = 'CA') The filter "state = 'CA'" removes 49,000 rows out of 50,000. It takes 0.24 milliseconds and returns 1,000 rows. This is fine. A full table scan of 50,000 rows is acceptable here.

Hash on customers The 1,000 CA customers are loaded into a hash table in memory (12.1 ms). This is necessary for the join and is fast.

Index Scan on orders (order_date >= '2024-01-01') The planner uses idx_orders_date to find 4,160 orders after January 2024. The index is being used efficiently (5.24 ms). Notice: actual rows (4,160) match the plan estimate almost exactly. The planner got this right.

Hash Join: orders + customers The 4,160 orders are joined to the 1,000 CA customers using a hash join. This takes 11.89 ms and produces 4,160 rows (all the matching orders).

Seq Scan on order_items: THE PROBLEM Look at this line: "Seq Scan on order_items oi (cost=0.00..98560.00 rows=2000000 width=16) (actual time=0.050..245.150 rows=2000000 loops=1)". The database reads all 2,000,000 line items. It scans every single row in the table. This operation takes 245 milliseconds. But we only need line items for the 4,160 orders we found. We are reading 480 times more rows than necessary.

Hash Join: order_items + orders The join itself (8,380 ms) is dominated by the Seq Scan on order_items. The hash table for the 4,160 orders is built, then matched against all 2,000,000 line items. Output: 28,950 rows (about 7 line items per matching order). The join itself is fast; the bottleneck is reading all 2,000,000 rows first.

Sort and GroupAggregate After the join, 28,950 rows must be sorted by (o.id, c.name) so the GROUP BY can aggregate them. This sort takes 8,420 ms and uses 2,450 kB of memory. This is not the root cause, but a consequence of the massive join output.

The real problem: the Seq Scan on order_items reads 2,000,000 rows when it should read only the ~30,000 line items we need. We need an index on order_items(order_id) so the database can jump directly to the relevant rows.

The Fix: Create the Missing Index

CREATE INDEX idx_order_items_order_id ON order_items(order_id);

Now run the query again with EXPLAIN ANALYZE:

Limit  (cost=5234.50..5234.63 rows=50 width=52) (actual time=185.240..185.265 rows=50 loops=1)
  ->  Sort  (cost=5234.50..5234.63 rows=50 width=52) (actual time=185.220..185.240 rows=50 loops=1)
        Sort Key: (sum(oi.price * oi.quantity)) DESC
        Sort Space Used: 4 kB
        ->  GroupAggregate  (cost=5209.72..5231.92 rows=50 width=52) (actual time=184.120..185.210 rows=50 loops=1)
              Group By Key: o.id, c.name
              Filter: (sum(oi.price * oi.quantity) > 1000)
              Plans:
                ->  Sort  (cost=5209.72..5216.92 rows=2876 width=52) (actual time=184.100..184.300 rows=28950 loops=1)
                      Sort Key: o.id, c.name
                      Sort Space Used: 2450 kB
                      ->  Nested Loop  (cost=352.50..5130.50 rows=28950 width=52) (actual time=0.100..80.240 rows=28950 loops=1)
                            ->  Hash Join  (cost=352.50..1200.50 rows=4160 width=36) (actual time=0.100..11.890 rows=4160 loops=1)
                                  Hash Cond: (o.customer_id = c.id)
                                  ->  Index Scan using idx_orders_date on orders o  (cost=0.42..875.00 rows=4160 width=8) (actual time=0.050..5.240 rows=4160 loops=1)
                                        Index Cond: (order_date >= '2024-01-01')'
                                  ->  Hash  (cost=250.00..250.00 rows=1000 width=28) (actual time=0.030..0.030 rows=1000 loops=1)
                                        ->  Seq Scan on customers c  (cost=0.00..250.00 rows=1000 width=28) (actual time=0.010..0.240 rows=1000 loops=1)
                                              Filter: (state = 'CA')
                                              Rows Removed by Filter: 49000
                            ->  Index Scan using idx_order_items_order_id on order_items oi  (cost=0.29..0.95 rows=7 width=16) (actual time=0.010..0.015 rows=7 loops=4160)

Look at the difference in the order_items step:

Metric	Before Index	After Index
Operation	Seq Scan (all 2M rows)	Index Scan (specific rows)
Rows Read	2,000,000	~28,950 (via nested loop)
Per-Loop Time	245 ms total	0.015 ms per iteration
Loops	1	4,160 (one per order)
Total Query Time	8,421 ms	185 ms

The join strategy changed from Hash Join to Nested Loop. With the index, the database can now say: "For each of these 4,160 orders, look up the order_items directly in the index." Each lookup takes 0.015 ms and finds 7 rows on average. 4,160 orders × 7 rows = 28,950 rows. The sort still takes time (184 ms), but that is unavoidable for ordering.

Total improvement: 8,421 ms to 185 ms. A 45x speedup. The query is now acceptable.

Understanding Key Metrics in EXPLAIN Output

cost=X..Y The planner's estimate of work. X is startup cost, Y is total cost. These are abstract units, not milliseconds. Larger costs usually mean more I/O or processing, but they are relative, not absolute. Use them to compare plans, not to predict real time.

actual time=X..Y Only shown in EXPLAIN ANALYZE. X is time to return the first row, Y is time to return all rows. Measured in milliseconds. This is real timing data and is what you care about.

rows The actual number of rows returned by this operation. Compare it to the estimated rows in the plan. If actual is much higher than estimated, the planner is bad at guessing, and you may need to update table statistics with ANALYZE.

loops How many times this operation ran. A Nested Loop that scans order_items with loops=4160 means it ran 4,160 times, once for each order. loops=1 means it ran once.

Filter vs Index Cond Index Cond is applied by the index itself (fast). Filter is applied after the index returns rows (slower). If you see Filter on a table scan, that condition was not used in the index, so the database had to read the rows first.

Common Mistakes When Reading EXPLAIN

Mistake 1: Focusing on cost instead of actual time Many developers see cost=145223 and assume the operation is slow. Cost is a planner heuristic. Real time comes from the actual time field. A high cost might be justified if the result set is small.

Mistake 2: Assuming Seq Scan is always bad A Seq Scan reading 50,000 rows in 240 ms is completely fine and often the right choice. Seq Scans are fast for small tables or when the cost of an index scan (with random I/O) would be higher. The problem in our example was reading 2,000,000 rows, not the Seq Scan itself.

Mistake 3: Missing the actual bottleneck because you stop reading early Developers see a Hash Join and assume it is slow, then add indexes randomly. The actual bottleneck is often in the child operations (the Seq Scan that feeds the join), not the join itself.

Mistake 4: Not updating ANALYZE If the planner estimates are wildly off (actual rows vs planned rows differ by 10x), run ANALYZE on the table to update statistics. The planner makes decisions based on these statistics.

ANALYZE order_items;

Mistake 5: Confusing loops with parallelism loops=10 does not mean the operation ran in parallel. It means it ran 10 times sequentially, probably in a nested loop. Parallelism is shown differently (Parallel Seq Scan).

When to Add Indexes vs Other Fixes

EXPLAIN ANALYZE tells you if an index would help, but not whether you should add one. Consider:

Add an index if: You see a Seq Scan on a large table where the output is much smaller than the input (many rows filtered out), and the query is slow. The index scan cost (including random I/O) should be lower than reading all rows.

Do not add an index if: The table is small (under 1,000 rows) or the index would be unused (no matching filter conditions). Indexes add write overhead to INSERT, UPDATE, and DELETE operations.

Consider rewriting the query if: The plan shows a massive join output followed by a sort and group, like in our example. Sometimes a different join order or aggregation strategy is faster. This is where a database consultant can help. If you manage a production database that regularly has slow query problems, consulting on query optimization can save significant engineering time.

Tools for Real-World Debugging

EXPLAIN is built into every SQL database, but parsing output by hand is tedious. Tools like pgAdmin for PostgreSQL, or DBeaver (which works across databases) make this easier by highlighting slow operations and suggesting optimizations. See recommended tools for a list of debuggers and analyzers that actually work.

For production systems, log slow queries and their EXPLAIN plans automatically. PostgreSQL has log_min_duration_statement to log queries over a threshold. MySQL has the slow query log. Use these logs to find the real bottlenecks in your system, not guesses from development.

FAQ

What does a Seq Scan mean?

Seq Scan (sequential scan) means the database reads every row in the table, in order. It does not use an index. This is not always bad. It is fast for small tables or when filtering most rows. It is a problem when you have millions of rows and the WHERE condition could use an index but does not.

Why do actual rows differ from planned rows?

The planner estimates based on table statistics, which become stale. Run ANALYZE to update statistics after large data changes. If estimates are still wrong, you may have columns with non-uniform data distribution, or the statistics table settings need tuning.

What is a Nested Loop?

A Nested Loop join runs the inner operation once for each row from the outer operation. It is fast when the inner operation is cheap (like an index lookup) and the outer result set is small. It is slow when combined with Seq Scans because you read the inner table repeatedly.

Can I force the planner to use a specific index?

In PostgreSQL, you can use the enable_* parameters or hint extensions like pg_hint_plan. In MySQL, you can use USE INDEX or FORCE INDEX. Avoid this unless the planner is clearly wrong. It is usually a sign that your statistics are outdated or the schema needs restructuring.

How do I know if an index will slow down writes?

Every index adds overhead to INSERT, UPDATE, and DELETE. Run a write-heavy workload before and after adding an index and measure throughput. For OLTP systems, the read speedup usually justifies the write cost. For write-heavy systems (logs, event streams), indexes must be selective.

Should I index every column used in a WHERE clause?

No. Index the columns that filter the most rows first, then add multi-column indexes if the planner still uses Seq Scans. Too many indexes slow down writes and waste storage. Read the EXPLAIN plan to see which columns actually need indexes.

What is the difference between EXPLAIN and EXPLAIN ANALYZE?

EXPLAIN shows the plan without running the query (fast, estimates only). EXPLAIN ANALYZE actually runs the query and shows real numbers (slow, but accurate). Always use EXPLAIN ANALYZE for troubleshooting.

TLS and mTLS Handshake Explained With a Runnable Example

Bar Dror — Wed, 30 Sep 2026 08:08:41 +0000

Originally published on Code Beneath.

A TLS handshake happens every time your browser connects to an HTTPS site. A mutual TLS or mTLS handshake is the same process, except the server also verifies the client's certificate. Most developers understand the concept but struggle when building systems that require mTLS authentication, because they've never seen what actually crosses the wire. This article walks through a real mtls handshake example with packet-level detail using openssl s_client, then provides a complete, runnable Go server and client you can execute immediately.

Why mTLS Matters

Standard TLS (one-way TLS) proves the server's identity to the client. The client trusts the server because a Certificate Authority signed the server's certificate. But what if you're running microservices on your own network and want to ensure only your authorized services can connect to each other? That's where mTLS comes in. Each service has its own certificate, signed by your private CA. When service A connects to service B, service B demands to see A's certificate and validates it. Service A, in turn, validates service B's certificate. Both sides prove their identity. This is standard in service mesh implementations (Istio, Linkerd) and in Kubernetes clusters.

The TLS Handshake: ClientHello to Finished

Before we look at mTLS specifically, let's trace a standard TLS handshake. The client sends a ClientHello with supported cipher suites, TLS versions, and extensions. The server responds with ServerHello, picks a cipher suite, and sends its certificate chain. The client validates the chain against a trusted CA. Both sides compute a session key. The server says "done," the client says "done," and encrypted communication begins.

In mTLS, after the server sends its certificate, the server sends a CertificateRequest message telling the client "I also want your certificate." The client sends its certificate. The server validates it. Then both parties send a Finished message, and the handshake completes. The entire conversation is still in plaintext until the Finished message, which is encrypted with the derived session key to prove both sides computed the same key and no one tampered with the handshake.

Seeing the Handshake Byte-by-Byte with openssl s_client

Let's first create a self-signed certificate and a minimal HTTPS server to examine. Then we'll use openssl to spy on what happens.

# Generate a self-signed server certificate valid for 365 days
openssl req -x509 -newkey rsa:2048 -nodes \
  -out server.crt -keyout server.key -days 365 \
  -subj "/CN=localhost"

Now create a minimal Python HTTPS server to serve that certificate:

import ssl
import http.server
import socketserver

PORT = 8443

class MyHTTPSHandler(http.server.SimpleHTTPRequestHandler):
    def do_GET(self):
        self.send_response(200)
        self.send_header("Content-type", "text/plain")
        self.end_headers()
        self.wfile.write(b"Hello from HTTPS\n")

context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
context.load_cert_chain("server.crt", "server.key")

with socketserver.TCPServer(("127.0.0.1", PORT), MyHTTPSHandler) as httpd:
    print(f"Listening on https://127.0.0.1:{PORT}")
    httpd.serve_forever()

Run the server in one terminal:

python3 https_server.py

In another terminal, use openssl s_client with verbose output to watch the handshake:

openssl s_client -connect 127.0.0.1:8443 -servername localhost -showcerts < /dev/null

You'll see output similar to this (simplified):

CONNECTED(00000003)
depth=0 CN = localhost
verify error:num=18:self signed certificate
verify return:1
---
Certificate chain
 0 s:CN = localhost
   i:CN = localhost
-----BEGIN CERTIFICATE-----
MIICpDCCAYwCCQC+...
-----END CERTIFICATE-----
subject=CN = localhost

issuer=CN = localhost

---
No client certificate CA names sent
---
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: ...

Notice "No client certificate CA names sent" because the server is not requiring client authentication. That's standard TLS. Now let's build mTLS.

Building mTLS: Generate Client and Server Certificates

For mTLS, we need a CA, a server certificate signed by that CA, and a client certificate signed by the same CA.

# Step 1: Create a self-signed CA
openssl genrsa -out ca.key 2048
openssl req -new -x509 -days 365 -key ca.key -out ca.crt \
  -subj "/CN=MyCA"

# Step 2: Create a server certificate signed by the CA
openssl genrsa -out server.key 2048
openssl req -new -key server.key -out server.csr \
  -subj "/CN=localhost"
openssl x509 -req -days 365 -in server.csr \
  -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out server.crt

# Step 3: Create a client certificate signed by the CA
openssl genrsa -out client.key 2048
openssl req -new -key client.key -out client.csr \
  -subj "/CN=clientuser"
openssl x509 -req -days 365 -in client.csr \
  -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out client.crt

Now we have six files: ca.key, ca.crt (the CA), server.key, server.crt (server credentials), and client.key, client.crt (client credentials).

mTLS Server and Client in Go

Here's a complete, runnable Go HTTPS server that requires client certificates:

package main

import (
    "crypto/tls"
    "crypto/x509"
    "fmt"
    "io/ioutil"
    "log"
    "net"
    "net/http"
)

func handler(w http.ResponseWriter, r *http.Request) {
    clientCert := r.TLS.PeerCertificates[0]
    fmt.Fprintf(w, "Hello, %s!\n", clientCert.Subject.CommonName)
    fmt.Fprintf(w, "You connected with certificate serial: %s\n", clientCert.SerialNumber)
}

func main() {
    // Load CA certificate to verify client certificates
    caCertPEM, err := ioutil.ReadFile("ca.crt")
    if err != nil {
        log.Fatal(err)
    }

    caCertPool := x509.NewCertPool()
    if !caCertPool.AppendCertsFromPEM(caCertPEM) {
        log.Fatal("Failed to parse CA certificate")
    }

    // Load server certificate and key
    serverCert, err := tls.LoadX509KeyPair("server.crt", "server.key")
    if err != nil {
        log.Fatal(err)
    }

    // Create TLS configuration for the server
    tlsConfig := &tls.Config{
        Certificates: []tls.Certificate{serverCert},
        ClientAuth:   tls.RequireAndVerifyClientCert,
        ClientCAs:    caCertPool,
    }

    // Create a listener with TLS
    listener, err := tls.Listen("tcp", "127.0.0.1:8443", tlsConfig)
    if err != nil {
        log.Fatal(err)
    }
    defer listener.Close()

    // Wrap listener to extract peer info for logging
    http.HandleFunc("/", handler)

    server := &http.Server{
        TLSConfig: tlsConfig,
    }

    fmt.Println("mTLS server listening on https://127.0.0.1:8443")
    fmt.Println("Press Ctrl+C to stop")

    if err := server.Serve(listener); err != nil && err != http.ErrServerClosed {
        log.Fatal(err)
    }
}

Save this as mtls_server.go. Now the client:

package main

import (
    "crypto/tls"
    "crypto/x509"
    "fmt"
    "io/ioutil"
    "log"
    "net/http"
)

func main() {
    // Load CA certificate to verify server
    caCertPEM, err := ioutil.ReadFile("ca.crt")
    if err != nil {
        log.Fatal(err)
    }

    caCertPool := x509.NewCertPool()
    if !caCertPool.AppendCertsFromPEM(caCertPEM) {
        log.Fatal("Failed to parse CA certificate")
    }

    // Load client certificate and key
    clientCert, err := tls.LoadX509KeyPair("client.crt", "client.key")
    if err != nil {
        log.Fatal(err)
    }

    // Create TLS configuration for the client
    tlsConfig := &tls.Config{
        Certificates: []tls.Certificate{clientCert},
        RootCAs:      caCertPool,
    }

    // Create HTTP client with custom TLS config
    client := &http.Client{
        Transport: &http.Transport{
            TLSClientConfig: tlsConfig,
        },
    }

    // Make request to server
    resp, err := client.Get("https://127.0.0.1:8443/")
    if err != nil {
        log.Fatal(err)
    }
    defer resp.Body.Close()

    body, err := ioutil.ReadAll(resp.Body)
    if err != nil {
        log.Fatal(err)
    }

    fmt.Println(string(body))
}

Save this as mtls_client.go. Now test it:

# Terminal 1: Run the server
go run mtls_server.go

# Terminal 2: Run the client
go run mtls_client.go

Output from the client:

Hello, clientuser!
You connected with certificate serial: ...

The handshake succeeded. Now let's see what openssl reports about client certificates being requested:

openssl s_client -connect 127.0.0.1:8443 -cert client.crt -key client.key \
  -CAfile ca.crt < /dev/null

Look for the line "Verify return code: 0 (ok)" and notice now it says "Client certificate requested" in the output. The handshake includes the CertificateRequest message.

What Happens If the Client Has No Certificate

Try connecting without a client certificate:

openssl s_client -connect 127.0.0.1:8443 < /dev/null 2>&1 | head -20

You'll see an error during the handshake. The server expects a certificate, and the connection drops. This is the security boundary: only clients with a valid certificate signed by your CA can connect.

Comparison: One-Way TLS vs. mTLS

Aspect	One-Way TLS	mTLS (Mutual TLS)
Server proves identity	Yes	Yes
Client proves identity	No	Yes
Server validates client cert	No	Yes
Use case	Public websites (HTTPS)	Microservices, internal APIs, service mesh
ClientAuth setting	tls.NoClientCert or tls.RequestClientCert	tls.RequireAndVerifyClientCert
Handshake message count	~8 messages	~10 messages (includes CertificateRequest, client cert, client cert verify)

Common Mistakes

Forgetting to set ClientCAs. If you set ClientAuth to RequireAndVerifyClientCert but don't load the CA certificates into ClientCAs, every client certificate validation will fail. The server has nothing to verify against.

Mixing up RootCAs and ClientCAs. In a client, RootCAs is for verifying the server's certificate. In a server, ClientCAs is for verifying the client's certificate. They often contain the same CA cert, but the role is different. The Go docs can be confusing here.

Using tls.InsecureSkipVerify in production. This disables all certificate validation and opens you to man-in-the-middle attacks. Never do this outside of local testing.

Forgetting certificate rotation. Certificates expire. In a service mesh or large deployment, you need a process to rotate certificates before they expire. This is where tools like cert-manager (in Kubernetes) come in, but that's a separate topic.

Not understanding the CN vs. SANs issue. In modern TLS, the CommonName (CN) is deprecated; use Subject Alternative Names (SANs) instead. If you need your certificate to work for both localhost and 127.0.0.1, add both as SANs when generating the certificate. For this article's simple example, CN is fine.

Integration with Token-Based Security

mTLS and token-based authentication are complementary. mTLS secures the transport layer and proves the client's identity at the network level. On top of that, you can layer token-based mechanisms (JWT, OAuth) for application-level authorization. For a detailed treatment of how tokens work in web applications, see token-based security in web applications.

Common Pitfalls with Certificate Management

In a real system, managing certificates securely is critical. Never commit private keys to version control. Treat them like passwords. See managing password safety on Linux for best practices on securing sensitive files. The same principles apply to certificate keys.

What Happens Inside the Handshake

Here's the step-by-step message flow for mTLS:

ClientHello: Client sends supported cipher suites, TLS versions, and random nonce.
ServerHello: Server picks a cipher suite and sends its random nonce.
Certificate: Server sends its certificate chain.
CertificateRequest: Server tells the client it must also send a certificate. This message is absent in standard TLS.
ServerKeyExchange or direct key agreement: Server sends key agreement parameters (or skips this in TLS 1.3).
ServerHelloDone: Server says "your turn."
Certificate: Client sends its certificate.
ClientKeyExchange: Client sends key agreement parameters.
CertificateVerify: Client signs a hash of the handshake transcript with its private key, proving it owns the certificate. This is absent in standard TLS.
Finished: Client sends an encrypted message proving the session key is correct.
Finished: Server responds with its own encrypted Finished message.

The CertificateRequest, client Certificate, and CertificateVerify messages are the three additions that make mTLS mutual instead of one-way.

Testing mTLS with curl

Once your server is running, you can also test with curl:

curl --cert client.crt --key client.key \
  --cacert ca.crt \
  https://127.0.0.1:8443/

This is useful for manual testing of mTLS endpoints. curl requires the --cert flag (client certificate), --key flag (client private key), and --cacert flag (CA certificate to verify the server). Without all three, the handshake will fail.

Performance Considerations

mTLS adds negligible overhead to each request after the initial handshake. The handshake itself takes a few milliseconds. In a service mesh where services are constantly opening new connections (or reusing them via connection pooling), the handshake cost is amortized. The real cost is CPU for certificate validation and TLS encryption/decryption, but modern CPUs with AES-NI instructions handle this efficiently. For internal microservices, mTLS is the right choice; the security benefit far outweighs the performance cost.

FAQ

Why is mTLS needed if I'm already using TLS for HTTPS?

Standard HTTPS proves the server's identity to the client. mTLS also proves the client's identity to the server. In a public website, you don't need to verify the client because millions of strangers visit. In a private network of microservices, you want each service to authenticate itself. mTLS is a control boundary: only services with valid certificates can connect.

Can I use self-signed certificates with mTLS?

Yes, as shown in this article. For production, you typically run your own CA (private CA) or use a tool like cert-manager to manage certificates. Self-signed certificates are fine for development and testing.

How do I know if a connection is using mTLS?

On the server side, if r.TLS.PeerCertificates is not empty and has at least one certificate, the client authenticated with mTLS. On the client side, if you load a certificate into the TLS config and send it, you're using mTLS. You can also use openssl s_client and look for "Client certificate requested" in the output.

What is the difference between ClientAuth: tls.RequestClientCert and tls.RequireAndVerifyClientCert?

RequestClientCert means "ask the client for a certificate, but allow the handshake to succeed if the client doesn't send one." RequireAndVerifyClientCert means "demand a certificate and fail the handshake if one is not provided or is invalid." For security-critical paths (microservices), use RequireAndVerifyClientCert.

Can I verify the client certificate's CN instead of relying on the CA pool?

Not recommended. Always verify using the CA chain (ClientCAs pool). If you manually check the CN, you bypass Go's robust certificate validation logic and can introduce vulnerabilities. Trust the standard library's verification.

How do I handle certificate expiration?

Monitor certificate expiration dates and rotate them before they expire. In Kubernetes, cert-manager automates this. For manual systems, set a calendar reminder 30 days before expiration. When you rotate, generate new certificates, deploy them to your services, and restart the services to pick up the new certificates.

Is mTLS the same as mutual authentication?

mTLS is mutual authentication at the transport layer using certificates. There are other forms of mutual authentication (API keys, OAuth) at the application layer. mTLS specifically refers to mutual certificate-based TLS.