DEV Community: Bar Dror

Why Your SQL Query Is Slow: Reading EXPLAIN

Bar Dror — Fri, 02 Oct 2026 07:00:56 +0000

Originally published on Code Beneath.

You run a query and it takes 12 seconds. You know it should take 200 milliseconds. The problem is somewhere in the database, not your application code. But where? This is where knowing how to read SQL EXPLAIN plan output becomes the difference between a quick fix and hours of guessing. An EXPLAIN plan shows you exactly what the database engine is doing: which tables it reads, in what order, whether it uses an index, how many rows it examines, and where the time actually goes. This is not optional knowledge for backend engineers. It is the foundation of real database performance work.

Most developers learn the basics: "use an index" or "add a WHERE clause." But they do not understand what the EXPLAIN output actually means, so they add indexes randomly and hope. They see a Seq Scan and think it is always bad. They see a Hash Join and assume it is slow. They miss the actual bottleneck because they do not know where to look. This article walks through a real slow query, shows you exactly how to read the EXPLAIN ANALYZE output line by line, and demonstrates the precise index that fixes it with before-and-after timings.

What an EXPLAIN Plan Actually Shows

An EXPLAIN plan is a tree of operations. The database tells you which operation runs first, what data it reads, how many rows it expects versus how many it actually finds, and how much time each step consumes. The indentation shows nesting: outer operations depend on inner ones.

In PostgreSQL, the two most important commands are EXPLAIN and EXPLAIN ANALYZE. EXPLAIN shows the estimated plan without running the query. EXPLAIN ANALYZE actually runs the query and shows real numbers. Always use EXPLAIN ANALYZE for real troubleshooting, because estimates can be wildly wrong.

Each line in an EXPLAIN ANALYZE output tells you:

The operation (Seq Scan, Index Scan, Hash Join, Sort, Aggregate, etc.)
The table or index involved
The filter conditions applied
Rows output by this step (actual)
Rows estimated by the planner (planned)
Time spent in this step
Total time including child operations

A Real Slow Query: Orders Report

Consider a typical SaaS database with orders, customers, and line items. A report needs to find all orders placed after January 2024 by customers in California with a total value over 1000 dollars. Here is the schema:

CREATE TABLE customers (
  id BIGINT PRIMARY KEY,
  name VARCHAR(255),
  state VARCHAR(2),
  created_at TIMESTAMP
);

CREATE TABLE orders (
  id BIGINT PRIMARY KEY,
  customer_id BIGINT NOT NULL,
  order_date TIMESTAMP NOT NULL,
  created_at TIMESTAMP,
  FOREIGN KEY(customer_id) REFERENCES customers(id)
);

CREATE TABLE order_items (
  id BIGINT PRIMARY KEY,
  order_id BIGINT NOT NULL,
  price NUMERIC(10, 2),
  quantity INT,
  FOREIGN KEY(order_id) REFERENCES orders(id)
);

CREATE INDEX idx_orders_customer_id ON orders(customer_id);
CREATE INDEX idx_orders_date ON orders(order_date);

The query:

SELECT
  o.id,
  c.name,
  SUM(oi.price * oi.quantity) as total
FROM orders o
JOIN customers c ON c.id = o.customer_id
JOIN order_items oi ON oi.order_id = o.id
WHERE c.state = 'CA'
  AND o.order_date >= '2024-01-01'
  AND (oi.price * oi.quantity) IS NOT NULL
GROUP BY o.id, c.name
HAVING SUM(oi.price * oi.quantity) > 1000
ORDER BY total DESC;

Assume the database has 500,000 orders, 50,000 customers, and 2,000,000 line items. The query takes 8.4 seconds. Here is the actual EXPLAIN ANALYZE output:

Limit  (cost=145223.50..145223.63 rows=50 width=52) (actual time=8421.340..8421.365 rows=50 loops=1)
  ->  Sort  (cost=145223.50..145223.63 rows=50 width=52) (actual time=8421.320..8421.340 rows=50 loops=1)
        Sort Key: (sum(oi.price * oi.quantity)) DESC
        Sort Space Used: 4 kB
        ->  GroupAggregate  (cost=145198.72..145221.92 rows=50 width=52) (actual time=8420.120..8421.310 rows=50 loops=1)
              Group By Key: o.id, c.name
              Filter: (sum(oi.price * oi.quantity) > 1000)
              Plans:
                ->  Sort  (cost=145198.72..145205.92 rows=2876 width=52) (actual time=8420.100..8420.300 rows=28950 loops=1)
                      Sort Key: o.id, c.name
                      Sort Space Used: 2450 kB
                      ->  Hash Join  (cost=1252.50..145120.50 rows=28950 width=52) (actual time=12.250..8380.240 rows=28950 loops=1)
                            Hash Cond: (oi.order_id = o.id)
                            ->  Seq Scan on order_items oi  (cost=0.00..98560.00 rows=2000000 width=16) (actual time=0.050..245.150 rows=2000000 loops=1)
                            ->  Hash  (cost=1200.50..1200.50 rows=4160 width=36) (actual time=12.100..12.100 rows=4160 loops=1)
                                  ->  Hash Join  (cost=352.50..1200.50 rows=4160 width=36) (actual time=0.100..11.890 rows=4160 loops=1)
                                        Hash Cond: (o.customer_id = c.id)
                                        ->  Index Scan using idx_orders_date on orders o  (cost=0.42..875.00 rows=4160 width=8) (actual time=0.050..5.240 rows=4160 loops=1)
                                              Index Cond: (order_date >= '2024-01-01')
                                        ->  Hash  (cost=250.00..250.00 rows=1000 width=28) (actual time=0.030..0.030 rows=1000 loops=1)
                                              ->  Seq Scan on customers c  (cost=0.00..250.00 rows=1000 width=28) (actual time=0.010..0.240 rows=1000 loops=1)
                                                    Filter: (state = 'CA')
                                                    Rows Removed by Filter: 49000

Reading the EXPLAIN Output Step by Step

Start at the innermost operation and work outward. The database engine executes from the bottom up.

Innermost: Seq Scan on customers (state = 'CA') The filter "state = 'CA'" removes 49,000 rows out of 50,000. It takes 0.24 milliseconds and returns 1,000 rows. This is fine. A full table scan of 50,000 rows is acceptable here.

Hash on customers The 1,000 CA customers are loaded into a hash table in memory (12.1 ms). This is necessary for the join and is fast.

Index Scan on orders (order_date >= '2024-01-01') The planner uses idx_orders_date to find 4,160 orders after January 2024. The index is being used efficiently (5.24 ms). Notice: actual rows (4,160) match the plan estimate almost exactly. The planner got this right.

Hash Join: orders + customers The 4,160 orders are joined to the 1,000 CA customers using a hash join. This takes 11.89 ms and produces 4,160 rows (all the matching orders).

Seq Scan on order_items: THE PROBLEM Look at this line: "Seq Scan on order_items oi (cost=0.00..98560.00 rows=2000000 width=16) (actual time=0.050..245.150 rows=2000000 loops=1)". The database reads all 2,000,000 line items. It scans every single row in the table. This operation takes 245 milliseconds. But we only need line items for the 4,160 orders we found. We are reading 480 times more rows than necessary.

Hash Join: order_items + orders The join itself (8,380 ms) is dominated by the Seq Scan on order_items. The hash table for the 4,160 orders is built, then matched against all 2,000,000 line items. Output: 28,950 rows (about 7 line items per matching order). The join itself is fast; the bottleneck is reading all 2,000,000 rows first.

Sort and GroupAggregate After the join, 28,950 rows must be sorted by (o.id, c.name) so the GROUP BY can aggregate them. This sort takes 8,420 ms and uses 2,450 kB of memory. This is not the root cause, but a consequence of the massive join output.

The real problem: the Seq Scan on order_items reads 2,000,000 rows when it should read only the ~30,000 line items we need. We need an index on order_items(order_id) so the database can jump directly to the relevant rows.

The Fix: Create the Missing Index

CREATE INDEX idx_order_items_order_id ON order_items(order_id);

Now run the query again with EXPLAIN ANALYZE:

Limit  (cost=5234.50..5234.63 rows=50 width=52) (actual time=185.240..185.265 rows=50 loops=1)
  ->  Sort  (cost=5234.50..5234.63 rows=50 width=52) (actual time=185.220..185.240 rows=50 loops=1)
        Sort Key: (sum(oi.price * oi.quantity)) DESC
        Sort Space Used: 4 kB
        ->  GroupAggregate  (cost=5209.72..5231.92 rows=50 width=52) (actual time=184.120..185.210 rows=50 loops=1)
              Group By Key: o.id, c.name
              Filter: (sum(oi.price * oi.quantity) > 1000)
              Plans:
                ->  Sort  (cost=5209.72..5216.92 rows=2876 width=52) (actual time=184.100..184.300 rows=28950 loops=1)
                      Sort Key: o.id, c.name
                      Sort Space Used: 2450 kB
                      ->  Nested Loop  (cost=352.50..5130.50 rows=28950 width=52) (actual time=0.100..80.240 rows=28950 loops=1)
                            ->  Hash Join  (cost=352.50..1200.50 rows=4160 width=36) (actual time=0.100..11.890 rows=4160 loops=1)
                                  Hash Cond: (o.customer_id = c.id)
                                  ->  Index Scan using idx_orders_date on orders o  (cost=0.42..875.00 rows=4160 width=8) (actual time=0.050..5.240 rows=4160 loops=1)
                                        Index Cond: (order_date >= '2024-01-01')'
                                  ->  Hash  (cost=250.00..250.00 rows=1000 width=28) (actual time=0.030..0.030 rows=1000 loops=1)
                                        ->  Seq Scan on customers c  (cost=0.00..250.00 rows=1000 width=28) (actual time=0.010..0.240 rows=1000 loops=1)
                                              Filter: (state = 'CA')
                                              Rows Removed by Filter: 49000
                            ->  Index Scan using idx_order_items_order_id on order_items oi  (cost=0.29..0.95 rows=7 width=16) (actual time=0.010..0.015 rows=7 loops=4160)

Look at the difference in the order_items step:

Metric	Before Index	After Index
Operation	Seq Scan (all 2M rows)	Index Scan (specific rows)
Rows Read	2,000,000	~28,950 (via nested loop)
Per-Loop Time	245 ms total	0.015 ms per iteration
Loops	1	4,160 (one per order)
Total Query Time	8,421 ms	185 ms

The join strategy changed from Hash Join to Nested Loop. With the index, the database can now say: "For each of these 4,160 orders, look up the order_items directly in the index." Each lookup takes 0.015 ms and finds 7 rows on average. 4,160 orders × 7 rows = 28,950 rows. The sort still takes time (184 ms), but that is unavoidable for ordering.

Total improvement: 8,421 ms to 185 ms. A 45x speedup. The query is now acceptable.

Understanding Key Metrics in EXPLAIN Output

cost=X..Y The planner's estimate of work. X is startup cost, Y is total cost. These are abstract units, not milliseconds. Larger costs usually mean more I/O or processing, but they are relative, not absolute. Use them to compare plans, not to predict real time.

actual time=X..Y Only shown in EXPLAIN ANALYZE. X is time to return the first row, Y is time to return all rows. Measured in milliseconds. This is real timing data and is what you care about.

rows The actual number of rows returned by this operation. Compare it to the estimated rows in the plan. If actual is much higher than estimated, the planner is bad at guessing, and you may need to update table statistics with ANALYZE.

loops How many times this operation ran. A Nested Loop that scans order_items with loops=4160 means it ran 4,160 times, once for each order. loops=1 means it ran once.

Filter vs Index Cond Index Cond is applied by the index itself (fast). Filter is applied after the index returns rows (slower). If you see Filter on a table scan, that condition was not used in the index, so the database had to read the rows first.

Common Mistakes When Reading EXPLAIN

Mistake 1: Focusing on cost instead of actual time Many developers see cost=145223 and assume the operation is slow. Cost is a planner heuristic. Real time comes from the actual time field. A high cost might be justified if the result set is small.

Mistake 2: Assuming Seq Scan is always bad A Seq Scan reading 50,000 rows in 240 ms is completely fine and often the right choice. Seq Scans are fast for small tables or when the cost of an index scan (with random I/O) would be higher. The problem in our example was reading 2,000,000 rows, not the Seq Scan itself.

Mistake 3: Missing the actual bottleneck because you stop reading early Developers see a Hash Join and assume it is slow, then add indexes randomly. The actual bottleneck is often in the child operations (the Seq Scan that feeds the join), not the join itself.

Mistake 4: Not updating ANALYZE If the planner estimates are wildly off (actual rows vs planned rows differ by 10x), run ANALYZE on the table to update statistics. The planner makes decisions based on these statistics.

ANALYZE order_items;

Mistake 5: Confusing loops with parallelism loops=10 does not mean the operation ran in parallel. It means it ran 10 times sequentially, probably in a nested loop. Parallelism is shown differently (Parallel Seq Scan).

When to Add Indexes vs Other Fixes

EXPLAIN ANALYZE tells you if an index would help, but not whether you should add one. Consider:

Add an index if: You see a Seq Scan on a large table where the output is much smaller than the input (many rows filtered out), and the query is slow. The index scan cost (including random I/O) should be lower than reading all rows.

Do not add an index if: The table is small (under 1,000 rows) or the index would be unused (no matching filter conditions). Indexes add write overhead to INSERT, UPDATE, and DELETE operations.

Consider rewriting the query if: The plan shows a massive join output followed by a sort and group, like in our example. Sometimes a different join order or aggregation strategy is faster. This is where a database consultant can help. If you manage a production database that regularly has slow query problems, consulting on query optimization can save significant engineering time.

Tools for Real-World Debugging

EXPLAIN is built into every SQL database, but parsing output by hand is tedious. Tools like pgAdmin for PostgreSQL, or DBeaver (which works across databases) make this easier by highlighting slow operations and suggesting optimizations. See recommended tools for a list of debuggers and analyzers that actually work.

For production systems, log slow queries and their EXPLAIN plans automatically. PostgreSQL has log_min_duration_statement to log queries over a threshold. MySQL has the slow query log. Use these logs to find the real bottlenecks in your system, not guesses from development.

FAQ

What does a Seq Scan mean?

Seq Scan (sequential scan) means the database reads every row in the table, in order. It does not use an index. This is not always bad. It is fast for small tables or when filtering most rows. It is a problem when you have millions of rows and the WHERE condition could use an index but does not.

Why do actual rows differ from planned rows?

The planner estimates based on table statistics, which become stale. Run ANALYZE to update statistics after large data changes. If estimates are still wrong, you may have columns with non-uniform data distribution, or the statistics table settings need tuning.

What is a Nested Loop?

A Nested Loop join runs the inner operation once for each row from the outer operation. It is fast when the inner operation is cheap (like an index lookup) and the outer result set is small. It is slow when combined with Seq Scans because you read the inner table repeatedly.

Can I force the planner to use a specific index?

In PostgreSQL, you can use the enable_* parameters or hint extensions like pg_hint_plan. In MySQL, you can use USE INDEX or FORCE INDEX. Avoid this unless the planner is clearly wrong. It is usually a sign that your statistics are outdated or the schema needs restructuring.

How do I know if an index will slow down writes?

Every index adds overhead to INSERT, UPDATE, and DELETE. Run a write-heavy workload before and after adding an index and measure throughput. For OLTP systems, the read speedup usually justifies the write cost. For write-heavy systems (logs, event streams), indexes must be selective.

Should I index every column used in a WHERE clause?

No. Index the columns that filter the most rows first, then add multi-column indexes if the planner still uses Seq Scans. Too many indexes slow down writes and waste storage. Read the EXPLAIN plan to see which columns actually need indexes.

What is the difference between EXPLAIN and EXPLAIN ANALYZE?

EXPLAIN shows the plan without running the query (fast, estimates only). EXPLAIN ANALYZE actually runs the query and shows real numbers (slow, but accurate). Always use EXPLAIN ANALYZE for troubleshooting.

TLS and mTLS Handshake Explained With a Runnable Example

Bar Dror — Wed, 30 Sep 2026 08:08:41 +0000

Originally published on Code Beneath.

A TLS handshake happens every time your browser connects to an HTTPS site. A mutual TLS or mTLS handshake is the same process, except the server also verifies the client's certificate. Most developers understand the concept but struggle when building systems that require mTLS authentication, because they've never seen what actually crosses the wire. This article walks through a real mtls handshake example with packet-level detail using openssl s_client, then provides a complete, runnable Go server and client you can execute immediately.

Why mTLS Matters

Standard TLS (one-way TLS) proves the server's identity to the client. The client trusts the server because a Certificate Authority signed the server's certificate. But what if you're running microservices on your own network and want to ensure only your authorized services can connect to each other? That's where mTLS comes in. Each service has its own certificate, signed by your private CA. When service A connects to service B, service B demands to see A's certificate and validates it. Service A, in turn, validates service B's certificate. Both sides prove their identity. This is standard in service mesh implementations (Istio, Linkerd) and in Kubernetes clusters.

The TLS Handshake: ClientHello to Finished

Before we look at mTLS specifically, let's trace a standard TLS handshake. The client sends a ClientHello with supported cipher suites, TLS versions, and extensions. The server responds with ServerHello, picks a cipher suite, and sends its certificate chain. The client validates the chain against a trusted CA. Both sides compute a session key. The server says "done," the client says "done," and encrypted communication begins.

In mTLS, after the server sends its certificate, the server sends a CertificateRequest message telling the client "I also want your certificate." The client sends its certificate. The server validates it. Then both parties send a Finished message, and the handshake completes. The entire conversation is still in plaintext until the Finished message, which is encrypted with the derived session key to prove both sides computed the same key and no one tampered with the handshake.

Seeing the Handshake Byte-by-Byte with openssl s_client

Let's first create a self-signed certificate and a minimal HTTPS server to examine. Then we'll use openssl to spy on what happens.

# Generate a self-signed server certificate valid for 365 days
openssl req -x509 -newkey rsa:2048 -nodes \
  -out server.crt -keyout server.key -days 365 \
  -subj "/CN=localhost"

Now create a minimal Python HTTPS server to serve that certificate:

import ssl
import http.server
import socketserver

PORT = 8443

class MyHTTPSHandler(http.server.SimpleHTTPRequestHandler):
    def do_GET(self):
        self.send_response(200)
        self.send_header("Content-type", "text/plain")
        self.end_headers()
        self.wfile.write(b"Hello from HTTPS\n")

context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
context.load_cert_chain("server.crt", "server.key")

with socketserver.TCPServer(("127.0.0.1", PORT), MyHTTPSHandler) as httpd:
    print(f"Listening on https://127.0.0.1:{PORT}")
    httpd.serve_forever()

Run the server in one terminal:

python3 https_server.py

In another terminal, use openssl s_client with verbose output to watch the handshake:

openssl s_client -connect 127.0.0.1:8443 -servername localhost -showcerts < /dev/null

You'll see output similar to this (simplified):

CONNECTED(00000003)
depth=0 CN = localhost
verify error:num=18:self signed certificate
verify return:1
---
Certificate chain
 0 s:CN = localhost
   i:CN = localhost
-----BEGIN CERTIFICATE-----
MIICpDCCAYwCCQC+...
-----END CERTIFICATE-----
subject=CN = localhost

issuer=CN = localhost

---
No client certificate CA names sent
---
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: ...

Notice "No client certificate CA names sent" because the server is not requiring client authentication. That's standard TLS. Now let's build mTLS.

Building mTLS: Generate Client and Server Certificates

For mTLS, we need a CA, a server certificate signed by that CA, and a client certificate signed by the same CA.

# Step 1: Create a self-signed CA
openssl genrsa -out ca.key 2048
openssl req -new -x509 -days 365 -key ca.key -out ca.crt \
  -subj "/CN=MyCA"

# Step 2: Create a server certificate signed by the CA
openssl genrsa -out server.key 2048
openssl req -new -key server.key -out server.csr \
  -subj "/CN=localhost"
openssl x509 -req -days 365 -in server.csr \
  -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out server.crt

# Step 3: Create a client certificate signed by the CA
openssl genrsa -out client.key 2048
openssl req -new -key client.key -out client.csr \
  -subj "/CN=clientuser"
openssl x509 -req -days 365 -in client.csr \
  -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out client.crt

Now we have six files: ca.key, ca.crt (the CA), server.key, server.crt (server credentials), and client.key, client.crt (client credentials).

mTLS Server and Client in Go

Here's a complete, runnable Go HTTPS server that requires client certificates:

package main

import (
    "crypto/tls"
    "crypto/x509"
    "fmt"
    "io/ioutil"
    "log"
    "net"
    "net/http"
)

func handler(w http.ResponseWriter, r *http.Request) {
    clientCert := r.TLS.PeerCertificates[0]
    fmt.Fprintf(w, "Hello, %s!\n", clientCert.Subject.CommonName)
    fmt.Fprintf(w, "You connected with certificate serial: %s\n", clientCert.SerialNumber)
}

func main() {
    // Load CA certificate to verify client certificates
    caCertPEM, err := ioutil.ReadFile("ca.crt")
    if err != nil {
        log.Fatal(err)
    }

    caCertPool := x509.NewCertPool()
    if !caCertPool.AppendCertsFromPEM(caCertPEM) {
        log.Fatal("Failed to parse CA certificate")
    }

    // Load server certificate and key
    serverCert, err := tls.LoadX509KeyPair("server.crt", "server.key")
    if err != nil {
        log.Fatal(err)
    }

    // Create TLS configuration for the server
    tlsConfig := &tls.Config{
        Certificates: []tls.Certificate{serverCert},
        ClientAuth:   tls.RequireAndVerifyClientCert,
        ClientCAs:    caCertPool,
    }

    // Create a listener with TLS
    listener, err := tls.Listen("tcp", "127.0.0.1:8443", tlsConfig)
    if err != nil {
        log.Fatal(err)
    }
    defer listener.Close()

    // Wrap listener to extract peer info for logging
    http.HandleFunc("/", handler)

    server := &http.Server{
        TLSConfig: tlsConfig,
    }

    fmt.Println("mTLS server listening on https://127.0.0.1:8443")
    fmt.Println("Press Ctrl+C to stop")

    if err := server.Serve(listener); err != nil && err != http.ErrServerClosed {
        log.Fatal(err)
    }
}

Save this as mtls_server.go. Now the client:

package main

import (
    "crypto/tls"
    "crypto/x509"
    "fmt"
    "io/ioutil"
    "log"
    "net/http"
)

func main() {
    // Load CA certificate to verify server
    caCertPEM, err := ioutil.ReadFile("ca.crt")
    if err != nil {
        log.Fatal(err)
    }

    caCertPool := x509.NewCertPool()
    if !caCertPool.AppendCertsFromPEM(caCertPEM) {
        log.Fatal("Failed to parse CA certificate")
    }

    // Load client certificate and key
    clientCert, err := tls.LoadX509KeyPair("client.crt", "client.key")
    if err != nil {
        log.Fatal(err)
    }

    // Create TLS configuration for the client
    tlsConfig := &tls.Config{
        Certificates: []tls.Certificate{clientCert},
        RootCAs:      caCertPool,
    }

    // Create HTTP client with custom TLS config
    client := &http.Client{
        Transport: &http.Transport{
            TLSClientConfig: tlsConfig,
        },
    }

    // Make request to server
    resp, err := client.Get("https://127.0.0.1:8443/")
    if err != nil {
        log.Fatal(err)
    }
    defer resp.Body.Close()

    body, err := ioutil.ReadAll(resp.Body)
    if err != nil {
        log.Fatal(err)
    }

    fmt.Println(string(body))
}

Save this as mtls_client.go. Now test it:

# Terminal 1: Run the server
go run mtls_server.go

# Terminal 2: Run the client
go run mtls_client.go

Output from the client:

Hello, clientuser!
You connected with certificate serial: ...

The handshake succeeded. Now let's see what openssl reports about client certificates being requested:

openssl s_client -connect 127.0.0.1:8443 -cert client.crt -key client.key \
  -CAfile ca.crt < /dev/null

Look for the line "Verify return code: 0 (ok)" and notice now it says "Client certificate requested" in the output. The handshake includes the CertificateRequest message.

What Happens If the Client Has No Certificate

Try connecting without a client certificate:

openssl s_client -connect 127.0.0.1:8443 < /dev/null 2>&1 | head -20

You'll see an error during the handshake. The server expects a certificate, and the connection drops. This is the security boundary: only clients with a valid certificate signed by your CA can connect.

Comparison: One-Way TLS vs. mTLS

Aspect	One-Way TLS	mTLS (Mutual TLS)
Server proves identity	Yes	Yes
Client proves identity	No	Yes
Server validates client cert	No	Yes
Use case	Public websites (HTTPS)	Microservices, internal APIs, service mesh
ClientAuth setting	tls.NoClientCert or tls.RequestClientCert	tls.RequireAndVerifyClientCert
Handshake message count	~8 messages	~10 messages (includes CertificateRequest, client cert, client cert verify)

Common Mistakes

Forgetting to set ClientCAs. If you set ClientAuth to RequireAndVerifyClientCert but don't load the CA certificates into ClientCAs, every client certificate validation will fail. The server has nothing to verify against.

Mixing up RootCAs and ClientCAs. In a client, RootCAs is for verifying the server's certificate. In a server, ClientCAs is for verifying the client's certificate. They often contain the same CA cert, but the role is different. The Go docs can be confusing here.

Using tls.InsecureSkipVerify in production. This disables all certificate validation and opens you to man-in-the-middle attacks. Never do this outside of local testing.

Forgetting certificate rotation. Certificates expire. In a service mesh or large deployment, you need a process to rotate certificates before they expire. This is where tools like cert-manager (in Kubernetes) come in, but that's a separate topic.

Not understanding the CN vs. SANs issue. In modern TLS, the CommonName (CN) is deprecated; use Subject Alternative Names (SANs) instead. If you need your certificate to work for both localhost and 127.0.0.1, add both as SANs when generating the certificate. For this article's simple example, CN is fine.

Integration with Token-Based Security

mTLS and token-based authentication are complementary. mTLS secures the transport layer and proves the client's identity at the network level. On top of that, you can layer token-based mechanisms (JWT, OAuth) for application-level authorization. For a detailed treatment of how tokens work in web applications, see token-based security in web applications.

Common Pitfalls with Certificate Management

In a real system, managing certificates securely is critical. Never commit private keys to version control. Treat them like passwords. See managing password safety on Linux for best practices on securing sensitive files. The same principles apply to certificate keys.

What Happens Inside the Handshake

Here's the step-by-step message flow for mTLS:

ClientHello: Client sends supported cipher suites, TLS versions, and random nonce.
ServerHello: Server picks a cipher suite and sends its random nonce.
Certificate: Server sends its certificate chain.
CertificateRequest: Server tells the client it must also send a certificate. This message is absent in standard TLS.
ServerKeyExchange or direct key agreement: Server sends key agreement parameters (or skips this in TLS 1.3).
ServerHelloDone: Server says "your turn."
Certificate: Client sends its certificate.
ClientKeyExchange: Client sends key agreement parameters.
CertificateVerify: Client signs a hash of the handshake transcript with its private key, proving it owns the certificate. This is absent in standard TLS.
Finished: Client sends an encrypted message proving the session key is correct.
Finished: Server responds with its own encrypted Finished message.

The CertificateRequest, client Certificate, and CertificateVerify messages are the three additions that make mTLS mutual instead of one-way.

Testing mTLS with curl

Once your server is running, you can also test with curl:

curl --cert client.crt --key client.key \
  --cacert ca.crt \
  https://127.0.0.1:8443/

This is useful for manual testing of mTLS endpoints. curl requires the --cert flag (client certificate), --key flag (client private key), and --cacert flag (CA certificate to verify the server). Without all three, the handshake will fail.

Performance Considerations

mTLS adds negligible overhead to each request after the initial handshake. The handshake itself takes a few milliseconds. In a service mesh where services are constantly opening new connections (or reusing them via connection pooling), the handshake cost is amortized. The real cost is CPU for certificate validation and TLS encryption/decryption, but modern CPUs with AES-NI instructions handle this efficiently. For internal microservices, mTLS is the right choice; the security benefit far outweighs the performance cost.

FAQ

Why is mTLS needed if I'm already using TLS for HTTPS?

Standard HTTPS proves the server's identity to the client. mTLS also proves the client's identity to the server. In a public website, you don't need to verify the client because millions of strangers visit. In a private network of microservices, you want each service to authenticate itself. mTLS is a control boundary: only services with valid certificates can connect.

Can I use self-signed certificates with mTLS?

Yes, as shown in this article. For production, you typically run your own CA (private CA) or use a tool like cert-manager to manage certificates. Self-signed certificates are fine for development and testing.

How do I know if a connection is using mTLS?

On the server side, if r.TLS.PeerCertificates is not empty and has at least one certificate, the client authenticated with mTLS. On the client side, if you load a certificate into the TLS config and send it, you're using mTLS. You can also use openssl s_client and look for "Client certificate requested" in the output.

What is the difference between ClientAuth: tls.RequestClientCert and tls.RequireAndVerifyClientCert?

RequestClientCert means "ask the client for a certificate, but allow the handshake to succeed if the client doesn't send one." RequireAndVerifyClientCert means "demand a certificate and fail the handshake if one is not provided or is invalid." For security-critical paths (microservices), use RequireAndVerifyClientCert.

Can I verify the client certificate's CN instead of relying on the CA pool?

Not recommended. Always verify using the CA chain (ClientCAs pool). If you manually check the CN, you bypass Go's robust certificate validation logic and can introduce vulnerabilities. Trust the standard library's verification.

How do I handle certificate expiration?

Monitor certificate expiration dates and rotate them before they expire. In Kubernetes, cert-manager automates this. For manual systems, set a calendar reminder 30 days before expiration. When you rotate, generate new certificates, deploy them to your services, and restart the services to pick up the new certificates.

Is mTLS the same as mutual authentication?

mTLS is mutual authentication at the transport layer using certificates. There are other forms of mutual authentication (API keys, OAuth) at the application layer. mTLS specifically refers to mutual certificate-based TLS.