DEV Community

ZeroTrust Architect
ZeroTrust Architect

Posted on • Originally published at cacheguard.com

IKEv2, ESP, and StrongSwan: Dissecting an IPsec VPN from Handshake to Packet

#security #networking #linux #devops

IPsec is a suite of protocols, not a single protocol. Understanding a production deployment means understanding how its components compose: IKEv2 for key exchange, ESP for data encryption, and a PKI for peer authentication.

What is IPsec?

IKEv2 in detail

IKEv2 (RFC 7296) negotiates Security Associations (SAs) — agreements between peers on cryptographic parameters and keys. It runs over UDP/500 (or UDP/4500 when NAT is detected).

IKE_SA_INIT exchange 

Initiator                              Responder
  |                                        |
  |-- HDR, SAi1, KEi, Ni --------------->|
  |     (proposed algorithms, DH pubkey,   |
  |      nonce)                            |
  |<-- HDR, SAr1, KEr, Nr, [CERTREQ] ----|
  |     (chosen algorithm, DH pubkey,      |
  |      nonce, optional cert request)     |
Enter fullscreen mode Exit fullscreen mode

Both sides compute the shared DH secret and derive the IKE SA keys. All subsequent IKEv2 messages are encrypted with these keys.

IKE_AUTH exchange 

Initiator                              Responder
  |                                        |
  |-- HDR, SK{IDi, [CERT], AUTH,         |
  |          SAi2, TSi, TSr} ----------->|
  |     (identity, certificate, auth,      |
  |      child SA proposal, traffic        |
  |      selectors)                        |
  |<-- HDR, SK{IDr, [CERT], AUTH,        |
  |           SAr2, TSi, TSr} -----------|
Enter fullscreen mode Exit fullscreen mode

AUTH contains a signature over the IKE_SA_INIT messages using the peer's private key. Both sides verify the other's certificate chain against a trusted CA before accepting the SA.

Certificate-based authentication: the PKI chain

In a production deployment, each VPN endpoint has:

  • A root CA certificate (shared across all participants, used for verification)
  • A server certificate signed by the root CA (presented by the VPN server)
  • Client certificates signed by the root CA (one per device or user)

When client A connects to the server:

  1. Server presents its certificate → client verifies signature chain to root CA
  2. Client presents its certificate → server verifies signature chain to root CA
  3. Both compute AUTH payload from their private keys
  4. Both verify the other's AUTH payload using the other's public key

Compromise of one client certificate requires only revoking that certificate (via CRL or OCSP). The root CA and other client certificates are unaffected.

ESP in tunnel mode: the encapsulation format

Once the IKE_AUTH exchange completes, a Child SA (ESP SA) is established. All data flows as ESP packets.

In tunnel mode, the entire original IP packet is encrypted and wrapped:

[New IP header][ESP header][Encrypted: Original IP header + payload][ESP trailer][ESP auth]
Enter fullscreen mode Exit fullscreen mode

The original source and destination IPs are inside the encrypted portion — invisible to observers on the transit network. The new IP header shows only the VPN endpoints.

ESP header structure 

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|               Security Parameters Index (SPI)                 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                      Sequence Number                          |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                    Payload Data (encrypted)                   |
Enter fullscreen mode Exit fullscreen mode

SPI identifies which SA to use for decryption. Sequence numbers prevent replay attacks.

StrongSwan configuration (server side) 

# /etc/ipsec.conf
conn remote-access
    keyexchange=ikev2
    left=%any                    # Server's IP (any interface)
    leftcert=server-cert.pem     # Server certificate
    leftca=root-ca.pem           # CA for client verification
    leftid=@vpn.example.com
    right=%any                   # Accept any client IP
    rightsourceip=10.10.0.0/24   # Virtual IP pool for clients
    rightdns=10.10.0.1
    ike=aes256-sha256-ecp256!
    esp=aes256-sha256!
    auto=add
Enter fullscreen mode Exit fullscreen mode

CacheGuard wraps StrongSwan configuration behind a web interface, manages certificate generation via an integrated mini-PKI, and produces ready-to-import client profiles for iOS, macOS, Android, Windows, and Linux — eliminating manual StrongSwan and PKI configuration.

→ https://www.cacheguard.com/what-is-ipsec/

Originally published on the CacheGuard Blog. CacheGuard is free and open source — GitHub.

Top comments (0)