Both IPsec and TLS VPNs encrypt traffic between endpoints. The difference is where in the protocol stack they operate and what that implies for performance, firewall traversal, and failure modes.
IPsec: network-layer encryption
IPsec operates at OSI Layer 3. It encrypts IP packets entirely, including headers in tunnel mode, and delivers them inside new IP packets. The protocol suite has three components:
- IKE (Internet Key Exchange): Handles authentication and SA (Security Association) negotiation. Uses UDP/500 for initial contact, switches to UDP/4500 if NAT is detected (NAT-T).
- ESP (Encapsulating Security Payload): Provides encryption and authentication of the payload. IP protocol number 50.
- AH (Authentication Header): Authentication only, no encryption. Protocol number 51. Rarely used in modern deployments.
IKE phase negotiation
IKEv2 (RFC 7296) reduces the handshake to two exchanges:
IKE_SA_INIT: Peers negotiate cryptographic algorithms, exchange nonces, and perform a Diffie-Hellman key exchange. Result: an IKE SA protecting subsequent messages.
IKE_AUTH: Peers authenticate each other (via certificates or PSK) and establish the first Child SA (the actual IPsec tunnel). Result: an ESP SA ready to carry traffic.
Initiator Responder
|-- IKE_SA_INIT request -->|
|<- IKE_SA_INIT response --|
|-- IKE_AUTH request ------>|
|<- IKE_AUTH response ------|
|=== ESP tunnel active ====|
NAT traversal
IPsec was designed before NAT was widespread. When a NAT device sits between peers, ESP packets can be mangled (NAT rewrites IP headers but ESP authenticates them, causing verification failure). NAT-T solves this by encapsulating ESP inside UDP/4500, making the payload opaque to NAT devices.
Detection happens during IKE_SA_INIT: if either peer detects a NAT (via NAT-D payload hash comparison), both switch to UDP/4500 for all subsequent communication.
TLS VPN: application-layer tunnelling
TLS VPNs (OpenVPN, SSL VPN) run over TCP or UDP at the application layer. They establish a TLS session first, then tunnel IP traffic through it.
Key difference: TLS uses TCP/443 or UDP/443 by default — the same ports as HTTPS. Firewalls and restrictive networks almost never block port 443, making TLS VPNs significantly more traversal-friendly than IPsec.
The trade-off is performance. TLS over TCP introduces TCP-over-TCP — when the tunnelled TCP connection retransmits, the outer TCP also retransmits, causing retransmit storms under packet loss. TLS over UDP avoids this but is less universally available.
Port and firewall traversal comparison
| Protocol | Ports used | Firewall blocked? | NAT behaviour |
|---|---|---|---|
| IKEv2/IPsec | UDP/500, UDP/4500, ESP (IP 50) | Sometimes | Requires NAT-T |
| IKEv1/IPsec | UDP/500, UDP/4500 or AH/ESP | Sometimes | Requires NAT-T |
| OpenVPN (UDP) | UDP/1194 (configurable) | Sometimes | Generally fine |
| TLS VPN | TCP/443 or UDP/443 | Rarely | Always fine |
When IPsec fails and TLS doesn't
Corporate guest networks, hotel WiFi, and some mobile carriers actively block UDP/500 and UDP/4500, preventing IKEv2 negotiation from completing. ESP (protocol 50) may also be filtered. In these environments, IPsec connections time out silently while TLS-based connections on port 443 succeed.
When to use which
IPsec is the right choice for:
- Site-to-site tunnels between fixed endpoints (no port blocking concerns)
- Remote access on managed device fleets where client software can be deployed
- High-throughput requirements (lower per-packet overhead than TLS)
- Environments where all platforms need to use the native OS VPN client (iOS, macOS, Windows, Android all have native IKEv2 clients)
TLS VPN is the right choice for:
- Environments with aggressive outbound filtering (hotels, corporate guest networks, restrictive countries)
- Clientless access where users connect via a browser
- Scenarios where per-user application-level access control is needed (TLS VPNs make per-app policy easier)
CacheGuard implements IPsec VPN using IKEv2 with ESP in tunnel mode, powered by StrongSwan. It supports certificate-based and PSK authentication, generates client profiles for all major platforms, and includes DynDNS integration for deployments without a fixed public IP.
→ https://www.cacheguard.com/ipsec-vs-ssl-vpn/
Originally published on the CacheGuard Blog. CacheGuard is free and open source — GitHub.
Top comments (0)