DEV Community

Cover image for What is IPsec? | How IPsec VPNs Work Explained!!
Pavankumar Hegde
Pavankumar Hegde

Posted on • Edited on

What is IPsec? | How IPsec VPNs Work Explained!!

Lesson Contents

Table of Contents

IKE (Internet Key Exchange)

IPsec (Internet Protocol Security) is a framework that helps us to protect IP traffic on the network layer. Why? because the IP protocol itself doesn’t have any security features at all. IPsec can protect our traffic with the following features:

Image description


1. Confidentiality: by encrypting our data, nobody except the sender and receiver will be able to read our data.

2. Integrity: we want to make sure that nobody changes the data in our packets. By calculating a hash value, the sender and receiver will be able to check if changes have been made to the packet.

3. Authentication: the sender and receiver will authenticate each other to make sure that we are really talking with the device we intend to.

As a framework, IPsec uses a variety of protocols to implement the features I described above. Here’s an overview:

Image description


IPsec can be used on many different devices, it’s used on routers, firewalls, hosts, and servers. Here are some examples of how you can use it:

  • Between two routers to create a site-to-site VPN that “bridges” two LANs together.
  • Between a firewall and Windows host for remote access VPN.
  • Between two Linux servers to protect an insecure protocol like telnet.

IPsec is pretty complex and there are a lot of different ways to implement it. In this lesson, I will start with an overview and then we will take a closer look at each of the components.

Before we can protect any IP packets, we need two IPsec peers that build the IPsec tunnel.

Introduction to ISAKMP protocol

The protocol used in IPSec for key management is called ISAKMP/Oakley. ISAKMP stands for the Internet Security Association and Key Management Protocol. It is a protocol platform used for key management. It defines the procedure and packet formats for negotiating, establishing, modifying, and deleting SAs. ISAKMP messages can be transmitted via the TCP or UDP transport protocol. Port number 500 of TCP and UDP are reserved for ISAKMP protocol. The initial version of ISAKMP mandated the use of the Oakley protocol. Oakley is based on the Diffie Hellman key exchange protocol.

So first, we will discuss the Oakley and then will discuss the ISAKMP.

Oakley protocol

Oakley protocol is a refined version of the Diffie Hellman key exchange protocol. Diffie Hellman offers two main features the creation of secret keys as and when required and no requirement for any pre-existing infrastructure. However, the Diffie Hellman key exchange protocol also has some problems. Oakley protocol is designed to solve these problems.

Features of Oakley are as follows:

  • It enables the exchange of Diffie Hellman public key values
  • To defeat the congestion attacks, it implements a mechanism which is called cookies.
  • To defeat the man-in-the-middle attack, it provides an authentication mechanism.

Below are the approaches that are taken by the Oakley protocol to solve the problems of the Diffie Hellman key protocol.

1. Authentication: Oakley protocol supports three authentication mechanisms which follow

  • Digital signatures: Generation of a message digest and its encryption with the private key of the sender.

  • Public key encryption: Encryption of information such as the user ID of the sender and the public key of the receiver.

  • Secret key encryption: Key derived by using some out-of-band mechanism.

2. dealing with congestion attacks: to defeat the congestion attack, Oakley uses the cookie concept. In this type of attack, an attacker forges the source address of another legitimate user and sends a public Diffie Hellman key to another legitimate user. Then the receiver performs some calculations to calculate the secret key. Several calculations. Performed rapidly one after another causing the congestion of the victim’s computer. To defeat this, each side in Oakley must send the pseudo-random number, which is called a cookie, in the initial message, which the other side must acknowledge. This acknowledgment must be repeated in the first message of the Diffie key exchange protocol. If an attacker forges the source address, he does get the acknowledgment cookie from the victim, and her attacks fail.

3. ISAKMP: This protocol defines the procedure and the format for establishing, maintaining, and deleting the SA information. ISAKMP message contains an ISAKMP header followed by one or more payloads. The entire block is encapsulated inside the transport segment.

> ISAKMP Header

Image description


To establish an IPsec tunnel, we use a protocol called IKE (Internet Key Exchange).

There are two phases to build an IPsec tunnel:

  1. IKE Phase 1
  2. IKE Phase 2

IKE Phase 1

In IKE phase 1, two peers will negotiate about the encryption, authentication, hashing, and other protocols that they want to use and some other parameters that are required. In this phase, an ISAKMP (Internet Security Association and Key Management Protocol) session is established. This is also called the ISAKMP tunnel or IKE phase 1 tunnel.

The collection of parameters that the two devices will use is called a SA (Security Association).

The main purpose of IKE Phase 1 is to establish a secure tunnel that we can use for IKE Phase 2.

We can break down Phase 1 in three simple steps:

Step 1: Negotiation

The peer that has traffic that should be protected will initiate the IKE phase 1 negotiation. The two peers will negotiate about the following items:

Hashing: we use a hashing algorithm to verify the integrity, we use MD5 or SHA for this.

Authentication: each peer has to prove who he is. Two commonly used options are a pre-shared key or digital certificates.

DH (Diffie Hellman) group: the DH group determines the strength of the key that is used in the key exchange process. The higher group numbers are more secure but take longer to compute.

Lifetime: how long does the IKE phase 1 tunnel stand up? the shorter the lifetime, the more secure it is because rebuilding it means we will also use new keying material. Each vendor uses a different lifetime, a common default value is 86400 seconds (1 day).

Encryption: what algorithm do we use for encryption? For example, DES, 3DES, or AES.

Step 2: DH Key Exchange

Once the negotiation has succeeded, the two peers will know what policy to use. They will now use the DH group that they negotiated to exchange keying material. The end result will be that both peers will have a shared key.

Step 3: Authentication

The last step is that the two peers will authenticate each other using the authentication method that they agreed upon on in the negotiation. When the authentication is successful, we have completed IKE phase 1. The end result is a IKE phase 1 tunnel (aka ISAKMP tunnel) which is bidirectional. This means that both peers can send and receive on this tunnel.


The three steps above can be completed using two different modes:

  • Main mode
  • Aggressive mode

Main mode uses six messages while aggressive mode only uses three messages. Main mode is considered more secure. Let’s take a look at closer look at both modes.

Image description

Main Mode

IKEv1 main mode uses 6 messages. I will show you these in Wireshark and I’ll explain the different fields.

- Message 1

Image description


The initiator (peer that wants to build the tunnel) will send the first message. This is a proposal for the security association. Above you can see that the initiator uses IP address 192.168.12.1 and is sending a proposal to the responder (peer we want to connect to) 192.168.12.2. IKE uses UDP port 500 for this. In the output above you can see an initiator SPI (Security Parameter Index), this is a unique value that identifies this security association.

We can see the IKE version (1.0) and that we are using main mode. The domain of interpretation is IPsec and this is the first proposal. In the transform payload, you can find the attributes that we want to use for this security association.

- Message 2

Image description


When the responder receives the first message from the initiator, it will reply. This message is used to inform the initiator that we agree upon the attributes in the transformed payload. You can also see that the responder has set its own SPI value.

- Message 3

Image description


Since our peers agree on the security association to use, the initiator will start the Diffie Hellman key exchange. In the output above you can see the payload for the key exchange and the nonce.

- Message 4

Image description

The responder will also send his/her Diffie Hellman nonces to the initiator, our two peers can now calculate the Diffie Hellman shared key.

- Message 5

Image description


The last two messages are encrypted so we can’t see its contents anymore. These two are used for identification and authentication of each peer. The initiator starts.

- Message 6

Image description


And above we have the 6th message from the responder with its identification and authentication information. IKEv1 main mode has now been completed and we can continue with IKE phase 2.

Before we continue with phase 2, let me show you aggressive mode first.

Aggressive mode

IKEv1 aggressive mode only requires three messages to establish the security association. It’s quicker than main mode since it adds all the information required for the DH exchange in the first two messages. Main mode is considered more secure since identification is encrypted, aggressive mode does this in clear text.

Let’s take a look at the different messages.

- Message 1

Image description


The first message is from the initiator (192.168.12.1) to the responder (192.168.12.2). You can see the transformed payload with the security association attributes, DH nonces, and the identification (in clear text) in this single message.

- Message 2

Image description


The responder now has everything it needs to generate the DH shared key and sends some nonces to the initiator so that it can also calculate the DH shared key. It also calculates a hash that is used for authentication.

- Message 3

Image description


Both peers have everything they need, the last message from the initiator is a hash that is used for authentication.

Note: PFS is optional and forces the peers to run the DH exchange again to generate a new shared key in each IKE phase 2 quick mode.

IKE Phase 2

Quick Mode

The IKE phase 2 tunnel (IPsec tunnel) will be actually used to protect user data. There is only one mode to build the IKE phase 2 tunnel which is called quick mode.

Just like in IKE phase 1, our peers will negotiate about a number of items:

  • IPsec Protocol: do we use AH or ESP?
  • Encapsulation Mode: transport or tunnel mode?
  • Encryption: what encryption algorithm do we use? DES, 3DES or AES?
  • Authentication: what authentication algorithm do we use? MD5 or SHA?
  • Lifetime: how long is the IKE phase 2 tunnel valid? When the tunnel is about to expire, we will refresh the keying material.
  • (Optional) DH exchange: used for PFS (Perfect Forward Secrecy).

This negotiation happens within the protection of our IKE phase 1 tunnel so we can’t see anything. Just for the sake of completeness, here’s what it looks like in Wireshark:


- Message 1

Image description


- Message 2

Image description


- Message 3

Image description


Once IKE phase 2 has been completed, we are finally ready to protect some user data.

Conclusion

As we conclude our exploration of IKE Phase 1 & 2 messages, we've laid the foundation for a secure and authenticated connection. The intricacies of Main Mode and the efficiency of Aggressive Mode have paved the way for the next chapter in our journey. Join us in the upcoming blog where we delve into the fascinating realm of IPsec Protocols, unlocking the layers of authentication, encryption, and secure communication. Stay tuned for a comprehensive guide that demystifies the protocols ensuring the robustness of our networks. Secure your seat as we unravel the intricacies of IPsec in our next installment!


Connect with Me

Feel free to reach out for collaborations, questions, or just to say hello!


References

Thanks for reading this far. That's my basic understanding of IPSec and VPN.

Thanks again!! 😊


Recent Blog Posts

  1. Unlocking the Secrets: A Guide to TCP 3-Way Handshake

  2. Decoding the Layers: A Journey Through the OSI Model!

Top comments (0)