how IPsec works

How IPSec Works

IPSec involves many component technologies and encryption methods. Yet IPSec's operation can be broken down into five main steps:

1. "Interesting traffic" initiates the IPSec process. Traffic is deemed interesting when the IPSec security policy configured in the IPSec peers starts the IKE process.

2. IKE phase 1. IKE authenticates IPSec peers and negotiates IKE SAs during this phase, setting up a secure channel for negotiating IPSec SAs in phase 2.

3. IKE phase 2. IKE negotiates IPSec SA parameters and sets up matching IPSec SAs in the peers.

4. Data transfer. Data is transferred between IPSec peers based on the IPSec parameters and keys stored in the SA database.

5. IPSec tunnel termination. IPSec SAs terminate through deletion or by timing out.

This five-step process is shown in Figure 3.

Figure 3 The five steps of IPSec.

Step 1—Defining Interesting Traffic

What type of traffic is deemed interesting is determined as part of formulating a security policy for use of a VPN. The policy is then implemented in the configuration interface for each particular IPSec peer. For example, in Cisco routers and PIX Firewalls, access lists are used to determine the traffic to encrypt. The access lists are assigned to a cryptography policy; the policy's permit statements indicate that the selected traffic must be encrypted, and deny statements indicate that the selected traffic must be sent unencrypted. With the Cisco Secure VPN Client, you use menu windows to select connections to be secured by IPSec. When interesting traffic is generated or transits the IPSec client, the client initiates the next step in the process, negotiating an IKE phase 1 exchange.

Step 1 is shown in Figure 4.

Figure 4 Defining "interesting traffic."

Step 2—IKE Phase 1

The basic purpose of IKE phase 1 is to authenticate the IPSec peers and to set up a secure channel between the peers to enable IKE exchanges. IKE phase 1 performs the following functions:

• Authenticates and protects the identities of the IPSec peers

• Negotiates a matching IKE SA policy between peers to protect the IKE exchange

• Performs an authenticated Diffie-Hellman exchange with the end result of having matching shared secret keys

• Sets up a secure tunnel to negotiate IKE phase 2 parameters

IKE phase 1 occurs in two modes: main mode and aggressive mode. These modes are described in the following sections.

Main Mode

Main mode has three two-way exchanges between the initiator and the receiver.

• First exchange: The algorithms and hashes used to secure the IKE communications are agreed upon in matching IKE SAs in each peer.

• Second exchange: Uses a Diffie-Hellman exchange to generate shared secret keying material used to generate shared secret keys and to pass nonces—random numbers sent to the other party and then signed and returned to prove their identity.

• Third exchange: Verifies the other side's identity. The identity value is the IPSec peer's IP address in encrypted form. The main outcome of main mode is matching IKE SAs between peers to provide a protected pipe for subsequent protected ISAKMP exchanges between the IKE peers. The IKE SA specifies values for the IKE exchange: the authentication method used, the encryption and hash algorithms, the Diffie-Hellman group used, the lifetime of the IKE SA in seconds or kilobytes, and the shared secret key values for the encryption algorithms. The IKE SA in each peer is bi-directional.

Aggressive Mode

In aggressive mode, fewer exchanges are made, and with fewer packets. On the first exchange, almost everything is squeezed into the proposed IKE SA values: the Diffie-Hellman public key; a nonce that the other party signs; and an identity packet, which can be used to verify identity via a third party. The receiver sends everything back that is needed to complete the exchange. The only thing left is for the initiator to confirm the exchange. The weakness of using the aggressive mode is that both sides have exchanged information before there's a secure channel. Therefore, it's possible to "sniff" the wire and discover who formed the new SA. However, it is faster than main mode.

Step 2 is shown in Figure 5.

Figure 5 IKE phase 1.

Step 3—IKE Phase 2

The purpose of IKE phase 2 is to negotiate IPSec SAs to set up the IPSec tunnel. IKE phase 2 performs the following functions:

• Negotiates IPSec SA parameters protected by an existing IKE SA

• Establishes IPSec security associations

• Periodically renegotiates IPSec SAs to ensure security

• Optionally performs an additional Diffie-Hellman exchange

IKE phase 2 has one mode, called quick mode. Quick mode occurs after IKE has established the secure tunnel in phase 1. It negotiates a shared IPSec policy, derives shared secret keying material used for the IPSec security algorithms, and establishes IPSec SAs. Quick mode exchanges nonces that provide replay protection. The nonces are used to generate new shared secret key material and prevent replay attacks from generating bogus SAs.

Quick mode is also used to renegotiate a new IPSec SA when the IPSec SA lifetime expires. Base quick mode is used to refresh the keying material used to create the shared secret key based on the keying material derived from the Diffie-Hellman exchange in phase 1.

Perfect Forward Secrecy

If perfect forward secrecy (PFS) is specified in the IPSec policy, a new Diffie-Hellman exchange is performed with each quick mode, providing keying material that has greater entropy (key material life) and thereby greater resistance to cryptographic attacks. Each Diffie-Hellman exchange requires large exponentiations, thereby increasing CPU use and exacting a performance cost.

Step 4—IPSec Encrypted Tunnel

After IKE phase 2 is complete and quick mode has established IPSec SAs, information is exchanged via an IPSec tunnel. Packets are encrypted and decrypted using the encryption specified in the IPSec SA. This IPSec encrypted tunnel can be seen in Figure 6.

Figure 6 IPSec encrypted tunnel.

Step 5—Tunnel Termination

IPSec SAs terminate through deletion or by timing out (see Figure 7). An SA can time out when a specified number of seconds have elapsed or when a specified number of bytes have passed through the tunnel. When the SAs terminate, the keys are also discarded. When subsequent IPSec SAs are needed for a flow, IKE performs a new phase 2 and, if necessary, a new phase 1 negotiation. A successful negotiation results in new SAs and new keys. New SAs can be established before the existing SAs expire, so that a given flow can continue uninterrupted.

Figure 7 Tunnel termination.

This brings us to the end of the fourth part of this five-part series of articles covering IPSec. Be sure to catch the next installment.

Return to BeginningPrevious Section