What is an IPSec VPN? IPSec protocol explained

An IPSec VPN is a Virtual Private Network service using IPSec as its security protocol. Its addition to online communications helps secure data sent over networks. 

Two of its biggest responsibilities are encryption and authentication. Encryption means it converts IP packets into a secret code. Authentication verifies the source of packets. 

So, IPSec is a preferred set of networking protocols to establish more secure connections to the internet. Let’s discover how it works and what security benefits it brings to VPNs. 

What IPSec is

Within the term IPSec, IP is Internet Protocol, while Sec stands for Secure or Security. Thus, IPSec is Internet Protocol Security. The name suggests the protocol aims to protect the IP routing protocol. 

So, the IP determines the destination for data packets, and the Sec protects and authenticates the process. 

IPSec protocol suite 

IPSec is a set of protocols, so we discuss several protocols for securing communications: 

• The Authentication Header (AH) checks the data source and payload for modifications during the transfer from the source to the destination. So, it guarantees data integrity and that it remains unchanged until it reaches the final stop. 
• Encapsulating Security Payload (ESP) performs data encryption of packets traveling from one device to another. 
• Security Association (SA) determines the security properties communicating hosts agree upon.

Why IPSec is so necessary for secure communications 

IPSec guarantees that data is much more secure based on these actions: 

• It encrypts data packets, preventing snooping from unknown entities. 
• IPSec verifies that data comes from trusted sources. 
• It guarantees that data reaches its destination without interference from unwanted actors. 

Also, TCP/IP mainly focuses on data arriving at its destinations successfully. IPSec encrypts and authenticates each packet, ensuring more security. So, it plays an essential role in safeguarding our communications. 

How IPSec affects MSS and MTU

First, let’s look at the meaning of MSS and MTU. 

MSS stands for maximum segment size and examines the size of data packets’ payloads.

MTU (maximum transition unit) looks at the size of the whole packet, including the payload and header. 

IPSec protocols add more weight to data packets. However, the addition should only be a few bytes. Nonetheless, connections using IPSec need to perform adjustments for MSS and MTU.

For instance, proper MSS and MTU values settings should prevent fragmentation and packet retransmission. 

How do VPNs use IPSec? 

An IPSec VPN uses the IPSec protocol suite to build the privacy of digital communications between devices and software. Then, a VPN can authenticate data packets and ensure untainted information reaches users. 

Also, it can encrypt data, preventing bad actors from intercepting and reading information. So, it minimizes the risks of man-in-the-middle attacks and attempts to modify online requests. 

In simpler terms, IPSec empowers VPN connections to offer high protection for your internet traffic. 

How IPSec works 

IPSec secures data through a series of particular stages: 

• IPSec works at the network layer of the OSI model and directly on top of IP. 
• Internet Key Exchange (IKE) is the protocol initiating the Security Association process. As a result, connecting devices share keys for encrypting and decrypting messages. 
• IPSec uses Diffie-Hellman: an asymmetric key algorithm.
• The process divides data into smaller parts called packets. Each of them has the payload (actual data transferred) and the header. The latter contains instructions on how to handle the packet. 
• The authentication steps verify the source of data and its integrity. 
• IPSec does encrypt data (or the payload) and the header. However, there are exceptions, like with the transport mode. 
• Encrypted IPSec packets use a UDP transport protocol and travel to their destinations. It differs from regular traffic, and UDP has the benefit of getting through firewalls successfully. 
• After packets arrive at their destination, their information gets decrypted. 

Advantages and disadvantages of IPSec 

While IPSec or IPSec VPN is a secure choice, you might wish to see its main strengths and potential pitfalls: 

Advantages 

1. A highly secure protocol that improves internet connections with its authentication and encryption. 
2. IPSec supports multiple encryption algorithms like AES, ChaCha, Triple DES, and Blowfish. 
3. It is also flexible, assisting various VPN connection types, including site-to-site, client-to-site, and client-to-client. 

Disadvantages 

1. IPSec can bring higher CPU usage as it deals with data encryption and decryption. 
2. Internet speeds might slightly decrease because of complex encryption and routing procedures. In most cases, the slower speed might not be noticeable to users. 
3. The complexity of IPSec might make it more challenging to set up and troubleshoot. 

Which VPN protocol employs IPSec as its data encryption mechanism?

L2TP, or Layer 2 Tunneling Protocol, is the most common VPN protocol to use with IPSec. The addition of IPSec is crucial as the LT2P does not encrypt the data it traverses. Thus, it requires assistance from IPSec to support secure communications. 

However, IPSec VPN can use other tunneling protocols as well. For instance, IKEv2 is also a popular choice. IKEv2 is a well-known protocol for stable and faster connections. IPSec is also its security fuel, adding more protection for internet traffic.

What are the IPSec modes? 

IPSec can work in several modes: 

• Tunnel mode is more suitable for protecting and securing data on public networks. It relates to the process of encrypting the data itself and its header. 
• Transport mode only encrypts the data itself but leaves headers unchanged. Thus, it is more suitable for communication via trusted networks. 

IPSec VPN and SSL VPN comparison 

Note: security experts have spoken against using SSL because of its vulnerabilities. Thus, SSL VPN usually means TLS VPN since TLS replaced the deprecated SSL. 

IPSec VPN and SSL VPN work differently, and we see that from their characteristics: 

• IPSec operates on the network layer, while SSL works on the application layer. Thus, IPSec runs directly on IP. SSL instead encrypts HTTP traffic. 
• SSL VPN offers better conditions for custom access control. It is possible to configure access privileges on an app-by-app basis. With IPSec VPNs, setting up multiple VPNs for different access rights is necessary. 
• SSL VPN usually works with browser-based applications. However, IPSec VPN can protect any IP-based app. 
• IPSec requires users to install VPN software to create a tunnel between the client and server. SSL VPNs can work on web browsers, which could mean faster connections. 

Overall, the security of a VPN requires the analysis of all its components. Thus, it is crucial to pay attention to the whole picture. 

However, SSL VPNs are more suitable for remote access control. IPSec VPNs are more suitable as personal solutions, like protecting your privacy and hiding IP addresses. 

Atlas VPN uses IKEv2/IPSec

Atlas VPN uses IKEv2/IPSec protocols to securely tunnel data between clients and servers. It is one of the best protocol combinations nowadays. However, you can also pick WireGuard®, which is also a highly efficient protocol. 

*WireGuard® is a registered trademark of Jason A. Donenfeld. 

How do you select IKEv2/IPSec for Atlas VPN connections? 

You can pick IKEv2/IPSec after opening settings in Atlas VPN applications. Let the app pick it for you if you are unsure which to choose. 

However, you might wish to choose IKEv2/IPSec for its stability, resistance to network changes, and robust security. WireGuard offers similar benefits, but one of its most significant advantages is potentially faster internet speeds.