IPSec Tunneling, Transfer, AH, and ESP Explained – IPSec vs SSL Differences

a tunnel

Quick Answer

IPSec is a set of protocols that create secure connections over the internet. It is commonly associated with the use of VPNs but is used in other use cases as well. IPSec runs directly on top of the Internet Protocol (IP).

The Authentication Header (AH), Encapsulating Security Protocol (ESP), and Security Association (SA) protocols collaborate to create IPsec.

The AH protocol adds a cryptographic authentication header to each packet which ensures that the data remains unchanged and verifies the sender’s identity. While using AH the packet’s contents are not encrypted, but it does confirm that the data has not been tampered with during transmission.

ESP can operate in two primary modes: transport mode, where only the payload is encrypted, and tunnel mode, where the entire original packet (including the IP header) is encapsulated within a new IP packet with its own header.

You would typically choose either AH or ESP based on your specific security needs.

Tunnel mode is often used for securing communications between network gateways or for remote access VPNs. Transport mode encrypts the payload of IP packets while keeping the destination IP address and other header details visible. This means that even though the content is protected, the routing information remains visible.

IPSec and SSL both serve as protocols for securing network communications, each operating at distinct layers of the OSI model. IPSec functions at the network layer, providing encryption directly to IP packets atop the IP protocol.

SSL operates at the transport layer (some argue it can be placed in layers 4-7) focusing on encrypting HTTP traffic, without the inherent capability to secure lower-level communications. IPsec often demands more configuration and is typically employed for enterprise-level security, while SSL finds its primary use in safeguarding web browsing sessions.

IPsec Explained

IPSec is a technology designed to guarantee secure networking by protecting against threats on both private networks and the Internet. Its configuration and control are in the hands of administrators who establish security policies. IPSec functions according to predefined rules and policies established by these administrators. While end users enjoy the advantages of having secure connections when they use a network, they do not have the authority or ability to decide when or how IPSec is activated. IPSec is automatically integrated into the network’s security setup and is always active to ensure security.

transport/tunnel mode

It has two main goals: protecting the data within IP packets and enhancing network security through packet filtering. IPSec achieves these goals using encryption, security protocols, and dynamic key management, making it adaptable for various types of secure communications. It can also be used to selectively block specific types of traffic.

IPSec operates on a security model that establishes trust and security between the source and destination IP addresses. Instead of treating IP addresses as hosts, it verifies the identity of the devices behind those addresses through authentication. Only the sending and receiving devices need to be aware of the secured communication, and they handle security on their ends. This approach can be applied in different scenarios, such as securing local networks, wide-area networks, and remote access.

IPSec provides protection against various types of common network attacks. The level of protection is dictated by the security settings in your IPSec policy. It addresses threats like eavesdropping by encrypting the content of IP packets using the Encapsulating Security Payload (ESP) protocol.

How IPSec works?

  1. IPSec Policy Configuration: Initially, you define security policies that govern how IP network traffic should be handled. These policies consist of packet filters, which specify actions like permitting, blocking, or negotiating security. These filters are integrated into the computer’s TCP/IP protocol stack to oversee incoming and outgoing IP packets.
  2. Internet Key Exchange (IKE) Security Associations (SA): IKE is a protocol for securely establishing trust between computers. It negotiates security parameters and generates cryptographic keys for data protection. This process results in the creation of two types of security associations: one for securing the negotiation process itself and another for protecting application traffic.
  3. IPSec Network Traffic Processing: Let’s see how IPSec works in practice with two computers on a LAN. It begins with computer-1 transmitting an application packet to computer-2. The IPSec driver on computer-1 evaluates outbound filters, initiates security negotiations, and establishes trust via IKE. IPSec security associations are now established, leading to the encryption and protection of the data. Throughout this process, data undergoes encryption, decryption, and validation to guarantee both confidentiality and integrity.
  4. Network Device Considerations: While routers and switches simply forward encrypted IP packets to their destinations, devices like firewalls or security routers may need special configuration to allow IPSec and IKE traffic to pass through. Even if packets are not encrypted, these devices can still inspect packet contents. If any tampering or modifications are detected by the receiving IPSec-enabled computer, they will discard the compromised packets.

IPSec Tunnel and Transport Mode

Transport mode serves as the default mode for securing end-to-end communications. In transport mode, IPsec provides authentication and integrity to the IP payload, which typically consists of TCP segments, UDP messages, or ICMP messages. This mode offers protection through the use of either an Authentication Header (AH) or Encapsulating Security Payload (ESP) header.

AH in transport mode provides authentication and integrity to the IP header and the data payload but does not encrypt them. While the data remains readable, AH uses keyed hashing to sign the authentication header and places it within the IP header as an extension header. AH provides authentication and integrity, but not confidentially as it does not encrypt data. AH can be used alone or in combination with ESP.

ah transport mode

ESP in transport mode provides not only authentication and integrity, but also confidentiality for the IP payload. It encrypts and signs only the IP payload, not the IP header, ensuring data remains secure and confidential during transmission.

esp transport mode

Transport mode is a versatile choice in IPsec, offering different levels of security based on whether AH or ESP is selected, and is commonly used for securing point-to-point communications within a network.

Tunnel Mode encrypts both the IP header and payload of an entire IP packet. In this mode, a new IP header is added, designating the tunnel endpoints, while the original IP header inside contains the source and destination addresses. Tunnel mode is particularly valuable for protecting traffic between different networks, especially when it travels through an untrusted intermediate network.

Within tunnel mode, there are two primary protocols: AH and ESP. AH tunnel mode encapsulates an IP packet with AH and an additional IP header, confirming packet integrity and authentication.

ah tunnel mode

In contrast, ESP tunnel mode encapsulates an IP packet with both ESP and IP headers, along with an ESP authentication trailer. This approach provides not only integrity and authentication but also confidentiality for the protected data. When using tunnel mode, the entire packet beyond the ESP header is treated as data, making it all part of the encrypted payload.

esp tunnel mode

IPSec tunnels are typically configured to secure IP traffic between specific addresses or subnets. Configuration involves setting rules for both outbound and inbound traffic, defining filter lists, tunnel endpoints, authentication methods, and other necessary settings. These configurations are managed under the supervision of network administrators.

What Role Does IPSec Play in a VPN?


A VPN creates a secure tunnel through which data can travel, ensuring that it remains encrypted and confidential.

At its core, a VPN is designed to provide three fundamental functionalities: confidentiality, integrity, and authenticity. VPNs achieve these goals by encrypting data, using various cryptographic protocols, and employing secure authentication mechanisms.

IPSec is a natural choice for securing VPN connections. It operates at the network layer (Layer 3) of the OSI model, making it an ideal choice for securing IP packets. When data is transmitted over the internet, it is vulnerable to malicious actors. IPSec steps in to mitigate these risks.

IPsec tunnel mode plays a vital role in VPNs, making your online activities more secure. Within the ESP tunnel mode, the complete IP packet is both encrypted and encapsulated within an outer packet, thereby enhancing security. This way, your information stays confidential, unchanged, and trustworthy.


IPsec and SSL VPNs are two widely used technologies that offer secure remote access to network resources, but they operate differently and cater to distinct use cases.

network layer

IPsec VPNs operate at the network layer. They establish secure tunnels between entities based on their IP addresses, commonly connecting remote hosts to network VPN servers.

IPsec provides strong encryption and authentication mechanisms, allowing for secure data transmission between any systems identifiable by IP addresses. While it offers robust security, it can be complex to set up and configure, often requiring specific client software installations. IPsec VPNs are well-suited for scenarios where broader network access is required, such as connecting remote offices or enabling remote workers to access an entire network.

transport layer

SSL VPNs, on the other hand, work at the transport layer, encrypting data exchanged between processes identified by port numbers.

Typically, SSL VPNs don’t need additional software installations as they often operate through web browsers. SSL VPNs provide precise control over network connections, enabling organizations to restrict access to particular services within their network. This capability makes SSL VPNs well-suited for situations where meticulous access management is needed, such as specific applications or services like email and web applications.

Each VPN type has its strengths and weaknesses, making it essential to align the choice with the organization’s specific security needs and deployment considerations.