Tribhuvan University

Institute of Science and Technology

Bachelor in Computer Science and Information Technology (BSC CSIT)
7Th Semester
Network Security (CS-416 )
Unit-6: IP Security 6hrs

Instructor
Tekendra Nath Yogi
Tekendranath@gmail.com
 Contents
• Unit 6: IP Security (6 Hrs.)

– 6.1. IP Security Overview

– 6.2. IP Security Policy

– 6.3. Authentication Header

– 6.4. Encapsulating Security Payload

– 6.5. Security Associations

– 6.6. Internet Key Exchange

7/2/2025 By: Tekendra Nath Yogi 2

 6.1. IP Security Overview

• IPsec provides security at the network layer or IP level.

• Unlike application-specific security protocols e.g., SSL, S/MIME, Kerberos.

• Offering protection for all IP traffic regardless of the application.

• It is useful for securing data in transit over LANs, WANs, and the Internet.

• Defined by Internet Architecture Board (IAB) in RFC 1636 (1994).

7/2/2025 By: Tekendra Nath Yogi 3

 Cont’d…
• Why IPsec?

• Protects communication across public and private networks.

• Ensures security for both security-aware and security-ignorant applications.

• Offers organization-wide protection without modifying individual

applications.

7/2/2025 By: Tekendra Nath Yogi 4

 Cont’d…
• Main Functional Areas of IPSec:

• Authentication: Verifies the identity of the sender and Ensures the packet
was not altered in transit.

• Confidentiality: Encrypts data to prevent unauthorized access or

eavesdropping.

• Key Management: Manages secure exchange of cryptographic keys.

7/2/2025 By: Tekendra Nath Yogi 5

 Cont’d…
• Applications of IPSec:

• Secure VPNs: Encrypt connections between remote branches or users over

the Internet.

• Remote Access: Secure access for employees connecting via ISPs.

• Intranets & Extranets: Authenticate and encrypt communication between

organizations.

• E-commerce: Adds a layer of security beyond application-level encryption

like HTTPS.

7/2/2025 By: Tekendra Nath Yogi 6

 Cont’d…

Figure: IP Security Uses

7/2/2025 By: Tekendra Nath Yogi 7
 Cont’d…
• Key Protocols and Documents:
Component Purpose RFC
Concepts, definitions,
Architecture RFC 4301
requirements
Provides message integrity and
AH (Authentication Header) RFC 4302
authentication (deprecated)
ESP (Encapsulating Security Provides encryption and/or
RFC 4303
Payload) authentication
RFC 7296
IKE (Internet Key Exchange) Manages secure key exchange
(IKEv2)
Defines encryption and hashing Multiple
Crypto Algorithms
methods RFCs
Overview of all IPsec-related
Document Roadmap RFC 6071
specs

7/2/2025 By: Tekendra Nath Yogi 8

 Cont’d…
• Security Services Provided by IPsec (RFC 4301):

– Access Control

– Connectionless Integrity

– Data Origin Authentication

– Replay Attack Protection

– Confidentiality

– Traffic Flow Confidentiality (limited)

7/2/2025 By: Tekendra Nath Yogi 9

 6.2. IP Security Policy
• Fundamental Operation of IPSec:

– Every IP packet is subjected to a security policy that decides how it

should be handled?

• Protected,

• Bypassed, or

• Discarded.

– This decision is made using two databases:

• Security Association Database (SAD)

• Security Policy Database (SPD)

7/2/2025 By: Tekendra Nath Yogi 10

 Cont’d…

Figure: IPsec Architecture

7/2/2025 By: Tekendra Nath Yogi 11

 Cont’d…
• Security Associations (SAs):

– A Security Association (SA) is a one-way secure connection between

two IPsec-enabled systems.

– Two SAs are needed for two-way communication.

– SA is uniquely identified by:

• SPI (Security Parameter Index) – 32-bit number in AH/ESP header

• Destination IP Address

• Security Protocol (AH or ESP)

7/2/2025 By: Tekendra Nath Yogi 12

 Cont’d…
• Security Association Database (SAD): Contains parameters for active
SAs:

– Security Parameter Index (SPI)

– Sequence Number Counter: Tracks the packet order

– Anti-Replay Window: Prevents packet replay attacks

– AH/ESP Info: Algorithms, keys, lifetimes

– Lifetime of SA: Time or data limit before expiration

– Mode: Tunnel or transport

– Path MTU: Packet size info for fragmentation handling

• Note: Key management is linked to SPI, but IPsec separates

authentication/encryption from key exchange for flexibility.
7/2/2025 By: Tekendra Nath Yogi 13
 Cont’d…
• Security Policy Database (SPD): Determines what action to apply to each
packet:
– PROTECT: Apply IPsec processing (encryption/authentication)

– BYPASS: Allow packet without IPsec

– DISCARD: Drop the packet

• SPD uses selectors to match traffic:

– Local/Remote IP address

– Next-layer protocol (e.g., TCP, UDP)Local/Remote ports

– User ID (if available)

• Example: Host 1.2.3.10 allowed to communicate securely with server

1.2.4.10 on port 500 (IKE), where port 500 traffic may be set to BYPASS.

7/2/2025 By: Tekendra Nath Yogi 14

 Cont’d…
• IP Traffic Processing: IPsec is executed on a packet-by-packet basis.
When IPsec is implemented:

– Each outbound IP packet is processed by the IPsec logic before

transmission, and

– Each inbound packet is processed by the IPsec logic after reception and
before passing the packet contents on to the next higher layer (e.g., TCP
or UDP).

7/2/2025 By: Tekendra Nath Yogi 15

 Cont’d…
• IP Traffic Processing: Outbound packets

Figure: Processing Model for Outbound Packets

7/2/2025 By: Tekendra Nath Yogi 16
 Cont’d…
• IP Traffic Processing: Inbound packets

Figure: Processing Model for Inbound Packets

7/2/2025 By: Tekendra Nath Yogi 17

 6.3. Authentication Header (AH)
• Provides support for data integrity & authentication of IP packets

– End system such as router can authenticate user or app

– Prevents address spoofing attacks by tracking sequence numbers

• Based on use of a MAC

– HMAC-MD5-96 or HMAC-SHA-1-96

• Parties must share a secret key

• However, It does not provide encryption; payload data remains visible.

7/2/2025 By: Tekendra Nath Yogi 18

 Cont’d…
• Figure below shows the Authentication Header fields:

7/2/2025 By: Tekendra Nath Yogi 19

 Cont’d…
• Next Header (8 bits): Identifies the type of header immediately following
this header

• Payload Length (8 bits): Length of Authentication Header in 32-bit words.

• Reserved (16 bits): For future use

• Security Parameters Index (32 bits): Identifies a security association

• Sequence Number (32 bits): A monotonically increasing counter value

• Authentication Data (variable): A variable-length field (must be an integral

number of 32-bit words) that contains the Integrity Check Value (ICV), or
MAC, for this packet

7/2/2025 By: Tekendra Nath Yogi 20

 Cont’d…
• Transport & Tunnel Modes: Figure below shows the end-to-end (transport)
mode and end-to-intermediate (tunnel) mode.

7/2/2025 By: Tekendra Nath Yogi 21

 Cont’d…
• In transport mode, AH authenticates the entire IP payload and immutable
fields of the IP header, including the AH header itself.

• In IPv4, the packet structure includes: Original IP header, AH header and

Payload.

• In IPv6, the structure includes IPv6 base header, Extension headers, AH

header and Payload.

• Used for host-to-host communication where authentication and integrity are

needed.

• Often combined with ESP for both authentication and encryption.

7/2/2025 By: Tekendra Nath Yogi 22

 Cont’d…
• In tunnel mode, AH authenticates the entire original IP packet (header and
payload) but not the new outer IP header.

• The structure becomes: New outer IP header, AH header and Original IP

packet.

• Suitable for gateway-to-gateway or gateway-to-host communication in

VPNs requiring only authentication.

• Tunnel mode helps protect internal network structure and secures the
authenticity and integrity of the encapsulated packet.

• Limitations include lack of encryption and vulnerability to traffic analysis.

• Compared to transport mode, tunnel mode increases packet size due to

encapsulation but provides stronger protection for internal routing.

7/2/2025 By: Tekendra Nath Yogi 23

 6.4. Encapsulating Security Payload (ESP)
• ESP can be used to Provide:
– Confidentiality,

– Data origin authentication,

– Connectionless integrity,

– Anti-replay service and

– (limited) traffic flow confidentiality.

• Supports range of ciphers, modes, padding

– including DES, Triple-DES, RC5, IDEA etc

– CBC & other modes

– Padding needed to fill block size, fields, for traffic flow

7/2/2025 By: Tekendra Nath Yogi 24

 Cont’d…
• Figure below shows the format of an ESP packet:

Figure: Top-level format of an ESP Packet

7/2/2025 By: Tekendra Nath Yogi 25

 Cont’d…
• Each ESP packet contains the following fields:
• Security Parameters Index (32 bits): Identifies a security association

• Sequence Number (32 bits): A monotonically increasing counter value;

this provides an anti-replay function.

• Payload Data (variable): This is a transport-level segment (transport mode)

or IP packet (tunnel mode) that is protected by encryption. May include IV
(Initialization Vector) for some encryption algorithms.

• Padding (0–255 bytes): Ensures block alignment for encryption

algorithms, Assures 32-bit alignment, Helps hide the actual payload size
(Traffic Flow Confidentiality - TFC padding).

7/2/2025 By: Tekendra Nath Yogi 26

 Cont’d…
• Pad Length (8 bits): Specifies how many bytes of padding are present.

• Next Header (8 bits): Indicates the type of data in the Payload (e.g., TCP,
UDP, IPv6 extension headers).

• Integrity Check Value (ICV- variable multiple of 32:

– Optional.

– Used if integrity protection is selected.

– Computed over the ESP packet excluding the ICV field.

– Comes after encryption to enable DoS resistance and parallel

processing.

7/2/2025 By: Tekendra Nath Yogi 27

 Cont’d…

Figure: Substructure of payload data

7/2/2025 By: Tekendra Nath Yogi 28

 Cont’d…
• Encryption and Authentication Behavior:

• The following fields are encrypted:

– Payload Data

– Padding

– Pad Length

– Next Header

• ICV is added after encryption and used to verify integrity.

• If a combined mode (e.g., AES-GCM) is used, the ICV may be omitted and
integrity is checked within the ciphertext.

7/2/2025 By: Tekendra Nath Yogi 29

 Cont’d…
• Anti-Replay Protection Mechanism: Replay Attack: A valid packet is captured
and resent by an attacker. It could disrupt services.

• Sender Behavior:
– Sequence number starts at 0, first packet uses 1.

– If anti-replay is enabled (default), sequence number must not wrap around (i.e., max = 2³² − 1).

– Once max is reached, the SA must be terminated and re-negotiated.

• Receiver Behavior:
– Maintains a sliding window (default size = 64).

– Tracks which sequence numbers have already been accepted.

– Packet processing:
• Within window and new then Authenticate, Accept and Mark slot.

• Right of window and new then Authenticate, Advance window and Mark slot.

• Left of window or authentication fails then Discard packet and keep Log event record.

7/2/2025 By: Tekendra Nath Yogi 30

 Cont’d…
• Transport and Tunnel Mode: Transport Mode - ESP in transport mode
protects the payload of the IP packet, such as TCP, UDP, or ICMP data.

Figure: Scope of ESP Encryption and Authentication: In Transport Mode

7/2/2025 By: Tekendra Nath Yogi 31

 Cont’d…
• Protocol Operation for ESP: Transport Mode

• Only the transport layer data and ESP trailer are encrypted; the original IP header remains
unencrypted.
• If authentication is enabled, it covers the ESP header and the ciphertext (encrypted
transport data plus ESP trailer).
7/2/2025 By: Tekendra Nath Yogi 32
 Cont’d…
• In IPv4, the ESP header is placed before the transport header; the ESP trailer follows
the transport payload, and optional authentication data comes after the trailer.

• In IPv6, the ESP header appears after the base IPv6 header and any extension
headers except destination options.

• The encrypted payload includes the transport segment, ESP trailer, and possibly
destination options if placed after the ESP header.

• The source encrypts the transport segment and ESP trailer to form the ciphertext,
then optionally adds authentication data.

• The packet is sent with the original IP header visible, allowing intermediate routers
to process it.

• The destination uses the Security Parameters Index (SPI) to decrypt and extract the
original transport layer data.

7/2/2025 By: Tekendra Nath Yogi 33

 Cont’d…
• It is typically used for host-to-host communication, like between two PCs or
a client-server.

Figure: End-to-end IPsec Transport-Mode Encryption

• This mode is useful for providing end-to-end security between hosts that
support ESP, avoiding the need to secure each application individually.
• A limitation is that it exposes the source and destination IP addresses,
making it susceptible to traffic analysis.
7/2/2025 By: Tekendra Nath Yogi 34
 Cont’d…
• Transport and Tunnel Mode: Tunnel Mode - ESP in tunnel mode protects
the entire original IP packet, including its header and payload.

Figure: Scope of ESP Encryption and Authentication: In Tunnel Mode

7/2/2025 By: Tekendra Nath Yogi 35

 Cont’d…
• Protocol Operation for ESP: Tunnel mode

7/2/2025 By: Tekendra Nath Yogi 36

 Cont’d…
• It is commonly used between gateways (e.g., firewall-to-firewall) or between a
gateway and a host.
• The entire original IP packet is encrypted and encapsulated; a new outer IP
header is added.
• If authentication is enabled, it covers the ESP header and the encrypted inner
packet.
• The firewall or router checks outgoing packets and, if required, wraps the full
original IP packet with ESP.A new IP header is created with the source and
destination addresses of the gateways involved.
• The encrypted packet travels over public networks with the outer IP header
visible.
• At the receiving gateway, the outer IP header is removed, and the original IP
packet is restored and forwarded.
7/2/2025 By: Tekendra Nath Yogi 37
 Cont’d…

Figure: Example of Virtual Private Network Implemented with IPsec Tunnel Mode
7/2/2025 By: Tekendra Nath Yogi 38
 Cont’d…
• It is widely used in VPNs to connect remote networks securely over the
Internet.

• Enables internal devices to communicate securely without each one needing

to implement IPsec.

• Offers full confidentiality of the entire IP packet, including the internal IP

addresses.

• Hides internal network structure and simplifies key management by limiting

the number of endpoints.

• Offers better protection against traffic analysis than transport mode.

7/2/2025 By: Tekendra Nath Yogi 39

 Cont’d…
• Summary: Tunnel Mode and Transport Mode Functionality

7/2/2025 By: Tekendra Nath Yogi 40

 6.5. Security Associations (SA)
• An individual SA can implement either the AH or ESP protocol but not
both.

• However, Sometimes a particular traffic flow requires the services provided

by both AH and ESP.

• For example, traffic flow may require IPsec services between hosts and, for
that same flow, separate services between security gateways, such as
firewalls.

• In such scenarios, Multiple SAs must be employed for the same traffic flow
to achieve the desired IPsec services.

• a sequence of SAs through which traffic must be processed to provide a

desired set of IPsec services – Security Association Bundle.

7/2/2025 By: Tekendra Nath Yogi 41

 Cont’d…
• Security associations may be combined into bundles in two ways:
– Transport Adjacency

• Applies AH and ESP on the same IP packet without tunneling.

• All processing is done at the same endpoint (e.g., host-to-host).

• Example: Apply ESP (for encryption) first. Then apply AH (for authentication) in
transport mode.

• Limitation: Only one layer of combination is useful (no nesting).

– Iterated Tunneling

• Uses multiple layers of tunnels.

• Each layer may terminate at different IPsec endpoints (e.g., gateway-to-gateway and
host-to-host).

• Allows multi-level protection (e.g., encrypt at gateway, authenticate at host).

• Example: ESP tunnel from host A to firewall B, AH transport between A and B.

7/2/2025 By: Tekendra Nath Yogi 42
 Cont’d…
• Combining AH and ESP for both authentication and Confidentiality:
Three Approaches:

– ESP with Authentication Option

– Transport Adjacency: AH Outside ESP

– Transport-Tunnel Bundle: AH Inside, ESP Outside

7/2/2025 By: Tekendra Nath Yogi 43

 Cont’d…
• ESP with Authentication Option: Single ESP SA, with both encryption
and optional authentication.

– Transport Mode:

• Encrypts transport layer payload.

• Authenticates the ciphertext and ESP header.

• IP header not authenticated.

– Tunnel Mode:

• Encrypts entire original IP packet.

• Authenticates ciphertext and ESP header.

• Hides and protects full IP payload.

7/2/2025 By: Tekendra Nath Yogi 44

 Cont’d…
• Transport Adjacency: AH Outside ESP

– Uses two SAs:

• Inner SA is ESP that provides encryption only

• Outer SA is AH provides authentication

– Mode:

• Both in transport mode.

• AH covers: IP header, ESP header and encrypted payload.

• Better authentication (includes source/destination IPs).

– Authenticates more of the packet (including IP header).

– Slightly higher overhead (due to two SAs).

7/2/2025 By: Tekendra Nath Yogi 45

 Cont’d…
• Transport-Tunnel Bundle: AH Inside, ESP Outside

– Inner SA is AH in transport mode that authenticates IP header and

payload.

– Outer SA is ESP in tunnel mode that encrypts entire inner authenticated

packet.

– Authentication-before-encryption

– Keeps the authentication data confidential.

– Easier storage of authentication info (e.g., for logging/audit).

7/2/2025 By: Tekendra Nath Yogi 46

 Cont’d…
• Four Basic Security Architecture Cases:

1. Case 1: Host-to-Host Security (Both Hosts Support IPsec)

2. Case 2: Gateway-to-Gateway Security (VPN between

Firewalls/Routers)

3. Case 3: Host-to-Host + Gateway-to-Gateway (Nested Security)

4. Case 4: Remote Host Accessing Internal Network

7/2/2025 By: Tekendra Nath Yogi 47

 Cont’d…
• Case 1: Host-to-Host Security (Both Hosts Support IPsec)
– All SAs terminate at end systems.

– Can use:
• AH in transport mode

• ESP in transport mode

• ESP and AH in transport mode

• Either of the above nested inside tunnel mode SA

7/2/2025 By: Tekendra Nath Yogi 48

 Cont’d…
• Case 2: Gateway-to-Gateway Security (VPN between Firewalls/Routers)
– Hosts do not implement IPsec.

– One ESP or AH tunnel mode SA is sufficient.

– VPN Scenario.

7/2/2025 By: Tekendra Nath Yogi 49

 Cont’d…
• Case 3: Host-to-Host and Gateway-to-Gateway (Nested Security)
– Gateway provides VPN-like tunnel (AH or ESP).

– Hosts add end-to-end SAs (AH or ESP transport mode).

– Good for securing specific applications while still using a VPN.

7/2/2025 By: Tekendra Nath Yogi 50

 Cont’d…
• Case 4: Remote Host Accessing Internal Network
– Remote host → tunnel mode SA to firewall.

– Firewall decrypts, then sends packet internally.

– Between remote and local hosts: additional transport mode SA may exist.

– Common in remote access VPNs.

7/2/2025 By: Tekendra Nath Yogi 51

 6.6. Internet Key Exchange
• Handles key generation & distribution

• Typically need 2 pairs of keys

– 2 per direction for AH & ESP

• Manual key management

– sysadmin manually configures every system

• Automated key management

– automated system for on demand creation of keys for SA’s in large

systems

– has Oakley & ISAKMP elements

7/2/2025 By: Tekendra Nath Yogi 52

 Cont’d…
• Oakley:

– a key exchange protocol

– based on Diffie-Hellman key exchange

– adds features to address weaknesses

• cookies, groups (global params), nonces, DH key exchange with

authentication

– can use arithmetic in prime fields or elliptic curve fields

7/2/2025 By: Tekendra Nath Yogi 53

 Cont’d…
• ISAKMP:

– Internet Security Association and Key Management Protocol

– provides framework for key management

– defines procedures and packet formats to establish, negotiate, modify, &

delete SAs

– independent of key exchange protocol, encryption alg, & authentication

method

7/2/2025 By: Tekendra Nath Yogi 54

 Cont’d…
• ISAKMP:

7/2/2025 By: Tekendra Nath Yogi 55

 Cont’d…
• ISAKMP Payloads & Exchanges:

– have a number of ISAKMP payload types:

• Security, Proposal, Transform, Key, Identification, Certificate,

Certificate, Hash, Signature, Nonce, Notification, Delete

– ISAKMP has framework for 5 types of message exchanges:

• base, identity protection, authentication only, aggressive,

informational

7/2/2025 By: Tekendra Nath Yogi 56

 Thank You !

7/2/2025 By: Tekendra Nath Yogi 57