Computer Security: Principles and

Practice
Fifth Edition

Chapter 10
Buffer Overflow

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Table 10.1
A Brief History of Some Buffer Overflow Attacks

1988 The Morris Internet Worm used a buffer overflow exploit in “fingerd” as one of its attack
mechanisms.

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Table 10.1
A Brief History of Some Buffer Overflow Attacks

1988 The Morris Internet Worm used a buffer overflow exploit in “fingerd” as one of its attack
mechanisms.
1995 A buffer overflow in N C S A httpd 1.3 is discovered and published on the Bugtraq
Mailing list by Thomas Lopatic.

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Table 10.1
A Brief History of Some Buffer Overflow Attacks

1988 The Morris Internet Worm used a buffer overflow exploit in “fingerd” as one of its attack
mechanisms.
1995 A buffer overflow in N C S A httpd 1.3 is discovered and published on the Bugtraq
Mailing list by Thomas Lopatic.
1996 Aleph One publishes “Smashing the Stack for Fun and Profit” in Phrack magazine, giving
a step-by-step introduction to exploiting stack-based buffer overflow vulnerabilities.

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Table 10.1
A Brief History of Some Buffer Overflow Attacks

1988 The Morris Internet Worm used a buffer overflow exploit in “fingerd” as one of its attack
mechanisms.
1995 A buffer overflow in N C S A httpd 1.3 is discovered and published on the Bugtraq
Mailing list by Thomas Lopatic.
1996 Aleph One publishes “Smashing the Stack for Fun and Profit” in Phrack magazine, giving
a step-by-step introduction to exploiting stack-based buffer overflow vulnerabilities.
2001 The Code Red worm exploits a buffer overflow in Microsoft I I S

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Table 10.1
A Brief History of Some Buffer Overflow Attacks

1988 The Morris Internet Worm used a buffer overflow exploit in “fingerd” as one of its attack
mechanisms.
1995 A buffer overflow in N C S A httpd 1.3 is discovered and published on the Bugtraq
Mailing list by Thomas Lopatic.
1996 Aleph One publishes “Smashing the Stack for Fun and Profit” in Phrack magazine, giving
a step-by-step introduction to exploiting stack-based buffer overflow vulnerabilities.
2001 The Code Red worm exploits a buffer overflow in Microsoft I I S

2003 The Slammer worm exploits a buffer overflow in Microsoft S Q L Server 2000.

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Table 10.1
A Brief History of Some Buffer Overflow Attacks

1988 The Morris Internet Worm used a buffer overflow exploit in “fingerd” as one of its attack
mechanisms.
1995 A buffer overflow in N C S A httpd 1.3 is discovered and published on the Bugtraq
Mailing list by Thomas Lopatic.
1996 Aleph One publishes “Smashing the Stack for Fun and Profit” in Phrack magazine, giving
a step-by-step introduction to exploiting stack-based buffer overflow vulnerabilities.
2001 The Code Red worm exploits a buffer overflow in Microsoft I I S

2003 The Slammer worm exploits a buffer overflow in Microsoft S Q L Server 2000.

2004 The Sasser worm exploits a buffer overflow in Microsoft Windows 2000/X P Local Security
Authority Subsystem Service (L S A S S).

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Buffer Overflow
A buffer overflow, also known as a buffer overrun, is defined
in the N I S T Glossary of Key Information Security Terms
as follows:

“A condition at an interface under which more input can be

placed into a buffer or data holding area than the capacity
allocated, overwriting other information. Adversaries exploit
such a condition to crash a system or to insert specially
crafted code that allows them to gain control of the system.”

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Buffer Overflow
• A very common attack mechanism

– First widely used by the Morris Worm in 1988

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Buffer Overflow
• A very common attack mechanism

– First widely used by the Morris Worm in 1988

• Prevention techniques known

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Buffer Overflow
• A very common attack mechanism

– First widely used by the Morris Worm in 1988

• Prevention techniques known

• Still of major concern

– Legacy of buggy code in widely deployed operating

systems and applications

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Buffer Overflow
• A very common attack mechanism

– First widely used by the Morris Worm in 1988

• Prevention techniques known

• Still of major concern

– Legacy of buggy code in widely deployed operating

systems and applications

– Continued careless programming practices by

programmers
Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Buffer Overflow Basics
• Programming error when a process attempts to store data
beyond the limits of a fixed-sized buffer

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Buffer Overflow Basics
• Programming error when a process attempts to store data
beyond the limits of a fixed-sized buffer

• Overwrites adjacent memory locations

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Buffer Overflow Basics
• Programming error when a process attempts to store data
beyond the limits of a fixed-sized buffer

• Overwrites adjacent memory locations

– Locations can hold other program variables,

parameters, or program control flow data

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Buffer Overflow Basics
• Programming error when a process attempts to store data
beyond the limits of a fixed-sized buffer

• Overwrites adjacent memory locations

– Locations can hold other program variables,

parameters, or program control flow data

• Buffer can be located on the stack, in the heap, or in the

data section of the process

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Buffer Overflow Basics
• Consequences:

– Corruption of program data

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Buffer Overflow Basics
• Consequences:

– Corruption of program data

– Unexpected transfer of control

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Buffer Overflow Basics
• Consequences:

– Corruption of program data

– Unexpected transfer of control

– Memory access violations

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Buffer Overflow Basics
• Consequences:

– Corruption of program data

– Unexpected transfer of control

– Memory access violations

– Execution of code chosen by attacker

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Figure 10.1
Basic Buffer Overflow Example

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Figure 10.2
Basic Buffer Overflow Stack Values

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Buffer Overflow Attacks
• To exploit a buffer overflow an attacker needs:

– To identify a buffer overflow vulnerability in some

program that can be triggered using externally sourced
data under the attacker’s control

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Buffer Overflow Attacks
• To exploit a buffer overflow an attacker needs:

– To identify a buffer overflow vulnerability in some

program that can be triggered using externally sourced
data under the attacker’s control

– To understand how that buffer is stored in memory and

determine potential for corruption

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Buffer Overflow Attacks
• Identifying vulnerable programs can be done by:

– Inspection of program source

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Buffer Overflow Attacks
• Identifying vulnerable programs can be done by:

– Inspection of program source

– Tracing the execution of programs as they process

oversized input

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Buffer Overflow Attacks
• Identifying vulnerable programs can be done by:

– Inspection of program source

– Tracing the execution of programs as they process

oversized input

– Using tools such as fuzzing to automatically identify

potentially vulnerable programs

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Programming Language History
• At the machine level, data manipulated by machine
instructions and executed by the computer processor are
stored in either the processor’s registers or in memory

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Programming Language History
• At the machine level, data manipulated by machine
instructions and executed by the computer processor are
stored in either the processor’s registers or in memory

• Assembly language programmer is responsible for the

correct interpretation of any saved data value

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Programming Language History
• Modern high-level languages have a strong notion of type
and valid operations

– Not vulnerable to buffer overflows

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Programming Language History
• Modern high-level languages have a strong notion of type
and valid operations

– Not vulnerable to buffer overflows

– Does incur overhead, some limits on use

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Programming Language History
• C and related languages have high-level control
structures, but allow direct access to memory

– Hence are vulnerable to buffer overflow

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Programming Language History
• C and related languages have high-level control
structures, but allow direct access to memory

– Hence are vulnerable to buffer overflow

– Have a large legacy of widely used, unsafe, and hence

vulnerable code

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Stack Buffer Overflows
• Occur when buffer is located on stack

– Also referred to as stack smashing

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Stack Buffer Overflows
• Occur when buffer is located on stack

– Also referred to as stack smashing

– Used by Morris Worm

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Stack Buffer Overflows
• Occur when buffer is located on stack

– Also referred to as stack smashing

– Used by Morris Worm

– Exploits included an unchecked buffer overflow

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Stack Buffer Overflows
• Occur when buffer is located on stack

– Also referred to as stack smashing

– Used by Morris Worm

– Exploits included an unchecked buffer overflow

• Are still being widely exploited

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Stack Buffer Overflows
• Stack frame

– When one function calls another it needs somewhere

to save the return address

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Stack Buffer Overflows
• Stack frame

– When one function calls another it needs somewhere

to save the return address

– Also needs locations to save the parameters to be

passed in to the called function and to possibly save
register values

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Figure 10.3
Example Stack Frame with Functions P and Q

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Figure 10.4
Program Loading into Process Memory

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Figure 10.5
Basic Stack Overflow Example

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Figure 10.6
Basic Stack Overflow Stack Values

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Figure 10.7a
Another Stack Overflow Example

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Figure 10.7b
Another Stack Overflow Example

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Table 10.2
Some Common Unsafe C Standard Library Routines

gets(char *str) read line from standard input into

str

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Table 10.2
Some Common Unsafe C Standard Library Routines

gets(char *str) read line from standard input into

str
sprintf(char *str, char *format, …) create s t r according to supplied
format and variables

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Table 10.2
Some Common Unsafe C Standard Library Routines

gets(char *str) read line from standard input into

str
sprintf(char *str, char *format, …) create s t r according to supplied
format and variables
s t rcat(char *dest, char *s r c) append contents of string s r c to
string dest

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Table 10.2
Some Common Unsafe C Standard Library Routines

gets(char *str) read line from standard input into

str
sprintf(char *str, char *format, …) create s t r according to supplied
format and variables
s t rcat(char *dest, char *s r c) append contents of string s r c to
string dest
s t r c p y(char *dest, char *s r c) copy contents of string s r c to
string dest

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Table 10.2
Some Common Unsafe C Standard Library Routines

gets(char *str) read line from standard input into

str
sprintf(char *str, char *format, …) create s t r according to supplied
format and variables
s t rcat(char *dest, char *s r c) append contents of string s r c to
string dest
s t r c p y(char *dest, char *s r c) copy contents of string s r c to
string dest
vsprintf(char *str, char *f m t, v create s t r according to supplied
a_list a p) format and variables

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Shellcode
• Code supplied by attacker

– Often saved in buffer being overflowed

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Shellcode
• Code supplied by attacker

– Often saved in buffer being overflowed

– Traditionally transferred control to a user command-

line interpreter (shell)

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Shellcode
• Machine code

– Specific to processor and operating system

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Shellcode
• Machine code

– Specific to processor and operating system

– Traditionally needed good assembly language skills to

create

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Shellcode
• Machine code

– Specific to processor and operating system

– Traditionally needed good assembly language skills to

create

– More recently a number of sites and tools have been

developed that automate this process

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Shellcode
• Metasploit Project

– Provides useful information to people who perform

penetration, I D S signature development, and exploit
research

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Figure 10.8
Example U N I X Shellcode

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Table 10.3
Some Common x86 Assembly Language Instructions
M O V s r c, dest copy (move) value from s r c into dest
L E A s r c, dest copy the address (load effective address) of s r c into dest
ADD / SUB s r c, dest add / sub value in s r c from dest leaving result in dest
AND / OR / XOR s r c, dest logical and / or / x o r value in s r c with dest leaving result in dest
C M P val1, val2 compare val1 and val2, setting C P U flags as a result
J M P / J Z / J N Z addr jump / if zero / if not zero to addr
PUSH s r c push the value in s r c onto the stack
POP dest pop the value on the top of the stack into dest
CALL addr call function at addr
LEAVE clean up stack frame before leaving function
RET return from function
I N T num software interrupt to access operating system function
NOP no operation or do nothing instruction

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Table 10.4
Some x86 Registers
32 bit 16 bit 8 bit (high) 8 bit (low) Use

%e a x %a x %a h %a l Accumulators used for arithmetical and I/O operations

and execute interrupt calls
%e b x %b x %b h %b l Base registers used to access memory, pass system
call arguments, and return values
%e c x %c x %c h %c l Counter registers

%e d x %d x %d h %d l Data registers used for arithmetic operations, interrupt

calls, and I O Operations
Blank Blank Blank

%e b p Base Pointer containing the address of the current

stack frame
Blank Blank Bank

%e i p Instruction Pointer or Program Counter containing the

address of the next instruction to be executed
Blank Blank Blank

%e s i Source Index register used as a pointer for string or

array operations
Blank Blank Blank

%e s p Stack Pointer containing the address of the top of

stack

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Figure 10.9
Example Stack Overflow Attack

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Stack Overflow Variants
• Targets:

– A trusted system utility

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Stack Overflow Variants
• Targets:

– A trusted system utility

– Network service daemon

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Stack Overflow Variants
• Targets:

– A trusted system utility

– Network service daemon

– Commonly used library code

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Stack Overflow Variants
• Shellcode functions:

– Launch a remote shell when an attacker connects to it

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Stack Overflow Variants
• Shellcode functions:

– Launch a remote shell when an attacker connects to it

– Create a reverse shell that connects back to the

hacker

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Stack Overflow Variants
• Shellcode functions:

– Launch a remote shell when an attacker connects to it

– Create a reverse shell that connects back to the

hacker

– Use local exploits that establish a shell

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Stack Overflow Variants
• Shellcode functions:

– Launch a remote shell when an attacker connects to it

– Create a reverse shell that connects back to the

hacker

– Use local exploits that establish a shell

– Flush firewall rules that currently block other attacks

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Stack Overflow Variants
• Shellcode functions:

– Launch a remote shell when an attacker connects to it

– Create a reverse shell that connects back to the

hacker

– Use local exploits that establish a shell

– Flush firewall rules that currently block other attacks

– Break out of a chroot (restricted execution)

environment, giving full access to the system
Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Buffer Overflow Defenses
• Two broad defense approaches

– Compile-time

▪ Aim to harden programs to resist attacks in new

programs

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Buffer Overflow Defenses
• Two broad defense approaches

– Run-time

▪ Aim to detect and abort attacks in existing

programs

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Compile-Time Defenses: Programming
Language
• Use a modern high-level language

– Not vulnerable to buffer overflow attacks

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Compile-Time Defenses: Programming
Language
• Use a modern high-level language

– Not vulnerable to buffer overflow attacks

– Compiler enforces range checks and permissible

operations on variables

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Compile-Time Defenses: Programming
Language
• Disadvantages

– Additional code must be executed at run time to

impose checks

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Compile-Time Defenses: Programming
Language
• Disadvantages

– Additional code must be executed at run time to

impose checks

– Flexibility and safety comes at a cost in resource use

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Compile-Time Defenses: Programming
Language
• Disadvantages

– Distance from the underlying machine language and

architecture means that access to some instructions
and hardware resources is lost

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Compile-Time Defenses: Programming
Language
• Disadvantages

– Distance from the underlying machine language and

architecture means that access to some instructions
and hardware resources is lost

– Limits these languages’ usefulness in writing code,

such as device drivers, that must interact with such
resources

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Compile-Time Defenses: Safe Coding
Techniques
• C designers placed much more emphasis on space
efficiency and performance considerations than on type
safety

– Assumed programmers would exercise due care in

writing code

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Compile-Time Defenses: Safe Coding
Techniques
• Programmers need to inspect the code and rewrite any
unsafe coding

– An example of this is the Open B S D project

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Compile-Time Defenses: Safe Coding
Techniques
• Programmers have audited the existing code base,
including the operating system, standard libraries, and
common utilities

– Resulted in one of the safest operating systems in

widespread use

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Figure 10.10
Examples of Unsafe C Code

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Compile-Time Defenses: Language
Extensions/Safe Libraries
• Handling dynamically allocated memory is more
problematic because the size information is not available
at compile time

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Compile-Time Defenses: Language
Extensions/Safe Libraries
• Handling dynamically allocated memory is more
problematic because the size information is not available
at compile time

– Requires an extension and the use of library routines

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Compile-Time Defenses: Language
Extensions/Safe Libraries
• Handling dynamically allocated memory is more
problematic because the size information is not available
at compile time

– Requires an extension and the use of library routines

▪ Programs and libraries need to be recompiled

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Compile-Time Defenses: Language
Extensions/Safe Libraries
• Handling dynamically allocated memory is more
problematic because the size information is not available
at compile time

– Requires an extension and the use of library routines

▪ Programs and libraries need to be recompiled

▪ Likely to have problems with third-party applications

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Compile-Time Defenses: Language
Extensions/Safe Libraries
• Concern with C is use of unsafe standard library routines

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Compile-Time Defenses: Language
Extensions/Safe Libraries
• Concern with C is use of unsafe standard library routines

– One approach has been to replace these with safer

variants

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Compile-Time Defenses: Language
Extensions/Safe Libraries
• Concern with C is use of unsafe standard library routines

– One approach has been to replace these with safer

variants

▪ Libsafe is an example

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Compile-Time Defenses: Language
Extensions/Safe Libraries
• Concern with C is use of unsafe standard library routines

– One approach has been to replace these with safer

variants

▪ Libsafe is an example

▪ Library is implemented as a dynamic library

arranged to load before the existing standard
libraries

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Compile-Time Defenses: Stack Protection
• Add function entry and exit code to check stack for signs
of corruption

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Compile-Time Defenses: Stack Protection
• Random canary:

– Unpredictable value

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Compile-Time Defenses: Stack Protection
• Random canary:

– Unpredictable value

– Should be different on different systems

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Compile-Time Defenses: Stack Protection
• Stackshield and Return Address Defender (R A D):

– G C C extensions that include additional function entry

and exit code

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Compile-Time Defenses: Stack Protection
• Stackshield and Return Address Defender (R A D):

– G C C extensions that include additional function entry

and exit code

▪ Function entry writes a copy of the return address

to a safe region of memory

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Compile-Time Defenses: Stack Protection
• Stackshield and Return Address Defender (R A D):

– G C C extensions that include additional function entry

and exit code

▪ Function entry writes a copy of the return address

to a safe region of memory

▪ Function exit code checks the return address in the

stack frame against the saved copy

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Compile-Time Defenses: Stack Protection
• Stackshield and Return Address Defender (R A D):

– G C C extensions that include additional function entry

and exit code

▪ Function entry writes a copy of the return address

to a safe region of memory

▪ Function exit code checks the return address in the

stack frame against the saved copy

▪ If change is found, aborts the program

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Run-Time Defenses: Executable
Address Space Protection
• Make regions of memory non-executable:

– Requires support from memory management unit

(MMU)

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Run-Time Defenses: Executable
Address Space Protection
• Make regions of memory non-executable:

– Requires support from memory management unit

(MMU)

– Long existed on S P A R C / Solaris systems

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Run-Time Defenses: Executable
Address Space Protection
• Make regions of memory non-executable:

– Requires support from memory management unit

(MMU)

– Long existed on S P A R C / Solaris systems

– Recent on x86 Linux/Unix/Windows systems

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Run-Time Defenses: Executable
Address Space Protection
• Issues

– Support for executable stack code

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Run-Time Defenses: Executable
Address Space Protection
• Issues

– Support for executable stack code

– Special provisions are needed

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Run-Time Defenses: Address Space
Randomization
• Manipulate location of key data structures

– Stack, heap, global data

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Run-Time Defenses: Address Space
Randomization
• Manipulate location of key data structures

– Stack, heap, global data

– Using random shift for each process

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Run-Time Defenses: Address Space
Randomization
• Manipulate location of key data structures

– Stack, heap, global data

– Using random shift for each process

– Large address range on modern systems means

wasting some has negligible impact

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Run-Time Defenses: Address Space
Randomization
• Manipulate location of key data structures

– Stack, heap, global data

– Using random shift for each process

– Large address range on modern systems means

wasting some has negligible impact

• Randomize location of heap buffers

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Run-Time Defenses: Address Space
Randomization
• Manipulate location of key data structures

– Stack, heap, global data

– Using random shift for each process

– Large address range on modern systems means

wasting some has negligible impact

• Randomize location of heap buffers

• Random location of standard library functions

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Run-Time Defenses: Guard Pages
• Place guard pages between critical regions of memory

– Flagged in M M U as illegal addresses

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Run-Time Defenses: Guard Pages
• Place guard pages between critical regions of memory

– Flagged in M M U as illegal addresses

– Any attempted access aborts process

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Run-Time Defenses: Guard Pages
• Place guard pages between critical regions of memory

– Flagged in M M U as illegal addresses

– Any attempted access aborts process

• Further extension places guard pages between stack

frames and heap buffers

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Run-Time Defenses: Guard Pages
• Place guard pages between critical regions of memory

– Flagged in M M U as illegal addresses

– Any attempted access aborts process

• Further extension places guard pages between stack

frames and heap buffers

– Cost in execution time to support the large number of

page mappings necessary

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Replacement Stack Frame
• Variant that overwrites buffer and saved frame pointer
address

– Saved frame pointer value is changed to refer to a

dummy stack frame

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Replacement Stack Frame
• Variant that overwrites buffer and saved frame pointer
address

– Saved frame pointer value is changed to refer to a

dummy stack frame

– Current function returns to the replacement dummy

frame

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Replacement Stack Frame
• Variant that overwrites buffer and saved frame pointer
address

– Saved frame pointer value is changed to refer to a

dummy stack frame

– Current function returns to the replacement dummy

frame

– Control is transferred to the shellcode in the

overwritten buffer

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Replacement Stack Frame
• Off-by-one attacks

– Coding error that allows one more byte to be copied

than there is space available

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Replacement Stack Frame
• Defenses

– Any stack protection mechanisms to detect

modifications to the stack frame or return address by
function exit code

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Replacement Stack Frame
• Defenses

– Any stack protection mechanisms to detect

modifications to the stack frame or return address by
function exit code

– Use non-executable stacks

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Replacement Stack Frame
• Defenses

– Any stack protection mechanisms to detect

modifications to the stack frame or return address by
function exit code

– Use non-executable stacks

– Randomization of the stack in memory and of system

libraries

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Return to System Call
• Stack overflow variant replaces return address with
standard library function

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Return to System Call
• Stack overflow variant replaces return address with
standard library function

– Response to non-executable stack defenses

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Return to System Call
• Stack overflow variant replaces return address with
standard library function

– Response to non-executable stack defenses

– Attacker constructs suitable parameters on stack

above return address

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Return to System Call
• Stack overflow variant replaces return address with
standard library function

– Response to non-executable stack defenses

– Attacker constructs suitable parameters on stack

above return address

– Function returns and library function executes

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Return to System Call
• Stack overflow variant replaces return address with
standard library function

– Response to non-executable stack defenses

– Attacker constructs suitable parameters on stack

above return address

– Function returns and library function executes

– Attacker may need exact buffer address

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Return to System Call
• Stack overflow variant replaces return address with
standard library function

– Response to non-executable stack defenses

– Attacker constructs suitable parameters on stack

above return address

– Function returns and library function executes

– Attacker may need exact buffer address

– Can even chain two library calls

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Heap Overflow
• Attack buffer located in heap

– Typically located above program code

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Heap Overflow
• Attack buffer located in heap

– Typically located above program code

– Memory is requested by programs to use in dynamic

data structures (such as linked lists of records)

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Heap Overflow
• No return address

– Hence no easy transfer of control

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Heap Overflow
• No return address

– Hence no easy transfer of control

– May have function pointers can exploit

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Heap Overflow
• No return address

– Hence no easy transfer of control

– May have function pointers can exploit

– Or manipulate management data structures

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Heap Overflow
• Defenses

– Making the heap non-executable

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Heap Overflow
• Defenses

– Making the heap non-executable

– Randomizing the allocation of memory on the heap

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Figure 10.11 (1 of 2)
Example Heap Overflow Attack

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Figure 10.11 (2 of 2)
Example Heap Overflow Attack

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Global Data Overflow
• Defenses

– Non executable or random global data region

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Global Data Overflow
• Defenses

– Non executable or random global data region

– Move function pointers

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Global Data Overflow
• Defenses

– Non executable or random global data region

– Move function pointers

– Guard pages

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Global Data Overflow
• Can attack buffer located in global data

– May be located above program code

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Global Data Overflow
• Can attack buffer located in global data

– May be located above program code

– If has function pointer and vulnerable buffer

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Global Data Overflow
• Can attack buffer located in global data

– May be located above program code

– If has function pointer and vulnerable buffer

– Or adjacent process management tables

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Global Data Overflow
• Can attack buffer located in global data

– May be located above program code

– If has function pointer and vulnerable buffer

– Or adjacent process management tables

– Aim to overwrite function pointer later called

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Figure 10.12 (1 of 2)
Example Global Data Overflow Attack

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Figure 10.12 (2 of 2)
Example Global Data Overflow Attack

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Summary
• Stack overflows

– Buffer overflow basics

– Stack buffer overflows

– Shellcode

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Summary
• Defending against buffer overflows

– Compile-time defenses

– Run-time defenses

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Summary
• Other forms of overflow attacks

– Replacement stack frame

– Return to system call

– Heap overflows

– Global data area overflows

– Other types of overflows

Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved
Copyright
This work is protected by United States
copyright laws and is provided solely for
the use of instructors in teaching their
courses and assessing student learning.
Dissemination or sale of any part of this
work (including on the World Wide Web)
will destroy the integrity of the work and is
not permitted. The work and materials from
it should never be made available to
students except by instructors using the
accompanying text in their classes. All
recipients of this work are expected to
abide by these restrictions and to honor the
intended pedagogical purposes and the
needs of other instructors who rely on
these materials.
Copyright © 2024, 2018, 2015 Pearson Education, Inc. All Rights Reserved