OPERATING SYSTEM HARDENING:

AN INTRODUCTION

PART 01: INTRODUCTION TO OPERATING

SYSTEM HARDENING

2
INTRODUCTION

• Operating systems are the software that provides access to the various hardware resources.

• Computer programs need more than just the CPU to work. They also need access to other parts of
the computer system, like the hard drive, memory, screen, keyboard, and mouse. The operating
system acts like a manager, giving programs access to these resources when they need them. For
example, think of a program that asks you to enter your password. Here's how the operating system
helps:

✓ Hard Drive: The operating system finds the program on the hard drive and loads it into memory.

✓ Memory: The operating system accesses the device memory to load the program, enabling its execution.

✓ Screen: The operating system displays the password prompt on the screen.

✓ Keyboard/Mouse: You use the keyboard or mouse to type in your password.

INTRODUCTION

• A successful operating system has three primary jobs:

✓ Manage resources: Provide tools like file systems and memory

management to efficiently use hardware components.
✓ Schedule fairly: Switch between different programs running
concurrently, ensuring each gets a fair share of resources.
✓ Maintain security: Control access to resources, preventing one
program from negatively affecting another.

4
INTRODUCTION

• Secure Execution of Processes requires proper resource and scheduling

mechanisms.

1. Resource mechanisms:
✓ Must have boundaries between objects.
✓ Operations should not interfere with each other.
✓ Example: File system preventing one file from overwriting another.

2. Scheduling mechanisms:
✓ Must ensure resource availability to prevent denial of service attacks.
✓ Example: Algorithms guaranteeing all processes get to run.

INTRODUCTION

• Challenges in Operating System Security:

• Designing security mechanisms to protect program execution and

data in complex environments.
• Two main types of systems:
✓ Constrained systems: High security, limited applications and
functionality.
✓ General-purpose systems: Prioritize function and flexibility,
harder to secure.
• Recent advances are improving both types of systems.

6
SECURE OPERATING SYSTEMS

• The goal of operating system security is the development of a secure operating system. A
"secure operating system" is an ideal concept, but no complex system can be truly 100%
secure due to potential programming errors.

• Why Study Secure Operating Systems?

• Even though perfect security is impractical, studying how to build an ideally secure
system helps us assess the security of real-world operating systems.
• By comparing real systems to the ideal, we can:
✓ Understand the strengths and weaknesses of common security approaches.
✓ Identify challenges in implementing secure systems.
✓ Evaluate how different approaches achieve security in practice.
✓ Gain insights to develop better security mechanisms in the future.

SECURE OPERATING SYSTEMS

• Key Considerations for Building Secure Systems:

✓ Threat model: How the system will withstand potential attacks.

✓ Trust model: Which software components need to be trusted to
enforce security (this is called the Trusted Computing Base).

8
THREAT MODEL

• Defines potential attacker actions.

• Assumes a powerful attacker:

✓ Can inject operations from the network.

✓ Actively works to violate security goals.
✓ etc

TRUST MODEL

• Defines the software and data a system relies on for security.

• Ideally, includes minimal software needed for security
enforcement.
• In practice, involves a wider range of software, like:

✓ Operating system code

✓ Authentication programs (e.g., login, SSH)
✓ etc

10
SECURITY GOALS

1. Defines the operations that can be executed by a system while still preventing unauthorized
access.

2. Guide system design and implementation.

3. Focus on confidentiality, integrity, and availability.

4. Can be defined based on security requirements or by limiting functionality. For example:

✓ Bell-LaPadula model's simple-security property: Prevents a process from reading information

classified higher than its own secrecy level. This focuses on secrecy.

✓ Principle of least privilege: Grants a process only the minimum necessary permissions to
perform their tasks. This is a functional goal that may improve security but doesn't guarantee
it.

11

HARDENING LEVELS

• OS’s need security measures (hardening) to protect sensitive data.

• The appropriate hardening level depends on how sensitive the data is
and how strong the access controls are (assessing the strength of access
controls helps determine the appropriate level of security measures
needed. If the access controls are weak, a higher level of hardening might
be necessary to address the vulnerabilities).

12
HARDENING LEVELS

• Start with the minimal level and add more layers as needed.
• Higher levels offer more security but require more effort and expertise.
• If not done properly, high-level hardening can make things worse.
• Choose the right security level for your needs and resources.

13

THREATS TO OPERATING SYSTEMS

• Availability Attacks: Disrupting system operations by overloading

resources or exploiting vulnerabilities. This can prevent legitimate
users from accessing services or data. Example: Denial of service
attack on a log server.
• Confidentiality Attacks: Stealing sensitive data like business
information or user credentials. This can lead to financial loss or
identity theft. Example: Data exfiltration through ransomware.
• Integrity Attacks: Modifying or corrupting the system for malicious
purposes, such as defacing websites, mining cryptocurrency.

14
WHY DO SECURITY BREACHES HAPPEN?

• Security bugs:

✓ Vulnerabilities in the operating system or applications.

✓ Often preventable with timely security updates.
• Poor server configuration:

✓ Default settings are often insecure.

✓ Lack of trained personnel to properly configure servers.
• Insecure IoT devices:

✓ Many devices have security vulnerabilities due to improper

configuration (people just don’t know how to configure them
securely).

15

PROTECTING AGAINST THREATS

• Strengthening the security of both applications and the operating system.

• Keeping software up-to-date to patch vulnerabilities.
• Implementing mechanisms to detect malicious activity.
• Conducting penetration tests and security audits to identify weaknesses.

16
KEEPING UP WITH SECURITY NEWS

• Staying informed about security threats and vulnerabilities is important

for maintaining a secure system.
• Resources:

✓ Specialized websites: Packet Storm Security, The Hacker News.

✓ General tech news sites: Ars Technica, Fudzilla, The Register, ZDNet.
✓ Linux news sites: LXer.
✓ YouTube channels: BeginLinux Guru.

• Remember: No operating system is completely secure. Regular updates

and security awareness are essential for maintaining a safe computing
environment.
17

GENERAL PRINCIPLES OF SECURITY AND HARDENING

• Principle of minimization

✓ Keep systems simple to reduce vulnerabilities and make management easier. This includes removing
unnecessary software, services, and configurations to minimize potential attack vectors and simplify
maintenance tasks.
• Principle of least privilege

✓ Give users and applications only the necessary permissions to do their jobs. This limits the potential
damage from attacks and makes it harder for attackers to gain control of the system. It involves carefully
analyzing the tasks and responsibilities of each user and process and assigning permissions accordingly.
• Defense in depth principle

✓ Create multiple layers of security to make it harder for attackers to compromise the system. Each layer
provides an opportunity to detect and respond to attacks, even if one layer is breached. This can include
firewalls, intrusion detection systems, access controls, and encryption, among other measures.
18
 PART 02: MULTICS OPERATING SYSTEM

19

INTRODUCTION TO THE MULTICS OPERATING SYSTEM

• The Multics system is the first modern operating system.

• Multics project introduced many fundamental concepts for both operating systems and
secure operating systems.

• The Multics project started in 1963. The goal was to create an operating system that could
be used by many people at the same time. This kind of operating system is called a
timesharing operating system. Multics was designed to allow multiple users to share the
computer's resources, such as its processor, memory, and storage. To do this, Multics
provided functions for managing these resources, including:

✓ Multiplexing: Sharing a single resource among multiple users.

✓ Scheduling: Determining which user gets to use a resource at any given time.
✓ Communication: Allowing users to communicate with each other and the system.
✓ Protection: Preventing one user from interfering with another user's work.

20
MULTICS: PIONEERING FEATURES

• Multics was a pioneering operating system that introduced several significant features. These included:

✓ Segmented and virtual memory: This allowed for more efficient use of memory and better
isolation between programs.

✓ Shared memory multiprocessors: Multiple processors could access the same memory,
improving performance.

✓ Hierarchical file systems: Files were organized in a tree-like structure, making them easier to
manage.

✓ Online reconfiguration: Changes to the system could be made without taking it offline,
improving availability.

21

MULTICS SYSTEM ARCHITECTURE

1. Processes: Executable contexts that run program code.

2. Segments: Hold all code, data, and I/O devices accessible to a process.

• Organized in a hierarchy of directories.

• Accessible locally or by name (like file paths).

• Process Protection Domain

✓ Defines the segments a process can access and the operations it can
perform. Each segment has associated accessibility rules (who can access
and what they can do).

22
MULTICS SECURITY FUNDAMENTALS

• The Multics system's security relies on several concepts:

✓ The Multics Supervisor is the core component of the Multics system, responsible for
enforcing security and managing system resources. It's the most trusted component,
running in the most privileged ring (ring 0).
✓ Protection Rings create a hierarchical structure of privilege levels. The supervisor
runs in the most privileged ring (ring 0), while user processes run in less privileged
rings. This helps to isolate the supervisor from potentially malicious user processes.
✓ Segment Descriptor Words (SDWs): These are used to describe segments and
define their access permissions. Each segment has an associated SDW that stores
information about its size, location, and access rights. The supervisor uses SDWs to
control access to segments and ensure that processes only interact with segments
they are authorized to access.

23

MULTICS SECURITY FUNDAMENTALS

The Multics login process. The user’s password is submitted to the Multics answering service which must check the

password against the entries in the password segment. The Multics supervisor in the privileged protection ring 0
authorizes access to this segment and adds a SDW for it to the answering service’s descriptor segment. The answering

service cannot modify its own descriptor segment.

24
MULTICS PROTECTION SYSTEM MODELS

• The Multics protection system uses three different models that work together to protect the
system:

• Access Control List (ACL):

✓ Each object (segment or directory) has an associated ACL.
✓ ACL entries specify which users (or groups of users) can perform which operations on
that object.
✓ Segments allow read (r), write (w), or execute (e) operations.
✓ Directories allow status checks (s), modifications (m), or appending entries (a).
• Rings and Brackets:
✓ Multics limits access based on protection rings, which are hierarchical levels of
privilege.
✓ Each segment has a ring bracket that defines the range of rings allowed to access it.
25

MULTICS PROTECTION SYSTEM MODELS

• Multilevel Security (MLS):

✓ Prevents unauthorized information flow between different
security levels.
✓ Each directory has a mapping of segments to secrecy levels.
✓ Processes can only access segments at or below their own
secrecy level.

26
 PART 03: OPERATING SYSTEM INSTALLATION

27

INITIAL SETUP AND PATCHING OF AN OPERATING SYSTEM

28
INITIAL SETUP AND PATCHING OF AN OPERATING SYSTEM

1. System Security Begins with the Installation: This emphasizes that security should be
integrated from the very beginning, starting with the operating system installation itself.

2. Ideally New Systems Should be Constructed on a Protected Network: This helps ensure
that the base system is not compromised from the outset.

3. The Initial Installation Should Comprise the Minimum Necessary:

✓ Install essential components only: Only install the core operating system
components and essential services required for the system's intended function. This
minimizes the potential attack surface by reducing the number of software packages
that could contain vulnerabilities.
✓ Add more only if needed: Additional software packages should be included only if
they are necessary for the system's operation. Avoid installing unnecessary software
that could increase the risk of security breaches.
29

1. Gg

2. K
INITIAL
3. K
SETUP AND PATCHING OF AN OPERATING SYSTEM

4. The Overall Boot Process Must Also be Secured: This involves securing the boot process, which is the
sequence of steps the system takes to start up. Securing the boot process prevents tampering with the
system during startup and ensures that only authorized software is loaded.

5. Care is Also Required with the Selection and Installation of Any Additional Device Driver Code:

✓ Validate driver code: Device drivers are software components that allow the operating system to
interact with hardware devices. It’s important to carefully check the integrity and source of any
additional device driver code to ensure it's legitimate and doesn't introduce vulnerabilities.

✓ Drivers run with high privileges: Device drivers often execute with full kernel-level privileges,
meaning they have access to the core of the operating system. This makes them a potential target
for attackers, so it's important to ensure they are secure.

30