Chapter 3: Protection Mechanisms of Operating
Systems
Chapter 3: Protection Mechanisms of Operating Systems 1 / 52
Table of Contents
1 Introduction
2 1. Authentication and Identification
3 2. Access Control Mechanisms
4 3. File System Protection
5 4. Process and Application Isolation
6 6. Protection Against Code Manipulation Attacks
7 7. Protection Mechanisms Against External Threats: Antivirus and
Firewall
8 Conclusion
9 Additional Useful Information
Chapter 3: Protection Mechanisms of Operating Systems 2 / 52
Introduction to Operating System Protection
Modern operating systems are designed to ensure the security of
computer resources against growing threats.
System protection relies on a set of essential mechanisms.
These mechanisms aim to ensure the confidentiality (keeping
information secret), integrity (ensuring data is accurate and
unchanged), and availability (making sure resources are accessible
when needed) of data and processes.
They help prevent unauthorized access, control permissions, and
protect critical files.
This chapter explores these fundamental mechanisms in detail.
Chapter 3: Protection Mechanisms of Operating Systems 3 / 52
Authentication and Identification: 1.1 Distinction between
Identification and Authentication
Identification is the process by which a user presents themselves to
the system (e.g., by entering a username).
Authentication, on the other hand, is the process of proving that the
user is who they claim to be.
Chapter 3: Protection Mechanisms of Operating Systems 4 / 52
Authentication and Identification: 1.2 Example: Windows
and Active Directory
Identification: The user enters their username on the Windows login
screen. This declares their identity.
Authentication: Active Directory, a directory service by Microsoft,
then verifies the provided password by comparing it to a hashed
version stored in its database. This confirms the user’s claimed
identity.
If the information matches, the user is authenticated and can access
their session.
Chapter 3: Protection Mechanisms of Operating Systems 5 / 52
Authentication and Identification: 1.3 Authentication
Methods and Their Security
Authentication by password (traditional method, vulnerabilities).
Two-Factor Authentication (2FA) for enhanced security.
Chapter 3: Protection Mechanisms of Operating Systems 6 / 52
Authentication and Identification: 1.3.1 Authentication by
Password
The most common method, based on a shared secret between the
user and the system.
However, it is vulnerable to various attacks:
Brute-force attacks: Trying numerous combinations to guess the
password.
Dictionary attacks: Using lists of common words to attempt password
cracking.
Phishing: Deceiving users into revealing their passwords through fake
websites or emails.
Database breaches: If the password database is compromised by
attackers.
Chapter 3: Protection Mechanisms of Operating Systems 7 / 52
1.3.1: Example: Password Policy on Linux
Linux systems use PAM (Pluggable Authentication Modules) for
flexible authentication management. PAM allows administrators to
configure various authentication methods and policies.
It is possible to configure a robust password policy via PAM,
imposing:
Complexity requirements: Minimum length, combination of
uppercase and lowercase letters, numbers, and symbols.
Regular checks: Encouraging users to change their passwords
periodically.
Blocking of simple or previously used passwords: Enhancing
password security by preventing the use of easily guessable or reused
passwords.
Chapter 3: Protection Mechanisms of Operating Systems 8 / 52
Authentication and Identification: 1.3.2 Two-Factor
Authentication (2FA)
2FA combines a password with a second authentication factor to
verify a user’s identity.
This second factor can be something the user:
Has: A code generated by an app on a phone (like Google
Authenticator), a physical security key (like a YubiKey).
Is: A biometric data (fingerprint, facial recognition).
2FA makes it much harder for an attacker to access an account, even
if they know the password, as they would also need the second factor.
Chapter 3: Protection Mechanisms of Operating Systems 9 / 52
1.3.2: Example: Google Authenticator and SSH
On a Linux server, it is possible to enable two-factor authentication
for SSH (Secure Shell) connections, which are used for secure remote
access.
By installing Google Authenticator (or another OTP - One-Time
Password - application), the user must enter a unique and temporary
code in addition to their password to log in.
This code typically changes every 30 seconds, making it difficult to
intercept and reuse by attackers.
Chapter 3: Protection Mechanisms of Operating Systems 10 / 52
Access Control Mechanisms: 2.1 Introduction to Access
Control Mechanisms
Once a user is authenticated, the operating system needs to
determine what they are allowed to do.
This is governed by access control mechanisms, which define and
enforce the permissions of users over system resources.
These resources include files, directories, processes, memory, and
network interfaces.
Chapter 3: Protection Mechanisms of Operating Systems 11 / 52
Access Control Mechanisms: 2.2 Access Control Models
There are several models for organizing and applying access
permissions.
Each model has its own characteristics and is suited to different types
of environments and security requirements.
The main models include:
Discretionary Access Control (DAC)
Mandatory Access Control (MAC)
Role-Based Access Control (RBAC)
Attribute-Based Access Control (ABAC)
Other Models and Considerations
Policy-Based Access Control (PBAC): Often seen as an extension or
variant of ABAC, this model emphasizes the explicit formulation of
security policies—often centralized—to automate decision-making.
Hybrid Models: In practice, it is common to combine multiple models
(for example, using both the structure of RBAC and the flexibility of
ABAC) to address complex environments where requirements vary
greatly depending on context.
Chapter 3: Protection Mechanisms of Operating Systems 12 / 52
Access Control Mechanisms: 2.3 Discretionary Access
Control (DAC)
Principle: Access control is left to the discretion of the owner of the
object.
Main Feature: Each user can manage the rights of their own files or
resources and delegate some access rights to others.
Real-World Example in OS: UNIX/Linux File Permissions: In most
UNIX and Linux distributions, file permissions (read, write, execute)
are set by the file owner, who can grant or restrict access to other
users and groups. This is a classic example of DAC, where the
resource owner determines access rights.
Chapter 3: Protection Mechanisms of Operating Systems 13 / 52
2.3 Example: Unix/Linux Permissions
On a Linux system, the chmod (change mode) command allows the
owner of a file or directory to modify the permissions for three
categories of users:
The owner (user).
The group associated with the file/directory (group).
All other users on the system (others).
The possible permissions are:
Read (r): Allows viewing the contents of a file or listing the contents
of a directory.
Write (w): Allows modifying the contents of a file or creating/deleting
files in a directory.
Execute (x): Allows running a file as a program or entering a directory.
Chapter 3: Protection Mechanisms of Operating Systems 14 / 52
Access Control Mechanisms: 2.4 Mandatory Access
Control (MAC)
Principle: Access is regulated by centralized policies defined by an
administrator or according to predefined security levels.
Main Feature: Access decisions do not depend on individual users,
which strengthens security but limits flexibility.
Chapter 3: Protection Mechanisms of Operating Systems 15 / 52
2.4: Example: SELinux on Red Hat
Security-Enhanced Linux (SELinux) is a security module integrated
into the Linux kernel, commonly used in Red Hat distributions.
SELinux enforces very strict security policies based on the MAC
model.
It prevents even a user with superuser (root) privileges from
modifying certain critical files or performing specific actions without
explicit authorization defined by the SELinux policy. This provides a
strong layer of security against both malicious users and compromised
processes.
Chapter 3: Protection Mechanisms of Operating Systems 16 / 52
Access Control Mechanisms: 2.5 Role-Based Access
Control (RBAC)
Principle: Access is controlled based on roles assigned to users, and
each role possesses a specific set of permissions.
Main Feature: It simplifies security management in organizations by
grouping permissions according to job functions or positions.
This greatly simplifies the management of permissions, especially in
complex environments with many users and resources, as changes to
permissions only require updating the roles, not individual user
accounts.
Chapter 3: Protection Mechanisms of Operating Systems 17 / 52
2.5: Example: Permission Management in Windows Server
In a Windows Server environment using Active Directory [Windows
Active Directory and Group Policies], administrators can create
groups and assign permissions to these groups.
Users are then added to these groups based on their roles (e.g.,
Domain Administrator, Standard User, etc.).
The permissions associated with the group automatically apply to all
members of that group, streamlining the process of managing access
rights for numerous users.
Chapter 3: Protection Mechanisms of Operating Systems 18 / 52
Access Control Mechanisms: 2.6 Attribute-Based
Access Control (ABAC)
Principle: Access decisions are based on attributes associated with
users (subjects), resources (objects), actions, and the environmental
context.
Main Features:
Flexibility: ABAC allows for very granular and dynamic access control
policies by considering multiple criteria (for example, time of day,
location, device type, etc.).
Scalability: By managing permissions based on sets of attributes
rather than fixed roles, it is easier to adapt policies to complex and
rapidly changing environments.
Usage Example: In a company, access policies might allow access to
sensitive data only if the user has a “customer service” attribute and
the request is made during business hours from an approved device.
Chapter 3: Protection Mechanisms of Operating Systems 19 / 52
2.5: Real-World Example in OS/Environments:
XACML-based Solutions in Cloud Platforms: Many modern cloud
environments, such as AWS with its IAM policies, implement
concepts similar to ABAC. They allow administrators to craft policies
that consider attributes (like user tags, resource identifiers, or
contextual information) to grant or deny access, effectively creating
dynamic and context-aware security measures.
Chapter 3: Protection Mechanisms of Operating Systems 20 / 52
File System Protection: 3.1 Securing Stored Data
File systems contain the majority of the information stored on a
computer.
Their protection is absolutely crucial for data security.
Chapter 3: Protection Mechanisms of Operating Systems 21 / 52
File System Protection: 3.2 Protection of System Files
Critical operating system files must be protected against
unauthorized or accidental modification.
Any alteration of these files could compromise the stability and
security of the entire system.
Chapter 3: Protection Mechanisms of Operating Systems 22 / 52
3.2: Example: Windows File Protection (WFP)
Windows File Protection (WFP) is a feature of Windows that
prevents malicious programs or installation errors from replacing
important system files with incorrect or corrupted versions.
If a protected file is modified, WFP automatically restores the correct
version from a secure cache, ensuring the integrity of the operating
system.
Chapter 3: Protection Mechanisms of Operating Systems 23 / 52
File System Protection: 3.3 Data Encryption
Encryption is the process of transforming data into an unreadable
format (called ciphertext) using an algorithm and a key.
Only someone with the correct decryption key can revert the data to
its original format (called plaintext).
Encryption is an effective method to protect the confidentiality of
data, whether it is at rest (stored on a disk) or in transit (transmitted
over a network).
Chapter 3: Protection Mechanisms of Operating Systems 24 / 52
3.3: Example: BitLocker on Windows
BitLocker is a full-disk encryption feature integrated into certain
editions of Windows.
It allows encrypting the entire hard drive (or specific partitions),
making the data inaccessible without the correct password or recovery
key.
BitLocker is particularly useful for protecting data in case of theft or
loss of the computer.
Chapter 3: Protection Mechanisms of Operating Systems 25 / 52
Section 4: Introduction to Process and Application
Isolation
Process isolation and application isolation are important
techniques to limit the damage if a process or application is
compromised by an attacker.
The idea is to prevent a malicious or compromised application from
affecting the rest of the system or accessing sensitive data belonging
to other applications.
Chapter 3: Protection Mechanisms of Operating Systems 26 / 52
Process and Application Isolation: 4.1 Sandboxing
A sandbox is an isolated execution environment that allows running
an application or process in a restricted manner.
An application running in a sandbox has limited access to system
resources (files, memory, network, etc.).
This means that if the application is malicious or contains a
vulnerability, the damage it can cause is confined to the sandbox
environment, preventing it from harming the host system or other
applications.
Chapter 3: Protection Mechanisms of Operating Systems 27 / 52
4.1: Example: Google Chrome Sandbox
The Google Chrome web browser uses a sandbox architecture to
isolate each Browse tab.
If a malicious website attempts to exploit a vulnerability in the
browser, it will be confined to the sandbox environment of that tab
and will not have access to the operating system’s files or data from
other tabs, enhancing the security of the user’s Browse experience.
Chapter 3: Protection Mechanisms of Operating Systems 28 / 52
Section 5: Containerization and Virtualization
Containerization and virtualization are more advanced techniques
that create more complete isolated execution environments than
simple sandboxes.
Containerization: Packages an application and its dependencies into
a container that is isolated from the host operating system but shares
its kernel. Multiple containers can run on the same OS kernel.
Virtualization: Creates virtual machines (VMs) which are complete
simulations of computer systems, each with its own operating system,
kernel, etc., running on a physical host. This provides a higher level
of isolation.
Chapter 3: Protection Mechanisms of Operating Systems 29 / 52
5: Docker and Kubernetes
Docker is a popular containerization platform that allows creating
and managing application containers easily and efficiently. Each
application runs in its own isolated container, limiting the potential
impact of a security breach.
Kubernetes is an open-source container orchestration system that
automates the deployment, scaling, and management of containerized
applications, providing a robust and scalable environment for running
isolated applications.
Chapter 3: Protection Mechanisms of Operating Systems 30 / 52
Section 6: Introduction to Code Manipulation Attacks
Attacks aimed at manipulating the code during execution to gain
control of a program or system are a constant threat.
Several mechanisms exist to make these attacks more difficult to
carry out.
Chapter 3: Protection Mechanisms of Operating Systems 31 / 52
Code Manipulation Attacks: 6.1 Address Space Layout
Randomization (ASLR)
Address Space Layout Randomization (ASLR) is a security
technique that involves randomly arranging the positions of key
memory areas (such as the base of the executable, the stack, and the
heap) when a program is loaded into memory.
This makes it much harder for an attacker to predict the location of
critical instructions or data, complicating buffer overflow attacks and
other forms of code injection.
Chapter 3: Protection Mechanisms of Operating Systems 32 / 52
6.1: Example: ASLR Implementation on Linux and
Windows
ASLR is a standard security feature enabled by default in modern
versions of Windows and most Linux distributions.
The level of randomization can vary depending on the operating
system and configurations, but it significantly increases the difficulty
of exploiting memory-based vulnerabilities.
Chapter 3: Protection Mechanisms of Operating Systems 33 / 52
Code Manipulation Attacks: 6.2 Data Execution
Prevention (DEP)
Data Execution Prevention (DEP), sometimes called NX bit
(No-Execute), is a security feature at the processor and operating
system level that prevents code from being executed from memory
regions that are designated as being for data (such as the stack or the
heap).
This technique helps to counter many code injection attacks, where
an attacker tries to insert malicious code into a data area and then
trick the system into executing it.
Chapter 3: Protection Mechanisms of Operating Systems 34 / 52
6.2 : Example: DEP on Windows
The DEP feature is integrated into Windows and can be configured
through system settings.
It helps block the execution of malicious code in non-executable
memory areas, thereby strengthening the system’s security against
various types of exploits.
Chapter 3: Protection Mechanisms of Operating Systems 35 / 52
Protection Mechanisms Against External Threats: 7.1
Introduction to External Threats
External threats, originating from the internet or other networks,
pose a significant risk to the security of operating systems.
Two fundamental tools to protect against these threats are antivirus
software and firewalls.
Chapter 3: Protection Mechanisms of Operating Systems 36 / 52
Protection Mechanisms Against External Threats: 7.2
Antivirus: Threat Detection and Eradication
Antivirus software is designed to detect, block, and remove
malicious software (malware) such as viruses, worms, Trojans,
ransomware, spyware, etc.
Antivirus programs use several techniques to identify threats:
Signature-based analysis: Comparing files to a database of known
virus signatures.
Heuristic detection: Analyzing the behavior of files and programs to
identify suspicious activities that may indicate the presence of an
unknown malware.
Real-time monitoring: Continuously scanning system activity to
detect and block threats as they appear.
Sandboxing: Executing suspicious files in an isolated environment to
observe their behavior before allowing them to run normally.
Chapter 3: Protection Mechanisms of Operating Systems 37 / 52
Protection Mechanisms Against External Threats: 7.2:
Concrete Examples of Antivirus Implementation
Windows Defender (Microsoft Defender Antivirus): The
antivirus software integrated into Windows operating systems,
offering real-time protection and integration with Windows Security.
ClamAV: An open-source antivirus often used on Linux servers to
scan files and analyze emails for malicious content.
XProtect: The antivirus built into macOS, running in the
background to block malware.
Numerous other commercial and free antivirus solutions are available
for various operating systems.
Chapter 3: Protection Mechanisms of Operating Systems 38 / 52
7.2: Limitations and Circumvention of Antivirus
While antivirus software is essential, it is not foolproof.
Malware creators constantly develop new techniques to bypass
antivirus detection:
Encryption and obfuscation: To hide the malicious code and avoid
signature-based detection.
Zero-day attacks: Exploiting software vulnerabilities that are
unknown to antivirus vendors.
Rootkits: Malware that hides deeply within the operating system to
evade detection.
Therefore, it is crucial to supplement antivirus protection with other
security measures.
Chapter 3: Protection Mechanisms of Operating Systems 39 / 52
7.3: Firewalls: Network Traffic Filtering
A firewall is a security device that controls incoming and outgoing
network traffic to and from a computer or network.
It acts as a barrier that filters connections based on configured
security rules.
Chapter 3: Protection Mechanisms of Operating Systems 40 / 52
7.4: How a Firewall Works
Firewalls work by examining data packets that pass through the
network.
They can block or allow traffic based on various criteria, such as:
The source and destination IP address.
The TCP or UDP port being used.
The network protocol (e.g., HTTP, FTP, SSH).
Firewalls help protect against unauthorized access attempts, network
attacks, and the spread of malware.
Chapter 3: Protection Mechanisms of Operating Systems 41 / 52
7.5: Types of Firewalls
Software firewall: Installed directly on the operating system of an
individual computer (e.g., Windows Firewall, iptables on Linux).
Hardware firewall: Integrated into a router or a dedicated network
security appliance, protecting the entire network (e.g., Cisco ASA,
Netgear firewalls).
Chapter 3: Protection Mechanisms of Operating Systems 42 / 52
7.6: Concrete Examples of Firewall Implementation
iptables (Linux): A powerful command-line utility for configuring the
firewall integrated into the Linux kernel (netfilter).
Windows Defender Firewall: The firewall built into Windows
operating systems, offering a user-friendly graphical interface for
configuring rules.
pfSense: An open-source firewall software based on FreeBSD,
offering advanced features for businesses and experienced users.
Chapter 3: Protection Mechanisms of Operating Systems 43 / 52
7.7: Advanced Firewalls: IDS and IPS
Some advanced firewalls integrate intrusion detection and prevention
systems (IDS/IPS).
IDS (Intrusion Detection System): Detects suspicious activities
and alerts the administrator but does not actively block attacks.
IPS (Intrusion Prevention System): Goes further by actively
blocking malicious intrusion attempts.
Concrete example: Snort, a widely used open-source IDS/IPS.
Chapter 3: Protection Mechanisms of Operating Systems 44 / 52
Conclusion: A Multi-Layered Approach is Necessary
Modern operating systems incorporate a variety of protection
mechanisms to ensure the security of users and data.
From authentication to process isolation, including encryption and
access controls, each component plays a crucial role in a
comprehensive security strategy.
However, for effective cybersecurity, it is essential to adopt a
multi-layered approach, combining these mechanisms with rigorous
update management, proactive monitoring, and user awareness
training.
Chapter 3: Protection Mechanisms of Operating Systems 45 / 52
Additional Useful Information: Importance of Updates
Keeping your operating system and applications up to date is
fundamental for security.
Updates often contain patches for security vulnerabilities that could
be exploited by attackers.
Enable automatic updates whenever possible to ensure timely
protection against new threats.
Chapter 3: Protection Mechanisms of Operating Systems 46 / 52
Additional Useful Information: Password Best Practices
Use long and complex passwords that are different for each account
to prevent attackers from gaining access to multiple accounts if one is
compromised.
Consider using a password manager to securely store and manage
your passwords, making it easier to use strong, unique passwords.
Enable Two-Factor Authentication (2FA) wherever it is offered to add
an extra layer of security to your accounts.
Chapter 3: Protection Mechanisms of Operating Systems 47 / 52
Additional Useful Information: Beware of Phishing and
Social Engineering
Attackers often use manipulation techniques to obtain sensitive
information, such as passwords or personal details.
Be vigilant against suspicious emails, messages, and calls that may be
attempts to trick you into revealing confidential information.
Never click on links or open attachments from unknown sources, as
they may contain malware or lead to phishing websites.
Chapter 3: Protection Mechanisms of Operating Systems 48 / 52
Additional Useful Information: The Importance of Data
Backup
Regularly backing up your important data allows you to restore it in
case of loss due to an attack, hardware failure, or other unforeseen
events.
Use different backup methods (external hard drive, cloud storage,
etc.) to ensure redundancy and protect against various types of data
loss.
Chapter 3: Protection Mechanisms of Operating Systems 49 / 52
Additional Useful Information: Wi-Fi Network Security
Use a strong password (WPA3 if possible) for your Wi-Fi network to
prevent unauthorized access to your home or office network.
Consider hiding the SSID (network name) for an additional layer of
security, although this is not a primary security measure.
Be cautious when connecting to public Wi-Fi networks, as they may
not be secure. Consider using a VPN (Virtual Private Network) to
encrypt your connection on public networks.
Chapter 3: Protection Mechanisms of Operating Systems 50 / 52
Additional Useful Information: The Principle of Least
Privilege
Grant users and applications only the privileges necessary to perform
their tasks. This limits the potential damage if an account or
application is compromised by an attacker.
Chapter 3: Protection Mechanisms of Operating Systems 51 / 52
Additional Useful Information: Monitoring and Activity
Logs
Monitoring system activity logs can help detect suspicious activities or
intrusion attempts. Regularly reviewing logs can provide valuable
insights into the security posture of your system.
Log analysis tools can automate this process, making it easier to
identify potential security incidents.
Chapter 3: Protection Mechanisms of Operating Systems 52 / 52
