Faculty of Computer Engineering Informatics and

Communications

Department of Computer Engineering

Module: Parallel and Distributed Computing Code: HCE/HHE/HSE 311

Introduction to Parallel and Distributed Computing

The simultaneous growth in availability of big data and in the number of simultaneous users on
the Internet places particular pressure on the need to carry out computing tasks “in parallel,” or
simultaneously. Parallel and distributed computing occurs across many different topic areas in
computer science, including algorithms, computer architecture, networks, operating systems,
and software engineering. During the early 21st century there was explosive growth in
multiprocessor design and other strategies for complex applications to run faster. Parallel and
distributed computing builds on fundamental systems concepts, such as concurrency, mutual
exclusion, consistency in state/memory manipulation, message-passing, and shared-memory
models.

Parallel Computing:
In parallel computing multiple processors performs multiple tasks assigned to them
simultaneously. Memory in parallel systems can either be shared or distributed. Parallel
computing provides concurrency and saves time and money.

Distributed Computing:
In distributed computing we have multiple autonomous computers which seems to the user as
single system. In distributed systems there is no shared memory and computers communicate

HCE/HHE/HSE 311 by Ignatius Chikukuza

with each other through message passing. In distributed computing a single task is divided
among different computers.

Difference between Parallel Computing and Distributed Computing:

Parallel computing Distributed computing

1. Many operations are performed System components are located at different

simultaneously locations
2. Single computer is required Uses multiple computers
3. Multiple processors perform multiple Multiple computers perform multiple
operations operations
4. It may have shared or distributed It have only distributed memory
memory
5. Processors communicate with each Computers communicate with each
other through bus other through message passing.
6. Improves the system performance Improves system scalability, fault tolerance
and resource sharing capabilities

HCE/HHE/HSE 311 by Ignatius Chikukuza

Parallel Computing - It is the use of multiple processing elements simultaneously for
solving any problem. Problems are broken down into instructions and are solved concurrently
as each resource which has been applied to work is working at the same time.

Advantages of Parallel Computing over Serial Computing are as follows:

1. It saves time and money as many resources working together will reduce the time and
cut potential costs.
2. It can be impractical to solve larger problems on Serial Computing.
3. It can take advantage of non-local resources when the local resources are finite.
4. Serial Computing „wastes‟ the potential computing power, thus Parallel Computing
makes better work of hardware.

Types of Parallelism:
Bit-Level Parallelism
- Bit-level parallelism refers to the ability of a processor to perform operations on
multiple bits of data simultaneously.
- This type of parallelism is achieved by increasing the word size of the processor,
which allows it to process more data in parallel.
- Example: A 32-bit processor can perform operations on 32 bits of data
simultaneously, whereas a 64-bit processor can perform operations on 64 bits of data
simultaneously.

Instruction-Level Parallelism
- Instruction-level parallelism refers to the ability of a processor to execute multiple
instructions simultaneously.
- This type of parallelism is achieved by identifying independent instructions in a
program that can be executed concurrently.
- Techniques such as pipelining, superscalar execution, and out-of-order execution are
used to achieve instruction-level parallelism.

Task-Level Parallelism
- Task-level parallelism refers to the ability of a system to execute multiple tasks or
threads simultaneously.
- This type of parallelism is achieved by dividing a program into smaller tasks that can
be executed independently.
- Task-level parallelism can be achieved using multiple processing cores, multiple
processors, or distributed computing systems.

HCE/HHE/HSE 311 by Ignatius Chikukuza

Key Differences
- Granularity: Bit-level parallelism is the finest granularity, followed by instruction-
level parallelism, and then task-level parallelism.
- Dependency: Bit-level parallelism and instruction-level parallelism require minimal
dependency analysis, whereas task-level parallelism requires more complex
dependency analysis.
- Scalability: Task-level parallelism is more scalable than instruction-level parallelism
and bit-level parallelism, as it can take advantage of multiple processing cores and
distributed systems.

Why parallel computing?

 The whole real world runs in dynamic nature i.e. many things happen at a certain time
but at different places concurrently. This data is extensively huge to manage.
 Real world data needs more dynamic simulation and modeling, and for achieving the
same, parallel computing is the key.
 Parallel computing provides concurrency and saves time and money.
 Complex, large datasets, and their management can be organized only and only using
parallel computing‟s approach.
 Ensures the effective utilization of the resources. The hardware is guaranteed to be
used effectively whereas in serial computation only some part of hardware was used
and the rest rendered idle.
 Also, it is impractical to implement real-time systems using serial computing.

Applications of Parallel Computing:

1. Databases and Data mining.
2. Real time simulation of systems.
3. Science and Engineering.
4. Advanced graphics, augmented reality and virtual reality.

Limitations of Parallel Computing:

1. Its addresses such as communication and synchronization between multiple sub-tasks
and processes is difficult to achieve.
2. The algorithms must be managed in such a way that they can be handled in the
parallel mechanism.
3. The algorithms or program must have low coupling and high cohesion. But it‟s
difficult to create such programs.
4. More technically skilled and expert programmers can code a parallelism based
program well.

HCE/HHE/HSE 311 by Ignatius Chikukuza

Future of Parallel Computing: The computational graph has undergone a great transition
from serial computing to parallel computing. Tech giant such as Intel has already taken a step
towards parallel computing by employing multicore processors. Parallel computation will
revolutionize the way computers work in the future, for the better good. With the entire world
connecting to each other even more than before, Parallel Computing does a better role in
helping us stay that way. With faster networks, distributed systems, and multi-processor
computers, it becomes even more necessary.

HCE/HHE/HSE 311 by Ignatius Chikukuza

Distributed Computing - A distributed system contains multiple nodes that are physically
separate but linked together using the network. All the nodes in this system communicate
with each other and handle processes in tandem. Each of these nodes contains a small part of
the distributed operating system software.

A diagram to better explain the distributed system is –

Types of Distributed Systems

The nodes in the distributed systems can be arranged in the form of client/server systems or
peer to peer systems. Details about these are as follows −

Client/Server Systems
In client server systems, the client requests a resource and the server provides that resource.
A server may serve multiple clients at the same time while a client is in contact with only one
server. Both the client and server usually communicate via a computer network and so they
are a part of distributed systems.

Peer to Peer Systems

The peer to peer systems contains nodes that are equal participants in data sharing. All the
tasks are equally divided between all the nodes. The nodes interact with each other as required
as they share resources. This is done with the help of a network.

Advantages of Distributed Systems

 All the nodes in the distributed system are connected to each other. So nodes can
easily share data with other nodes.
 More nodes can easily be added to the distributed system i.e. it can be scaled as
required.
 Failure of one node does not lead to the failure of the entire distributed system. Other
nodes can still communicate with each other.
 Resources like printers can be shared with multiple nodes rather than being restricted

HCE/HHE/HSE 311 by Ignatius Chikukuza

 to just one.
Disadvantages of Distributed Systems
 It is difficult to provide adequate security in distributed systems because the nodes as
well as the connections need to be secured.
 Some messages and data can be lost in the network while moving from one node to
another.
 The database connected to the distributed systems is quite complicated and difficult to
handle as compared to a single user system.
 Overloading may occur in the network if all the nodes of the distributed system try to
send data at once.

Distributed Computing Models

There are certain technologies utilizing the power of distributed computing creating platforms
that are flexible, reliable and scalable. These technologies are listed below:

 Virtualization
 Service-Oriented Architecture (SOA)
 Grid Computing
 Utility Computing

Virtualization

Virtualization is a technique, which allows sharing single physical instance of an application

or resource among multiple organizations or tenants (customers). It does this by assigning a
logical name to a physical resource and providing a pointer to that physical resource when
demanded.

HCE/HHE/HSE 311 by Ignatius Chikukuza

The Multitenant architecture offers virtual isolation among the multiple tenants. Hence, the
organizations can use and customize their application as though they each have their
instances running.
.
Service-Oriented Architecture (SOA)
Service-Oriented Architecture helps to use applications as a service for other applications
regardless the type of vendor, product or technology. Therefore, it is possible to exchange the
data between applications of different vendors without additional programming or making
changes to services.

The cloud computing service oriented architecture is shown in the diagram below.

HCE/HHE/HSE 311 by Ignatius Chikukuza

Grid Computing

Grid Computing refers to distributed computing, in which a group of computers from

multiple locations are connected with each other to achieve a common objective. These
computer resources are heterogeneous and geographically dispersed.
Grid Computing breaks complex task into smaller pieces, which are distributed to CPUs that
reside within the grid.

HCE/HHE/HSE 311 by Ignatius Chikukuza

Utility Computing

Utility computing is based on Pay-per-Use model. It offers computational resources on

demand as a metered service. Cloud computing, grid computing, and managed IT services
are based on the concept of utility computing.

Examples of real-world applications

Scientific Simulations
1. Climate Modeling: Parallel computing is used to simulate complex climate models,
predicting weather patterns and climate change.
2. Fluid Dynamics: Distributed computing is used to simulate fluid dynamics, such as ocean
currents and turbulence.
3. Materials Science: Parallel computing is used to simulate the behavior of materials at the
molecular level.

Data Analytics
1. Big Data Processing: Distributed computing is used to process large datasets, such as those
generated by social media, sensors, and IoT devices.
2. Machine Learning: Parallel computing is used to train machine learning models, such as
neural networks and decision trees.
3. Data Mining: Distributed computing is used to analyze large datasets, identifying patterns
and trends.

Artificial Intelligence
1. Deep Learning: Parallel computing is used to train deep learning models, such as
convolutional neural networks (CNNs) and recurrent neural networks (RNNs).
2. Natural Language Processing: Distributed computing is used to analyze and process large
amounts of text data.
3. Computer Vision: Parallel computing is used to analyze and process visual data, such as
images and video.

Cryptography
1. Cryptanalysis: Distributed computing is used to break encryption algorithms, such as RSA
and AES.
2. Secure Communication Protocols: Parallel computing is used to implement secure
communication protocols, such as SSL/TLS.

Weather Forecasting
1. Numerical Weather Prediction: Parallel computing is used to simulate weather patterns,
predicting weather forecasts.
2. Climate Modeling: Distributed computing is used to simulate climate models, predicting
long-term climate trends.

HCE/HHE/HSE 311 by Ignatius Chikukuza

Finance and Economics
1. Risk Analysis: Distributed computing is used to analyze financial risk, predicting potential
losses and gains.
2. Portfolio Optimization: Parallel computing is used to optimize investment portfolios,
maximizing returns and minimizing risk.

Healthcare and Bioinformatics

1. Genomics: Distributed computing is used to analyze genomic data, identifying genetic
variants and predicting disease susceptibility.
2. Medical Imaging: Parallel computing is used to analyze medical images, such as MRI and
CT scans.

Other Applications
1. Gaming: Parallel computing is used to render graphics and simulate game physics.
2. Video Processing: Distributed computing is used to process video data, such as video
encoding and decoding.
3. Robotics: Parallel computing is used to control robots, simulating motion and predicting
behaviour.

HCE/HHE/HSE 311 by Ignatius Chikukuza