What is Deep Learning?

Introduction to
Module 1
Deep Learning

Learning Outcomes
By the end of this unit the learner will be able to:

 Define deep learning and understand its significance.

 Describe the history and evolution of deep learning.
 Differentiate between deep learning and other machine learning approaches.
 Explain the importance and applications of deep learning in modern society.

Copyrights © OHSC (Oxford Home Study Centre).All Rights Reserved. 1|16

What is Deep Learning?

Module 1
Introduction to Deep Learning
Overview of Deep Learning
Definition of Deep Learning
Deep learning, a subset of machine learning, has revolutionized the field of artificial
intelligence (AI) by enabling computers to learn from large amounts of data and perform
complex tasks with high accuracy. It mimics the human brain's neural networks, allowing
machines to recognize patterns, make decisions, and predict outcomes. In this section, we will
discuss in detail about the definition of deep learning, its underlying principles, and its
significance in the modern technological landscape:

Definition of Deep Learning

Neural Networks and Layers

Deep learning is based on artificial neural networks (ANNs), which are computational models
inspired by the human brain's neural structure. An ANN consists of layers of interconnected
nodes or "neurons," each representing a mathematical function. These layers can be broadly
categorized into:

 Input Layer: The first layer that receives the raw data input.

 Hidden Layers: Intermediate layers where the input data undergoes various
transformations through the weighted connections of neurons. These layers are where
the network "learns" by adjusting weights based on errors.

 Output Layer: The final layer that produces the output, such as a classification label or
a predicted value.

Output Layer

Hidden Layers

Input Layer

Fig1.1: Neural Networks and Layers

Copyrights © OHSC (Oxford Home Study Centre).All Rights Reserved. 2|16

What is Deep Learning?
In deep learning, the term "deep" refers to the presence of multiple hidden layers, enabling
the model to capture intricate patterns and relationships in the data.

Learning Process

The learning process in deep learning involves training the neural network using large
datasets. This training is typically supervised, meaning the model learns from labelled data.
The process can be summarized as follows:

1. Forward Propagation: Data is passed through the network layer by layer, with each
neuron applying a function to its inputs and passing the result to the next layer.

2. Loss Calculation: The model's output is compared to the true labels to calculate the
loss, a measure of how far the prediction is from the actual value.

3. Backpropagation: The error is propagated back through the network, adjusting the
weights of the connections to minimize the loss. This is done using optimization
techniques like gradient descent.

4. Iteration: The forward and backward propagation steps are repeated for many
iterations (epochs) until the model achieves satisfactory performance.

Activation Functions

Activation functions introduce non-linearity into the network, enabling it to learn and
represent complex patterns. Common activation functions include:

 ReLU (Rectified Linear Unit): Allows the network to handle non-linear relationships by
outputting the input directly if positive, and zero otherwise.

 Sigmoid: Squashes the input to a range between 0 and 1, useful for binary
classification.

 Softmax: Converts outputs into probabilities, commonly used in multi-class

classification problems.

Deep learning represents a significant leap in the field of AI, offering powerful techniques to
analyse and interpret complex data. By leveraging multi-layered neural networks and
sophisticated learning algorithms, deep learning models can achieve remarkable accuracy in
tasks such as image recognition, natural language processing, and autonomous driving. Its
ability to learn from vast amounts of data and improve over time makes deep learning an
indispensable tool in modern technology.

Copyrights © OHSC (Oxford Home Study Centre).All Rights Reserved. 3|16

What is Deep Learning?

History and Evolution of Deep Learning

Deep learning has emerged as a powerful subset of machine learning, enabling computers to
learn from data and perform tasks that traditionally required human intelligence. In this
section, we will discuss in detail about the history and evolution of deep learning, tracing its
development from its inception to its current state as a cornerstone of artificial intelligence
research and applications:

History and Evolution of Deep Learning

Deep learning is rooted in artificial neural networks (ANNs), which were first proposed in the
1940s as a computational model inspired by the human brain. However, it was not until the
1980s and 1990s that ANNs gained traction with advancements in computing power and
algorithms. During this period, backpropagation, a method for training neural networks, was
developed, allowing for more efficient learning.

The term "deep learning" specifically refers to neural networks with many layers (deep
networks), which became feasible to train with the availability of large datasets, powerful
GPUs, and algorithmic improvements in the mid-2000s. Key milestones include:

1. 2006 - 2010: Emergence of Deep Learning

 Geoff Hinton and collaborators demonstrated the effectiveness of deep

learning methods, such as deep belief networks and restricted Boltzmann
machines, in various applications like speech recognition and image
classification.

2. 2012: ImageNet Competition

 AlexNet, a deep convolutional neural network (CNN) developed by Alex

Krizhevsky, Ilya Sutskever, and Geoff Hinton, won the ImageNet Large Scale
Visual Recognition Challenge (ILSVRC), significantly outperforming traditional
methods and demonstrating the power of deep learning for image
classification.

3. 2014: Expansion and Application

 Deep learning gained further popularity as researchers applied it successfully

to various domains, including natural language processing (e.g., word
embedding’s, sequence-to-sequence models) and reinforcement learning.

4. Present: State-of-the-Art

 Today, deep learning is integral to many AI applications, from autonomous

driving and healthcare to finance and recommendation systems. Architectures
like transformers have advanced natural language understanding, while GANs
(Generative Adversarial Networks) have revolutionized image synthesis.

Copyrights © OHSC (Oxford Home Study Centre).All Rights Reserved. 4|16

What is Deep Learning?

2006 - 2010
• Emergence of Deep Learning

2012
• ImageNet Competition

2014
• Expansion and Application

Present
• State-of-the-Art
Fig 1.2: History and Evolution of Deep Learning

Deep learning has evolved from its theoretical origins in the 1940s to become a revolutionary
force in artificial intelligence. With its ability to automatically learn representations of data
through neural networks with many layers, deep learning has enabled unprecedented
progress in computer vision, natural language processing, and beyond. The field continues to
advance rapidly, driven by innovations in algorithms, hardware, and applications, promising
further breakthroughs and transformative impacts across industries

Importance and Applications of Deep Learning in Modern Society

Deep learning has emerged as a transformative technology in the field of artificial intelligence,
enabling computers to learn from large amounts of data and make decisions in a manner that
mimics human thought processes. In this section, we will discuss in detail about the
importance and applications of deep learning in modern society, highlighting its impact across
various domains:

Importance and Applications of Deep Learning in Modern Society

Deep learning is significant for several reasons:

1. Handling Large and Complex Data

 Deep learning excels in processing large-scale data, including unstructured and

complex data types such as images, videos, and natural language text. It can
automatically learn features from raw data, eliminating the need for manual
feature extraction.

Copyrights © OHSC (Oxford Home Study Centre).All Rights Reserved. 5|16

What is Deep Learning?
2. High Accuracy and Performance

 Deep learning algorithms have achieved state-of-the-art performance in

various tasks, surpassing traditional machine learning approaches. This
includes image and speech recognition, natural language processing, and even
complex games like Go.

3. Versatility Across Applications

 Deep learning is versatile and applicable to a wide range of domains:

 Computer Vision: Deep learning powers image and video recognition

systems, enabling applications in autonomous vehicles, medical
imaging, and security surveillance.

 Natural Language Processing (NLP): It enables machines to understand

and generate human language, facilitating applications such as
chatbots, sentiment analysis, and machine translation.

 Speech Recognition: Deep learning models have greatly improved

speech recognition accuracy, leading to advancements in virtual
assistants like Siri and Alexa.

 Recommendation Systems: Deep learning algorithms are used to

personalize recommendations in e-commerce, streaming services, and
social media platforms.

 Healthcare: Applications include disease diagnosis from medical

images, drug discovery, and personalized treatment plans.

 Finance: Deep learning is applied in fraud detection, algorithmic

trading, and risk assessment.

 Gaming and Robotics: Deep learning is used in gaming AI, robotics, and
autonomous systems for navigation and decision-making.

4. Continuous Advancements

 The field of deep learning is dynamic, with continuous research and

development leading to new architectures, algorithms, and applications.
Innovations like transformers for NLP and GANs for generative tasks have
expanded the capabilities of deep learning.

5. Impact on Society

 Deep learning is driving technological advancements that improve efficiency,

accuracy, and automation across industries, leading to economic growth and
societal benefits.

Copyrights © OHSC (Oxford Home Study Centre).All Rights Reserved. 6|16

What is Deep Learning?
Deep learning has revolutionized artificial intelligence by enabling computers to learn from
data and perform tasks that were previously only achievable by humans. Its impact spans
various domains, from improving healthcare outcomes and enhancing customer experiences
to transforming industries such as finance, transportation, and entertainment. As research in
deep learning continues to progress, the technology is expected to further enhance its
capabilities and find even more applications, solidifying its role as a cornerstone of modern

Core Concepts
Neurons and Neural Networks
Neurons and neural networks are fundamental concepts in deep learning, inspired by the
biological neural networks found in the human brain. These concepts form the building blocks
of artificial intelligence, enabling machines to learn from data and perform tasks that
traditionally required human intelligence. Understanding the structure and function of
neurons, as well as the architecture and operation of neural networks, is essential for
comprehending how deep learning models operate and achieve remarkable results across
various domains. Below we discuss in detail about this topic:

Neurons and Neural Networks: Core Concepts

1. Neurons

Neurons are the basic units of computation in neural networks, both in biological and
artificial systems.

 Biological Neurons: In the brain, neurons receive signals from other neurons
through dendrites, process these signals in the cell body (soma), and transmit
signals along the axon to other neurons through synapses.

 Dendrites: Receive signals from other neurons or from sensory organs.

 Soma (Cell Body): Integrates signals received from dendrites.

 Axon: Transmits signals to other neurons through synapses.

 Synapses: Connections between neurons where neurotransmitters are

released to pass signals.

 Artificial Neurons (Perceptrons): In artificial neural networks (ANNs), artificial

neurons are mathematical models inspired by biological neurons.

 Inputs: Each artificial neuron receives one or more inputs, represented as

numerical values.

 Weights: Each input is multiplied by a weight, which determines the

strength of the input signal.

Copyrights © OHSC (Oxford Home Study Centre).All Rights Reserved. 7|16

What is Deep Learning?
 Bias: A bias term is added to the weighted sum of inputs, helping the model
learn by allowing it to fit the data better.

 Activation Function: After computing the weighted sum of inputs and bias,
an activation function is applied to determine the output of the neuron.

2. Neural Networks

Neural networks are composed of layers of interconnected neurons and are designed to
process data in a way that mimics the human brain.

 Layers in Neural Networks

 Input Layer: Receives input data and passes it to the next layer.

 Hidden Layers: Layers between the input and output layers, where
computations occur.

 Output Layer: Produces the final output of the network.

 Connections (Edges)

 Neurons in adjacent layers are connected by edges, which have weights that
determine the strength of the connection between neurons.

 Activation Function

 Each neuron applies an activation function to the weighted sum of its inputs
and bias, introducing non-linearity into the network.

 Common activation functions include:

 Sigmoid: S-shaped function that outputs values between 0 and 1,

useful for binary classification problems.

 ReLU (Rectified Linear Unit): Outputs the input directly if positive,

otherwise outputs zero, making it computationally efficient.

 Tanh: S-shaped function that outputs values between -1 and 1, useful

for binary classification problems.

3. Training Neural Networks

Training neural networks involves optimizing their weights and biases so that the network
can learn to map inputs to outputs accurately.

 Forward Propagation

 During forward propagation, input data is passed through the network, with
each layer performing its computations to produce a prediction.

Copyrights © OHSC (Oxford Home Study Centre).All Rights Reserved. 8|16

What is Deep Learning?
 Loss Function

 A loss function measures the difference between the predicted output and the
actual target output.

 Backpropagation:

 Backpropagation is a crucial algorithm for training neural networks.

 It calculates the gradient of the loss function with respect to each weight in the
network, allowing the weights to be adjusted in a way that reduces the error
in prediction.

 Optimization Algorithms

 Optimization algorithms like gradient descent are used to update the weights
and biases of the network based on the gradients computed during
backpropagation.

 Variants of gradient descent include stochastic gradient descent (SGD), mini-

batch gradient descent, and Adam optimizer.

4. Types of Neural Networks

Neural networks can be categorized into different types based on their structure and use
cases.

 Feedforward Neural Networks (FNNs)

 The simplest type of neural network where information flows in one direction,
from input to output.

 Commonly used for tasks such as image classification, speech recognition, and
financial forecasting.

 Convolutional Neural Networks (CNNs)

 Specifically designed for processing grid-like data, such as images and videos.

 Consist of convolutional layers that automatically learn spatial hierarchies of

features.

 Widely used in tasks like image classification, object detection, and image
segmentation.

 Recurrent Neural Networks (RNNs)

 Suitable for sequential data, with connections between units forming directed
cycles, allowing for feedback loops.

Copyrights © OHSC (Oxford Home Study Centre).All Rights Reserved. 9|16

What is Deep Learning?
 Effective for tasks such as natural language processing (NLP), speech
recognition, and time series prediction.

 Long Short-Term Memory (LSTM) Networks

 A type of RNN designed to address the vanishing gradient problem, enabling it

to capture long-term dependencies in data.

 Used in applications such as speech recognition, language modelling, and

handwriting recognition.

5. Applications of Neural Networks

Neural networks have been applied to various domains and have demonstrated impressive
performance in numerous tasks.

 Computer Vision

 Image classification: Distinguishing objects within images.

 Object detection: Locating and classifying objects within images or videos.

 Image segmentation: Dividing images into segments to simplify analysis.

 Natural Language Processing (NLP)

 Sentiment analysis: Determining the sentiment expressed in text data.

 Machine translation: Translating text from one language to another.

 Named entity recognition: Identifying named entities such as persons,

organizations, or locations in text.

 Speech Recognition

 Converting spoken language into text, enabling applications such as virtual

assistants and transcription services.

 Healthcare

 Disease diagnosis from medical images, such as X-rays and MRI scans.

 Drug discovery and personalized medicine.

 Predicting patient outcomes based on electronic health records.

 Finance

 Stock market prediction and algorithmic trading.

 Fraud detection and credit scoring.

Copyrights © OHSC (Oxford Home Study Centre).All Rights Reserved. 10 | 1 6

What is Deep Learning?
 Gaming

 AI opponents in video games, adjusting difficulty levels based on player

performance.

 Game strategy optimization and simulation.

6. Ethical and Social Implications

As neural networks become more integrated into society, it's crucial to consider their
ethical implications:

 Bias in Data: Neural networks can perpetuate biases present in training data, leading
to biased decision-making in applications like hiring and lending.

 Privacy Concerns: Neural networks may process sensitive personal data, raising
concerns about data security and privacy.

 Job Displacement: Automation driven by neural networks may lead to job

displacement in certain sectors, necessitating retraining and job creation in new fields.

 Transparency and Accountability: Ensuring that neural networks are transparent in

their decision-making processes and holding developers accountable for their
algorithms' outcomes.

Neurons

Neural Networks

Training Neural
Networks

Types of Neural
Networks

Applications of
Neural Networks

Ethical and Social

Implications

Fig 1.3: Neurons and Neural Networks: Core Concepts

Copyrights © OHSC (Oxford Home Study Centre).All Rights Reserved. 11 | 1 6

What is Deep Learning?
Neurons and neural networks are foundational concepts in deep learning, enabling machines
to learn and make decisions in a way that mimics human thought processes. From the basic
structure of neurons to the complex architectures of neural networks, these concepts are
essential for understanding how deep learning models operate and achieve impressive results
across various domains. Understanding these core concepts is crucial for anyone interested in
exploring the capabilities and potential of deep learning in solving complex real-world
problems.

Deep vs. Shallow Learning

Deep learning and shallow learning are two distinct approaches to machine learning,
characterized by the depth of the models used in each method. Understanding the differences
between deep and shallow learning is essential for comprehending their respective strengths,
limitations, and applications in various domains. Below we discuss in detail about this topic:

Deep vs. Shallow Learning: Core Concepts

1. Shallow Learning

Shallow learning refers to traditional machine learning methods that typically involve
models with only a few layers (often one or two layers) of computational units. These
models are simpler in structure compared to deep learning models and do not involve a
deep hierarchy of layers.

Characteristics of Shallow Learning

 Feature Engineering: Shallow learning relies heavily on feature engineering, where

domain knowledge is used to manually extract relevant features from raw data.
These features are then fed into the learning algorithm.

 Linear Models: Common algorithms used in shallow learning include linear

regression, logistic regression, support vector machines (SVM), and decision trees.

 Less Computational Complexity: Shallow models are computationally less intensive

compared to deep learning models, making them faster to train and less resource-
demanding.

 Interpretable Results: Shallow models often provide more interpretable results

because the relationships between features and the output are more
straightforward.

 Limited to Moderate-Scale Problems: Shallow learning is effective for tasks where

the data has a relatively simple structure and the relationships between features
and the target variable are not too complex.

Examples of Shallow Learning Algorithms

Copyrights © OHSC (Oxford Home Study Centre).All Rights Reserved. 12 | 1 6

What is Deep Learning?
 Logistic Regression: A linear model used for binary classification problems.

 Support Vector Machines (SVM): Effective for both classification and regression tasks,
especially when the data is not linearly separable.

 Decision Trees: Non-linear models that partition the data into subsets based on
different attributes.

Shallow Deep
Learning Learning
Hierarchical
Feature
Feature
Engineering
Learning

Deep Neural
Linear Models Networks
(DNNs)

Less Non-linear
Computational Activation
Complexity Functions

High
Interpretable
Computational
Results
Complexity

Limited to State-of-the-
Moderate- Art
Scale Problems Performance

Fig 1.4: Deep vs. Shallow Learning: Core Concepts

2. Deep Learning

Deep learning, on the other hand, refers to a class of machine learning techniques based
on artificial neural networks (ANNs) that are composed of many layers of computational
units. These networks are capable of learning hierarchical representations of data, with
each layer of neurons learning progressively more abstract features.

Characteristics of Deep Learning

 Hierarchical Feature Learning: Deep learning models automatically learn

hierarchical representations of data, eliminating the need for manual feature
engineering.

 Deep Neural Networks (DNNs): Deep learning models consist of multiple layers of
neurons, where each layer learns features at different levels of abstraction.

Copyrights © OHSC (Oxford Home Study Centre).All Rights Reserved. 13 | 1 6

What is Deep Learning?
 Non-linear Activation Functions: Deep networks typically use non-linear activation
functions (e.g., ReLU, sigmoid, tanh) to introduce non-linearity, allowing the models
to learn complex mappings between inputs and outputs.

 High Computational Complexity: Training deep learning models requires significant

computational resources, including GPUs, due to their large number of parameters
and complex architectures.

 State-of-the-Art Performance: Deep learning has achieved state-of-the-art

performance in various tasks, including image and speech recognition, natural
language processing, and game playing.

Examples of Deep Learning Architectures

 Convolutional Neural Networks (CNNs): Specifically designed for processing spatial

data like images and videos, with convolutional layers that automatically learn spatial
hierarchies of features.

 Recurrent Neural Networks (RNNs): Suitable for sequential data, with connections
between units forming directed cycles, allowing for feedback loops.

 Long Short-Term Memory (LSTM) Networks: A type of RNN designed to address the
vanishing gradient problem, enabling it to capture long-term dependencies in data.

Key Differences between Deep and Shallow Learning

1. Representation Learning

 Shallow learning relies on feature engineering, where features are manually

crafted by experts.

 Deep learning performs automatic feature learning, where features are

learned from data directly.

2. Model Complexity

 Shallow learning models are simpler, with fewer layers and parameters.

 Deep learning models are more complex, with multiple layers and a large
number of parameters.

3. Computational Requirements

 Shallow learning models are less computationally intensive and can be trained
on simpler hardware.

 Deep learning models require more computational power, often necessitating

GPUs or specialized hardware for training.

Copyrights © OHSC (Oxford Home Study Centre).All Rights Reserved. 14 | 1 6

What is Deep Learning?
4. Task Complexity

 Shallow learning is suitable for tasks with simpler data representations and
fewer complex patterns.

 Deep learning excels in tasks with complex data representations and intricate
patterns, such as image recognition and natural language understanding.

5. Interpretability

 Shallow learning models often provide more interpretable results because of

their simpler structure.

 Deep learning models, especially those with many layers, are more complex
and less interpretable, often described as "black-box" models.

Applications of Deep Learning

Deep learning has been successfully applied to various domains and tasks, including:

 Computer Vision: Image classification, object detection, image segmentation.

 Natural Language Processing (NLP): Sentiment analysis, machine translation, text

generation.

 Speech Recognition: Speech-to-text conversion, voice assistants.

 Healthcare: Disease diagnosis from medical images, drug discovery, personalized

treatment plans.

 Finance: Stock market prediction, fraud detection.

 Gaming: AI opponents, game strategy optimization.

 Autonomous Vehicles: Object detection, path planning, real-time decision-making.

Deep learning and shallow learning represent two distinct paradigms within the field of
machine learning, characterized by the depth of the models used. Shallow learning relies on
simpler models and feature engineering, making it suitable for tasks with less complex data
representations. In contrast, deep learning leverages deep neural networks with multiple
layers to automatically learn hierarchical representations of data, enabling it to excel in tasks
with complex patterns and large-scale data. Understanding the differences between deep and
shallow learning is crucial for selecting the appropriate approach for different applications and
domains, taking into account factors such as data complexity, interpretability, computational
resources, and task requirements.

Deep learning represents a transformative branch of artificial intelligence, characterized by its

ability to process vast amounts of data and learn complex patterns. Originating from the study
of artificial neural networks, deep learning has evolved rapidly, fuelled by advancements in

Copyrights © OHSC (Oxford Home Study Centre).All Rights Reserved. 15 | 1 6

What is Deep Learning?
computing power and the availability of big data. Its applications span a wide range of fields,
including image and speech recognition, natural language processing, autonomous vehicles,
and healthcare diagnostics, among others. The fundamental components of deep learning,
such as neurons and neural networks, form the basis for its sophisticated learning capabilities,
distinguishing it from shallow learning methods. As deep learning continues to advance, it
promises to further revolutionize industries, enhance decision-making processes, and
contribute to the development of innovative technologies.

Further Reading:

 AI Foundations of Deep Learning: Easy To Read Guide Introducing the

Foundations Of Deep Learning and AI Book 2 of 9: AI Foundations | by Jon
Adams | Jan 30, 2024

 Introduction to Deep Learning: From Logical Calculus to Artificial Intelligence

(Undergraduate Topics in Computer Science) Part of: Undergraduate Topics in
Computer Science (113 books) | by Sandro Skansi | Feb 6, 2018

Copyrights © OHSC (Oxford Home Study Centre).All Rights Reserved. 16 | 1 6