Multimedia

Computing
C O N C EPT S & D EF IN IT IONS

Abdulmotaleb El Saddik
FRSC, FIEEE, FCAE, FEIC
Distinguished Professor
https://www.instagram.com/le.prof/ Multimedia Communications Research Lab
www.mcrlab.net
https://www.twitter.com/aelsaddik www.elsaddik.net

https://www.linkedin.com/in/abdulmotaleb-el-saddik-1a58a79/
Outline
• What is Multimedia
• The Convergence of Multimedia Technologies
• Multimedia Data Sources
• Multimodal Interactions
• Multimedia Big Data
• Artificial Intelligence for Multimedia
• Multimedia Communications and Networking
• Multimedia Technologies Convergence and
Cybersecurity
• Applications of Multimedia Technologies

© 2002 Multimedia Communications research Laboratory (MCRLab) 2

Objectives of the Course
• At the end of the course, YOU will be able to:
– Have an excellent understanding of Multimedia:
• Technologies
• Services
• Applications
– Understand the different traditional and emerging Media
– Master basic networking concepts and protocols
– Understand how multimedia and networking (communications)
play together
– Master the Digital Twin concept, and the convergence of its
enabling technologies

© 2002 Multimedia Communications research Laboratory (MCRLab) 3

© 2002 Multimedia Communications research Laboratory (MCRLab) 4
Definition: Multimedia
• Notion
– Multi
• Many
– Media
• "Things in the middle"

• These media can be in the form of:

• Graphics, animation, audio, video, text, haptics, smell, etc.
– by
• Computer, TV, phone, etc.

© 2002 Multimedia Communications research Laboratory (MCRLab) 5

Multimedia Applications
• Multimedia application can be either linear or
non-linear
• Linear
– Not interactive
– Starting at a beginning and running through the end
• Eg. Montage on TV

• Non-linear
– Interactive
– Ability of end user to interact and control the application
• Eg. Games software

© 2002 Multimedia Communications research Laboratory (MCRLab) 6

Multimedia Interpretations
• Perception: How do humans perceive Information?
– aural sense, visual sense
• Representation (Computer Science): How is information coded?
– ASCII, GIF, JPEG, ...
• Presentation (in Electronics): Which ‘medium’ is used to convey (I/O) info?
– paper, monitor, loudspeaker, ..
• Storage (for Publishers): Where is information stored?
– paper, Hard disk, CD-ROM, ...
• Transmission (in Data Communication): Which ‘medium’ is used to transmit
info.?
– coaxial, optical fiber, ..
• Distribution (for Content Providers): Which ‘medium’ is used to exchange info?
– CD, JAZ-Drives, optical fiber, ...
• Dissemination (in Public): which ‘medium’ conveys info to the public?
– Press, TV, Radio, (coming: Internet): term "mass media" coined in 1923

© 2002 Multimedia Communications research Laboratory (MCRLab) 7

Multimedia Systems
This is
Combination of:
Y plain text

Spread
Sheet

Discrete Media

&

Y
T
Continuous Media

X
© 2002 Multimedia Communications research Laboratory (MCRLab) 8
Multimedia Communications
• MULTIMEDIA COMMUNICATIONS is the
field referring to the:
– Representation,
– Storage,
– Retrieval, and
– Dissemination
• of
– Machine-processable information expressed in multiple media
such as:
• Text, Voice, Graphics, Images, Animations, Audio, Video, smell and touch

© 2002 Multimedia Communications research Laboratory (MCRLab) 9

Multimedia is everywhere

MULTIMEDIA

DATA PRESENTATION
Used in almost every field: Education; Entertainment; Marketing; Healthcare; Military, etc.

© 2002 Multimedia Communications research Laboratory (MCRLab) 10

Multimedia Content
• Exists in different formats, we are aware o:
– Text, Audio and Video
• Content processing devices can
– Capture
– Access
– Play / Display
– Interact with
the multimedia content

© 2002 Multimedia Communications research Laboratory (MCRLab) 11

Definition: Haptics
• Haptics: a technology that can create an experience
of touch. This includes vibrations, forces, motion,
etc.

Device Actuator

Haptic jacket
Haptic Force-Feedback

Haptic Arm Band Haptic Glove

Vibrating Sofa

El Saddik, A., Orozco, M., Eid, M., & Cha, J. (2011). Haptics technologies: Bringing touch to multimedia. Springer.

© 2002 Multimedia Communications research Laboratory (MCRLab) 12

Multimedia content

Social
IoT Networks

Medical Finance

© 2002 Multimedia Communications research Laboratory (MCRLab) 13

Multimedia Data Sources
• Sensors (Hardware sensors):
A sensor is a device which detects or measures a physical
property and sends the information to other electronics.
• Increasing use of sensors:

Sensors

Actuators

IOT

© 2002 Multimedia Communications research Laboratory (MCRLab) 14

Examples of Sensors
• Just in a smartphone:
– Global Positioning System (GPS)
– Accelerometer
– Sound Sensor (Microphone)
– Magnetometer (compass)
– Proximity sensor
– Ambient Light Sensor
– Barometer Sensor
Sensors – Touchscreen Sensors
– Fingerprint Sensor
– Heart Rate Sensor
Actuators
– Pedometer
IOT – Barcode Sensor
– Gyroscope
– More ! (Thermometer, Air humidity, Geiger Counter

© 2002 Multimedia Communications research Laboratory (MCRLab) 15

Multimedia Data Sources (cont.)
• Hard sensors: defined earlier, produce part of the
data
• Soft Sensors: we include here data provided by
humans, such as through social media.

Sensors

Actuators

IOT

© 2002 Multimedia Communications research Laboratory (MCRLab) 16

Social Media as a Data Source
• Ever-increasing use of social media around the
world
• The most popular online activity
• Examples:
– Facebook has >2.5 billion monthly active users
– Instagram has >1 billion monthly active users

 High amounts of data from social media is made

available everyday.

© 2002 Multimedia Communications research Laboratory (MCRLab) 17

Multimedia Data Sources
• Sensors (Hardware sensors):
A sensor is a device which detects or measures a physical
property and sends the information to other electronics.
• Increasing use of sensors:

© 2002 Multimedia Communications research Laboratory (MCRLab) 18

Examples of Sensors
• Just in a smartphone:
– Global Positioning System (GPS)
– Accelerometer
– Gyroscope
– Magnetometer (compass)
– Proximity sensor
– Ambient Light Sensor
– Barometer Sensor
– Sound Sensor (Microphone)
– Touchscreen Sensors
– Fingerprint Sensor
– Heart Rate Sensor
– Pedometer
– Barcode Sensor
– More ! (Thermometer, Air humidity, Geiger Counter..)
© 2002 Multimedia Communications research Laboratory (MCRLab) 19
Multimedia Data Sources (cont.)
• Hard sensors: defined earlier, produce part of the
data
• Soft Sensors: we include here data provided by
humans, such as through social media.

© 2002 Multimedia Communications research Laboratory (MCRLab) 20

Social Media as a Data Source
• Ever-increasing use of social media around the
world
• The most popular online activity
• Examples:
– Facebook has 2.37 billion monthly active users
– Instagram has 1 billion monthly active users

 High amounts of data from social media is made

available everyday.

© 2002 Multimedia Communications research Laboratory (MCRLab) 21

Multimedia Intelligence

DL

ML

Artificial
Intelligence

© 2002 Multimedia Communications research Laboratory (MCRLab) 22

MM Artificial Intelligence
Intelligence
• Content Analysis and Understanding:
– This involves the automatic extraction of meaningful information from multimedia
content. Techniques like image recognition, video analysis, and natural language
processing are used to identify and categorize elements within the media, such as
objects in images, scenes in videos, or sentiments in text.
• Content Creation and Manipulation:
– AI-driven generation and modification of multimedia content, such as creating
realistic images, videos, or audio recordings.
• Multimedia Integration and Fusion:
– The combination of different types of media and information sources to provide a
more comprehensive understanding or experience. For example, integrating text,
images, and audio to create immersive educational content.
• Intelligent Multimedia Systems:
– Development of smart applications and systems that can autonomously perform
tasks like surveillance, content recommendation, or automated video editing.

© 2002 Multimedia Communications research Laboratory (MCRLab) 23

Multimedia Interactions
Audio,
Video
AR, VR

Robotics/Haptics

MMI

© 2002 Multimedia Communications research Laboratory (MCRLab) 24

Conveying Information: User Interaction

1. Through software actuation:

– For example: displaying a message, or notification
2. Through hardware actuation:
– Definition: An actuator is a device which takes as an input a
control command and outputs a change in the physical
environment
• Actuators can generate motion, force, heat,
flow, etc…
• For example:
– actuators can provide motion to a robot
– actuators can also produce motion of a automatic sliding door

© 2002 Multimedia Communications research Laboratory (MCRLab) 25

Multimodal Interactions
• Multiple modalities.
• A modality here is a channel by which
information is communicated between
the computer and the human
• Multimodal interaction: The use of
more than one modality to
communicate.
• Examples:
– Use of audio, video, 3D graphics, etc..
– Use of graphics and haptics

© 2002 Multimedia Communications research Laboratory (MCRLab) 26

Multimedia Networking

Communication: BT, Wi-

Fi, 5G, 6G, ...

© 2002 Multimedia Communications research Laboratory (MCRLab) 27

Multimedia and Networking

• Communicate the multimedia content over

a network
– Fast transmission
– High volumes of data
– Fast feedback
– Synchronization
– Compression
– Scalability

© 2002 Multimedia Communications research Laboratory (MCRLab) 28

Multimedia Security
Cybersecurity

© 2002 Multimedia Communications research Laboratory (MCRLab) 29

Security in MM Applications
• Security challenges concerning large scale
multimedia data, and cyber-enabled
multimedia applications:
– Authentication & Access Control
– Data Confidentiality & Data Integrity
– Encryption
– Digital Rights Management (DRM)
– Watermarking
– Secure Protocols
– Privacy protection

© 2002 Multimedia Communications research Laboratory (MCRLab) 30

Convergence of MM Technologies

Audio,
Video Cybersecurity
AR, VR
Haptics

Multimodal DL

?
Interaction
ML

Sensors Artificial
Intelligence

Actuators
Communication: BT, Wi-
Fi, 5G, 6G...
Internet of
Things

Leveraging Multimedia Communications Technologies

 For a better quality of life and well-being
© 2002 Multimedia Communications research Laboratory (MCRLab) 31
Convergence of MM Technologies

Audio,
Video Cybersecurity
AR, VR
Haptics

Multimodal DL
Interaction
Digital
Twin ML

Sensors Artificial
Intelligence

Actuators
Communication: BT, Wi-
Fi, 5G, 6G...
Internet of
Things

Leveraging Multimedia Communications Technologies

 For a better quality of life and well-being
© 2002 Multimedia Communications research Laboratory (MCRLab) 32
Digital Twin

© 2002 Multimedia Communications research Laboratory (MCRLab) 33

Digital Twin

© 2002 Multimedia Communications research Laboratory (MCRLab) 34

Definition
A digital twin is a digital replica
of a living or non-living
physical entity1.
By bridging the physical and the virtual world, data is
transmitted seamlessly allowing the virtual entity to exist
simultaneously with the physical entity.

“a digital twin facilitates the means to monitor, understand,

and optimize the functions of the physical entity and provides
continuous feedback to improve quality of life and well-being. A
digital twin is hence the convergence of several technologies,
such as AI, AR/VR and haptics, IoT, cybersecurity and
communication networks.”

1 El Saddik, A. (2018). Digital Twins: The Convergence of Multimedia Technologies. IEEE MultiMedia, 25(2), 87-92.

© 2002 Multimedia Communications research Laboratory (MCRLab) 35

 What makes the Digital Twin unique ?

© 2002 Multimedia Communications research Laboratory (MCRLab) 36

DT Convergence of MM Tech

Cybersecurity
(including biometric,
Data Social
privacy)
IoT Networks

Medical Finance

AI Machine MMI
Learning Audio/
Cognition AR/VR
Video
Ontology QoE-powered
Deep Communications
Learning Holograms Haptics/
(BT, Wi-Fi, 4G,
Rules H-Robots
5G, etc.)

© 2016, Prof. A. El Saddik, elsaddik@uottawa.ca, reproduce with permission

© 2002 Multimedia Communications research Laboratory (MCRLab) 37

Opportunities

© 2002 Multimedia Communications research Laboratory (MCRLab) 38

Digital Twin Benefits in Industry
• Reducing costs
– Predict failure and reduce maintenance cost
• Reducing production time
• Improving efficiency
• Predict opportunities and benefit from them

https://www.youtube.com/watch?v=2dCz3oL2rTw&t=111s

© 2002 Multimedia Communications research Laboratory (MCRLab) 39

Current DT Applications
• The manufacturing industry has the highest
number of applications.
• Applications of DT starting in the domains:
– Aerospace,
– Energy,
– Automotive,
– Agriculture.
–…

© 2002 Multimedia Communications research Laboratory (MCRLab) 40

DT Applications Example 1
• In the manufacturing industry:
– Operational data is gathered overtime, either for
the product or the process
– Example: engine manufacture
• Real time analysis of engine speed
• Fuel consumption
• Temperature monitoring

© 2002 Multimedia Communications research Laboratory (MCRLab) 41

DT Applications Example 2
• In the Automotive industry:
– Vehicle design
• Integrating data from previous years features usage, customer feedback,
components breaking…
– Vehicle manufacturing
• See previous slide
– Vehicle sales
• Digitally enabling cars features, depending on customers request
– Vehicle service
• Keeping records of service history by DT of the vehicle, even if the
vehicle is moved to a new area, or has a new ownership

© 2002 Multimedia Communications research Laboratory (MCRLab) 42

DT: Redefined by Gartner
• The redefinition of DT by El Saddik in 2018 has been matched by
Gartner in their “Top 10 Strategic Technology Trends for 2019”[1].
• The DT: one of the top 10 strategies for three years in a
row [1] where digitals twins are referred to as:
“digital representations of people, processes and things”.
• Humans: now one of the focuses of DT works.

[1] M. Kerremans, B. Burke, D. Cearley, and A. Velosa, “Top 10 Strategic Technology Trends for 2019: Digital Twins,” 2019.

© 2002 Multimedia Communications research Laboratory (MCRLab) 43

• In the rest of this course, the DT refers mainly to the
digital twin of a human real twin (RT).

© 2002 Multimedia Communications research Laboratory (MCRLab) 44

Analogy between DT and RT

Figure published in: El Saddik, A. (2018). Digital Twins: The Convergence of Multimedia Technologies. IEEE MultiMedia, 25(2), 87-92.

© 2002 Multimedia Communications research Laboratory (MCRLab) 45

Characteristics of DT
• Unique identifier. Digital twins would each have a unique identifier
in order to communicate with their twin.
• Sensors and actuators. Real twins could be equipped with sensors
so that digital twins could replicate their senses—sight, hearing, taste,
smell, and touch—using the appropriate actuators, depending on
application needs.
• AI. Digital twins are equipped with a controller embedded with
ontologies, machine learning, and deep learning techniques in order to
make fast and intelligent decisions on behalf of their real twin.
• Communication. Digital twins should be able to interact in near real
time with the environment, real twins, and/or other digital twin.
Communication, including the sense of touch (haptics), must occur
within 1 ms and thus must follow 5G and Tactile Internet standards.

© 2002 Multimedia Communications research Laboratory (MCRLab) 46

Characteristics of DT (cont.)
Multimodal Interaction. Digital twins could have a virtual
representation as a 3D avatar, hologram, or even a humanoid social
robot but it could also be software components without a tangible
representation, depending on the application.

Privacy and security. Digitals twins should be able to protect the

identity and privacy of their real twin. This would require the use of
advanced cryptography algorithms and biometrics techniques
(ECG-biometrics, haptic biometrics, and so on) as well as the
resolution of regulatory and political issues.

© 2002 Multimedia Communications research Laboratory (MCRLab) 47

Ecosystem of the Digital Twin for health and well-being.

Published in IEEE MMTC Communications – Frontiers. “Dtwins: A Digital Twins Ecosystem For Health
And Well-Being” by El Saddik et al.
© 2002 Multimedia Communications research Laboratory (MCRLab) 48
DT for quality of life
• What are the possible benefits?
– For example, a stroke could be predicted before it occurs,
enabling preventive steps to be taken.
– Machine and deep learning techniques could also be used
to detect lifestyle patterns and predict potential health
problems.
– In addition, contextual data such as information about the
environment, age, emotional state, and preferences could
be gathered and analyzed to fully understand and
characterize a user’s holistic condition.

© 2002 Multimedia Communications research Laboratory (MCRLab) 49

Applications of DT - 1

• Digital Immortality:
– Through its AI evolution,
• your DT can interact with your relatives and friends decide what info
to give to whom.
• It can also decide when to delete itself and hence delete the
complete e-legacy of the real twin

• Well-being:
– DTs can play an important role in well-being. In terms of stress relief, the
digital twin can detect stress level using sensory technology and can
determine the causes for stress.

© 2002 Multimedia Communications research Laboratory (MCRLab) 50

Applications of DT - 2

• Relationships:
– With digital twins, people will automatically receive information regarding
how their relationships affect their well-being. Digital twins can also
simulate real-life situations giving the real twins insights on how a friend or
colleague will react to a certain conversation or situation.

• Health:
– DT can show what is happening inside their RT’s body.
– predict the occurrence of an illness by analyzing the symptoms
– DT has all the necessary elements for the prediction:
• personal history, health related and non-health related
• Context: location, time, activities…

© 2002 Multimedia Communications research Laboratory (MCRLab) 51

Additional Readings
• Must read: “Digital Twins: The Convergence of Multimedia
Technologies” by El Saddik.
https://ieeexplore.ieee.org/document/8424832

• Suggested: “C2PS: A Digital Twin Architecture Reference

Model for the Cloud-Based Cyber-Physical Systems” by Alam
and El Saddik.
https://ieeexplore.ieee.org/document/7829368

© 2002 Multimedia Communications research Laboratory (MCRLab) 52

© 2002 Multimedia Communications research Laboratory (MCRLab) 53