8/16/2021
Data and Types of Data
Data Science
• Multi-disciplinary field that uses scientific methods,
processes, algorithms and systems to extract
knowledge and insight from structured and
unstructured data
• Central concept is gaining insight from data
• Machine learning uses data to extract knowledge
Data Modeling Inference
Data Collection (Machine
Learning)
Data Preprocessing
Data
Feature
Database Cleaning and
Representation
Cleansing
2
1
8/16/2021
Data Science
• Multi-disciplinary field that uses scientific methods,
processes, algorithms and systems to extract
knowledge and insight from structured and
unstructured data
• Central concept is gaining insight from data
• Machine learning uses data to extract knowledge
Data Modeling Inference
Data Collection (Machine
Learning)
Data Preprocessing
Data
Feature
Database Cleaning and
Representation
Cleansing
3
Data Collection
• Data manifests itself in many different forms
• Different forms of data require different ways to
collect them and different storage solutions
• Collection of data may consists of sending out
surveys, polls or doing other experiments
• Data based on the way it is collected:
– Data that comes from surveys
• Usually textual form of data or mixed
2
8/16/2021
Data Collection
• Data manifests itself in many different forms
• Different forms of data require different ways to
collect them and different storage solutions
• Collection of data may consists of sending out
surveys, polls or doing other experiments
• Data based on the way it is collected:
– Data that comes from surveys
• Usually textual form of data or mixed
– Data entered in a database as system entry
• E.g. Student information entered on academic automation
system etc.
– Data in the form of signals (comes from sensors)
• Speech/Audio, Images and videos, Temperature readings,
Humidity, Seismic data, EEG (all bio-type signals) etc.
• According to the objective of the task, the way the
data is collected will change
Types of Data: Based on Organization
1. Unstructured data:
– Rawest form of data
– Example: Any type of files like texts, images, sounds or
videos etc.
– This type of data stored in a repository of files
• Well organised directories on the computer hard drive
3
8/16/2021
Types of Data: Based on Organization
2. Structured data:
– It is a tabular data (rows and columns), which are very
well defined
– Stored in databases
• Spreadsheets [Comma Separated Value (CSV) format]
• Oracle
• DB2
• MySQL etc.
7
Types of Data: Based on Organization
3. Semi-Structured data:
– Anywhere between unstructured and structured data
– A consistent format is defined, however there is no strict
structure and parts of data may be incomplete or
different type
– Example: Data in the form of XML and JSON
• Stored in document oriented databases
4
8/16/2021
Types of Data: Based on Organization
3. Semi-Structured data:
– Anywhere between unstructured and structured data
– A consistent format is defined, however there is no strict
structure and parts of data may be incomplete or
different type
– Example: Data in the form of XML and JSON
• Stored in document oriented databases
Type of Data: Based on Variables
(Value) found in Data
• Mainly in Structured Data:
1. Numerical data:
– Data represented as numbers
– Data in which information is measurable
– This type of data is called quantitative data as its
describes a quantity
– Two types based on the values taken:
• Continuous valued data:
– Numbers does not have logical end
– Range lies in the natural limit of what we are measuring
– Example: Cost of the books, atmospheric temperature etc.
• Discrete valued data:
– Numbers have logical end
– There is a specific limit on the range of the values
– Example: number of members of family, number of days in a
month, number of colours in flag etc.
10
5
8/16/2021
Type of Data: Based on Variables
(Value) found in Data
2. Categorical data:
– Data that is not a number. It can be string of text or
date
– It describe an item or event to one of few different
categories
– Example: Ethnicity, gender, eye colour, etc.
– This type of data is called qualitative data as its
describes a quality
– Three types values they hold:
• Ordinal values: Values that have a set order to them
– Example: Severity of a alarm as “Critical”, “Medium” and
“”Low”, Ranking of a running race as “ First”, Second”, Third”
• Nominal values: Values that have no set order to them
– Example: Values for the variables “Marital Status”, “Country”,
“Eye Colour” etc.
• Binary values: Special type of categorical data
– Have only two values – “Yes” and “No” OR “True” and “False”
OR “1” and “0” 11
Type of Data: Based on Variables
(Value) found in Data
3. Time series data:
– Series of data. It involve time and some kind of value
– Example: Temperature at every hour
– It is clearly structured and numeric in nature
– Special case of numerical data
– This type of data is important because of IoT and
sensors
– Data from sensors are almost always time-series in
nature
12
6
8/16/2021
Data, Types of Data and Data
Collection using Sensors
Need for Data Preprocessing
Summery of Previous Class:
Data Science
• Multi-disciplinary field that uses scientific methods,
processes, algorithms and systems to extract
knowledge and insight from structured and
unstructured data
• Central concept is gaining insight from data
• Machine learning uses data to extract knowledge
Data Modeling Inference
Data Collection (Machine
Learning)
Data Preprocessing
Data
Feature
Database Cleaning and
Representation
Cleansing
14
7
8/16/2021
Summery of Previous Class:
Types of Data: Based on Organization
1. Unstructured data:
2. Structured data:
– It is a tabular data (rows and columns), which are very
well defined
– Each row is finite ordered list (sequence) of elements,
where each element in a column is belonging to an
attribute of specific type
– Example: Spreadsheets [Comma Separated Value
(CSV) format]
3. Semi-structured data:
15
Summery of Previous Class:
Type of Data: Based on Variables (Value)
found in Data
• Mainly in Structured Data:
1. Numerical data:
– Two types based on the values taken:
• Continuous valued data:
• Discrete valued data:
2. Categorical data:
– Three types values they hold:
• Ordinal values:
• Nominal values:
• Binary values:
3. Time series data:
16
8
8/16/2021
Summery of Previous Class:
Data Collection
• Data manifests itself in many different forms
• Different forms of data require different ways to
collect them and different storage solutions
• Collection of data may consists of sending out
surveys, polls or doing other experiments
• Data based on the way it is collected:
– Data that comes from surveys
• Usually textual form of data or mixed
– Data entered in a database as system entry
• E.g. Student information entered on academic automation
system etc.
– Data in the form of signals (comes from sensors)
• Speech/Audio, Images and videos, Temperature readings,
Humidity, Seismic data, EEG (all bio-type signals) etc.
• According to the objective of the task, the way the
data is collected will change
Data Collection from Sensors
• Sensors are the devices that respond to the
environment around it and convert the physical
parameters into a signal (e.g., optical, electrical,
mechanical ) suitable for processing
Surrounding signal
Sensor electrical,
environment
optical or
mechanical
• Example: a temperature sensor outputs an electrical
signal whose voltage or current can be used to
identify the temperature around it
• Sensors can be an electrical/mechanical component, a
module or a subsystem
18
9
8/16/2021
Different Types of Sensors
• Acoustic, sound sensors (e.g., microphone)
• Visual sensors (e.g. cameras)
• Environmental sensors (e.g., temperature, humidity,
pressure etc.)
• Chemical sensors (e.g., Diesel Nitrogen Oxide (Nox)
sensors to measure engine-out NOx gas concentration)
• Flow sensors (e.g., water flow sensors)
• Motion sensors (e.g., gyroscope)
• Proximity or presence sensor (e.g., Passive Infrared
(PIR) )
• Biosensors (e.g., glucose monitor)
• And many more …
19
IIT Mandi Weather Station: Environmental Data
(Temperature, Humidity, Pressure etc) Collection
High-Level Overview
Zigbee/802.15.4
network
Environmental Sensor
SQL Database
+ PHP (web
access)
IITMandi
intranet
This is running
inside a
Raspberry Pi
WiFi/802.11b
Source: Dr. Siddhartha Sarma 20
10
8/16/2021
High-Level Overview: Environmental Data
(Temperature, Humidity, Pressure etc) Collection
Zigbee/802.15.4
network
Environmental Sensor
SQL Database
+ PHP (web
access)
IITMandi
intranet
This is running
inside a
Raspberry Pi
WiFi/802.11b
Source: Dr. Siddhartha Sarma 21
Land Slide Monitoring System (LMS)
• LMSs that rely on Internet of Things (IoT) and low-cost Micro-
Electro-Mechanical Systems (MEMS) sensors
Source: Dr. Varun Dutt (SCEE) and Dr. Uday Kala (SE) 22
11
8/16/2021
Components of LMS
• The LMS monitors a number of weather and soil
parameters via sensors on deployment location
GY 61 Pin Diagram of YL 69 Soil SIM 900A GSM E
Accelerometer GY-61 Moisture Sensor Module Force Sensor
Sensor
F G H I
Humidity Sensor Light Sensor Temperature and Tipping Rain Gauge
DHT 22 BH-1750 Pressure Sensor
BMP-180
Source: Dr. Varun Dutt (SCEE) and Dr. Uday Kala (SE) 23
Architecture and Features of LMS
• The LMS monitors a number of weather and soil
parameters via sensors on deployment location
Temperature & Barometric Rainfall Light Intensity
Humidity Pressure Intensity (0 - 65535 Lux)
(-40 C to +80 C & (300-1100 (in mm)
0-100 %) mb)
Soil force Soil moisture
Soil movement
(0-100N) (0-100 %)
(±2000°/sec rotational & ±16g
gravitational acceleration)
Source: Dr. Varun Dutt (SCEE) and Dr. Uday Kala (SE) 24
12
8/16/2021
Architecture and Features of LMS
• The LMS monitors a number of weather and soil
parameters via sensors on deployment location
Architecture diagram of LMS
The LMS will alert people via traffic lights, SMSs, or smart-apps on mobile
phones about the danger of impending landslides
Source: Dr. Varun Dutt (SCEE) and Dr. Uday Kala (SE) 25
Architecture and Features of LMS
• The LMS monitors a number of weather and soil
parameters via sensors on deployment location
Architecture diagram of LMS
The LMS will alert people via traffic lights, SMSs, or smart-apps on mobile
phones about the danger of impending landslides
Source: Dr. Varun Dutt (SCEE) and Dr. Uday Kala (SE) 26
13
8/16/2021
Data Preprocessing
Data Science
• Multi-disciplinary field that uses scientific methods,
processes, algorithms and systems to extract
knowledge and insight from structured and
unstructured data
• Central concept is gaining insight from data
• Machine learning uses data to extract knowledge
Data Modeling Inference
Data Collection (Machine
Learning)
Data Preprocessing
Data
Feature
Database Cleaning and
Representation
Cleansing
28
14
8/16/2021
Need for Data Preprocessing
• Real world data are tend to be incomplete, noisy and
inconsistent due to their huge size and their likely
origin from multiple heterogeneous sources
• Preprocessing is important to clean the data
• Low quality data will lead to low quality of analysis
results
• If the users believe the data is of low quality (dirty),
they are unlikely to trust the results of any data
analytics that has been applied to
• Low quality data can cause confusion for analytic
procedure using machine learning techniques,
resulting in unreliable output
• Incomplete, noisy and inconsistent data are common
properties of large real world databases
Tuple (Record) in Structured Data
• A tuple (record) is finite ordered list (sequence) of
elements, where each element is belonging to an
attribute
Tuple
(record)
• Each row is a tuple
15
8/16/2021
Incomplete Data
• Many tuple (records) have no recorded value for
several attributes
• Example:
Incomplete Data
• Many tuple (records) have no recorded value for
several attributes
• Reasons for incomplete data:
– User forgot to fill in a field
– User chose not to fill out the field as it was not
considered important at the time of the entry
– Relevant data may not be recorded due to
malfunctioning of equipment
– Data might have lost while transferring from recorded
place
– Data may not be recorded due to programming error
– Data might not be recorded due to technology
limitations like limited memory
16
8/16/2021
Noisy Data
• Many tuple (records) have incorrect value for several
attributes
• Reasons for noisy data:
– There may be human or computer error occurring in
data entry
– The data collection instruments used may be faulty
– Error in data transmission
– There may be technology limitation such as limited
buffer size for coordinating synchronised data transfer
and consumption
Inconsistent Data
• Data containing discrepancies in stored values for
some attributes
• Reasons for inconsistent data:
– It may result from inconsistencies in
• name conventions or
– Example: “Dept_ID”, “Department_ID”
“Roll_No”, “Registation_No”
• data codes used (mismatch in writing values) or
– Example: For department – “SCEE”, “School of Computing and
EE”
• inconsistent formats of input fields such as date
– Example: “dd-mm-yy”, “dd-mm-yyyy”, “mm/dd/yyyy”
– Inconsistency in name convention or formats of input
fields while integrating
– Example: While Integrating temperature records from
different locations, if the name conventions are different
– Inconsistent data may be due to human or computer
error occurring in data entry
17
