Data Mining:
Data
Preprocessing
Data Preprocessing
Why preprocess the data?
Data cleaning
Data integration and transformation
Data reduction
Discretization and concept hierarchy generation
Summary
Why Data Preprocessing?
Data in the real world is dirty
incomplete: lacking attribute values, lacking certain
attributes of interest, or containing only aggregate
data
noisy: containing errors or outliers
inconsistent: containing discrepancies in codes or
names
No quality data, no quality mining results!
Quality decisions must be based on quality data
Data warehouse needs consistent integration of
quality data
Multi-Dimensional Measure of
Data Quality
A well-accepted multidimensional view:
Accuracy
Completeness
Consistency
Timeliness
Believability
Value added
Interpretability
Accessibility
Major Tasks in Data
Preprocessing
Data cleaning
Fill in missing values, smooth noisy data, identify or remove outliers, and
resolve inconsistencies
Data integration
Integration of multiple databases, data cubes, or files
Data transformation
Normalization and aggregation
Data reduction
Obtains reduced representation in volume but produces the same or
similar analytical results
Data discretization
Part of data reduction but with particular importance, especially for
numerical data
Forms of data
preprocessing
Data Preprocessing
Why preprocess the data?
Data cleaning
Data integration and transformation
Data reduction
Discretization and concept hierarchy generation
Summary
Data Cleaning
Data cleaning tasks
Fill in missing values
Identify outliers and smooth out noisy data
Correct inconsistent data
Missing Data
Data is not always available
E.g., many tuples have no recorded value for several
attributes, such as customer income in sales data
Missing data may be due to
equipment malfunction
inconsistent with other recorded data and thus deleted
data not entered due to misunderstanding
certain data may not be considered important at the time of
entry
not register history or changes of the data
Missing data may need to be inferred.
How to Handle
Missing Data?
Ignore the tuple: usually done when class label is missing
(assuming the tasks in classification—not effective when the
percentage of missing values per attribute varies considerably)
Fill in the missing value manually: tedious + infeasible?
Use a global constant to fill in the missing value: e.g., “unknown”, a
new class?!
Use the attribute mean to fill in the missing value
Use the most probable value to fill in the missing value: inference-
based such as Bayesian formula or decision tree
Noisy Data
Noise: random error or variance in a measured variable
Incorrect attribute values may due to
faulty data collection instruments
data entry problems
data transmission problems
technology limitation
inconsistency in naming convention
Other data problems which requires data cleaning
duplicate records
incomplete data
inconsistent data
How to Handle Noisy
Data?
Binning method:
first sort data and partition into (equi-depth) bins
then smooth by bin means, smooth by bin median,
smooth by bin boundaries, etc.
Clustering
detect and remove outliers
Combined computer and human inspection
detect suspicious values and check by human
Regression
smooth by fitting the data into regression functions
Simple Discretization
Methods: Binning
Equal-width (distance) partitioning:
It divides the range into N intervals of equal size: uniform grid
if A and B are the lowest and highest values of the attribute, the
width of intervals will be: W = (B-A)/N.
The most straightforward
But outliers may dominate presentation
Skewed data is not handled well.
Equal-depth (frequency) partitioning:
It divides the range into N intervals, each containing
approximately same number of samples
Good data scaling
Managing categorical attributes can be tricky.
Binning Methods for Data
Smoothing
* Sorted data for price (in dollars): 4, 8, 9, 15, 21, 21, 24, 25, 26, 28, 29,
34
* Partition into (equi-depth) bins:
- Bin 1: 4, 8, 9, 15
- Bin 2: 21, 21, 24, 25
- Bin 3: 26, 28, 29, 34
* Smoothing by bin means:
- Bin 1: 9, 9, 9, 9
- Bin 2: 23, 23, 23, 23
- Bin 3: 29, 29, 29, 29
* Smoothing by bin boundaries:
- Bin 1: 4, 4, 4, 15
- Bin 2: 21, 21, 25, 25
- Bin 3: 26, 26, 26, 34
Data Preprocessing
Why preprocess the data?
Data cleaning
Data integration and transformation
Data reduction
Discretization and concept hierarchy generation
Summary
Data Integration
Data integration:
combines data from multiple sources into a
coherent store.
Careful integration can help reduce and avoid
redundancies and inconsistencies in resulting data
set.
This can help improve the accuracy and speed of
the subsequent data mining process.
Data Integration
There are a number of issues to consider during data
integration. Schema integration and object matching
can be tricky.
How can equivalent real-world entities from
multiple data sources be matched up?
This is referred to as the entity identification
problem.
For example, how can the data analyst or the computer
be sure that customer-id in one database and cust-
number in another refer to the same attribute?
Data Integration
When matching attributes from one database
to another during integration, special attention
must be paid to the structure of the data.
For example, in one system, a discount may be
applied to the order, whereas in another system
it is applied to each individual line item within
the order.
If this is not caught before integration, items in
the target system may be improperly discounted.
Handling Redundant Data
Redundant data occur often when integration of multiple
databases
The same attribute may have different names in
different databases Careful integration of the data
from multiple sources may help reduce/avoid
redundancies and inconsistencies and improve mining
speed and quality
Data Transformation
Strategies for data transformation are:
Smoothing: remove noise from data
Attribute Construction: new attributes are constructed
Aggregation: summarization, data cube construction
Generalization: concept hierarchy climbing
Normalization: scaled to fall within a small, specified
range.
Discretization: raw values of a numeric attribute (e.g.,
age) are replaced by interval labels (e.g., 0–10, 11–20, etc.)
or conceptual labels (e.g., youth, adult, senior ).
Data Transformation by
Normalization
The measurement unit used can affect the data
analysis.
For example, changing measurement units from
meters to inches for height, or from kilograms to
pounds for weight, may lead to very different results.
To help avoid dependence on the choice of
measurement units, the data should be normalized or
standardized.
This involves transforming the data to fall within a
smaller or common range such as [−1,1] or [0.0, 1.0].
Data Transformation by
Normalization
Methods for Normalization:
min-max normalization
v minA
v' (new _ maxA new _ minA) new _ minA
maxA minA
z-score normalization
v mean A
v'
stand _ dev A
normalization by decimal scaling
v
v' j Where j is the smallest integer such that Max(| v ' |)<1
10
Data Transformation by
Normalization
Let A be the numeric attribute with n observed values v1, v2…………vn
v minA
v' (new _ maxA new _ minA) new _ minA
maxA minA
Data Transformation by
Normalization
v meanA
v'
stand _ devA
Data Transformation by
Normalization
Discretization
The raw values of a numeric attribute (e.g., age) are
replaced by interval labels (e.g., 0–10, 11–20, etc.) or
conceptual labels (e.g., youth, adult, senior ).
Three types of attributes:
Nominal — values from an unordered set
Ordinal — values from an ordered set
Continuous — real numbers
Discretization:
divide the range of a continuous attribute into intervals
Some classification algorithms only accept categorical attributes.
Reduce data size by discretization
Prepare for further analysis
Discretization
Discretization techniques can be categorized based on
how the discretization is performed, such as whether it
uses class information or which direction it proceeds
(i.e., top-down vs. bottom-up).
If the discretization process uses class information, then
we say it is supervised discretization.
If the process starts by first finding one or a few points
to split the entire attribute range, and then repeats this
recursively on the resulting intervals, it is called top-
down discretization or splitting.
Discretization
This contrasts with bottom-up discretization or
merging, which starts by considering all of the
continuous values as potential split-points, removes
some by merging neighborhood values to form intervals,
and then recursively applies this process to the resulting
intervals.
Data Reduction Strategies
Warehouse may store terabytes of data: Complex data, so
analysis/mining may take a very long time to run on the
complete data set
Data reduction
Obtains a reduced representation of the data set that is
much smaller in volume but yet produces the same (or
almost the same) analytical results
Data reduction strategies
Dimensionality reduction
Numerosity reduction
Data compression
Dimensionality Reduction
Feature selection (i.e., attribute subset selection):
Dimensionality reduction is the process of reducing the
number of random variables or attributes under
consideration.
It transform or project the original data onto a smaller
space.
Attribute subset selection is a method of dimensionality
reduction in which irrelevant, weakly relevant, or
redundant attributes or dimensions are detected and
removed
Numerosity Reduction
Numerosity reduction techniques replace the original data
volume by alternative, smaller forms of data representation.
These techniques may be parametric or
nonparametric.
For parametric methods, a model is used to estimate
the data, so that typically only the data parameters need
to be stored, instead of the actual data. (Outliers may
also be stored.)
Nonparametric methods for storing reduced
representations of the data include histograms,
clustering, sampling etc
Data Compression
In data compression, transformations are applied so as to
obtain a reduced or “compressed” representation of the
original data.
If the original data can be reconstructed from the
compressed data without any information loss, the data
reduction is called lossless.
If, instead, we can reconstruct only an approximation of the
original data, then the data reduction is called lossy.
Dimensionality reduction and numerosity reduction
techniques can also be considered forms of data
compression.
Data Preprocessing
Why preprocess the data?
Data cleaning
Data integration and transformation
Data reduction
Discretization and concept hierarchy generation
Summary
Discretization and Concept
hierachy
Discretization
reduce the number of values for a given continuous
attribute by dividing the range of the attribute into
intervals. Interval labels can then be used to replace
actual data values.
Concept hierarchies
reduce the data by collecting and replacing low level
concepts (such as numeric values for the attribute
age) by higher level concepts (such as young,
middle-aged, or senior).
Discretization for numeric
data
Binning (see sections before)
Histogram analysis (see sections before)
Clustering analysis (see sections before)
