Time series analysis in neuroscience
Lecture 10. Clustering and Classification
Alexander Zhigalov / Dept. of CS, University of Helsinki and Dept. of NBE, Aalto University
Time series analysis in neuroscience 2
Outline / overview
Section 1. Machine learning
Section 2. Clustering
Section 3. Classification
Section 4. Regression
Time series analysis in neuroscience 3
Section 1. Machine learning
Time series analysis in neuroscience 4
Machine learning
Introducing Machine Learning - MathWorks
Section 1 Machine learning
Time series analysis in neuroscience 5
Machine learning
Unsupervised learning is useful when you want to explore
your data but do not yet have a specific goal or are not sure
what information the data contains.
A supervised learning algorithm takes a known set of input
data (the training set) and known responses to the data
(output), and trains a model to generate reasonable
predictions for the response to new input data.
Section 1 Machine learning
Time series analysis in neuroscience 6
Machine learning
Introducing Machine Learning - MathWorks
Section 1 Machine learning
Time series analysis in neuroscience 7
Section 2. Clustering
Time series analysis in neuroscience 8
Clustering
In cluster analysis, data is partitioned into groups based on
some measure of similarity or shared characteristic.
Clusters are formed so that objects in the same cluster are
very similar and objects in different clusters are very
distinct.
Section 2 Clustering
Time series analysis in neuroscience 9
K-means clustering
How it works
Partitions data into k number of mutually exclusive clusters.
How well a point fits into a cluster is determined by the
distance from that point to the cluster’s center.
Best used ...
• When the number of clusters is known
• For fast clustering of large data sets
Result
Cluster centers
Section 2 Clustering
Time series analysis in neuroscience 10
K-means clustering (1/2)
The number of clusters (K) is equal to the
number of sources.
# create copies
X[i] = np.tile(S[i, :], (R, 1)) +
np.random.randn(R, N) * SNR
# measurements
Y = X[np.random.permutation(M*R), :]
# clustering using sklearn
model = cluster.KMeans(n_clusters=K)
model.fit(Y)
# clustering outcome
labels = model.labels_
Z = model.cluster_centers_
inertia = model.inertia_
print(inertia) # within-cluster sum-of-squares
inertia = 0.0
See, “L10_clustering_kmeans.py”
Section 2 Clustering
Time series analysis in neuroscience 11
K-means clustering (1/2)
The number of clusters (K) is greater or less
than the number of sources.
# create copies
X[i] = np.tile(S[i, :], (R, 1)) +
np.random.randn(R, N) * SNR
# measurements
Y = X[np.random.permutation(M*R), :]
# clustering using sklearn
model = cluster.KMeans(n_clusters=K)
model.fit(Y)
# clustering outcome
labels = model.labels_
Z = model.cluster_centers_
inertia = model.inertia_
print(inertia)
inertia = 603.5 inertia = 0.0
See, “L10_clustering_kmeans.py”
Section 2 Clustering
Time series analysis in neuroscience 12
Noisy measurements (1/2)
How does it work in presence of noise?
# create copies
X[i] = np.tile(S[i, :], (R, 1)) +
np.random.randn(R, N) * 0.5
# measurements
Y = X[np.random.permutation(M*R), :]
inertia = 2550.0
See, “L10_clustering_kmeans.py”
Section 2 Clustering
Time series analysis in neuroscience 13
Noisy measurements (2/2)
Can the algorithm put noise to a single
cluster?
# create copies
X[i] = np.tile(S[i, :], (R, 1)) +
np.random.randn(R, N) * 0.5
# measurements
Y = X[np.random.permutation(M*R), :]
inertia = 3381.0 inertia = 2279.0
See, “L10_clustering_kmeans.py”
Section 2 Clustering
Time series analysis in neuroscience 14
Hierarchical clustering
How it works
Produces nested sets of clusters by analyzing similarities
between pairs of points and grouping objects into a binary,
hierarchical tree.
Best used ...
• When you do not know in advance how many clusters
are in your data
• You want visualization to guide your selection
Result
Dendrogram showing the hierarchical relationship
between clusters
Section 2 Clustering
Time series analysis in neuroscience 15
Hierarchical clustering (1/2)
What are the distance measures between
signals?
# pair-wise distance between signals
PX = np.zeros((MR, MR))
PX[np.triu_indices(MR, 1)] = pdist(X,
'euclidean')
# distance after permutation
PY = np.zeros((MR, MR))
PY[np.triu_indices(MR, 1)] = pdist(Y,
'euclidean')
See, “L10_clustering_hierarchical.py”
Section 2 Clustering
Time series analysis in neuroscience 16
Hierarchical clustering (2/2)
How does it work?
# clustering
model =
cluster.AgglomerativeClustering()
model.fit(Y)
labels = model.labels_
children = model.children_
See, “L10_clustering_hierarchical.py”
Section 2 Clustering
Time series analysis in neuroscience 17
Noisy measurements
How does noise affect the clustering
results?
# create copies
X[i] = np.tile(S[i, :], (R, 1)) +
np.random.randn(R, N) * 0.5
# measurements
Y = X[np.random.permutation(M*R), :]
See, “L10_clustering_hierarchical.py”
Section 2 Clustering
Time series analysis in neuroscience 18
Hierarchical clustering 2D (1/2)
Could we cluster the covariance matrix
instead of sources?
# covariance
CX = np.cov(X)
CY = np.cov(Y)
# clustering
model =
cluster.AgglomerativeClustering(n_clusters=K)
model.fit(Y)
labels = model.labels_
# sort matrix
indices = np.squeeze(np.argsort(labels))
CZ = CY
CZ = CZ[indices, :]
CZ = CZ[:, indices]
See, “L10_clustering_hierarchical_2D.py”
Section 2 Clustering
Time series analysis in neuroscience 19
Hierarchical clustering 2D (2/2)
Sub-optimal number of clusters …
# covariance
CX = np.cov(X)
CY = np.cov(Y)
# clustering
model =
cluster.AgglomerativeClustering(n_clusters=K)
model.fit(Y)
labels = model.labels_
# sort matrix
indices = np.squeeze(np.argsort(labels))
CZ = CY
CZ = CZ[indices, :]
CZ = CZ[:, indices]
See, “L10_clustering_hierarchical_2D.py”
Section 2 Clustering
Time series analysis in neuroscience 20
Gaussian Mixture Model
How it works
Partition-based clustering where data points come from
different multivariate normal distributions with certain
probabilities.
Best used ...
• When a data point might belong to more than one
cluster
• When clusters have different sizes and correlation
structures within them
Result
A model of Gaussian distributions that give probabilities of
a point being in a cluster
Section 2 Clustering
Time series analysis in neuroscience 21
Gaussian Mixture Model (1/2)
Why does it work?
# generate data
Z = [np.random.randn(1, N) * 0.5 + 0.0,
np.random.randn(1, N) * 1.25 + 4.0]
# scatter plot (joint distribution)
plt.scatter(Z[0], Z[1])
# gaussian PDF
b = np.linspace(-3, 10, 1000)
p0 = norm.pdf(b, np.mean(Z[0]), np.std(Z[0]))
p1 = norm.pdf(b, np.mean(Z[1]), np.std(Z[1]))
# multivariate gaussian PDF
x, y = np.meshgrid(np.arange(-10.0, 10.0,
delta), np.arange(-10.0, 10.0, delta))
z = mlab.bivariate_normal(x, y, np.std(Z[0]),
np.std(Z[1]), np.mean(Z[0]), np.mean(Z[1]))
plt.contour(x, y, z)
See, “L10_clustering_gmm.py”
Section 2 Clustering
Time series analysis in neuroscience 22
Gaussian Mixture Model (2/2)
How does it work?
# fit model
model = mixture.GaussianMixture(n_components=K)
model.fit(X)
# model properties
Y = model.predict(X)
model_mu = model.means_
model_cov = model.covariances_
See, “L10_clustering_gmm.py”
Section 2 Clustering
Time series analysis in neuroscience 23
Section 3. Classification
Time series analysis in neuroscience 24
Classification
Classification techniques predict discrete responses, for
example, whether an email is genuine or spam.
Classification models are trained to classify data
into categories.
Section 3 Classification
Time series analysis in neuroscience 25
Support Vector Machine
How it works
Classifies data by finding the linear decision boundary
(hyperplane) that separates all data points of one class from
those of the other class. The best hyperplane for an SVM is
the one with the largest margin between the two classes.
Best used ...
• For data that has exactly two classes
• For high-dimensional, nonlinearly separable data
• When you need a classifier that is simple, easy to
interpret and accurate
Result
Training/fitting transforms data and labels to coefficients,
while testing/prediction transforms data and coefficients to
labels.
Section 3 Classification
Time series analysis in neuroscience 26
Support Vector Machine
When it works
SVM works if data has exactly two classes.
Section 3 Classification
Time series analysis in neuroscience 27
Support Vector Machine (1/2)
SVM as any other classification approach
training testing
consists of two stages: training and testing.
# data
X = np.random.randn(M, N)
# binary labels
y = get_sequence(5, 0.8, N)
# induce some correlation between X and y
X = X + 2.0 * np.tile(y, (M, 1))
# training and testing datasets
L = N // 2
Y = y[:L] # training labels
U = y[L:] # testing labels
XY = X[:, :L] # training data
XU = X[:, L:] # testing data
# train classifier
model = SVC(kernel='linear')
model.fit(XY.T, Y)
See, “L10_classification_svm_2_signals.py”
Section 3 Classification
Time series analysis in neuroscience 28
Support Vector Machine (2/2)
The classifier gives the coefficients that can
training testing
be converted to the decision function.
# classifier outcome
coef = model._get_coef()
intercept = model.intercept_
# decision function
Z = np.zeros(N)
for i in range(0, N):
Z[i] = np.sum(X[:, i] * coef) + intercept
# testing
v = U
u = model.predict(XU.T)
u = u > 0.5
# accuracy
a = np.mean(v == u)
print('accuracy: %1.2f' % (a))
accuracy = 99%
See, “L10_classification_svm_2_signals.py”
Section 3 Classification
Time series analysis in neuroscience 29
Classification accuracy (1/2)
What if two datasets cannot be clearly
training testing
separated?
# data
X = np.random.randn(M, N)
# binary labels
y = get_sequence(5, 0.8, N)
# induce some correlation between X and y
X = X + 2.0 * np.tile(y, (M, 1))
# training and testing datasets
L = N // 2
Y = y[:L] # training labels
U = y[L:] # testing labels
XY = X[:, :L] # training data
XU = X[:, L:] # testing data
# train classifier
model = SVC(kernel='linear')
model.fit(XY.T, Y)
See, “L10_classification_svm_2_signals.py”
Section 3 Classification
Time series analysis in neuroscience 30
Classification accuracy (2/2)
The decision function looks like a random
training testing
noise.
# classifier outcome
coef = model._get_coef()
intercept = model.intercept_
# decision function
Z = np.zeros(N)
for i in range(0, N):
Z[i] = np.sum(X[:, i] * coef) + intercept
# testing
v = U
u = model.predict(XU.T)
u = u > 0.5
# accuracy
a = np.mean(v == u)
print('accuracy: %1.2f' % (a))
accuracy = 76%
See, “L10_classification_svm_2_signals.py”
Section 3 Classification
Time series analysis in neuroscience 31
Multichannel recordings (1/2)
More channels could provide a better fit of
training testing
the model but also overfitting may occur.
# data
X = np.random.randn(M, N)
# binary labels
y = get_sequence(5, 0.8, N)
# induce some correlation between X and y
X = X + 2.0 * np.tile(y, (M, 1))
accuracy = 100%
See, “L10_classification_svm.py”
Section 3 Classification
Time series analysis in neuroscience 32
Classification accuracy (2/3)
In case of weak correlation between data
and labels …
# data
X = np.random.randn(M, N)
# binary labels
y = get_sequence(5, 0.8, N)
# induce some correlation between X and y
X = X + 0.5 * np.tile(y, (M, 1))
accuracy = 67%
See, “L10_classification_svm.py”
Section 3 Classification
Time series analysis in neuroscience 33
Discriminant Analysis
How it works
Discriminant analysis classifies data by finding linear
combinations of features. Discriminant analysis assumes
that different classes generate data based on Gaussian
distributions.
Best used ...
• When you need a simple model that is easy to interpret
• When you need a model that is fast to predict
Result
Coefficients
Section 3 Classification
Time series analysis in neuroscience 42
Logistic regression
How it works
Fits a model that can predict the probability of a binary
response belonging to one class or the other. Because of its
simplicity, logistic regression is commonly used as a starting
point for binary classification problems.
Best used ...
• When data can be clearly separated by a single, linear
boundary
• As a baseline for evaluating more complex classification
methods
Result
Coefficients
Section X Classification
Time series analysis in neuroscience 34
Section 4. Regression
Time series analysis in neuroscience 35
Regression
Regression techniques predict continuous responses, for
example, changes in temperature or fluctuations in
electricity demand.
Section 4 Regression
Time series analysis in neuroscience 36
Linear regression
How it works
Linear regression is a statistical modeling technique used to
describe a continuous response variable as a linear function
of one or more predictor variables.
Best used ...
• When you need an algorithm that is easy to interpret
and fast to fit
• As a baseline for evaluating other, more complex,
regression models
Result
Coefficients
Section 4 Regression
Time series analysis in neuroscience 37
Nonlinear regression
How it works
Nonlinear regression is a statistical modeling technique that
helps describe nonlinear relationships in experimental data.
Best used ...
• When data has strong nonlinear trends and cannot be
easily transformed into a linear space
• For fitting custom models to data
Result
Coefficients
Section 4 Regression
Time series analysis in neuroscience 38
SVM regression
How it works
SVM regression algorithms work like SVM classification
algorithms, but are modified to be able to predict a
continuous response.
Best used ...
• For high-dimensional data with a large number of
predictor variables
Result
Coefficients
Section 4 Regression
Time series analysis in neuroscience 39
Literature
Time series analysis in neuroscience 40
• Python programming language
- http://www.scipy-lectures.org/, see “materials/L02_ScipyLectures.pdf”
• Data analysis
- Andreas Müller and Sarah Guido “Introduction to Machine Learning with Python: A Guide for Data Scientists”
Literature
