UNIT-IV

Unsupervised Machine Learning

 UNIT-IV
Unsupervised Machine Learning
Contents

• Different clustering methods (Distance, Density,

Hierarchical)
• Iterative distance-based clustering; K-Means
Clustering Algorithm and Image Quantization,
• Basics of Principal Component Analysis.
 BOOK -- Machine Learning by Saikat Dutt
https://youtu.be/3QE_SueBMzc?si=wVEco4z88BM-4XU3
(Statistical characteristics of a population,
such as age, gender, ethnicity, education,
and income, etc.)
 Image segmentation and
compression
(Image Quantization)
• Consider the problem of identifying groups, or clusters, of data points in a
multidimensional space.

• Suppose we have a data set {x1,...,xN} consisting of

• N observations of a random D-dimensional Euclidean variable x.
• Our goal is to partition the dataset into K clusters.

• We might think of a cluster as comprising a group of data points whose inter-

point distances are small compared with the distances to points outside of the
cluster.

• Formalize this notion by a set of D-dimensional vectors µk, where k=1,...,K, in

which µk is a prototype associated with the kth CLUSTER.
• µk as representing the centres of the clusters.

• Our goal is
• to find an assignment of data points to clusters and a set of vectors {µk}, such that the
sum of the squares of the distances of each data point to its closest vector µk, is a
minimum.
Source: Pattern Recognition and Machine Learning by Christopher Bishop
• Notation to describe the assignment of data points to
clusters.
• For each data point xn, we introduce a corresponding set of
binary indicator variables rnk∈{0,1}, where k=1,...,K describing
which of the K clusters the data point xn is assigned to, so that if
data point xn is assigned to cluster k then rnk=1, and rnj =0 for j≠k.
• This is known as the 1-of-K coding scheme.

• We can then define an objective function, sometimes

called a distortion measure, given by

• which represents the sum of the squares of the distances of

each data point to its assigned vector µk.
• Our goal is to find values for the {rnk} and the {µk} to minimize J.

• We can do this through an iterative procedure in which each

iteration involves two successive steps corresponding to
successive optimizations w.r.t the rnk and the µk.
• First phase,
• choose some initial values for the µk.
• Then minimize J w.r.t rnk, keeping the µk fixed.
• Second phase,
• minimize J w.r.t the µk, keeping rnk fixed.

• This two-stage optimization is then repeated until convergence.

• These two stages of updating rnk and updating µk correspond

respectively to the E(expectation) and M(maximization) steps of
the EM algorithm.
• We simply assign the nth data point to the closest cluster centre.
• Formally, this can be expressed as
• The two phases of re-assigning data points to
clusters and re-computing the cluster means are
repeated in turn until there is no further change in
the assignments (or until some maximum number of
iterations is exceeded).

• Because each phase reduces the value of the

objective function J, convergence of the algorithm is
assured.

• However, it may converge to a local rather than

global minimum of J.
• We have deliberately chosen poor initial values for the cluster centres so that the
algorithm takes several steps before convergence.
• In practice, a better initialization procedure would be to choose the cluster centres µ k
to be equal to random subset of K data points.
• It is also worth noting that the K-means algorithm itself is often used to initialize the
parameters in a Gaussian mixture model before applying the EM algorithm.
• A direct implementation of the K-means algorithm, as discussed here, can be relatively
slow because, in each E step, it is necessary to compute the Euclidean distance
between every prototype vector and every data point.
 Image segmentation and compression
(quantization)
• Illustration of the application of the K-means algorithm;
• Consider the related problems of image segmentation and image compression.
• The goal of segmentation is
• to partition an image into regions, each of which has a reasonably
homogeneous visual appearance or which corresponds to objects or parts of
objects.

• Each pixel in an image is a point in a 3-dimensional space comprising

the intensities of the red, blue, and green channels

• The segmentation algorithm treats each pixel in the image as a

separate data point.

• For a given value of K, the algorithm is representing the image using

only K colors.
• This use of K-means is not a sophisticated approach to image segmentation
because it takes no account of the spatial proximity of different pixels.
• We illustrate the result of running K-means to convergence, for any particular value of K, by re-
drawing the image and replacing each pixel vector with the {R,G,B} intensity triplet given by the
centre µk to which that pixel has been assigned.

• Results for various values of K are shown in the Figure.

• We can also use a clustering algorithm to perform data
compression.

• It is important to distinguish between lossless data compression

(reconstructing the original data exactly) and lossy data
compression (accepting some errors in the reconstruction).

• We can apply the K-means algorithm to the problem of lossy data

compression as follows.
• For each of the N data points, we store only the identity k of the cluster
to which it is assigned.
• We also store the values of the K cluster centres µk, which typically
requires significantly less data, provided we choose K<< N.
• Each data point is then approximated by its nearest centre µk.
• New data points can similarly be compressed by first finding the nearest
µk and then storing the label k instead of the original data vector.
• This framework is often called vector quantization, and the vectors µk
are called code-book vectors.
• Suppose the original image has N pixels comprising
{R,G,B} values, each of which is stored with 8 bits of
precision.
• Then to transmit the whole image directly would cost
24N bits.

• Suppose we first run K-means on the image data, and

then instead of transmitting the original pixel intensity
vectors, we transmit the identity of the nearest vector µk.
• Because there are K such vectors, this requires log2K bits
per pixel.
• We must also transmit the K code book vectors µk, which
requires 24K bits, so the total number of bits required to
transmit the image is 24K + N log2 K (rounding up to the
nearest integer).
• The original image shown in Figure 9.3 (above) has 240×180 =
43,200 pixels and so requires 24×43,200 = 1,036,800 bits to
transmit directly.

• Compressed images require 43,248 bits (K =2); 86,472 bits (K =3),

and 173,040bits (K =10),respectively, to transmit.

• These represent compression ratios compared to the original

image of 4.2%, 8.3%, and 16.7%, respectively.

• We see that there is a trade-off between the degree of

compression and image quality.

• If we had been aiming to produce a good image compressor, then

it would be more fruitful to consider small blocks of adjacent
pixels, for instance 5×5, and thereby exploit the correlations that
exist in natural images between nearby pixels.
https://youtu.be/0NNcVu9v3nw?si=q8yyJ9-0z8ypBLhW
Example –
NEXT SLIDE
https://youtu.be/YH0r47m0kFM?si=3oVMsdODjZLpfsHZ
Min distance
between 42 and 43
Merge 42 and 43

Merged 42 and 43
Min distance
between 25 and 27

Merge 25 and 27

Merged 42 and 43
Min distance between
22 and (25,27)

Merged 22 and (25,27)

 Min distance between
18 and (22,25,27)

Merged 18 and (22,25,27)

Min distance between
P3 and P6
Min distance between
P4 and (P3,P6)
Density-based
clustering methods
• When partitioning and hierarchical clustering methods are used, the
resulting clusters are spherical or nearly spherical in nature.

• In the case of the other shaped clusters such as S-shaped or uneven

shaped clusters, the above two types of method do not provide
accurate results.

• The density-based clustering approach provides a solution to identify

clusters of arbitrary shapes.

• The principle is based on identifying the dense area and sparse area
within the data set and then run the clustering algorithm.

• DBSCAN (Density-Based Spatial Clustering of Applications with Noise)

is one of the popular density-based algorithm which creates clusters
by using connected regions with high density.
 Why do we need a Density-Based
clustering algorithm like DBSCAN when
•
we already have K-means clustering?
K-Means clustering may cluster loosely related observations together. Every observation becomes a
part of some cluster even if the observations are scattered far away in the vector space.
• Since clusters depend on the mean value of cluster elements, each data point plays a role in forming the
clusters. A slight change in data points might affect the clustering outcome.
• This problem is greatly reduced in DBSCAN due to the way clusters are formed.
• Another challenge with k-means is that you need to specify the number of clusters (“k”) in order to
use it. Much of the time, we won’t know what a reasonable k value is a priori.
• In DBSCAN, you don’t have to specify the number of clusters to use it.
• All you need is a function to calculate the distance between values and some guidance for what
amount of distance is considered “close”. DBSCAN also produces more reasonable results than k-means
across a variety of different distributions.
• Below figure illustrates the fact:
(Density-Based Spatial Clustering of Applications with Noise)

https://youtu.be/
jqKAAVEwX9M?
si=cnvsppj2X-KFBWA6
A