Unit II

Image Segmentation
Dr. S. Manjula Gandhi
Associate Professor
Department of Computing
Coimbatore Institute of Technology
manjulagandhi@cit.edu.in

1
 Outline
• Active Contours
• Split and Merge
• Mean Shift and mode finding

2
 Contours
• Contours-curve joining all the continuous
points (along the boundary), having same
colour or intensity.
• Used for shape analysis, object detection and
recognition.

3
• To improve accuracy, use binary images.
• First apply threshold or canny edge detection.
• findcontours() – does not modify the source, returns a
modified image.
• In OpenCV, object to be found should be white and
background to be black.

4
• Parameters-Source image, contour retrieval
mode, contour approximation method
• Output- Modified image, contours, hierarchy
– Contours-a python list of all contours in the image,
each individual contour is a Numpy array of (x, y)
coordinates of boundary points of the object.

5
 How to draw contours?
• cv.drawContours()
• To draw all the contours in an image
– cv.drawContours(img, contours, -1, (0,255,0),3)
• To draw an individual contour (third contour)
– cv.drawContours(img, contours, 2, (0,255,0),3)
• Another Method
– cnt=contours[2]
– cv.drawContours(img, [cnt], (0,255,0),3)

6
 Contour Approximation method
• Third argument in cv.findContours()
• Contours are the boundaries of a shape with
same intensity.
• Stores (x, y)coordinates of the boundary of a
shape.

7
• cv.CHAIN_APPROX_NONE – all the boundary
points are stored.
• cv.CHAIN_APPROX_SIMPLE – removes all
redundant points

8
 Contour Features
1. Moments
Calculate center of mass of the object, area of
the object, etc.
cv.moments()

9
• Centroid
Cx=M10/M00, Cy=M01/M00

10
2. Contour Area
- cv.contourArea()

- using moments M[‘m00’]

11
3. Contour Perimeter
- Also called as arc length
- cv.arcLength()

- Second argument=
True→shape is a closed contour
False→Curve

12
4. Contour Approximation
- Approximates a contour shape to another
shape with less number of vertices depending
upon the precision specified.
- An implementation of Douglas Peucker
algorithm

13
• Tyring to find a square in an image, we didn’t get a
perfect square, we can approximate the shape

• Third argument →curve closed or not

– Second image, epsilon=10%

– Third Image, epsilon=1%
14
5. Convex Hull
- cv.convexHull() function checks a curve for
convexity defects and corrects it.
- Red line - convex hull of hand
- Double sided arrow marks -
convexity defects - local maximum
deviations of hull from contours

15
• points-are the contours we pass
• Hull- is the output, normally we avoid it
• Clockwise-Orientation flag, the output is oriented
clockwise/anti clockwise
• returnPoints-Default=True, returns the coordinates of the
hull points
– False, returns the contour points corresponding to the
hull points

16
• Example
– Find convexhull with returnPoints=True
• Returns four points of a rectangle
– (234, 202), (51,202), (51, 79), (234,79)
– returnPoints=False
• 129, 67, 0, 142
• These are the indices of the corresponding points in
contours
• Cnt[129]=[234,202]

17
6. Checking convexity
- To check if a curve is convex or not

18
7. Bounding Rectangle
- Straight Bounding Rectangle
- Rotated Rectangle

19
8. Minimum Enclosing Circle
- Finds the circumference of an object
- a circle which completely covers the object
with minimum area

20
9. Fitting an ellipse
Fit an ellipse to an object

21
10. Fitting a line
Given image contains a set of white points, we
can approximate a straight line to it.

22
Image segmentation

23
 The goals of segmentation
• Group together similar-looking pixels for efficiency of
further processing
– “Bottom-up” process
– Unsupervised

“superpixels”

24
 The goals of segmentation
• Separate image into coherent “objects”

image human segmentation

25
 Split and Merge
• A single threshold is rarely sufficient for the
whole image because of lighting and intra-
object statistical variations.
• Recursively split the whole image into pieces
based on region characteristics
• Merging pixels and regions together in a
hierarchical fashion.

26
 Algorithms
• Watershed
• Region splitting (divisive clustering)
• Region merging (agglomerative clustering)
• Mean Shift and mode finding

27
 Watershed Algorithm
• Interactive image segmentation algorithm
• Label the region which are sure of being the foreground
with one color
• Label the region which are sure of being the background
with another color
• Regions which we are not sure of anything are labelled as
zero
• Apply watershed algorithm
• Marker will be updated with the labels we gave, and the
boundaries of objects will have a value of -1

28
 Region Splitting
• Divisive clustering
• Compute a histogram for the whole image and
then find a threshold that best separates the
large peaks in the histogram.
• Process is repeated until regions are either
fairly uniform or below a certain size.

29
 Region Merging
• Agglomerative clustering
• Uses a dual grid for representing boundaries
between pixels.
• Merge regions based on their relative
boundary lengths and the strength of the
visible edges at these boundaries.

30
 Mean shift clustering and segmentation
• An advanced and versatile technique for
clustering-based segmentation

31
 Mean shift algorithm
• The mean shift algorithm seeks modes or
local maxima of density in the feature space
Feature space
image (L*u*v* color values)

32
Mean shift Search
window

Center of
mass

Mean Shift
vector

33
Mean shift Search
window

Center of
mass

Mean Shift
vector

34
Mean shift Search
window

Center of
mass

Mean Shift
vector

35
Mean shift Search
window

Center of
mass

Mean Shift
vector

36
Mean shift Search
window

Center of
mass

Mean Shift
vector

37
Mean shift Search
window

Center of
mass

Mean Shift
vector

38
Mean shift Search
window

Center of
mass

39
 Mean shift clustering
• Cluster: all data points in the attraction basin
of a mode
• Attraction basin: the region for which all
trajectories lead to the same mode

40
 Mean shift clustering/segmentation
• Find features (color, gradients, texture, etc)
• Initialize windows at individual feature points
• Perform mean shift for each window until convergence
• Merge windows that end up near the same “peak” or mode

Given image 41
• Pixels plotted in
L*u*v*space
• Clustered results
after 159 mean-
shift procedures
• Corresponding
trajectories with
peaks marked as
red dots.

42
Mean shift segmentation results

43
 Mean shift pros and cons
• Pros
– Does not assume spherical clusters
– Just a single parameter (window size)
– Finds variable number of modes
– Robust to outliers
• Cons
– Output depends on window size
– Computationally expensive
– Does not scale well with dimension of feature space

44
 Images as graphs
j
wij
i

• Node for every pixel

• Edge between every pair of pixels (or every pair of
“sufficiently close” pixels)
• Each edge is weighted by the affinity or similarity of
the two nodes 45
 Segmentation by graph partitioning

j
wij
i

A B C

• Break Graph into Segments

– Delete links that cross between segments
– Easiest to break links that have low affinity
• similar pixels should be in the same segments
• dissimilar pixels should be in different segments
46
 Measuring affinity
• Suppose we represent each pixel by a feature
vector x and define a distance function
appropriate for this feature representation.
• Then we can convert the distance between
two feature vectors into an affinity with the
help of a generalized Gaussian kernel:
 1 2
exp − 2 dist(xi , x j ) 
 2 
47
 Scale affects affinity
• Small σ: group only nearby points
• Large σ: group far-away points

48
 Graph cut

B
A

• Set of edges whose removal makes a graph

disconnected
• Cost of a cut: sum of weights of cut edges
• A graph cut gives us a segmentation
– What is a “good” graph cut and how do we find
one?

49
 Minimum cut
• We can do segmentation by finding the
minimum cut in a graph
– Efficient algorithms exist for doing this
Minimum cut example

50
 Minimum cut
• We can do segmentation by finding the
minimum cut in a graph
– Efficient algorithms exist for doing this
Minimum cut example

51
 Minimum cut
• Drawback: minimum cut tends to cut off very
small, isolated components

Cuts with
lesser weight
than the
ideal cut

Ideal Cut

52
 Normalized cut
• Drawback: minimum cut tends to cut off very
small, isolated components
• This can be fixed by normalizing the cut by
the weight of all the edges incident to the
segment
• The normalized cut cost is:

53
Normalized Cut

54
 Normalized cut
• Let W be the adjacency matrix of the graph
• Let D be the diagonal matrix with diagonal entries
D(i, i) = Σj W(i, j)
• Then the normalized cut cost can be written as
y (D − W ) y
T

T
y Dy
where y is an indicator vector whose value should
be 1 in the ith position if the ith feature point
belongs to A and a negative constant otherwise

55
 Normalized cut
• Finding the exact minimum of the normalized cut cost is
NP-complete, but if we relax y to take on arbitrary values,
then we can minimize the relaxed cost by solving the
generalized eigenvalue problem (D − W)y = λDy
• The solution y is given by the eigenvector corresponding to
the second smallest eigenvalue
• Intuitively, the ith entry of y can be viewed as a “soft”
indication of the component membership of the ith feature
– Can use 0 or median value of the entries as the splitting
point (threshold), or find threshold that minimizes the Ncut
cost

56
 Normalized cut algorithm
1. Represent the image as a weighted graph
G = (V,E), compute the weight of each edge,
and summarize the information in D and W
2. Solve (D − W)y = λDy for the eigenvector with
the second smallest eigenvalue
3. Use the entries of the eigenvector to
bipartition the graph
4. Recursively partition the segmented parts,
if necessary
57
 Example

• Input image and components returned by

normalized cut algorithm.
58
 Challenge
• How to segment images that are a “mosaic of
textures”?

59
 Using texture features for
segmentation
• Convolve image with a bank of filters

60
Using texture features for segmentation
• Convolve image with a bank of filters
• Find textons by clustering vectors of filter bank
outputs
Image Texton map

61
Using texture features for segmentation
• Convolve image with a bank of filters
• Find textons by clustering vectors of filter bank
outputs
• The final texture feature is a texton histogram
computed over image windows at some “local
scale”

62
Example results

63
 Normalized cuts: Pro and con
• Pros
– Generic framework, can be used with many
different features and affinity formulations
• Cons
– High storage requirement and time complexity
– Bias towards partitioning into equal segments

64