22/05/2021

Plan
• Edges
‒ Detection
‒ linking

Computer Vision • Feature extraction

‒ Global features
Chapter 4: Feature detection and Image
‒ Local features
matching
• Image Matching and Applications

1 2

What are edges/contours ?

• Local changes in the images
• Typically occur on the boundary between different
regions in an image
Surface
• Origins of edges:
orientation
Surface colours
Edge detection Depth
reflectance
Role of edges & general approach
Image gradient illumination
Edges detectors: sobel, prewitt, Canny detector

3 4
 22/05/2021

Edge is important? Edge is important?

• What do you see ? (A)Cave painting at
Chauvet, France,
about 30,000 B.C.;
(B)Aerial photograph of
the picture of a
monkey as part of
the Nazca Lines
geoglyphs, Peru,
about 700 – 200
B.C.;
(C)Shen Zhou (1427-
1509 A.D.): Poet on a
mountain top, ink on
paper, China;
(D)Line drawing by 7-
year old I. Lleras
(2010 A.D.).

5 6

Hubel & Wiesel, 1960s

Can we recognize these objects?

7 8
 22/05/2021

Edge detection How to find edges?

• Goal: Identify sudden changes (discontinuities) in • Intensity profile of an image is the set of intensity
an image values taken from regularly spaced points along a line
‒ Intuitively, most semantic and shape information from the segment or multi-line path in an image
image can be encoded in the edges
Edge
‒ More compact than pixels
• Why?
‒ Extract information, recognize objects
‒ Recover geometry and viewpoint Vertical vanishing
point
(at infinity)
Vanishing
line

Vanishing Vanishing
Source: J. Hayes point point

10 11

Image profile How to find edges?

• An edge is a place of rapid change in the image
intensity function
‒ Extrema of 1st derivate
‒ Zero-crossing of 2nd derivate

https://www.mathworks.com/help/images/intensity-profile-of-images.html
12 13
 22/05/2021

How to find edges? Image gradients

The first
derivative

The second
derivative

Source : Caroline Rougier. Traitement d'images (IFT2730). Univ. de Montréal.

14 15

Derivatives Types of Discrete derivative in 1D

• Derivative in 1D:

Backward

• Discrete derivative in 1D
Forward

Central

16 17
 22/05/2021

1D discrete derivate filters 1D discrete derivate example

• Backward filter: • Backward filter: [0 1 -1]
[0 1 -1]

• Forward:
[-1 1 0]
f(x) : 0 0 0 0 0 50 50 50 50 50
f'(x): 0 0 0 0 0 50 0 0 0 0
• Central:
[1 0 -1]

18 19

Discrete derivate in 2D Discrete derivate filters wrt. x and y

x

y
0
0 1 -1 1
-1

-1 1
-1 1 0 1 0 1 0 -1
0 -1

20 21
 22/05/2021

Image I (9 x 8)
0 0 0 0 0 0 0 0 0
Discrete derivate filters wrt. x and y 0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
Mx: Derivative filter wrt x
1 0 -1
0 0 0 0 0 0 0 0 0
• Robert filter (the first approximation filter for image derivative - 0 0 0 0 0 0 0 0 0 1
0 0 0 100 100 100 100 100 100
1965) 0
0 0 0 100 100 100 100 100 100
0 0 0 100 100 100 100 100 100 -1
0 0 0 100 100 100 100 100 100 My: Derivative filter wrt y
0 0 0 100 100 100 100 100 100
0 0 0 100 100 100 100 100 100

1 0 -1 1 1 1 Ix = I * Mx = the 1st derivative wrt x

• Prewitt filter 1/3 x 1 0 -1 1/3 x 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
1 -0 -1 -1 -1 -1 0 0 0 0 0 0 0
0 0 0 0 0 0 0
1 0 -1 1 2 1 0 100 100 0 0 0 0
0 100 100 0 0 0 0
• Sobel filter 1/4 x 2 0 -2 1/4 x 0 0 0 0 100 100 0 0 0 0
1 -0 -1 -1 -2 -1 0 100 100 0 0 0 0
0 100 100 0 0 0 0
22 23

Image I (9 x 8)
0 0 0 0 0 0 0 0 0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
Mx: Derivative filter wrt x
1 0 -1
3x3 image gradient filters
0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 1
0 0 0 100 100 100 100 100 100
0
0 0 0 100 100 100 100 100 100
0 0 0 100 100 100 100 100 100 -1
0 0 0 100 100 100 100 100 100 My: Derivative filter wrt y
0 0 0 100 100 100 100 100 100
0 0 0 100 100 100 100 100 100

Iy = I * My = the 1st derivative wrt y

0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 100 100 100 100 100
0 0 100 100 100 100 100
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
24 25
 22/05/2021

Image gradient Image gradient

• Derivative of image wrt. x + Derivative wrt. y • The gradient of an image:

Image gradient: magnitude and direction

• Gradient magnitude: gradient intensity for each

pixel (mostly used) • The gradient vector points in the direction of most rapid increase in intensity

• The gradient direction is given by

𝐺 = (𝐺 +𝐺 ) ≈ 𝐺 + 𝐺
• how does this relate to the direction of the edge?

• Gradient Direction: main direction of each pixel • The edge strength is given by the gradient magnitude

𝜃 = tan 𝐺 /𝐺
Source: Steve Seitz

26 27

Example Edge detector

Original Gradient
Image magnitude

x-direction y-direction

• Which one is the gradient in the x-direction? How about y-direction?

CS231: Lecture 3 Juan Carlos Niebles and Ranjay Krishna

28 29
 22/05/2021

Type of edges Effects of noise

Step Ramp Roof

Source : Gonzalez and Woods. Digital Image Processing 3ed. Prentice-Hall, 2008. Source : Gonzalez and Woods. Digital Image Processing 3ed. Prentice-Hall, 2008.

30 31

Effects of noise Effects of noise

• Consider a single row or column of the image • Solution: smoothing the image
‒ Plotting intensity as a function of position gives a signal

Where is the edge? Where is the edge?

Source: S. Seitz Source: S. Seitz

32 33
 22/05/2021

Solution: smooth first Derivative theorem of convolution

• This theorem gives us a very useful property:
f

• This saves us one operation:

f*g f

d d
g
( f  g) dx
dx

d d
• To find edges, look for peaks in ( f  g) f g
dx dx
Source: S. Seitz
Source: S. Seitz

34 35

Derivative theorem of convolution Sobel Operator

• This theorem gives us a very useful property: • Gaussian Smoothing + differentiation
Gaussian
smoothing

1 0 −1 1
• This saves us one operation: 𝐺 = 2 0 −2 = 2 1 0 −1
f
1 0 −1 1
differentiation
d
g
1 2 1 1
dx 𝐺 = 0 0 0 = 0 1 2 1
−1 −2 −1 −1
d
f g
dx
 Less sensible to noise
Source: S. Seitz

36 37
 22/05/2021

Prewitt Operator Simple edge detector with 1st derivative

• Mean smoothing + differentiation • Convolve the original image with 2 kernels to calculate
approximations
1 0 −1 1 of the derivatives
𝐺𝑥 = 1 0 −1 = 1 1 0 −1
1 0 −1 1
• Compute the gradient magnitude
1 1 1 1
𝐺𝑦 = 0 0 0 = 0 1 1 1
−1 −1 −1 −1

• Thesholding: choose edges to be the pixel above a

 Less sensible to noise threshold T

38 39

Simple edge detector with 1st derivative Problems

Gradient magnitude • Poor localization (multiple adjacent pixels)
Original
using
image
Sobel operator
• Thresholding value favors certain directions over
others
‒ Can miss oblique edges more than horizontal or vertical
edges  False negatives

Threshold
Threshold T = 60
T = 25

Source : Caroline Rougier. Traitement d'images (IFT2730). Univ. de Montréal.

40
 22/05/2021

Edge detector with 2nd derivative Edge detector with 2nd derivative
• 2nd derivative with Laplacian filter:
‒ convolution the image with one of 2 filters

• Edge detection
‒ Compute the 2nd derivative of the images
‒ Find the zero-crossing pixels  edges
Zero-crossing

42 43

Edge detector with 2nd derivative “Optimal” edge detector

Image Laplacian • Criteria:
‒ Good detection: the optimal detector must minimize the probability
of false positives (detecting spurious edges caused by noise), as well
as that of false negatives (missing real edges)
‒ Good localization: the edges detected must be as close as possible
to the true edges
‒ Single response: the detector must return one point only for each
true edge point; that is, minimize the number of local maxima around
the true edge

‒ Single response
‒ Sensible to noise

44
 22/05/2021

Canny detector Canny detector

• This is probably the most widely used edge detector in • Optimal:
computer vision ‒ Detection: weak edges detected
‒ Good location: close to the real edges
• Theoretical model: step-edges corrupted by additive ‒ Unique response: edge thickness = 1
Gaussian noise

• Canny has shown that the first derivative of the

Gaussian closely approximates the operator that
optimizes the product of signal-to-noise ratio and
localization

J. Canny, A Computational Approach To Edge Detection, IEEE Trans. PAMI, 8:679-714, 1986.

Canny detector: Steps Canny detector: Steps

1) Apply a gaussian filter on the image 4) Non-maxima suppression
‒ Lowpass filter to remove noise ‒ If the gradient magnitude of a pixel
2) Compute the gradient intensity in the image (x,y) is inferior to the one of its 2
‒ Sobel filter in X and Y
neighbors along the gradient
direction
‒ Compute the magnitude | G | = |Gx | + |Gy |
3) Compute image gradient direction
 set this magnitude for (x,y) to
‒ Gradient direction θ = arctan (Gy / Gx)
zero
‒ Round directions using multiples of 45°
 22/05/2021

Canny detector: Steps

Hysteresis thresholding
5) Edge thresholding (hysteresis)
• Using two thresholds: a theshold high (Sh) and a
threshold low (Sb)
strong edge pixel weak but connected
• For each pixel in the gradient magnitude: edge pixels

‒ IF magnitude(x,y) < Sb, THEN set the pixel to zero (non-edge)

strong edge pixel
‒ IF magnitude(x,y) > Sh, THEN the edge is an edge
‒ IF Sb ≤ magnitude(x,y) ≤ Sh, THEN the pixel is an edge IF it
is connected to another edge pixel

Source: S. Seitz

Canny detector 45 years of boundary detection

Input image Sobel

Non-maxima Thresholding
suppression

Source: Arbelaez, Maire, Fowlkes, and Malik. TPAMI 2011 (pdf)

 22/05/2021

Hough transform

Edge linking
Hough Transform
RANSAC

54 56

Hough transform Hough transform

• The Hough transform (HT)  Global approach to detect continuous edges
‒ can be used to detect lines  From the x-y plane to the parametric plane a-b

‒ was introduced in 1962 (Hough 1962) and first used to

find lines in images a decade later (Duda 1972)  x-y plane
 yi = a xi + b
‒ Goal: to find the location of lines in images.
 an infinity of lines going though one (xi, yi) pair
• Caveat: Hough transform can detect lines, circles  one sole line for the (a,b) pair
and other structures ONLY if their parametric
equation is known  a-b parametric plane
• It can give robust detection under noise and partial  b = - xi a + yi
one sole line for the (xi, yi) pair
occlusion


 an infinity of lines going through one (a,b) pair

58
 22/05/2021

x-y plane vs a-b plane Line vs Points

All the points (x,y) on a line in the x-y plane are going
x-y plane a-b parametric plane
through one sole point (a’, b’) in the a-b parametric plane
x-y plane a-b parametric plane

y i = a xi + b b = - xi a + yi
yi = a x i + b b = - xi a + y i
59 60

Main idea for the Hough transform Computing the Hough transform
 Accumulation cells - Matrix (a,b)  We compute the contour points of the input image
 Build an voting image  Sobel, Prewitt, Canny, …
 each point is voting for a particular line
 For each contour point, we compute a line (a,b)
 The lines receiving more votes are kept
 Result is one line in the a-b plane for each pixel (x,y)

 The maximum peaks in the a-b parametric plane

show the lines with the maximum of points in the
x-y plane
 The points indicating line crossing in the a-b plane show
the real lines existing in the x-y plane

61 62
 22/05/2021

Problem with the (a,b) space Example with 5 points

 Problem: for a vertical line, a=∞ !
C
 Solution: representing using polar coordinates (,θ) 1 2 2
 = x cos  + y sin 

D A E 1
3

3
4 B
F
θ = ±90 4 5
Image containing 5
5 points
Hough transform
Source : Gonzalez and Woods. Digital Image Processing 3ed. Prentice-Hall, 2008.
63 64

Hough transform (random points) Hough transform (straight line)

The transform for aligned points result in a line detection

The transform of random points does not give any precise results

65 66
 22/05/2021

Example Other example

Image Canny
Image Gradient

Hough Final
Lines
Hough Final

Source : Gonzalez and Woods. Digital Image Processing. Prentice-Hall, 2002. 67 Source : Gonzalez and Woods. Digital Image Processing. Prentice-Hall 3ed, 2008. 68

Ransac Ransac
• A model fitting method:
‒ A learning technique to estimate parameters of a model
by random sampling of observed data
‒ Used for :
• Line detection
• Correspondance problem (matching between 2 sets of
features)
• …

CS231: Ransac, Juan Carlos Niebles and Ranjay Krishna, Stanford Vision and Learning Lab

69 70
 22/05/2021

Example: Line Fitting Fitting as search in parametric space

• Why fit lines? • Choose a parametric model to represent a set of
‒ Many objects characterized by presence of straight features
lines • Membership criterion is not local
‒ Can’t tell whether a point belongs to a given model just by
looking at that point.
• Three main questions:
‒ What model represents this set of features best?
‒ Which of several model instances gets which feature?
‒ How many model instances are there?
• Computational complexity is important
‒ It is infeasible to examine every possible set of parameters
and every possible combination of features
Slide credit: Kristen Grauman
Source: L. Lazebnik

Difficulty of Line Fitting Voting

• Extra edge points (clutter), • It’s not feasible to check all combinations of features by
multiple models: fitting a model to each possible subset.
‒ Which points go with • Voting is a general technique where we let the features
which line, if any?
vote for all models that are compatible with it.
• Only some parts of each line
‒ Cycle through features, cast votes for model parameters.
detected, and some parts are
missing: ‒ Look for model parameters that receive a lot of votes.
‒ How to find a line that • Noise & clutter features will cast votes too, but typically
bridges missing evidence? their votes should be inconsistent with the majority of
• Noise in measured edge points, “good” features.
orientations: • Ok if some features not observed, as model can span
‒ How to detect true underlying multiple fragments.
parameters?
Slide credit: Kristen Grauman
Slide credit: Kristen Grauman
 22/05/2021

RANSAC [Fischler & Bolles 1981] RANSAC [Fischler & Bolles 1981]

• RANdom SAmple Consensus RANSAC loop:

1. Randomly select a seed group of points on which to
• Approach: we want to avoid the impact of outliers, base transformation estimate (e.g., a group of
so let’s look for “inliers”, and use only those matches)
2. Compute transformation from seed group
3. Find inliers to this transformation
• Intuition: if an outlier is chosen to compute the
current fit, then the resulting line won’t have much 4. If the number of inliers is sufficiently large, re-compute
least-squares estimate of transformation on all of the
support from rest of the points. inliers
• Keep the transformation with the largest number of
inliers
Slide credit: Kristen Grauman Slide credit: Kristen Grauman

RANSAC Line Fitting Example RANSAC Line Fitting Example

• Task: Estimate the best line • Task: Estimate the best line
‒ How many points do we need to estimate the line?

Sample two points

Slide credit: Kristen Grauman Slide credit: Kristen Grauman

 22/05/2021

RANSAC Line Fitting Example RANSAC Line Fitting Example

• Task: Estimate the best line • Task: Estimate the best line

Fit a line to them Total number of points

within a threshold of
line.

Slide credit: Kristen Grauman Slide credit: Kristen Grauman

RANSAC Line Fitting Example RANSAC Line Fitting Example

• Task: Estimate the best line • Task: Estimate the best line

“7 inlier points”

Total number of points Repeat, until we get a good

within a threshold of result.
line.

Slide credit: Kristen Grauman

Slide credit: Kristen Grauman
 22/05/2021

RANSAC Line Fitting Example

• Task: Estimate the best line

“11 inlier points”

Repeat, until we get a

good result.

Slide credit: Kristen Grauman

RANSAC: How many samples? RANSAC: Computed k (p=0.99)

• How many samples are needed?
‒ Suppose w is fraction of inliers (points from line)
Sample Proportion of outliers
‒ n points needed to define hypothesis (2 for lines) size
n 5% 10% 20% 25% 30% 40% 50%
‒ k samples chosen
2 2 3 5 6 7 11 17
‒ p: desired probability that we get a good sample
3 3 4 7 9 11 19 35
4 3 5 9 13 17 34 72
• Prob. that a single sample of n points is correct: wn 5 4 6 12 17 26 57 146
6 4 7 16 24 37 97 293
• Prob. that all k samples fail is: (1  w )
n k
7 4 8 20 33 54 163 588
8 5 9 26 44 78 272 1177
 Choose k high enough to keep this below desired
failure rate (1-p). p: desired probability that we get a good sample

Slide credit: David Lowe Slide credit: David Lowe

 22/05/2021

After RANSAC RANSAC: Pros and Cons

• RANSAC divides data into inliers and outliers and yields • Pros:
estimate computed from minimal set of inliers. ‒ General method suited for a wide range of model fitting
• Improve this initial estimate with estimation over all inliers (e.g. problems
with standard least-squares minimization). ‒ Easy to implement and easy to calculate its failure rate
• But this may change inliers, so alternate fitting with re- • Cons:
classification as inlier/outlier.
‒ Only handles a moderate percentage of outliers without
cost blowing up
‒ Many real problems have high rate of outliers (but
sometimes selective choice of random subsets can help)
• A voting strategy, the Hough transform can handle
high percentage of outliers
Slide credit: David Lowe

Plan References
• Edges • Lecture 3: CS231 - Juan Carlos Niebles and Ranjay
‒ Detection Krishna, Stanford Vision and Learning Lab
‒ linking • Vision par Ordinateur, Alain Boucher, IFI
• Feature extraction (next lesson)
‒ Global features
‒ Local features
• Image Matching and Applications (next lesson)

89 90
 22/05/2021

Thank you for

your attention!

91