Computer Vision April 21, 2021

Lecture 7
Spatial Image Processing

Dr. Muhammad Jawad Khan

Robotics and Intelligent Machine Engineering Department,

School of Mechanical and Manufacturing,
National University of Sciences and Technology,
Islamabad, Pakistan.
\
Robotics and Intelligent Machine Engineering, NUST 1 / ??
Acknowledgement

• Special thanks to Dr. Hasan Sajid

• 16-385 Computer Vision (CMU)

Robotics and Intelligent Machine Engineering, NUST 2 / ??

Spatial Image Processing: Correlation

• Applying Kernel to an image

Robotics and Intelligent Machine Engineering, NUST 3 / ??

• xcorr2 is not the best for finding the object
unless you know the exact peak auto
correlation value. A much more robust method
is to use normxcorr2() where you simply have
to look for the max. (If you look for the peak
using xcorr2(), it will not necessarily lie where
your template occurs in the image, it could be
over any old bright part of the image).

Robotics and Intelligent Machine Engineering, NUST 4 / ??

% Load images
onion = rgb2gray(imread('onion.png'));
peppers = rgb2gray(imread('peppers.png'));
imshowpair(peppers,onion,'montage')

c = normxcorr2(onion,peppers);
figure, surf(c), shading flat

[ypeak, xpeak] = find(c==max(c(:)));

% Compute translation from max location in correlation matrix
yoffSet = ypeak-size(onion,1);
xoffSet = xpeak-size(onion,2);

% Display matched area

figure
hAx = axes;
imshow(peppers,'Parent', hAx);
drawrectangle(hAx, 'Position', [xoffSet+1, yoffSet+1, size(onion,2), size(onion,1)]);

Robotics and Intelligent Machine Engineering, NUST 5 / ??

Demo code: For you to try
clc; % Clear the command window.
close all; % Close all figures (except those of imtool.)
imtool close all; % Close all imtool figures.
clear; % Erase all existing variables (which I know Jan hates).
workspace; % Make sure the workspace panel is showing.
format longg;
format compact;
fontSize = 20;
% Check that user has the Image Processing Toolbox installed.
hasIPT = license('test', 'image_toolbox');
if ~hasIPT
% User does not have the toolbox installed.
message = sprintf('Sorry, but you do not seem to have the Image Processing Toolbox.\nDo you want to try to
continue anyway?');
reply = questdlg(message, 'Toolbox missing', 'Yes', 'No', 'Yes');
if strcmpi(reply, 'No')
% User said No, so exit.
return;
end
end

Robotics and Intelligent Machine Engineering, NUST 6 / ??

% Read in a standard MATLAB color demo image.
folder = fullfile(matlabroot, '\toolbox\images\imdemos');
baseFileName = 'peppers.png';
% Get the full filename, with path prepended.
fullFileName = fullfile(folder, baseFileName);
if ~exist(fullFileName, 'file')
% Didn't find it there. Check the search path for it.
fullFileName = baseFileName; % No path this time.
if ~exist(fullFileName, 'file')
% Still didn't find it. Alert user.
errorMessage = sprintf('Error: %s does not exist.', fullFileName);
uiwait(warndlg(errorMessage));
return;
end
end

Robotics and Intelligent Machine Engineering, NUST 7 / ??

rgbImage = imread(fullFileName);
% Get the dimensions of the image. numberOfColorBands should be = 3.
[rows columns numberOfColorBands] = size(rgbImage);
% Display the original color image.
subplot(2, 2, 1);
imshow(rgbImage, []);
axis on;
title('Original Color Image', 'FontSize', fontSize);
% Enlarge figure to full screen.
set(gcf, 'units','normalized','outerposition',[0 0 1 1]);
smallSubImage = imcrop(rgbImage, [192 82 60 52]);
subplot(2, 2, 2);
imshow(smallSubImage, []);
axis on;
title('Template Image to Search For', 'FontSize', fontSize);
% Search the red channel for a match.
correlationOutput = normxcorr2(smallSubImage(:,:,1), rgbImage(:,:,1));
subplot(2, 2, 3);
imshow(correlationOutput, []);
title('Correlation Output', 'FontSize', fontSize);
[maxCorrValue, maxIndex] = max(abs(correlationOutput(:)));
[ypeak, xpeak] = ind2sub(size(correlationOutput),maxIndex(1));
corr_offset = [(xpeak-size(smallSubImage,2)) (ypeak-size(smallSubImage,1))];
subplot(2, 2, 4);
imshow(rgbImage);
hold on;
rectangle('position',[corr_offset(1) corr_offset(2) 50 50],...
'edgecolor','g','linewidth',2);
title('Template Image Found in Original Image', 'FontSize', fontSize);

Robotics and Intelligent Machine Engineering, NUST 8 / ??

Spatial Image Processing: Convolution

• Convolution is very similar to correlation

• Convolution and correlation are same if kernel

is symmetric which is in fact the case for
Gaussian kernel
• Operator form

Robotics and Intelligent Machine Engineering, NUST 9 / ??

Spatial Image Processing : Convolution

• Properties of Convolution *Convolution of 2

Gaussians is a Gaussian
– Commutative

– Associative

– Distributive

– Linear

Robotics and Intelligent Machine Engineering, NUST 10 / ??

Spatial Image Processing: Edge Detection

• Horizontal Gradient can be approximated as

Robotics and Intelligent Machine Engineering, NUST 11 / ??

Spatial Image Processing: Edge Detection

Robotics and Intelligent Machine Engineering, NUST 12 / ??

Spatial Image Processing: Edge Detection

• Horizontal Gradient Image  Vertical Lines

Robotics and Intelligent Machine Engineering, NUST 13 / ??

Spatial Image Processing: Edge Detection

• Detecting horizontal lines/edges?

• Sobel Kernel: for detecting horizontal and
vertical gradients simultaneously

Robotics and Intelligent Machine Engineering, NUST 14 / ??

Spatial Image Processing: Kernels

• Gaussian kernel is very common due to

smoothening property and removing noise
• Derivative kernel on an image results in
enhancing the noise

• We tackle this problem by reducing noise first

and then apply derivative kernel

Robotics and Intelligent Machine Engineering, NUST 15 / ??

Spatial Image Processing: Kernels

• Derivative of Gaussian Kernel

Robotics and Intelligent Machine Engineering, NUST 16 / ??

Spatial Image Processing: Kernels

• Second Derivative
– Laplacian is a second order isotropic derivative
• Provide the gradient maxima in both directions

Robotics and Intelligent Machine Engineering, NUST 17 / ??

Spatial Image Processing: Kernels

• Where the Laplacian Kernel coefficients

comes from?

Robotics and Intelligent Machine Engineering, NUST 18 / ??

Spatial Image Processing: Kernels

• Second Derivative
– More noisier than first derivative so we adopt the
same approach of reducing the noise followed by
gradient computation

Robotics and Intelligent Machine Engineering, NUST 19 / ??

Spatial Image Processing: Kernels

• Laplacian of Gaussian Kernel

Robotics and Intelligent Machine Engineering, NUST 20 / ??

Spatial Image Processing for Template Matching

• Goal is to find a particular object/pattern in an image

Robotics and Intelligent Machine Engineering, NUST 21 / ??

Spatial Image Processing for Template Matching

• We exploit spatial operator approach

Robotics and Intelligent Machine Engineering, NUST 22 / ??

Spatial Image Processing for Template Matching

• Similarity Measures
– Sum of Absolute Differences (SAD)

Gold Standard for

– Sum of Squared Differences (SSD) Template
Matching

– Zero Mean Normalized Cross Correlation (ZNCC)

Robotics and Intelligent Machine Engineering, NUST 23 / ??

Spatial Image Processing: Morphology

• Morphology: spatial processing sensitive to

shape
• Involves a structuring element with a specific
pattern
• Morphological Operations include
– Erosion
– Dilation

Robotics and Intelligent Machine Engineering, NUST 24 / ??

Mathematical Morphology

• Morphology means shape

• A type of spatial image processing that is
sensitive to shape i.e. the processed image
contains shapes with a structuring element S
• Structuring element is a pattern which
represents a subset of pixels within the input
window we intend to consider
• Example applications: erosion, dilation etc

Robotics and Intelligent Machine Engineering, NUST 25 / ??

Mathematical Morphology

• Erosion: output is true, if ALL pixels under the window are

true else zero.
• Output is true only if there is a shape/pattern match.

Window
In red

Robotics and Intelligent Machine Engineering, NUST 26 / ??

Mathematical Morphology

• Dilation: output is true, if ANY pixels under the window are

true else zero.

Window
In red

Robotics and Intelligent Machine Engineering, NUST 27 / ??

Mathematical Morphology

• Relationship between erosion and dilation

Assignment: Proof of above equation

To be submitted before next class

Robotics and Intelligent Machine Engineering, NUST 28 / ??

Mathematical Morphology

• Opening: erosion followed by dilation

Robotics and Intelligent Machine Engineering, NUST 29 / ??

Mathematical Morphology

• Closing: dilation followed by erosion

Robotics and Intelligent Machine Engineering, NUST 30 / ??

Mathematical Morphology

• Example Application: Finding diagonal lines

• Use a diagonal structuring element
Diagonal Structuring element Input Image

Robotics and Intelligent Machine Engineering, NUST 31 / ??

Mathematical Morphology

Robotics and Intelligent Machine Engineering, NUST 32 / ??

Mathematical Morphology

• Hit and miss structuring element

Robotics and Intelligent Machine Engineering, NUST 33 / ??

Mathematical Morphology

• Application: Skeletonization

Can be further processed to find intersections ……

Robotics and Intelligent Machine Engineering, NUST 34 / ??

Image Scaling

The process of
increasing/decreasing an
image’s size.
a. Original Image
b. Sub sampling every
7th pixel in both
directions
c. Sub sampling every
7th pixel in both
directions after
smoothing (i.e. first
remove high
frequency component
by blurring with a
gaussian kernel then
subsample)
d. Up sampling image in
c to original size by
pixel replication
Robotics and Intelligent Machine Engineering, NUST 35 / ??
Image Pyramid

• Same Image at multiple scales

Robotics and Intelligent Machine Engineering, NUST 36 / ??

Image Warping

• Image warping is a transformation of the pixel

coordinates rather than the pixel values.
• Warping can be used to scale an image up or
down in size, rotate an image or apply quite
arbitrary shape changes.
• The coordinates of a pixel in the new view (u
′,v′) are expressed as functions of the
coordinates in the original view

Robotics and Intelligent Machine Engineering, NUST 37 / ??

Image Warping

Original Image Scaling

Scaling and Shifting Rotation

Robotics and Intelligent Machine Engineering, NUST 38 / ??

Example: Scale and Shift

• Example: image reduction in size by a factor

of 4 in both directions and offset so that its
origin, its top-left corner, is shifted to the
coordinate (100, 200)

• Now, for every pixel in the output image the

corresponding coordinate in the input image is
given by the inverse of the functions f u and fv.

Robotics and Intelligent Machine Engineering, NUST 39 / ??

Example: Scale and Shift

Robotics and Intelligent Machine Engineering, NUST 40 / ??

Example: Scale and Shift

• Let’s suppose we want to find corresponding

input image coordinates (u, v) for particular
output image coordinates (u’, v’) = (303, 269),
we find using

Problematic?

Robotics and Intelligent Machine Engineering, NUST 41 / ??

Pixel Value Interpolation

• Take the closest value

i.e. 115 in this
particular case
• Weighted average
such as bilinear
interpolation

Robotics and Intelligent Machine Engineering, NUST 42 / ??

Example: Scale and Shift

• What will happen for (u’, v’) = (500, 300)?

Robotics and Intelligent Machine Engineering, NUST 43 / ??

 Feature Extraction

Robotics and Intelligent Machine Engineering, NUST 44 / ??

Image Features

• Images are simply large arrays of pixel values

• From this array we need to answer many
questions
– Is there any object in the scene?
– what is the pose of the object?
– what type of object is it?
– How fast is it moving? Etc
• Making sense of this array to answer such
questions is what we call as image features.

Robotics and Intelligent Machine Engineering, NUST 45 / ??

Image Features

• In contrast to image processing where the

output is an image, feature extraction process
returns image features (scalars as well as
vectors)
• Multiple classes of features exist
– Regions
– Lines
– Interest points

Robotics and Intelligent Machine Engineering, NUST 46 / ??

Image Features

• Region Features
– contiguous groups of pixels that are homogeneous
with respect to some pixel property. For example
the set of pixels that represent a red object against
a non-red background
• Line features
– describe straight lines in the world. Straight lines
are distinct and very common in man-made
environments – for example the edges of
doorways, buildings or roads

Robotics and Intelligent Machine Engineering, NUST 47 / ??

Image Features

• Interest Points
– distinctive points in a scene which can be reliably
detected in different views of the same scene

Robotics and Intelligent Machine Engineering, NUST 48 / ??

Line Features

• Lines are distinct visual features that are

particularly common in man-made
environments – for example the edges of
roads, buildings and doorways
• we have seen how image intensity gradients
can be used to find edges within an image,
• Now we focus on fitting line segments to such
edges

Robotics and Intelligent Machine Engineering, NUST 49 / ??

Line Features

• How can we group pixels into lines?

Are we done?

Robotics and Intelligent Machine Engineering, NUST 50 / ??

Hough Transform

• Based on simple voting system

Robotics and Intelligent Machine Engineering, NUST 51 / ??

Hough Transform

• Line is represented as
• Problematic for vertical lines where
• To handle infinity problems we represent line
with polar coordinates (ρ,θ) i.e. we work in
another space

Robotics and Intelligent Machine Engineering, NUST 52 / ??

Hough Transform

Polar representation of line

θ is the angle line makes with

horizontal axis

ρ is the perpendicular distance

between origin and a point on
line

Robotics and Intelligent Machine Engineering, NUST 53 / ??

Hough Transform

• Handling infinite number of lines through each

point is intractable, therefore we quantize the
polar coordinate space (ρ,θ)

Robotics and Intelligent Machine Engineering, NUST 54 / ??

Hough Transform

• Each cell represents a line in the image

• For every point (u, v) that lies on a line we vote for all the lines
(ρ,θ) that pass through it, such that
Robotics and Intelligent Machine Engineering, NUST 55 / ??
Example

Most of the array contains zero votes (dark blue) and the light curves are
trails of single votes corresponding to each of the five input points. These
curves intersect and those points correspond to lines with more than one
vote. We see four locations where two curves intersect, resulting in cells
with two votes, and these correspond to the lines joining the four outside
points
Robotics and Intelligent Machine Engineering, NUST 56 / ??
Hough Transform

• Failure cases:
– scene contains a lot of texture
– the edges are indistinct
• Inherent Limitation: Hough transform
estimates the direction of the line by fitting
lines to the edge pixels. It ignores rich
information about the direction of the edge at
each pixel

Robotics and Intelligent Machine Engineering, NUST 57 / ??

Motivation Again

• Data mismatch problem …….

Robotics and Intelligent Machine Engineering, NUST 58 / ??

Point Features

• These are visually distinct points in the image that

are known as interest points, salient points, key
points or corner points
• Interest points are quite distinct so they have a much
higher likelihood of being reliably detected in
different views of the same scene. They are
therefore key to multi-view techniques such as
stereo and motion estimation
• We will study both classical techniques and recent
scale-invariant techniques
Robotics and Intelligent Machine Engineering, NUST 59 / ??
Classical Corner Detector

• A point on a line has a strong gradient in a

direction normal to the line. However gradient
along the line is low which means that a pixel on
the line will look very much like its neighbors
along the line
• An interest point is a point that has a high image
gradient in orthogonal directions. It might be
single pixel that has a significantly different
intensity to all of its neighbors or it might literally
be a pixel on the corner of an object
Robotics and Intelligent Machine Engineering, NUST 60 / ??
Classical Corner Detector

• Two images of same scene but offset with

respect to each other

Robotics and Intelligent Machine Engineering, NUST 61 / ??

Classical Corner Detector

• Correspondence problem is a fundamental

vision problem

Robotics and Intelligent Machine Engineering, NUST 62 / ??

Classical Corner Detector

• Can we use intensity value to solve

correspondence problem?

Robotics and Intelligent Machine Engineering, NUST 63 / ??

Classical Corner Detector

• Can we always find corresponding point?

Robotics and Intelligent Machine Engineering, NUST 64 / ??

Classical Corner Detector

• Intuition

# of Unique points?

Robotics and Intelligent Machine Engineering, NUST 65 / ??

Classical Corner Detector

• Formal Definition

Similarity is measured between central window and shifted window in all eight cardinal
directions

Measure of Cornerness
Most basic corner detector a.k.a. Moravec’s Corner Detector

Robotics and Intelligent Machine Engineering, NUST 66 / ??

Moravec’s Corner Detector Limitation

• The main limitation of the Moravec detector is

that it is nonisotropic since it examines image
change, essentially gradient, in a limited
number of directions. Consequently the
detector can give a strong output for a point on
a line, which is not desirable.

Robotics and Intelligent Machine Engineering, NUST 67 / ??

Classical Corner Detector

• General Approach: add Gaussian weighting matrix to give more

weight-age to points closer to the center

Taylor Series approximation

Robotics and Intelligent Machine Engineering, NUST 68 / ??

Classical Corner Detector

Robotics and Intelligent Machine Engineering, NUST 69 / ??

Classical Corner Detector

which is a symmetric 2×2 matrix referred to variously as the structure tensor,

autocorrelation matrix or second moment matrix. It captures the intensity structure
of the local neighborhood and its eigenvalues provide a rotationally invariant
description of the neighborhood. The elements of the A matrix are computed from
the image gradients, squared or multiplied, and then smoothed using a weighting
matrix. The latter reduces noise and improves the stability and reliability of the
detector.

Robotics and Intelligent Machine Engineering, NUST 70 / ??

Classical Corner Detector

• An interest point (u,v) is one for which s(·) is

high for all directions of the vector (δu,δv).
That is, in whatever direction we move the
window it rapidly becomes dissimilar to the
original region.
Shi-Tomasi detector

Harris corner detector

Robotics and Intelligent Machine Engineering, NUST 71 / ??

Harris Corner Detector

Robotics and Intelligent Machine Engineering, NUST 72 / ??

Harris Corner Detector

Robotics and Intelligent Machine Engineering, NUST 73 / ??

Harris Corner Detector

• The Harris detector is computed from image

gradients and is therefore robust to offsets in
illumination, and the eigenvalues of the
structure tensor A are invariant to rotation.
• Not invariant to changes in scale . As we
zoom in the gradients around the corner points
become lower.

Robotics and Intelligent Machine Engineering, NUST 74 / ??

Scale-Space Corner Detectors

• What happens when we zoom in/out on an object in an

image?

• The size (scale) of the object changes.

– Zoom in causes the object to increase in scale but becomes blurry
– Zoom out causes the object to decrease in scale but becomes more
sharp
• Let’s use this property to make corner detector scale-
invariant !
Robotics and Intelligent Machine Engineering, NUST 75 / ??