CVT Assignment

BT22CSA040-Sujal Badgujar
1. Explain the concepts of foreground and background in image segmentation using graph
cut
methods. How are these represented in a graph?
Ans: In image segmentation using graph cut methods, the concepts of foreground and
background are crucial for effectively separating the main objects of interest from the rest
of the image.
Foreground and Background
Foreground: This refers to the main object or region of interest in an image that we want to
segment. For instance, in an image of a cat, the cat itself would be considered the
foreground.
Background: This includes everything else in the image that is not part of the foreground. In
the cat example, the background would be the floor, furniture, or any other objects in the
scene.

Representation in a Graph
In graph cut methods, both foreground and background are represented using a graph
structure:
Graph Structure:
Each pixel in the image is treated as a node in the graph.
Nodes are connected by edges that represent the relationship between neighbouring pixels
(often based on colour similarity or spatial proximity).

Source and Sink:

A source node is added to represent the foreground. This node is connected to nodes
corresponding to pixels that are likely part of the foreground.
A sink node represents the background. It is connected to nodes corresponding to pixels
likely belonging to the background.

Edge Weights:
Edges between neighbouring pixels (nodes) have weights that reflect the similarity between
those pixels. Lower weights typically indicate higher similarity (i.e., the pixels are more likely
to belong to the same segment).
Edges from the source node to foreground pixels and from background pixels to the sink
node also have weights, which represent the confidence of a pixel being foreground or
background.

Graph Cut Optimization

The goal is to find the minimum cut that separates the source from the sink. This cut
effectively partitions the graph into two sets: one corresponding to the foreground and the
other to the background. The edges that cross the cut (from foreground to background)
represent the least "cost" and thus define the boundary between the two regions.

Q2. Write a note on how minimum cuts are related to energy minimization in image
segmentation.
Minimum cuts and energy minimization are closely related concepts in image segmentation,
particularly within the framework of graph-based methods.
Energy Minimization in Image Segmentation
In image segmentation, the goal is to partition an image into segments (typically foreground
and background) while minimizing a certain cost associated with the segmentation. This
cost is often represented as an energy function that captures the overall quality of the
segmentation.
Components of the Energy Function
Data Term: This measures how well the pixels match their assigned labels (foreground or
background). It quantifies the likelihood that a pixel belongs to either segment based on
features such as colour or texture.
Smoothness Term: This encourages neighbouring pixels to have similar labels, promoting
spatial coherence. It discourages abrupt changes in labelling, which could lead to noisy
segmentations.
Relationship to Minimum Cuts
Graph Representation: As previously mentioned, each pixel is represented as a node in a
graph, with edges connecting neighbouring pixels. The source and sink nodes represent the
foreground and background, respectively.
Edge Weights: The edge weights between nodes (pixels) reflect the cost of separating those
pixels into different segments. The weights are derived from the data and smoothness
terms of the energy function.
Minimum Cut: Finding the minimum cut in the graph corresponds to identifying the
partition that minimizes the total edge weight crossing the cut. This means that the cut
represents the segmentation that has the lowest energy cost, effectively minimizing the
energy function.
Q3. Consider a small 4x4 image with pixel intensity values as follows:

Ans. Row-wise (horizontal connections):

● Row 1:
○ |10 - 12| = 2 (Edge between pixel (1,1) and (1,2))
○ |12 - 13| = 1 (Edge between pixel (1,2) and (1,3))
○ |13 - 15| = 2 (Edge between pixel (1,3) and (1,4))
● Row 2:
○ |20 - 22| = 2 (Edge between pixel (2,1) and (2,2))
○ |22 - 21| = 1 (Edge between pixel (2,2) and (2,3))
○ |21 - 19| = 2 (Edge between pixel (2,3) and (2,4))
● Row 3:
○ |23 - 25| = 2 (Edge between pixel (3,1) and (3,2))
○ |25 - 27| = 2 (Edge between pixel (3,2) and (3,3))
○ |27 - 30| = 3 (Edge between pixel (3,3) and (3,4))
● Row 4:
○ |18 - 16| = 2 (Edge between pixel (4,1) and (4,2))
○ |16 - 15| = 1 (Edge between pixel (4,2) and (4,3))
○ |15 - 12| = 3 (Edge between pixel (4,3) and (4,4))
Column-wise (vertical connections):
● Column 1:
○ |10 - 20| = 10 (Edge between pixel (1,1) and (2,1))
○ |20 - 23| = 3 (Edge between pixel (2,1) and (3,1))
○ |23 - 18| = 5 (Edge between pixel (3,1) and (4,1))
● Column 2:
○ |12 - 22| = 10 (Edge between pixel (1,2) and (2,2))
○ |22 - 25| = 3 (Edge between pixel (2,2) and (3,2))
○ |25 - 16| = 9 (Edge between pixel (3,2) and (4,2))
● Column 3:
○ |13 - 21| = 8 (Edge between pixel (1,3) and (2,3))
○ |21 - 27| = 6 (Edge between pixel (2,3) and (3,3))
○ |27 - 15| = 12 (Edge between pixel (3,3) and (4,3))
● Column 4:
○ |15 - 19| = 4 (Edge between pixel (1,4) and (2,4))
○ |19 - 30| = 11 (Edge between pixel (2,4) and (3,4))
○ |30 - 12| = 18 (Edge between pixel (3,4) and (4,4)
(1,1) --2-- (1,2) --1-- (1,3) --2-- (1,4)
| | | |
10 10 8 4
| | | |
(2,1) --2-- (2,2) --1-- (2,3) --2-- (2,4)
| | | |
3 3 6 11
| | | |
(3,1) --2-- (3,2) --2-- (3,3) --3-- (3,4)
| | | |
5 9 12 18
| | | |
(4,1) --2-- (4,2) --1-- (4,3) --3-- (4,4)
Q4. Explain the role of each term in this energy function. How does each term influence
the
segmentation result?
Ans.
Data Term E(data)(L):
The data term measures how well the segmentation fits the observed data, or how much
the labelling L aligns with the observed pixel intensities.
It encourages the segmentation to assign labels (foreground or background) to pixels based
on how similar their observed intensities are to the reference intensities of the respective
labels (foreground or background).
Smoothness term E(smoothness)(L):
The smoothness term penalizes label discontinuities between neighbouring pixels, where
adjacent pixels with different labels (one foreground, one background) incur a penalty.
This term enforces smoothness in the segmentation, discouraging abrupt changes between
neighbouring pixels unless supported by the image data.
It helps to create more homogeneous regions in the segmentation by avoiding noise and
small, isolated regions that are likely the result of random pixel intensity variations.

Influence on the Segmentation Result:

The data term pulls the segmentation toward fitting the observed image intensities as
closely as possible. If this term dominates, the result might overfit the image, leading to a
noisy segmentation with too much sensitivity to small intensity variations.
The smoothness term tries to make the segmentation smoother by encouraging
neighbouring pixels to have the same label. If this term dominates, the result will be a very
smooth segmentation, potentially over smoothing boundaries between foreground and
background objects.
Balancing these terms is critical for getting a good segmentation result that accurately
captures object boundaries without being too noisy or too smooth.

Q5. For the same 4x4 image provided in Question 2, calculate the data term E data (L) for
a
simple segmentation where the top-left half is labeled as the foreground and the bottom-
right
half is labeled as the background. Assume:
 Foreground pixels have a reference intensity of 15.
 Background pixels have a reference intensity of 25.
Ans: Calculating the Data Term Edata(L) for the 4x4 Image Let's assume the pixel intensity
values are 1 (x, y), and we have the following:
• Foreground pixels have a reference intensity of 1 FG = 15.
• Background pixels have a reference intensity of IBg = 25.
• The top-left half of the 4x4 image is labeled as foreground, and the bottom-right half is
labeled as background.
If we define the image intensities as follows:
• Top-left (first 2 rows and columns) is foreground.
• Bottom-right (last 2 rows and columns) is background.
The data term Edata (L) is computed as the sum of the squared differences between the
observed intensities and the reference intensities for each pixel:

Q6. Calculate the smoothness term E smoothness (L) for the same segmentation,
assuming that adjacent pixels with different labels incur a penalty of 5 units.
Ans: Calculating the Smoothness Term smoothness (L)
For the given 4x4 image:
• Adjacent pixels with different labels (foreground vs background) incur a penalty of 5 units.
• For a 4x4 image, check adjacent pairs of pixels both horizontally and vertically.
• Only consider the pairs where one pixel is labeled as foreground and the other as
background (across the boundary).

You can count the number of such discontinuities between adjacent pixels. If you have
access to the specific image segmentation or want guidance on how to calculate it, let me
know, and I can assist further.
In the simplest form:
Esmoothness (L) = 5 x (Number of adjacent pixels with different labels)
Q7. What is the GrabCut algorithm? How does it improve upon the basic graph cut
method?
Ans: The GrabCut algorithm is an advanced image segmentation technique that builds upon
the basic graph cut method, specifically designed to improve the segmentation of
foreground objects from the background in images. Here's an overview of how it works and
its enhancements over the basic graph cut approach.
Initialization: The user provides a rough bounding box around the object to be segmented.
This box defines the region of interest but does not require precise labeling.
Graph Construction: GrabCut constructs a graph where each pixel is a node. The graph
includes:
Source and Sink Nodes: Representing the foreground and background, respectively.
Edges: Weighted edges connect each pixel node to its neighbors and to the source and sink
nodes, based on color similarities.
Gaussian Mixture Models (GMM): Instead of using simple color histograms, GrabCut models
the color distributions of foreground and background using GMMs. This allows for more
accurate representation of pixel colors, accommodating variations and complexities in color
distributions.
Graph Cut Optimization: The algorithm computes the minimum cut of the graph to segment
the image, similar to basic graph cut methods. The cut separates the foreground from the
background based on the learned models.
Iterative Refinement: After the initial segmentation, GrabCut iteratively refines the GMMs
and updates the graph. This process allows for adjusting the model parameters based on
the current segmentation, leading to improved results.
User Interaction: If the initial segmentation is not satisfactory, the user can refine it by
providing additional foreground and background markings, allowing the algorithm to adapt
and improve.
Improvements Over Basic Graph Cut
Robustness to Color Variations: By using GMMs, GrabCut effectively captures more complex
color distributions, making it more robust to variations in the foreground and background
colors compared to simple histograms used in basic graph cuts.
Better Handling of Background Complexity: The algorithm can distinguish between similar
colors in the foreground and background more effectively, which is especially useful in
images with cluttered backgrounds.
Interactive Segmentation: The ability to refine the segmentation through user input
provides flexibility and improves accuracy, allowing the algorithm to adapt to the user's
needs and the specific characteristics of the image.
Iterative Updates: The iterative refinement of the segmentation process enhances
convergence to an optimal solution by adjusting the models based on the current state of
the segmentation, leading to more precise boundaries.
Q8. Using OpenCV (or any other suitable library), implement the GrabCut algorithm for a
sample image of your choice. You may use a rectangle to define the initial foreground.
• Include the source code for your implementation.
• Provide a detailed explanation of each step.
• Display the original image, the initial segmentation, and the final segmented result.
Ans:

1. Import Libraries:
o cv2: For image processing and GrabCut algorithm.
o numpy: For handling arrays and image masks.
o matplotlib.pyplot: For displaying images.
2. Load the Image:
o Use cv2.imread() to load the image.
o Convert the image to RGB format using cv2.cvtColor() for better visualization with
matplotlib.
3. Create Mask and Models:
o Initialize mask as a zero matrix with the same dimensions as the image. This mask will
be updated by the GrabCut algorithm.
o Initialize bg_model and fg_model as zero matrices, which will be updated by GrabCut.
4. Define Initial Foreground Rectangle:
o Set rect as a tuple specifying the initial rectangle around the foreground object.
Adjust these coordinates based on your specific image.
5. Apply GrabCut:
o Call cv2.grabCut() with the image, mask, rectangle, background model, foreground
model, number of iterations, and the mode cv2.GC_INIT_WITH_RECT to indicate that the
initialization is based on the rectangle.
6. Create Binary Mask:
o Update mask to differentiate between foreground and background. Values 2 and 0
are considered as background, while 1 and 3 are considered as foreground. Convert this into
a binary mask.
7. Segment the Image:
o Multiply the original image by the binary mask to extract the segmented foreground.
8. Display Results:
o Use matplotlib.pyplot to display the original image, initial segmentation mask, and
final segmented result side-by-side.

SECTION 2:
Q 1. In the steps involved in the Canny edge detection algorithm, What is the role of
Gaussian smoothing in this process? How do the upper and lower thresholds affect the
results of the Canny edge detector?
Ans. The Canny edge detection algorithm is a popular technique used to identify edges in an
image. The algorithm involves several key steps, and Gaussian smoothing plays a critical role
in one of these steps. Here’s a detailed look at the role of Gaussian smoothing and the
impact of upper and lower thresholds:
Role of Gaussian Smoothing
1. Noise Reduction:
● Purpose: Gaussian smoothing, also known as Gaussian blurring, is applied to the
image to reduce noise. Edges in an image can be affected by noise, which may result in false
edge detections. Smoothing helps to mitigate this issue by averaging pixel values within a
local neighbourhood.
● Implementation: This is achieved by convolving the image with a Gaussian filter. The
filter has a bell-shaped curve that assigns higher weights to pixels near the center and lower
weights to those further away. This process smooths out rapid intensity changes, which are
often caused by noise.
2. Edge Detection Preparation:
● Purpose: Smoothing helps in preparing the image for the next steps in edge detection
by ensuring that the gradient calculations are more stable and less affected by noise. A
smoother image allows for more reliable detection of actual edges.
● Implementation: The Gaussian filter is defined by its standard deviation (σ), which
controls the extent of the smoothing. A larger σ results in more smoothing and thus,
potentially, fewer detected edges but with less noise.
Impact of Upper and Lower Thresholds
1. Edge Tracking by Hysteresis:
● Purpose: After detecting edges using the gradient magnitude and direction, the
Canny algorithm applies a process known as edge tracking by hysteresis. This step involves
using two thresholds (upper and lower) to finalize the edges.
● Implementation: The algorithm first identifies potential edges using a high threshold.
Then, it performs a second pass to confirm edges based on the low threshold.
2. Upper Threshold:
● Purpose: The upper threshold is used to identify strong edges in the image. Pixels
with a gradient magnitude higher than this threshold are considered as part of an edge.
● Impact: If the upper threshold is set too high, some true edges may be missed
because their gradient magnitudes might not exceed this threshold. Conversely, setting it
too low may include too many false edges.
3. Lower Threshold:
● Purpose: The lower threshold is used for edge tracking. Pixels with gradient
magnitudes between the lower and upper thresholds are considered as part of an edge only
if they are connected to strong edges (i.e., pixels above the upper threshold).
● Impact: Setting the lower threshold too high can lead to missing weak edges that are
actually part of strong edges. Setting it too low might result in more false edges being
detected.
Q2. Explain how the Hough transform is used to detect lines in an image. What is the role
of the accumulator array? Apply the Hough transform to detect lines in an image
containing distinct geometric shapes (e.g., a synthetic image with lines or a real-world
image of a building). Provide the original image, your code, and the output showing the
detected lines.
Ans. Hough Transform for Line Detection
The Hough Transform is a feature extraction technique used to detect lines and other
shapes in an image. Here’s a detailed explanation of how it works for detecting lines:
Concept of the Hough Transform
1. Representation of Lines:
o In an image, a line can be represented in the Cartesian coordinate system as y=mx+b,
where m is the slope and b is the y-intercept.
o However, this representation can be problematic for vertical lines where mmm
becomes infinite. Instead, the Hough Transform uses a different parameterization: the polar
coordinate system.
2. Polar Coordinates:
o A line in polar coordinates can be represented as ρ=xcosθ+ysinθ, where ρ is the
distance from the origin to the line and θ is the angle of the line with respect to the x-axis.
o Here, ρ and θ are used to parameterize lines.
3. Accumulator Array:
o Role: The Hough Transform uses an accumulator array to keep track of potential lines
in the image. Each cell in the accumulator array corresponds to a specific ρ and θ value.
o Process:
▪ For each edge pixel in the image, the algorithm computes possible ρ values for
various θ values and increments the corresponding cells in the accumulator array.
▪ The cells in the accumulator array with the highest values correspond to the most
prominent lines in the image.
4. Detection:
o After filling the accumulator array, peaks in this array indicate the presence of lines in
the image.
o The detected lines are then mapped back to the image space.
Q3. What modifications are made to the basic Hough Transform to detect circles instead
of lines?
Ans. To detect circles using the Hough Transform, modifications are made to handle the
circular shape instead of straight lines. The Hough Transform for circles, often referred to as
the Hough Circle Transform, involves changes to both the parameterization and the
accumulator array. Here’s how it works:
Modifications for Circle Detection
1. Parameterization:
Circle Equation: A circle in Cartesian coordinates is defined as:
(x−a)^2+(y−b)^2=r^2
where (a,b) is the center of the circle and r is the radius.
Parameters: The Hough Circle Transform requires three parameters for each circle:
the x-coordinate of the center (a), the y-coordinate of the center (b), and the radius (r).
2. Accumulator Array:
● 3D Accumulator: Unlike the 2D accumulator array used in line detection, the Hough
Circle Transform uses a 3D accumulator array. The three dimensions correspond to the aaa,
bbb, and rrr parameters.
● Initialization: The accumulator array is initialized with zeros. It will be updated based
on the votes from edge points in the image.
3. Detection Process:
● Edge Detection: First, detect edges in the image using an edge detection method like
Canny.
● Voting Procedure:
For each edge pixel, iterate over a range of possible radii.
For each radius, calculate the possible circle centers (a,b)(a, b)(a,b) that could fit the
edge pixel using the circle equation.
Increment the corresponding cell in the 3D accumulator array for each possible (a,b,r)
(a, b, r)(a,b,r) combination.
● Finding Circles:
After populating the accumulator array, look for local maxima in the 3D array. These
maxima correspond to the most likely circles in the image.
Q4. What are the key steps involved in computing the Histogram of Oriented Gradients
(HOG) for object detection?
Ans. The Histogram of Oriented Gradients (HOG) is a feature descriptor used for object
detection, particularly effective for detecting objects like pedestrians. It captures the
distribution of gradient orientations in localized parts of an image, making it robust to
variations in lighting and pose. Here are the key steps involved in computing HOG:
Key Steps in Computing HOG
1. Preprocessing:
o Resize Image: Typically, HOG operates on an image of a fixed size to ensure uniformity
in the features. Resizing is important, especially in object detection tasks where the objects
need to be scaled.
o Grayscale Conversion: The image is often converted to grayscale since HOG is based
on gradient information, which does not require color data.
2. Gradient Computation:
o Compute Gradients: For each pixel, the gradients (derivatives) along the x-axis (Gx)
and y-axis (Gy) are computed using simple filters like the Sobel operator. These gradients
capture changes in intensity, which are important for detecting edges and shapes.
o Gradient Magnitude and Orientation:
Magnitude = (Gx^2+Gy^2)^1/2
Orientation = atan2(Gy,Gx)
The gradient magnitude represents the strength of the edge, while the orientation
represents the direction of the gradient (0 to 180 degrees, since we are interested in
unsigned angles).
3. Cell Division:
o Divide the Image into Cells: The image is divided into small, evenly spaced cells (e.g.,
8x8 pixels). Within each cell, a histogram of gradient orientations is created.
o Orientation Binning: For each pixel in a cell, the gradient magnitude contributes to an
orientation bin corresponding to the pixel’s gradient direction. The range of orientations (0-
180 degrees) is divided into a set number of bins (e.g., 9 bins, each representing a 20-
degree range).
4. Block Normalization:
o Group Cells into Blocks: To introduce local contrast normalization and improve
invariance to illumination, several adjacent cells are grouped into a block (e.g., a 2x2 cell
block). The HOG features for a block are the concatenated histograms of all cells within the
block.
o Normalize the Block: To handle variations in lighting and contrast, the histogram
values within each block are normalized. This ensures that the features are more robust to
changes in intensity.
▪ Several normalization methods can be used, such as:

where v is the vector of histogram values in the block, and ϵ\epsilonϵ is a small constant to
avoid division by zero.
5. Construct the HOG Descriptor:
o Flatten the Block Histograms: The normalized histograms from all blocks are
concatenated into a single feature vector, forming the HOG descriptor for the entire image
or region of interest.
o This descriptor is then used as input for a machine learning classifier (e.g., Support
Vector Machine) to detect objects like pedestrians.
6. Object Detection:
o Sliding Window: For object detection, a sliding window approach is used. The HOG
descriptor is computed for each window as it slides across the image. The classifier then
determines whether the window contains the object of interest.
Q5.How does Harris corner detection algorithm differentiate between edges, corners, and
flat regions? How does value of lambda defines the regions?
Ans: The Harris algorithm classifies regions based on how the intensity changes as you move
in different directions from a point. To differentiate between edges, corners, and flat
regions, it examines the eigenvalues of the structure tensor matrix (also known as the
autocorrelation or second-moment matrix), which captures the gradient changes in both
directions at a pixel.
Gradient Matrix (M) at a Point:
For each pixel, Harris computes a matrix M which is based on the image gradients:

where:
● Ix and Iy are the image gradients along the x and y directions, respectively.
● This matrix describes how the image intensity changes around a pixel.
Eigenvalues and Their Interpretation:
● Eigenvalues of the matrix M (denoted λ1 and λ2) give a measure of the change in
intensity in two orthogonal directions.
● The values of λ1 and λ2 are used to classify the region:
o Flat Region: If both eigenvalues are small, i.e., λ1≈0 and λ2≈0, there is little variation
in any direction. This corresponds to a flat region.
o Edge: If one eigenvalue is large and the other is small, i.e., λ1≫λ2 , there is significant
variation in one direction but not in the other. This corresponds to an edge.
o Corner: If both eigenvalues are large, i.e., λ1≫0 and λ2≫0, there is significant
variation in two orthogonal directions. This corresponds to a corner.
Harris Response Function (R)
To detect corners, the Harris algorithm defines a response function R, which combines the
eigenvalues. Instead of explicitly calculating eigenvalues, it approximates them using the
determinant and trace of the matrix M:
R = det(M) − k⋅(trace(M))^2
where:
● det(M)=λ1λ2=(Ix^2)(Iy^2)−(Ix Iy)^2 (product of the eigenvalues).
● trace(M)=λ1+λ2=Ix^2+Iy^2 (sum of the eigenvalues).
● k is an empirically determined constant, typically between 0.04 and 0.06.
The value of R helps distinguish between the types of regions:
● R≈0: Flat region.
● R<0: Edge.
● R>0: Corner.
Role of λ (Eigenvalues) in Defining Regions
● Flat Region: Both eigenvalues are small (e.g., λ1≈λ2≈0). This means there is little
intensity variation, so the region is flat.
● Edge: One eigenvalue is large, and the other is small (e.g., λ1≫λ2 or λ2≫λ1). This
indicates that there is strong intensity variation in one direction but not in the
perpendicular direction, meaning the region is an edge.
● Corner: Both eigenvalues are large (e.g., λ1≫0and λ2≫0). This indicates strong
intensity variation in all directions, meaning the region is a corner.
Q6. Compare the Harris corner detector and the Hessian affine detector. What are the
main differences between the two?
Ans. The Harris corner detector and the Hessian affine detector are both used for
identifying keypoints in images, but they differ in their approaches, performance, and
applications. Here's a comparison of the two detectors, highlighting their main differences:
1. Purpose and Keypoint Detection Approach
● Harris Corner Detector:
o The Harris corner detector is designed specifically for detecting corners, where there
is significant intensity variation in two perpendicular directions.
o It relies on gradient information and the second-order intensity variations at a pixel to
classify regions as corners, edges, or flat.
o It computes the structure tensor matrix and derives its eigenvalues to determine
whether a pixel is a corner based on the gradient changes.
o Harris corners are not inherently scale-invariant, meaning they work well for fixed-
scale detection but may not be robust to large scale changes.
● Hessian Affine Detector:
o The Hessian affine detector is an affine-invariant method designed to detect blob-like
structures (i.e., regions of interest that can vary in scale or orientation).
o It uses the Hessian matrix (second-order partial derivatives of image intensity) to
detect keypoints where the intensity changes sharply.
o The Hessian matrix allows for better detection of circular or blob-like features, rather
than corners, by analyzing the determinant of the matrix.
o It includes a scale selection step, making it scale-invariant and robust to large changes
in object size. The affine normalization helps deal with changes in viewpoint and non-
uniform scaling.
2. Underlying Mathematical Basis
● Harris Corner Detector:
o The Harris detector is based on the structure tensor (also called the second moment
matrix):

o Ix and Iy are image gradients along the x and y directions.

o The detector examines the eigenvalues of this matrix to identify regions of high
intensity variation in two directions (corners).
• Hessian Affine Detector:
● The Hessian affine detector is based on the Hessian matrix:

▪ Ixx, Ixy, and Iyy are second-order partial derivatives of the image intensity.
▪ The determinant of the Hessian matrix is used to identify regions with sharp changes
in intensity, which are typically blob-like or elliptical.
3. Scale and Affine Invariance
● Harris Corner Detector:
o The Harris corner detector is not scale-invariant, meaning it works well at a specific
scale but struggles when objects appear at different scales.
o It also lacks affine invariance, meaning that it is not robust to changes in object
orientation, skewing, or non-uniform scaling.
o While there is a scale-invariant variant of the Harris detector (Harris-Laplacian), the
basic Harris detector is limited in these aspects.
● Hessian Affine Detector:
o The Hessian affine detector is designed to be scale-invariant and affine-invariant.
o It uses multi-scale analysis and affine normalization to ensure that keypoints detected
are robust to changes in scale, rotation, and viewpoint, making it suitable for detecting
features in images where objects might be viewed from different angles or at different sizes.
4. Type of Features Detected
● Harris Corner Detector:
o Primarily detects corners (junctions of two edges), which are points of high intensity
variation in two directions.
o Corners are highly discriminative for tasks like image matching and tracking, but
Harris is not as effective for detecting blobs or circular features.
● Hessian Affine Detector:
o Detects blobs and affine-invariant regions, which are regions of uniform or smoothly
varying intensity with sharp intensity changes around the boundary.
o The Hessian detector is particularly good for finding blob-like features, which are
useful in detecting textures, keypoints in natural images, and object regions under affine
transformations.
5. Computational Complexity
● Harris Corner Detector:
o Harris is relatively simple and fast to compute since it mainly requires the
computation of image gradients and eigenvalues of the structure tensor.
o It is well-suited for real-time applications and works efficiently on small or fixed-size
images.
● Hessian Affine Detector:
o The Hessian affine detector is more computationally expensive due to the need for
second-order derivatives and the affine normalization process.
o It also requires multi-scale analysis, making it slower than the Harris corner detector,
but it is more robust for complex tasks involving scale and affine transformations.
6. Applications
● Harris Corner Detector:
o Best suited for detecting corners in tasks like image matching, object recognition, and
image stitching, where corners provide good discriminative features.
o Not ideal for detecting blobs or regions under scale and affine transformations.
● Hessian Affine Detector:
o Used in object detection and keypoint matching where scale and affine invariance is
required, such as in large-scale object recognition, texture recognition, and applications
involving multi-view images.
o Particularly effective in detecting blob-like structures in images with significant
viewpoint or scale changes.

Q7. How does SIFT achieve robustness to scale changes while generating key point
description?
Ans. The Scale-Invariant Feature Transform (SIFT) algorithm achieves robustness to scale
changes through a combination of scale-space analysis and orientation assignment. Here’s a
step-by-step explanation of how SIFT generates scale-invariant keypoints and descriptions:
1. Scale-Space Construction
To handle objects at different scales, SIFT constructs a scale-space representation of the
image by progressively smoothing it using Gaussian filters at different scales and then
finding keypoints that are invariant to scale changes.
● Gaussian Pyramid: The image is repeatedly convolved with Gaussian kernels of
varying sizes to create a pyramid of progressively blurred images.
L(x,y,σ)=G(x,y,σ)∗I(x,y)
where:
o L(x,y,σ) is the blurred image at scale σ.
o G(x,y,σ) is the Gaussian kernel with scale σ.
o I(x,y) is the original image.
● Difference of Gaussians (DoG): SIFT calculates the Difference of Gaussians by
subtracting two images at consecutive levels of the pyramid:
D(x,y,σ)=L(x,y,kσ)−L(x,y,σ)
This DoG function approximates the Laplacian of Gaussian, which highlights regions of the
image that have high intensity variation, helping identify candidate keypoints.
By detecting keypoints across different levels of this pyramid, SIFT ensures that keypoints
can be detected at multiple scales, making the algorithm robust to scale changes.
2. Keypoint Detection and Localization
SIFT identifies keypoints by finding extrema (minima or maxima) in the DoG across both the
spatial domain (x, y) and scale domain (σ).
● For each pixel, the DoG images at adjacent scales are compared, and if a pixel is an
extremum (greater or smaller than all its neighbors in both spatial and scale dimensions), it
is considered as a keypoint candidate.
● To refine the location and scale of the keypoints, SIFT fits a quadratic function to the
local neighborhood of each candidate extremum. This improves the accuracy of the
keypoint localization.
3. Scale Invariance via Scale-Normalized Keypoints
Each keypoint is associated with a particular scale at which it was detected. This ensures
that the keypoints are scale-normalized—the detected keypoints will remain stable even if
the object in the image appears at different scales. This scale normalization is a key step
toward making SIFT robust to scale changes.
4. Orientation Assignment (Rotation Invariance)
Once the keypoint has been detected, SIFT assigns an orientation to it based on the gradient
information around the keypoint at its detected scale. This ensures that the keypoint
descriptor is invariant to rotation.
● Gradient Calculation: The gradient magnitudes and orientations are computed for
each pixel in a region around the keypoint.
Magnitude = (Lx^2+Ly^2)^1/2
Orientation = atan2(Ly,Lx)
● where Lx and Ly are the gradients along the x and y axes.
● Dominant Orientation: A histogram of orientations is created, and the dominant
orientation (the peak in the histogram) is assigned to the keypoint. If there are multiple
strong peaks, multiple keypoints may be created at the same location with different
orientations, improving robustness.
The orientation assignment ensures rotation invariance, meaning the keypoints will match
correctly regardless of how the object is rotated in the image.
5. Keypoint Descriptor Generation (Scale-Invariant and Rotation-Invariant)
After determining the location, scale, and orientation of the keypoint, SIFT generates a
distinctive descriptor for each keypoint. The descriptor is designed to be both scale-
invariant and rotation-invariant.
● Local Gradient Histograms: Around each keypoint, SIFT selects a region (usually 16x16
pixels) and divides it into smaller sub-regions (e.g., 4x4). For each sub-region, a histogram of
gradient orientations is computed.
● The gradient magnitudes are weighted by a Gaussian window centered on the
keypoint, giving more importance to gradients closer to the keypoint's center.
● Descriptor Normalization: The histograms from all the sub-regions are concatenated
into a single vector, which is then normalized to reduce the effect of illumination changes.
This normalization ensures that the descriptor is robust to changes in contrast and
brightness.
Q8. What are the key differences between SIFT and SURF in terms of feature extraction
and computational efficiency?
Ans. SIFT (Scale-Invariant Feature Transform) and SURF (Speeded-Up Robust Features) are
both widely used algorithms for feature extraction in computer vision. While both are
designed to be scale and rotation invariant, there are significant differences in how they
achieve this and in their computational efficiency.
1. Feature Detection Approach
● SIFT:
o Feature Detection: SIFT uses a Difference of Gaussians (DoG) to approximate the
Laplacian of Gaussian. It detects keypoints as extrema in scale-space by finding points that
are maxima or minima across both spatial and scale dimensions.
o Scale Space: SIFT constructs a scale-space by progressively blurring the image with
different Gaussian kernels. The DoG is used to locate keypoints that are invariant to scale
changes.
o Orientation Assignment: SIFT assigns orientations to keypoints based on the gradient
information, making the descriptor rotation-invariant.
● SURF:
o Feature Detection: SURF uses a Hessian matrix approximation for keypoint detection,
relying on box filters to approximate second-order Gaussian derivatives. This makes SURF
faster than SIFT because box filters can be applied more efficiently using integral images.
o Scale Space: Instead of Gaussian pyramids, SURF uses Hessian matrix determinants
computed at different scales to detect keypoints.
o Orientation Assignment: Like SIFT, SURF assigns orientations to keypoints, but it uses
Haar wavelet responses in x and y directions to determine orientation. This method is
computationally faster than SIFT's gradient-based approach.
2. Feature Description
● SIFT:
o Descriptor: SIFT computes descriptors by dividing a region around each keypoint into
smaller cells (typically 4x4) and creating a histogram of gradient orientations within each
cell. The final descriptor is a 128-dimensional vector (4x4 cells, each with 8 orientation
bins).
o Keypoint Descriptor: The gradients are weighted by a Gaussian window to give more
importance to pixels near the center of the region.
o Robustness: SIFT's descriptors are highly robust to scale, rotation, illumination, and
small affine transformations, but the 128-dimensional descriptor makes SIFT slower in both
extraction and matching.
● SURF:
o Descriptor: SURF uses Haar wavelet responses to compute descriptors. It divides the
region around the keypoint into smaller sub-regions (4x4) and calculates Haar wavelet
responses for both x and y directions. The final descriptor is typically a 64-dimensional
vector (SURF-64), but there's also an extended version (SURF-128) for more robustness.
o Keypoint Descriptor: SURF uses fewer bins compared to SIFT, making its descriptors
smaller and faster to compute. Haar wavelets are faster to compute than gradient
histograms.
o Robustness: SURF is also robust to scale and rotation changes but may be slightly less
robust to illumination and perspective changes compared to SIFT.
3. Computational Efficiency
● SIFT:
o Speed: SIFT is more computationally intensive due to its use of Gaussian pyramids for
scale-space construction and the generation of 128-dimensional descriptors based on
gradient histograms. This makes it slower, especially for large images or real-time
applications.
o Time Complexity: The complexity of SIFT comes from its precise but slower keypoint
detection and descriptor generation methods.
● SURF:
o Speed: SURF is designed to be much faster than SIFT by using box filters for the
Hessian matrix and integral images for quick computation of these filters. The use of Haar
wavelets further reduces computational cost in descriptor generation.
o Time Complexity: SURF achieves significant speed improvements compared to SIFT,
making it more suitable for real-time applications.
o SURF typically runs 3-5 times faster than SIFT while maintaining similar accuracy in
feature detection.
4. Descriptor Size and Matching Efficiency
● SIFT:
o Descriptor Size: The standard SIFT descriptor has 128 dimensions, which makes it
more detailed but also larger. This results in higher computational cost during matching
because the distance between descriptors takes more time to compute.
o Matching: Because of the higher-dimensional descriptors, matching SIFT features
requires more processing time, especially when using large datasets or performing nearest-
neighbor search.
● SURF:
o Descriptor Size: The default SURF descriptor has 64 dimensions (SURF-64), which is
more compact than SIFT's. The smaller descriptor size makes SURF faster for both feature
extraction and matching.
o Matching: Due to the smaller descriptor size, SURF is quicker in feature matching,
especially in real-time or large-scale applications.
5. Robustness to Different Transformations
● SIFT:
o Scale and Rotation Invariance: SIFT is highly robust to scale and rotation changes, as
well as changes in illumination and small affine transformations. Its precise keypoint
localization and gradient-based descriptors make it more robust to complex deformations.
o Illumination and Affine Changes: SIFT tends to perform better than SURF in handling
changes in lighting conditions and larger affine distortions due to its finer gradient-based
descriptors.
● SURF:
o Scale and Rotation Invariance: SURF is also robust to scale and rotation, but it may be
slightly less accurate than SIFT when dealing with complex affine transformations or
significant changes in lighting.
o Illumination and Affine Changes: While SURF performs well in most conditions, it may
not be as robust as SIFT when dealing with dramatic illumination changes or complex
distortions.
6. Applications
● SIFT:
o Use Cases: SIFT is widely used in applications that require high accuracy and
robustness, such as object recognition, 3D reconstruction, and image stitching. However, its
slower performance makes it less suitable for real-time applications.
o When to Use: Best for applications where precision is critical, even if it requires more
computational power.
● SURF:
o Use Cases: SURF is often used in real-time applications like video processing, robotics,
and mobile vision systems due to its speed. It provides a good balance between accuracy
and computational efficiency.
o When to Use: Ideal for real-time systems or scenarios where computational resources
are limited but good feature detection is still required.

9. Choose any two feature extraction methods from this assignment (e.g., Canny edge
detection and HOG, or SIFT and SURF).
a) Compare the methods in terms of:
● Type of features extracted (edges, corners, gradients, etc.).
● Computational complexity.
● Robustness to noise, scale, and rotation.
b) Based on a chosen image, apply both methods and provide a visual comparison of the
results. Discuss which method is better suited for the specific image and task.
Ans. Let’s compare two feature extraction methods: Canny edge detection and Histogram of
Oriented Gradients (HOG). They are both used for feature extraction but serve different
purposes and work in different ways.
a) Comparison of Methods
1. Type of Features Extracted
● Canny Edge Detection:
o Type of Features: Extracts edges in an image. It focuses on detecting areas where the
intensity of the image changes sharply, identifying boundaries of objects within the image.
o Purpose: Often used in object boundary detection, segmentation, and contour
identification. It produces a binary image where edges are represented as white lines on a
black background.
● HOG (Histogram of Oriented Gradients):
o Type of Features: Extracts gradients and orientation information. HOG focuses on
capturing the shape and appearance of objects by computing the distribution of gradient
orientations within localized regions of the image.
o Purpose: Frequently used in object detection (e.g., pedestrian detection). It provides
a descriptor that represents the texture and shape of the object based on gradient patterns.
2. Computational Complexity
● Canny Edge Detection:
o Steps Involved:
1. Gaussian smoothing.
2. Gradient calculation.
3. Non-maximum suppression.
4. Double thresholding and edge tracking.
o Complexity: Canny is relatively efficient and works in real-time for many applications,
but the overall complexity depends on the image size. For a given image of size N×MN \
times MN×M, its complexity is approximately O(NM)O(NM)O(NM).
● HOG:
o Steps Involved:
1. Gradient computation.
2. Orientation binning.
3. Block normalization.
4. Histogram generation.
o Complexity: HOG is more computationally expensive than Canny because it calculates
gradients in multiple regions, generates histograms, and normalizes them. The complexity is
proportional to the number of cells and bins used, generally O(NM⋅k), where k is the
number of orientation bins. HOG can be computationally demanding, especially for large
images.
3. Robustness to Noise, Scale, and Rotation
● Canny Edge Detection:
o Noise: Moderately robust to noise due to the initial Gaussian smoothing step, which
helps reduce noise before edge detection. However, excessive noise can still impact edge
detection accuracy.
o Scale: Canny is not inherently scale-invariant. Edges detected at one scale may not
match those detected at another scale without appropriate adjustments to the parameters.
o Rotation: Canny is rotation-invariant since it detects edges regardless of the
orientation of the objects in the image.
● HOG:
o Noise: More robust to noise than Canny because it works on histograms of gradient
orientations rather than directly detecting edges. Small amounts of noise typically have a
minor effect on gradient orientations.
o Scale: HOG is not scale-invariant. The object’s size relative to the cell size affects the
computed gradient histograms, making it necessary to resize the image or use multi-scale
detection.
o Rotation: HOG can handle some degree of rotation, especially when rotation is within
the range of 45 degrees. However, significant rotation can degrade performance unless
specific adjustments are made to the algorithm.
b) Application to a Sample Image
1. Canny Edge Detection Code (using OpenCV)

2. HOG Feature Extraction Code (using OpenCV)

3. Visual Comparison
● Canny Edge Detection Result: The result will highlight the sharp edges in the image,
clearly delineating object boundaries such as the contours of the pedestrian and other
objects in the scene. This binary output shows where significant intensity changes occur.
● HOG Result: The HOG visualization will show the gradient patterns around the
objects, which represent the shape and structure of the pedestrian. The gradients are
distributed across different orientation bins, providing a detailed description of the object’s
form.
4. Which Method is Better?
● For Edge Detection: Canny is better suited if you are interested in detecting
boundaries of objects and contours in the image. It excels at finding edges but does not
capture more detailed information about texture or shape.
● For Object Detection: HOG is better suited for tasks that require shape-based
recognition. Its gradient orientation histograms provide more information about the
object’s appearance, making it ideal for detecting pedestrians or other objects based on
their structure.
● Task Suitability:
o Canny is ideal if the goal is to detect sharp object boundaries or edges, for example,
in image segmentation or contour detection tasks.
o HOG is more suitable for object detection tasks, where recognizing the shape and
form of the object is essential, such as detecting pedestrians in surveillance footage.
Overall, HOG provides a more detailed description of the object, while Canny focuses on
detecting boundaries, making them useful for different tasks depending on the application.