UNIT 4 – 3D VISION AND MOTION

1 Briefly discuss the various Methods of 3D Vision.

 3D Computer Vision is a branch of computer science that focuses on acquiring, image
processing, and analyzing three-dimensional visual data.
 It aims to reconstruct and understand the 3D structure of objects and scenes from two-
dimensional images or video data. 3D vision techniques use information from sources like
cameras or sensors to build a digital understanding of the shapes, structure, and properties of
objects in a scene.
 This has numerous applications in robotics, augmented/virtual reality, autonomous systems,
and many more.
What is 3D Computer Vision?
 3D computer vision systems extract, process, and analyze 2D visual data to generate their 3D
models.
 To do so, it employs different algorithms and data acquisition techniques that enable computer
vision models to reconstruct the dimensions, contours, and spatial relationships of objects
within a given visual setting.
 The 3D CV techniques combine principles from multiple disciplines, such as computer vision,
photogrammetry, geometry, and machine learning to derive valuable three-dimensional
information from images, videos, or sensor data.
An Example of 3D Computer Vision Technique [Source]
Fundamental Concepts in 3D Computer Vision
1. Depth Perceptions
Depth perception is the ability to estimate the distance between objects and the camera or sensor. This
is accomplished through methods like stereo vision, where two cameras are used to calculate depth or
by analyzing cues such as shading, texture changes, and motion differences in single-camera images
or video sequences.
 Depth Estimation in 3D Computer Vision
2. Spatial Dimensions
Spatial dimensions refer to the three orthogonal axes (X, Y, and Z) that make the 3D coordinate
system. These dimensions capture the height, width, and depth values of objects. Spatial coordinates
facilitate the representation, examination, and manipulation of 3D data like point clouds, meshes, or
voxel grids essential for applications such as robotics, augmented reality, and 3D reconstruction.

Spatial Dimensions
3. Homogeneous Coordinates and 3D Projective Geometry
 3D projective geometry and homogeneous coordinates offer a structure for representing and
handling 3D points, lines, and planes.
 Homogeneous coordinates represent points in space using an additional coordinate to allow
geometric transformations like rotation, translation, and scaling through matrix operations.
 On the other hand, 3D projective geometry deals with the mathematical representation and
manipulation of 3D objects along with their projections onto 2D image planes.

3D Projective
Geometry
4. Camera Models and Calibration Techniques for 3D Models
 The appropriate selection of camera models and their calibration techniques play a crucial role
in 3D CV to precisely reconstruct 3D models from 2D images.
 The use of high-definition camera models improves the geometric relationship between 3D
points in the real world and their corresponding 2D projections on the image plane.
 Meanwhile, accurate camera calibration helps estimate the camera’s intrinsic parameters, such
as focal length and principal point, as well as extrinsic parameters, including position and
orientation.
 These parameters are crucial for correcting distortions, aligning images, and triangulating 3D
points from multiple views to ensure accurate reconstruction of 3D models.
5. Stereo Vision
 Stereo vision is a method in 3D CV that utilizes two or more 3D machine vision cameras to
capture images of the same scene from slightly different angles.
 This technique works by finding matching points in both images and then calculating their 3D
locations using the known camera geometry.
  Stereo vision algorithms analyze the disparity or the difference in the positions of
corresponding points to estimate the depth of points in the scene. This depth data allows the
accurate reconstruction of industry 3D models, which can be useful for tasks like robotic
navigation, augmented reality, and 3D mapping.

Stereo Vision in 3D Image Reconstruction

Passive Techniques 3D Reconstruction

Passive imaging techniques directly analyze images or videos captured by existing light sources.
They achieve this without projecting or emitting any additional controlled radiation. Examples of
these techniques include:
Shape from Shading
 In 3D computer vision, shape from shading reconstructs an object’s 3D shape using just a
single 2D image.
 This technique analyzes how light hits the object (shading patterns) and how bright different
areas appear (intensity variations).
 By understanding how light interacts with the object’s surface, this vision technique estimates
its 3D shape.
 Shape from shading assumes we know the surface properties of objects (especially how they
reflect light) and the lighting conditions.
 Then, it uses special algorithms to find the most likely 3D shape of that object that explains
the shading patterns seen in the image.
 3D Shape Reconstruction Using Shape from Shading Technique
Shape from Texture
 Shape from texture is a method used in computer vision to determine the three-dimensional
shape of an object based on the distortions found in its surface texture.
 This technique relies on the assumption that the surface possesses a textured pattern with
known characteristics.
 By analyzing how this texture appears deformed in a 2D image, this technique can estimate
the 3D orientation and shape of the underlying surface.
 The fundamental concept is that the texture will be compressed in areas facing away from the
camera and stretched in areas facing toward the camera.

3D
Image Reconstruction Using Shape from Texture Technique
Depth from Defocus
 Depth from defocus is a process that calculates the depth or three-dimensional structure of a
scene by examining the degree of blur or defocus present in areas of an image.
  It works on the principle that objects situated at distances, from the camera lens will exhibit
varying levels of defocus blur. By comparing these blur levels throughout the image, DfD can
generate depth maps or three-dimensional models representing the scene.

Focus and Defocus Imaging Process for 3D Image Reconstruction

Structure from Motion (SfM)
 Structure from Motion (SfM) reconstructs the 3D structure of a scene from a set of 2D images.
It captures a set of overlapping 2D images as input.
 We can capture these images with a regular camera or even a drone.
 The first step identifies common features across these images, such as corners, edges, or
specific patterns.
 SfM then estimates the position and orientation (pose) of the camera for each image based on
the identified features and how they appear from different viewpoints.
 By having corresponding features in multiple images and the camera poses, it performs
triangulation to determine the 3D location of these features in the scene.
 Lastly, the SfM algorithms use the 3D positioning of these features to build a 3D model of the
scene which can be a point cloud representation or a more detailed mesh model.
 Structu
re from Motion (SfM) technique in 3D computer vision
Active Techniques 3D Reconstruction
Active 3D reconstruction methods project any kind of radiation, like light, sound, or radio waves onto
the object. It then analyzes their reflections, echoes, or distortions to reconstruct the 3D structure of
that object. Examples of such techniques may include:
Structured Lighting
 Structured light is an active 3D CV technique where a specifically designed light pattern or
beam is projected onto a visual scene.
 This light pattern can be in various forms including grids, stripes, or even more complex
designs.
 As the light pattern strikes objects that have varying shapes and depths, the light beams get
deformed.
 Therefore by analyzing how the projected beams bend and deviate on the object’s surface, a
vision system calculates the depth information of different points on the object.
 This depth data allows for reconstructing a 3D representation of the visual object that is under
observation.
Time-of-Flight (ToF) Sensors
 Time-of-flight (ToF) sensor is another active vision technique that measures the time it takes
for a light signal to travel from the sensor to an object and back.
 Common light sources for ToF sensors are lasers or infrared (IR) LEDs.
 The sensor emits a light pulse and then calculates the distance based on the time-of-flight of
the reflected light beam.
  By capturing this time for each pixel in the sensor array, a 3D depth map of the scene is
generated. Unlike regular cameras that capture color or brightness,
 ToF sensors provide depth information for every point which essentially helps in building a
3D image of the surroundings.

Time of
Flight (ToF) Sensor Technique

LiDAR
 LiDAR (Light Detection and Ranging) is a remote sensing 3D vision technique that uses laser
light to measure object distances.
 It emits laser pulses towards objects and measures the time it takes for the reflected light to
return.
 This data generates precise 3D representations of the surroundings. LiDAR systems create
high-resolution 3D maps that are useful for applications like autonomous vehicles, surveying,
archaeology, and atmospheric studies.

Deep Learning Approaches to 3D Vision (Advanced Techniques)

Recent advancements in deep learning have significantly impacted the field of 3D Computer Vision.
It has achieved remarkable results in various tasks such as:
3D CNNs
  3D convolutional neural networks, also known as 3D CNNs are a form of 3D deep learning
model crafted for analyzing three-dimensional visual data.
 In contrast to traditional CNN approaches that process 2D data, 3D CNNs leverage unique
filters to extract key features directly from volumetric data, such as 3D medical scans or
object models in three dimensions.
 This capability to process data in three dimensions enables this learning approach to capture
spatial relationships (such as object positioning) and temporal details (like motion progression
in videos).
 As a result, 3D CNNs prove effective for tasks like 3D object recognition, video analysis, and
precise segmentation of medical images for accurate diagnoses.

2D vs 3C CNNs

2. Explain shape from shading in detail.

Shape from shading (SfS) is a technique in computer vision used to determine the three-dimensional
shape of a surface from a single image. This method relies on the variations in shading (light and dark
areas) to infer the shape of the surface. Here’s a detailed explanation:
Concept and Theory
1. Basic Principle:
o Shape from shading exploits the way light interacts with a surface. When light hits an object,
it creates different shades depending on the surface’s orientation relative to the light source.
By analyzing these shades, we can infer the surface’s shape.
2. Mathematical Model:
 o The brightness ( E ) of a surface point can be expressed as:
E=ρ(l⋅n)
where:
 ( \rho ) is the reflectance (albedo) of the surface.
 ( \mathbf{l} ) is the unit vector in the direction of the light source.
 ( \mathbf{n} ) is the unit normal vector to the surface at the point.
3. Assumptions:
o Lambertian Surface: The surface reflects light uniformly in all directions.
o Known Light Source: The direction and intensity of the light source are known.
o Single Light Source: There is only one light source affecting the surface.
 When you look at images of smooth shaded objects, such as the ones shown in Figure 12.2, you can
clearly see the shape of the object from just the shading variation. How is this possible?
 The answer is that as the surface normal changes across the object, the apparent brightness changes as
a function of the angle between the local surface orientation and the incident illumination.
 The problem of recovering the shape of a surface from this intensity variation is known as shape
from shading and is one of the classic problems in computer vision.
 Most shape from shading algorithms assume that the surface under consideration is of a uniform
albedo and reflectance, and that the light source directions are either known or can be calibrated by
the use of a reference object.
 Under the assumptions of distant light sources and observer, the variation in intensity (irradiance
equation) become purely a function of the local surface orientation,
where (p, q) = (zx, zy) are the depth map derivatives and R(p, q) is called the reflectance map. For
example, a diffuse (Lambertian) surface has a reflectance map that is the (non

which arises naturally, since for a valid depth map z(x, y) with (p, q) = (zx, zy), we have py = zxy = zyx =
qx.
 3.Triangulation:
The problem of determining a point’s 3D position from a set of corresponding image locations and known
camera positions is known as triangulation.
 4.Bundle adjustment
Bundle adjustment is a crucial optimization technique in computer vision and photogrammetry. It refines
the 3D coordinates of scene geometry, camera positions, and camera parameters to minimize errors in 3D
reconstruction of images.
 The name “bundle adjustment” originates from the concept of adjusting the “bundles” of light rays
that travel from 3D points in the scene to the camera’s optical center.
 In essence, these bundles represent the paths of light rays captured by the camera from various points
in the scene.
 During the bundle adjustment process, the goal is to optimize these light ray bundles to minimize the
reprojection error, which is the difference between the observed image points and the projected points
from the 3D model. By refining the 3D coordinates of the scene, camera positions, and other
parameters, the process ensures a more accurate and reliable 3D reconstruction.
1. Introduction
In this tutorial, we’ll talk about bundle adjustments and their role in photogrammetry and computer vision.
2. The Role in Computer Vision and Photogrammetry
Bundle adjustment (BA) enhances the accuracy and reliability of 3D scene reconstructions from multiple
images and camera views:
  We use it to correct errors from the initial 3D reconstruction process, including inaccuracies in
camera pose, scene structure, or feature tracking.
 To do so, this technique optimizes the parameters of 3D reconstructions and refines the camera
calibration parameters.

2.1. Applications

We use it for object tracking, augmented reality, and scene understanding.

Additionally, it contributes to developing lifelike 3D environments, elevating user immersion
and interaction in virtual reality and simulation applications.

3. Workflow and Key Components of Bundle Adjustment

The workflow of Bundle Adjustment (BA) consists of a series of steps designed to refine the
parameters of a 3D reconstruction model from several 2D images:
3.1. Data Collection and Initial Reconstruction
 We start with a collection of 2D images capturing the scene or object of interest from different
viewpoints, ideally with overlapping features or key points recognizable in multiple images.
 To initialize the 3D scene structure and camera parameters, we do an initial structure-from-
motion (SfM) estimation. SfM algorithms analyze the image correspondences and camera
positions to provide an initial guess of the scene’s 3D structure and camera poses.
 However, this initial reconstruction often has errors. This is where bundle adjustment comes
into play.
3.2. Reprojection Errors and Objective Function Formulation
 First, we compute the reprojection errors for each image, measuring the disparity between
observed 2D features and their current 3D structure-based projected positions.
 That way, we asses our 3D model’s accuracy.
 We also define an objective function to optimize in further steps. Usually, it’s the sum of
squared (reprojection) errors across all images. It quantifies the overall error in the current
reconstruction.
3.3. Nonlinear Optimization and Simultaneous Parameter Updates
 Nonlinear optimization techniques like Levenberg-Marquardt or Gauss-Newton are used
iteratively to minimize the objective function, refining the 3D scene structure and camera
parameters.
 We apply simultaneous updates to the 3D structure and camera parameters throughout each
optimization iteration, ensuring that changes in one reconstruction aspect don’t adversely
affect others, thus promoting global consistency.
3.4. Refined Reconstruction and Post-processing

 The result of bundle adjustment is a refined 3D reconstruction model with an improved

accuracy. This model’s 3D scene structure and camera parameters produce a more precise scene
representation.
 Subsequently, we undertake post-processing steps, such as mesh generation, texture mapping, or
additional filtering, contingent on the particular application requirements.
 The outcome is a highly accurate 3D model suitable for various applications, including 3D mapping,
computer vision, robotics, and virtual reality.

4.Tools and Software Libraries:

There are many tools and software libraries for bundle adjustment:

 OpenCV provides various functions and classes for camera calibration and bundle adjustment,
encompassing sparse and dense bundle adjustment. functions
 COLMAP, an open-source software designed for 3D reconstruction from images, incorporates
bundle adjustment as an integral part of its pipeline, supporting both sparse and dense reconstruction.
 Bundler, a user-friendly tool for structure-from-motion (SfM), estimates camera parameters and 3D
structure from 2D images, making it a suitable choice for basic bundle adjustment tasks.
 g2o, an open-source C++ library specializing in optimizing graph-based nonlinear least-squares
problems, enjoys widespread adoption in computer vision and robotics for bundle adjustment
purposes.

5. Benefits of Using Bundle Adjustment

 First, it enhances the accuracy of camera pose estimation and 3D reconstruction.

 Furthermore, it ensures internal consistency among all camera poses and 3D points.
 The alignment of observations across different images reduces errors resulting from individual
camera calibrations or noisy measurements.

5.Shape from texture and Shape from focus

Shape from texture
 Texture can be very helpful to the human eye in permitting depth to be perceived.
 Although textured patterns can be very complex, even the simplest textural elements can carry depth
information.
 To disentangle such textured images sufficiently to deduce depths within the scene, it is first
necessary to find the horizon line reliably.
 This is achieved by taking all pairs of texture elements and deducing from their areas where the
horizon line would have to be. To proceed, we make use of the rule:

 which applies since circles at various depths would give a square law, although the progressive
eccentricity also reduces the area linearly in proportion to the depth.
 This information is accumulated in a separate image space and a line is then fitted to these data: false
alarms are eliminated automatically by this Hough-based procedure.
 At this stage the original data—the ellipse areas—provide direct information on depth, although some
averaging is required to obtain accurate results.
 Although this type of method has been demonstrated in certain instances, it is in practice highly
restricted unless very considerable amounts of computation are performed.
 Hence it is doubtful whether it can be of general practical use in machine vision applications.

Shape From Focus:

 Shape from Focus (SFF) is one of the passive techniques that uses focus information to
estimate the three-dimensional shape of an object in the scene. Images are taken at multiple
positions along the optical axis of the imaging device and are stored in a stack.
 A strong cue for object depth is the amount of blur, which increases as the object’s surface
moves away from the camera’s focusing distance.
 A number of techniques have been developed to estimate depth from the amount of defocus.
 In order to make such a technique practical, a number of issues need to be addressed:
• The amount of blur increase in both directions as you move away from the focus plane.
Therefore, it is necessary to use two or more images captured with different focus
distance settings or to translate the object in depth and look for the point of maximum
sharpness.
• The magnification of the object can vary as the focus distance is changed or the object is
moved. This can be modeled either explicitly (making correspondence more difficult) or using
telecentric optics, which approximate an orthographic camera and require an aperture in front
of the lens .
• The amount of defocus must be reliably estimated.
 A simple approach is to average the squared gradient in a region but this suffers from several
problems, including the image magnification problem mentioned above.
A better solution is to use carefully designed rational filter.
Figure 12.4 shows an example of a real-time depth from defocus sensor, which employs two
imaging chips at slightly different depths sharing a common optical path, as well as an active
illumination system that projects a checkerboard pattern from the same direction. As you can
see in Figure 12.4b–g, the system produces high-accuracy real-time depth maps for both static
and dynamic scenes.

6.Translational alignment
 Translational alignment in computer vision involves aligning images by shifting them horizontally
and vertically without any rotation or scaling. This process is crucial for several applications,
including:
1. Image Stitching: Combining multiple images to create a seamless panoramic view.
2. Video Stabilization: Reducing the effects of camera shake in video sequences.
3. Motion Tracking: Following the movement of objects across frames in a video.
 A commonly used method for achieving translational alignment is the Lucas-Kanade
algorithm.
 This algorithm works by estimating the displacement of image patches and minimizing the
difference between these patches in consecutive frames. f
 It’s widely used in motion-compensated video compression schemes like MPEG and H.263.
 The simplest way to establish an alignment between two images or image patches is to shift one
image relative to the other.
 Given a template image I0(x) sampled at discrete pixel locations {xi = (xi , yi)}, we wish to find
where it is located in image I1(x).
 A least squares solution to this problem is to find the minimum of the sum of squared differences
(SSD) function.

where u = (u, v) is the displacement and ei = I1(xi + u) − I0(xi) is called the residual error.
 The assumption that corresponding pixel values remain the same in the two images is often called the
brightness constancy constraint.
Hierarchical motion estimation:
 Hierarchical motion estimation is a technique used in computer vision and video processing to
estimate motion between frames of a video sequence.
 Pyramid Structure: The process involves creating a pyramid of images at multiple resolutions. The
motion is first estimated at the coarsest level (lowest resolution) and then refined at progressively
finer levels (higher resolutions).
 The motion estimate from one level of the pyramid is then used to initialize a smaller local search at
the next finer level.
 Coarse-to-Fine Strategy: By starting with a low-resolution image, large motions can be captured
more easily. As the resolution increases, finer details and smaller motions are refined, leading to a
more accurate overall motion estimation .
 Applications: This technique is widely used in various applications such as video compression,
object tracking, and 3D reconstruction. It helps in reducing the computational load while maintaining
high accuracy
Fourier-based alignment:
Fourier-based alignment is a technique used in image processing to align images by leveraging the
properties of the Fourier transform.
The Fourier-based alignment algorithm consists of the following steps:
1. For two color channels C1 and C2, compute corresponding Fourier transforms FT1 and FT2.
2. Compute the conjugate of FT2 (denoted as FT2*), and compute the product of FT1 and FT2*.
3. Take the inverse Fourier transform of this product and find the location of the maximum value in the
output image. Use the displacement of the maximum value to obtain the offset of C2 from C1.
Incremental refinement:
Incremental refinement in translational alignment is a technique used to improve the accuracy of aligning
images or sequences by iteratively adjusting the alignment parameters. This method is particularly useful
in fields like computer vision and bioinformatics.
In Computer Vision
In computer vision, incremental refinement is often applied in tasks such as image registration and video
stabilization. The process involves:
1. Initial Alignment: Starting with a rough alignment of the images.
2. Error Metric Calculation: Using metrics like Sum of Squared Differences (SSD) to measure the
alignment error.
The Sum of Squared Differences (SSD) is a common metric used to measure the similarity between two
images or signals. It is particularly useful in image processing tasks like image registration, template
matching, and motion estimation.
How SSD Works
1. Pixel-by-Pixel Comparison: SSD involves comparing corresponding pixels of two images.
2. Difference Calculation: For each pixel, the difference between the pixel values of the two images is
calculated.
3. Squaring the Differences: Each difference is squared to ensure all values are positive and to
emphasize larger differences.
4. Summing Up: The squared differences are summed up to produce a single value, which represents
the SSD.
3. Iterative Adjustment: Gradually refining the alignment by minimizing the error metric through
iterative updates1.