Image processing basics

Many slides adapted from

Andy Warhol, Marilyn Diptych, 1962 (source) Alyosha Efros, Derek Hoiem
Image processing basics: Outline
• Images as sampled functions
• Sampling and reconstruction, aliasing
• Image resampling, interpolation
• Image transformations
 Image formation (preview)
• What determines the brightness of an image pixel?
Light source(s)

Image plane
Scene
Light energy arriving at
surface
!′ per unit time is
quantified by irradiance
(Watts per sq. meter)

Strictly speaking, we need

spectral irradiance to
Camera (lenses, shutter)
account for light across
different wavelengths
Images as sampled functions
Digital color image
R

B
 Images in Python
im = cv2.imread(filename) # read image
im = cv2.cvtColor(im, cv2.COLOR_BGR2RGB) # order channels as RGB
im = im / 255 # values range from 0 to 1
RGB image im is a H x W x 3 matrix (numpy.ndarray)
im[0,0,0] is the top-left pixel value in R-channel
im[y, x, c] is the value y+1 pixels down, x+1 pixels to right in the cth channel
im[H-1, W-1, 2] is the bottom-right pixel in B-channel

column
row 0.92 0.93 0.94 0.97 0.62 0.37 0.85 0.97 0.93 0.92 0.99 R
0.95 0.89 0.82 0.89 0.56 0.31 0.75 0.92 0.81 0.95 0.91
0.89 0.72 0.51
0.92
0.55
0.93
0.51
0.94
0.42
0.97
0.57
0.62
0.41
0.37
0.49
0.85
0.91
0.97
0.92
0.93 0.92 0.99 G
0.96 0.95 0.88 0.94 0.56 0.46 0.91 0.87 0.90 0.97 0.95
0.71 0.81
0.95
0.81
0.89
0.89
0.87
0.72
0.82
0.57
0.51
0.92
0.89
0.37
0.55
0.93
0.56
0.80
0.51
0.94
0.31
0.88
0.42
0.97
0.75
0.89
0.57
0.62
0.92
0.79
0.41
0.37
0.81
0.85
0.49
0.85
0.95
0.91
0.97
0.91
0.92
0.93 0.92 0.99
B
0.49 0.62 0.60 0.58 0.50 0.60 0.58 0.50 0.61 0.45 0.33
0.96 0.95 0.88 0.94 0.56 0.46 0.91 0.87 0.90 0.97 0.95
0.86 0.84 0.74 0.58 0.95
0.51 0.89
0.39 0.82
0.73 0.89
0.92 0.56
0.91 0.31
0.49 0.75
0.74 0.92 0.81 0.95 0.91
0.71 0.81 0.81 0.87 0.57 0.37 0.80 0.88 0.89 0.79 0.85
0.96 0.67 0.54 0.85 0.89
0.48 0.72
0.37 0.51
0.88 0.55
0.90 0.51
0.94 0.42
0.82 0.57
0.93 0.41 0.49 0.91 0.92
0.69 0.49
0.49
0.56
0.86
0.62
0.66
0.84
0.60
0.96
0.43
0.74
0.71
0.58
0.95
0.42
0.58
0.81
0.50
0.88
0.77
0.51
0.81
0.60
0.94
0.73
0.39
0.87
0.58
0.56
0.71
0.73
0.57
0.50
0.46
0.90
0.92
0.37
0.61
0.91
0.99
0.91
0.80
0.45
0.87
0.49
0.88
0.33
0.90
0.74
0.89
0.97
0.79
0.95
0.85
How are the three
0.79 0.73 0.90 0.67 0.33 0.61 0.69 0.79 0.73 0.93 0.97
0.91 0.94
0.96
0.89
0.69
0.67
0.49
0.49
0.54
0.49
0.41
0.56
0.85
0.62
0.78
0.66
0.48
0.60
0.78
0.43
0.37
0.58
0.77
0.42
0.88
0.50
0.89
0.77
0.90
0.60
0.99
0.73
0.94
0.58
0.93
0.71
0.82
0.50
0.90
0.93
0.61
0.99
0.45 0.33 color channels
0.86 0.84 0.74 0.58 0.51 0.39 0.73 0.92 0.91 0.49 0.74
0.79
0.91
0.73
0.94
0.90
0.96
0.89
0.67
0.67
0.49
0.33
0.54
0.41
0.61
0.85
0.78
0.69
0.48
0.78
0.79
0.37
0.77
0.73
0.88
0.89
0.93
0.90
0.99
0.97
0.94
0.93
0.82 0.93 acquired?
0.69 0.49 0.56 0.66 0.43 0.42 0.77 0.73 0.71 0.90 0.99
0.79 0.73 0.90 0.67 0.33 0.61 0.69 0.79 0.73 0.93 0.97
0.91 0.94 0.89 0.49 0.41 0.78 0.78 0.77 0.89 0.99 0.93
How are the three channels acquired?

Bayer grid (1976) Why more green?

Human Luminance Sensitivity Function

Demosaicing:
Estimation of missing
components from
neighboring values

Source: Wikipedia
 Images as sampled functions
• We like to think of a digital image as a sampled representation
of a continuous function !(#, %) defined over a continuous 2D
domain
Sampling and reconstruction
• Sampling: recording the • Reconstruction: converting a
function’s values at a sampled representation back into a
discrete set of locations continuous function by “guessing”
what happens between the
samples

Source: S. Marschner (via A. Efros)

1D example: Digital audio
• Recording: sound to analog to samples to disc
• Playback: disc to samples to analog to sound again

Source: S. Marschner (via A. Efros)

Sampling and reconstruction
• Simple example: a sine wave

Source: S. Marschner (via A. Efros)

Sampling and reconstruction
• Simple example: a sine wave
• What if we “missed” things between the samples?
• Unsurprising result: information is lost

Source: S. Marschner (via A. Efros)

Sampling and reconstruction
• Simple example: a sine wave
• What if we “missed” things between the samples?
• Unsurprising result: information is lost
• Surprising result: indistinguishable from lower frequencies

Source: S. Marschner (via A. Efros)

Sampling and reconstruction
• Simple example: a sine wave
• What if we “missed” things between the samples?
• Unsurprising result: information is lost
• Surprising result: indistinguishable from lower frequencies
(or even higher frequencies)

Source: S. Marschner (via A. Efros)

Sampling and reconstruction
• Simple example: a sine wave
• What if we “missed” things between the samples?
• Unsurprising result: information is lost
• Surprising result: indistinguishable from lower frequencies
(or even higher frequencies)
• Aliasing: signal “traveling in disguise” as other frequencies

Source: S. Marschner (via A. Efros)

Wagon wheel effect

Actual motion

Perceived motion

https://en.wikipedia.org/wiki/Wagon-wheel_effect
https://maycontainmaths.wordpress.com/2015/02/22/the-wagon-wheel-effect/
Wagon wheel effect

Actual motion

Perceived motion

https://en.wikipedia.org/wiki/Wagon-wheel_effect
https://maycontainmaths.wordpress.com/2015/02/22/the-wagon-wheel-effect/
Aliasing in images
Aliasing “in the wild”
Disintegrating textures Moire patterns, false color

Source

Source Source
Nyquist-Shannon sampling theorem
• When sampling a signal at discrete intervals, the sampling
frequency must be at least twice the maximum frequency of
the input signal to allow us to reconstruct the original perfectly
from the sampled version

bad

good

https://en.wikipedia.org/wiki/Nyquist-Shannon_sampling_theorem
Anti-aliasing
• What are possible solutions?
• Sample more often (if you can)
• Get rid of all frequencies that are greater than half the new sampling
frequency
• Will lose information, but that’s better than aliasing
• How to get rid of high frequencies?
• Apply a smoothing or low-pass filter (later)
Why should you care about anti-aliasing?

R. Zhang. Making convolutional networks shift-invariant again. ICML 2019

Image processing basics: Outline
• Images as sampled functions
• Sampling and reconstruction, aliasing
• Image resampling, interpolation
Subsampling an image
• How do we reduce the size of an image by a factor of two?
Subsampling an image
• How do we reduce the size of an image by a factor of two?

How about throwing

away every other
row and column to
create a half-size
image?
Subsampling without pre-filtering

1/2 1/4 (2x zoom) 1/8 (4x zoom)

Source: S. Seitz (via D. Hoiem)

Subsampling with pre-filtering

1/2 1/4 1/8

• Image is smoothed with a Gaussian filter before subsampling
Source: S. Seitz (via D. Hoiem)
Upsampling an image
• How do we increase the size of an image by a factor of two?

Let’s increase the

resolution of the
sampling grid!
Upsampling an image
• How do we increase the size of an image by a factor of two?

What should this

value be?

Need to interpolate!
Upsampling an image
• How do we increase the size of an image by a factor of two?

(0,0) (1,0) (2,0) (3,0)

( 0.5, 0.5 =?
(0,1) (1,1) (2,1) (3,1)

(0,2) (1,2) (2,2) (3,2)

(0,3) (1,3) (2,3) (3,3)

Bilinear interpolation
• Let ! 0, 0 = %, ! 1, 0 = ', ! 0, 1 = (, ! 1, 1 = )

A B

1
! 0.5, 0 = (% + ')
2
C D
Bilinear interpolation
• Let ! 0, 0 = %, ! 1, 0 = ', ! 0, 1 = (, ! 1, 1 = )

A B

1
! 0, 0.5 = (% + ()
2
C D
Bilinear interpolation
• Let ! 0, 0 = %, ! 1, 0 = ', ! 0, 1 = (, ! 1, 1 = )

A B

! 0.5, 0.5 =
C D 1
(% + ' + ( + ))
4
Application: Demosaicing
Bilinear interpolation more generally
B

A
D

http://en.wikipedia.org/wiki/Bilinear_interpolation
Bilinear interpolation more generally

"' , $' "* , $'

A B

! ", $ = &'' ( + &*' + +&'* , + &** -

C D
"' , $* "* , $* "* − " $* − $ " − "' $* − $
&'' = &*' =
"* − "' $* − $' "* − "' $* − $'
"* − " $ − $' (" − "' ) $ − $'
&'* = &** =
"* − "' $* − $' "* − "' $* − $'

http://en.wikipedia.org/wiki/Bilinear_interpolation
Bilinear interpolation: Basis function view
• Interpolated function is sum of basis functions or “bumps”
centered at the four adjacent grid points, weighted by the
image values at the corresponding points
B Bilinear basis function

A
D

C
Other kinds of interpolation

Source: Wikipedia
Interpolation and function extrema
• When you use linear interpolation, extrema of the image
function can only occur at the original sample points
• What about nonlinear interpolation?

Image source
Image processing basics: Outline
• Images as sampled functions
• Sampling and reconstruction, aliasing
• Image resampling, interpolation
• Image transformations
Image transformations

Downsampling

T
Image transformations

Upsampling

T
Image transformations

Contrast change

T
Image transformations

Blurring/sharpening

T
Image transformations

Warping

T
Point processing
• Change range of image: ! = #(%)

f g
T
x x

• Negative: ! =1−%

Image source
Point processing
• Change range of image: ! = #(%)

f g
T
x x

• Affine contrast adjustment: ! = '% + )

f g
T
Point processing
• Change range of image: ! = #(%)

f g
T
x x

• Piecewise-linear contrast adjustment:

Point processing
• Change range of image: ! = #(%)

f g
T
x x

• Gamma correction: ! = '% (

) = 1/2 Original )=2

Source
Image filtering
• Roughly speaking, replace image value at ! with some
function of values in its spatial neighborhood "(!):

%(!) = '((("(!)))

f g
T
N(x) x
Image filtering
• Roughly speaking, replace image value at ! with some
function of values in its spatial neighborhood "(!):

%(!) = '((("(!)))

f g
T

• Examples: smoothing, sharpening, edge detection, etc.

Image warping
• Change domain of image:

!" = $ ! , &(!′) = *(!)

f g
T
x T(x)
Image warping
• Change domain of image:

!" = $ ! , &(!′) = *(!)

f g
T
Image warping
• Examples of global parametric warps:

scaling (uniform or
translation rotation non-uniform)

perspective
affine cylindrical
homography
(not covered
in this class)
Translation

! " = ! + %&
' " = ' + %(

%! = 2
%' = 1
Uniform scaling

$% = ! ∗ $
'% = ! ∗ '

! = 2
Non-uniform scaling
In matrix form:
% & = !" ∗ %
%′ !" 0 %
( & = !# ∗ ( = 0 !# (
(′

!" = 2,
!# = 0.5
Rotation
• Rotate the image by an angle of ! about the origin:

!
Rotation
• Rotate the image by an angle of ! about the origin:

#′ = * cos(( + !) = * cos(() cos(!) – * sin(() sin(!) = # cos(!) − % sin(!)

%′ = * sin(( + !) = * sin(() cos(!) + * cos(() sin(!) = # sin(!) + % cos(!)

(#′, %′) Apply trig identities Substitute

# = * cos(()
(#, %) % = * sin(()

! *
(
Rotation
• Rotate the image by an angle of ! about the origin:

#′ = # cos(!) − % sin(!)
%′ = # sin(!) + % cos(!)
(#′, %′)

(#, %) In matrix form:

#′ cos(!) −sin(!) #
= %
! %′ sin(!) cos(!)
Rotation
• 2D rotation in matrix form:

)′ cos(%) −sin(%) )
= +
+′ sin(%) cos(%)

,(%)
• Note: even though cos(%) and sin(%) are nonlinear functions of %,
)′ and +′ are linear combinations of ) and +
• What is the inverse transformation?
• Rotation by –%
• For rotation matrices, ,-. = ,0
Affine transformation
• Combination of translation, non-uniform scaling, rotation, and
shear
!′ % & ! *
= # + +
#′ ' (
Projective homography
• A transformation that preserves straight lines, but not
parallelism
"
$! + &' + ( "
-! + .' + /
! = , ' =
)! + ℎ' + + )! + ℎ' + +
Warping pixels
• Given a coordinate transform ("′, %′) = ((", %) and a source
image )(", %), how do we compute a transformed image
*("′, %′) = )(((", %))?

T(x,y)

y %′
x f(x,y) "′ *("′, %′)
Forward warping
• Send each pixel !(#, %) to its corresponding location
(#′, %′) = )(#, %) in the second image

T(x,y)

y %′
x f(x,y) #′ *(#′, %′)
Forward warping
• Send each pixel !(#, %) to its corresponding location
(#′, %′) = )(#, %) in the second image
• What if (#′, %′) is not an integer location on the pixel grid?
• We can “distribute” colors among the neighboring grid locations –
known as splatting

T(x,y)

y %′
x f(x,y) #′ *(#′, %′)
Inverse warping
• For each pixel grid location ("′, %′) in the second image, get
the color from its corresponding location ", % = ( )* " + , % +
in the first image

( )* ("′, %′)

y %′
x f(x,y) "′ ,("′, %′)
Inverse warping
• For each pixel grid location ("′, %′) in the second image, get
the color from its corresponding location ", % = ( )* " + , % +
in the first image
• What if ", % is not an integer location on the original pixel grid?
• Interpolate!

( )* ("′, %′)

y %′
x f(x,y) "′ ,("′, %′)
Forward vs. inverse warping
• Which is better?
• Usually inverse: more efficient, doesn’t have a problem with holes
• However, it requires an invertible warp function, which is not always
possible