KAZAKH-BRITISH TECHNICAL UNIVERSITY

Electrical and Computer Science epartment

Lecture Note!
"or t#e cour!e
Machine Vision
Lecturer: Associate professor Kornev A.
Almaty 2014
Obectives
!hrou"h theoretical stu#y$ problem solvin" an# laboratory practice$ the course provi#es a
foun#ation in machine vision theory. After successfully completin" the course$ stu#ents are able to:
apply an analysis to ima"e an# vi#eo #ata%
perform calculations base# on machine vision al"orithms$ e.". filterin"$ smoothin"$ se"mentation
an# etc%
use an applie# ima"e soft&are$ e.". Open'V.
'ameras an# ima"e processin" are becomin" important in in#ustry$ me#ical science an# security.
(y un#erstan#in" the basics of machine vision an# ima"e processin"$ stu#ents are able to solve
machine vision problems.
'ourse materials
!e)tboo*s:
+orsyth ,.A.$ -once .. 'omputer Vision$ A Mo#ern Approach. -rentice /all$ 2n# e#.$ 2012.
0on*a M.$ /lavac V.$ (oyle 1. 2ma"e -rocessin"$ Analysis an# Machine Vision. !homson$ 2003. 415
pp.
0upplementary
,avies 6. 1. 'omputer Machine Vision. Aca#emic -ress$ 2012. 712 pp.
/ornber" A. 86#.9 /an#boo* of Machine Vision. :iley$ 200;. 42< pp.
'ourse content
-arts of a machine vision system. 'apturin" #i"ital ima"es. 'amera mo#els. Lenses.
(asics of #i"ital ima"e processin".
2ma"e se"mentation. 2ma"e filterin" an# transformation.
2ma"e morpholo"y an# feature e)traction.
Vi#eo an# motion analysis.
/uman$ 'omputer an# Machine Vision
/uman colour vision 8provi#e# by the ;3 million cones in the eye9 is concentrate# in the centre of the
visual fiel# of the eye 8*no&n as the macula9. !he rest of your retina is ma#e up of aroun# 120 million
ro#s 8cells that are sensitive to visible li"ht of any &avelen"th=colour9. 2n a##ition$ each eye has a
rather lar"e blin# spot &here the optic nerve attaches to the retina.
!he ro#s are e)tremely sensitive photoreceptors$ capable of respon#in" to a sin"le photon$ but they
yiel# relatively poor spatial #etail #espite their hi"h number because many ro#s conver"e to the same
neuron &ithin the retina.
2n contrast$ cones become active at hi"her li"ht levels$ but the si"nal output by each cone in the fovea
is enco#e# by several neurons$ yiel#in" a hi"h resolution in that area.
-ractical Applications of Machine Vision 8MV9
'omputer vision has many applications in in#ustry$ particularly allo&in" the automatic inspection of
manufacture# "oo#s at any sta"e in the pro#uction line. +or e)ample$ it has been use# to:
2nspect printe# circuits boar#s to ensure that trac*s an# components are place# correctly.
2nspect print >uality of labels.
2nspect bottles to ensure they are properly fille#.
!he automatic rea#in" of license plates as they pass throu"h toll"ates on maor roa#s.
Au"mentin" sports broa#casts by #eterminin" #istances for penalties$ alon" &ith a ran"e of other
statistics.
(iometric security chec*s in airports usin" ima"es of faces an# ima"es of fin"erprints.
Au"mentin" movies by the insertion of virtual obects into vi#eo se>uences.
Assistin" #rivers by &arnin" them &hen they are #riftin" out of lane.
'reatin" <, mo#els of a #estroye# buil#in" from multiple ol# photo"raphs.
+uture Apps
:e e)pect machine vision to allo& robotic vehicles to be #riven autonomously%
become an inte"ral part of "eneral computer interfaces an# robotics%
provi#e reliable #ia"noses of me#ical con#itions from me#ical ima"es an# me#ical recor#s%
provi#e increase# levels of security throu"h biometric analysis%
automatically #etermine the i#entity of criminals throu"h the forensic analysis of vi#eo.
MV soft&are
-ersonal = professional pro#uctivety tools
A#obe -hotoshop$ 'orel ,ra&
Online photo e#itors : -i)lr$ +oto+le)er$ -icasa
Me#ical = biolo"ical ima"e analysis tools
Amira
2!K=V!K
?60!AL!
(io2ma"e@,
,2'OM
Auality an# 2nspection soft&are
+oo# inspection
0atellite an# aerial ima"e #ata processin"
<, mo#el$ vi#eo processin" an# animation
Auto#es* Maya$ <,Ma)
-i)ar = 1en#erMan
A#obe After 6ffects B -remiere
MV har#&are
-hoto B vi#eo
'ameras$ Lenses
Au"mente# an# Virtual 1eality 6>uipment
?lasses$ /ea#sets
Au"mente#Creality car &in#o&s
0ensors
MultiCspectral : @Cray$ '!$ ultraCsonic$ infrare#$ ...
0ensors fusion
-hysics of 'olor. Li"ht. 0pectral 1epresentation.
'olor 1epresentation Mo#els.
6M spectrum
6lectroma"netic ra#iation is characteriDe# by its &avelen"th 8or fre>uency9 an# its intensity.
Most cameras an# eyes have several #ifferent types of receptor$ &hich is sensitive to #ifferent
&avelen"ths. 'omparin" the response of several types of sensor yiel#s information about the
#istribution of ener"y &ith &avelen"th for the incomin" li"ht% this is color information.
0everal #ifferent mechanisms result in colore# li"ht. +irst$ li"ht sources can pro#uce #ifferent
amounts of li"ht at #ifferent &avelen"ths. !his is &hat ma*es li"hts loo* oran"e or yello&$ an#
fluorescent li"hts loo* bluish. 0econ#$ for most #iffuse surfaces$ reflectance #epen#s on &avelen"th$
so that some &avelen"ths may be lar"ely absorbe# an# others lar"ely reflecte#. !his means that most
surfaces &ill loo* colore# &hen lit by a &hite li"ht.
!he color is the >uality of an obect or substance &ith respect to li"ht reflecte# by the obect or
emitte# by the obect.
!he li"ht reflecte# from a colore# surface is affecte# by both the color of the li"ht fallin" on the
surface$ an# by the surface. +or e)ample$ a &hite surface lit by re# li"ht &ill reflect re# li"ht$ an# a re#
surface lit by &hite li"ht &ill also reflect re# li"ht.
!he li"ht comin" out of sources or reflecte# from surfaces has more or less ener"y at #ifferent
&avelen"ths$ #epen#in" on the processes that pro#uce# the li"ht. !his #istribution of ener"y &ith
&avelen"th is sometimes calle# a spectral ener"y #ensity C see !ab. 1.
Li"ht
!he visual system respon#s to li"ht in a ran"e of &avelen"ths from appro)imately 400nm to
appro)imately 300nm. !he colors seen at #ifferent &avelen"ths have a set of conventional names$
&hich ori"inate &ith 2saac Ee&ton 8the se>uence from 300nm to 400nm "oes 1e# Oran"e Fello&
?reen (lue 2n#i"o Violet9 C see !ab 2.
!he most important natural li"ht source is the sun. !he sun is usually mo#ele# as a #istant$
bri"ht point. Li"ht from the sun is scattere# by the air. 2n particular$ li"ht can leave the sun$ be
scattere# by the air$ stri*e a surface$ an# be reflecte# into the camera or the eye.
+or clear air$ the intensity of ra#iation scattere# by a unit volume #epen#s on the fourth po&er
of the fre>uency% this means that li"ht of a lon" &avelen"th can travel much farther before bein"
scattere# than li"ht of a short &avelen"th 8this is *no&n as 1aylei"h scatterin"9. !his means that$ &hen
the sun is hi"h in the s*y$ the sun loo*s yello& an# the s*y loo*s blue. !here are stan#ar# mo#els of
the spectral ener"y #ensity of the s*y at #ifferent times of #ay an# latitu#e$ too.
+re>uency$ /D G10
17
H10
17
$10
1;
I H10
13
$10
15
I H10
15
$10
14
H10
14
$10
11
H10
12
$10
10
H10
4
$10
;
:avelen"th$m J10
C10
H10
C12
$10
C4
I H10
C7
$10
C4
I H10
C3
$10
C;
I H10
C5
$10
C<
I H10
C<
$10
C2
I H10
0
$10
2
I
6mission KCrays @Crays ultraviolet li"ht infrare# ra#ar !V$+M
ra#io
!able 1. ,ivision of the &hole electroma"netic spectrum.
L$ nm <40 4<5 500 520 5;5 570 ;25 340
'olor violet blue cyan "reen yello& oran"e re#
!able 2. :avelen"th of the spectrum visible to humans.
!he 'olor of 0urfaces
!he color of surfaces is a result of a lar"e variety of mechanisms$ inclu#in" #ifferential
absorption at #ifferent &avelen"ths$ refraction$ #iffraction$ an# bul* scatterin". !here are t&o
pre#ominant physical mechanisms #escribin" &hat happens &hen a surface is irra#iate#.
+irst$ the surface reflection reboun# incomin" ener"y in a similar &ay to a mirror.
!he spectrum of the reflecte# li"ht remains the same as that of the illuminant an# it is in#epen#ent of
the surface C recall that shiny metals M#o not have a colorM.
0econ#$ the ener"y #iffuses into the material an# reflects ran#omly from the internal pi"ment in the
matter. !he mechanism is calle# bo#y reflection an# is pre#ominant in #ielectrics as plastic or paints.
'olors are cause# by the properties of pi"ment particles &hich absorb certain &avelen"ths from
the incomin" &avelen"th spectrum. Most sensors use# for color capture$ e.". in cameras$ #o not have
#irect access to color$ e)cept a spectrophotometer &hich resembles Ee&tonMs prism.
'olor representation
,escribin" colors accurately is a matter of "reat commercial importance. Many pro#ucts are
closely associate# &ith specific colors for e)ample$ the famous lo"os$ the color of various popular
mobile "a#"ets an# manufacturers are &illin" to ensure that #ifferent batches have the same color.
Linear 'olor 0paces
!here is a natural mechanism for representin" color: a"ree on a stan#ar# set of primaries$ an#
then #escribe any colore# li"ht by the three values of &ei"hts that people &oul# use to match the li"ht
usin" those primaries. +irst international stan#ar# &as establishe# in 17<1 by '26 82nternational
'ommission on 2llumination9$ Lausanne$ 0&itDerlan# C @FN color space. !he actual color is a mi)ture
of
' O c) @ P cF F PcN N$
Fig. 1. Spectral reflectance for a variety of natural surfaces, Forsyth&Ponce.
&here 0 J c)$ cF$ cN J 1 are &ei"hts 8intensities9 in the mi)ture.
!he subspace of colors for humans is calle# the color "amut.
1?( color space
0everal #ifferent primary colors an# correspon#in" color spaces are use# in practice$ an# these spaces
can be transforme# into each other.
!he 1?( color space has its ori"in in color !V &here '1! 8'atho#e 1ay !ubes9 &ere use#. !he 1?(
color space is a linear color space that formally uses sin"le &aveClen"th primaries 8;45.1; nm for 1e#$
52;.<2 nm for ?reen$ an# 444.44 nm for (lue9. :hen #isplayin" the ima"e$ a combination of these
three channels is presente# to the user. 2t is aroun# 1;.4 million 825; 25; 25;9 possible colors usin"
this representation.
!he transformation bet&een 1?( an# @FN is
1?( color information can easily be converte# to "rayscale usin" a formula such as:
F O 0.277 1 P 0.543 ? P 0.114 (.
2t is also &orth pointin" out that in most cameras 1?( values are interpolate#. !his means that
the ima"e receive# is not even a proper samplin" of the continuous ima"e$ but rather is interpolate#
from the #ata receive# by the sensor elements.
[
X
Y
Z
]
=
[
0.41 0.<; 0.14
0.21 0.32 0.03
0.02 0.12 0.75
][
R
G
B
]
Fig. 2. RGB colour image (left an! the same image in grayscale (right.
'yan Ma"enta Fello& 8'MF9 space
!he 'MF mo#el is base# on the secon#ary colors 81?( are the primary colors9$ an# is a
subtractive color scheme% that is the values of the '$ M an# F are subtracte# from pure &hite in or#er
to "et the re>uire# color. 2t is often employe# as a color mo#el &ithin printers &here &hite is the
startin" point.
'onversion from an 1?( ima"e is strai"htfor&ar#:
' O 255 C 1$ M O 255 C ?$ F O 255 C (.
FQV space
!he FQV color mo#el is use# for analo"ue television si"nals 8-AL$ E!0' ...9 an# is comprise#
of luminance 8F9 to"ether &ith t&o color components: blue minus luminance 8Q9 an# re# minus
luminance 8V9 !he transformation from 1?( is a"ain >uite strai"htfor&ar#
F O 0.277 1 P 0.543 ? P 0.114 ($
Q O 0.472 8( C F9$
V O 0.433 81 R F9.
!he human vision system is much more sensitive to luminance than it is to chrominance$ an#
this fact is e)ploite# &hen ima"es are enco#e# in television si"nals in or#er to re#uce the amount of
#ata that nee#s to be transmitte#.
+or e)ample$ in FQV 420p format 4 bytes of luminance 8F9 are transmitte# for every 2 bytes
of chrominance 81 Q an# 1 V9.
/ue Luminance 0aturation 8/L09 space
!he /L0 mo#el is fre>uently use# in computer vision because$ as &ell as separatin" the
luminance an# the chrominance$ it separates the chrominance into hue an# saturation$ "ivin" us
>uantities &hich &e can tal* about easily 8e.". #ar* blue$ li"ht re#$ etc.9. !he luminance typically
ran"es from 0 to 1.
!he hue #escribes the color an# ran"es from 0 to <;0. !he saturation 0 is the #e"ree of stren"th
or purity of the color an# ran"es from 0 to 1. 2n practical implementations$ all of these >uantities are
typically scale# to the 0 to 255 ran"e.
!he /L0 mo#el is an e)ample of nonClinear color space.
Other 'olor 0paces
1. /0V. /ue$ 0aturation an# Value$ is similar to /L0 but the #efinitions of channels #iffer.
2. F'r'b is a scale# version of FQV$ &hich is often use# in ima"e an# vi#eo compression.
0ome 'olor Applications
2n some applications &e nee# to i#entify &hich pi)els represent a particular color. +or e)ample$ to
locate roa# si"ns &e &ill be particularly intereste# in re#$ yello&$ blue$ blac* an# &hite colors.
0*in ,etection
0*in #etection can be performe# by simply analysin" pi)els values. !hrou"h simple
e)perimentation it &as foun# that
80aturation GO 0.29 AE,
80.5 J Luminance S 0aturation J <.09 AE,
8/ue JO 24 O1 /ue GO <<0 9 &ill i#entify many s*in pi)els.
1e# 6ye ,etection
A similar evaluation can be performe# to i#entify re# eye pi)els:
8Luminance GO 0.259 AE, 80aturation GO 0.49 AE,
80.5 J Luminance S 0aturation J 1.59 AE, 8/ue JO 14 O1 /ue GO <249.
!his &as #etermine# e)perimentally$ but it &as foun# to be a "oo# startin" point for the i#entification
of re# eye.
-alette ima"es
-alette ima"es 8calle# also in#e)e# ima"es9 provi#e a simple &ay to re#uce the amount of #ata
nee#e# to represent an ima"e. !he pi)el values set a lin* to loo*up table calle# a color table$ color
map$ or palette. !he loo*up table contains typically 25; entries 4 bits. 6ach entry of the table maps the
pi)el value to the color.
!he palette ima"es are &i#ely use# in ima"e file formats for raster ima"es such as !2++$ ?2+$
-E?.
Key points
6lectroma"netic ra#iation is characteriDe# by its &avelen"th 8or fre>uency9 an# its intensity.
!he color is the >uality of an obect or substance &ith respect to li"ht reflecte# by the obect or emitte#
by the obect.
!his #istribution of ener"y &ith &avelen"th is sometimes calle# a spectral ener"y #ensity.
!he color space is a mechanism for representin" color by three primaries$ e.". 1?(.
!he actual color is a mi)ture of ' O c) @ P cF F PcN N.
2ma"e 1epresentation. Eoise. /isto"rams. !hreshol#in"
2ma"e 1epresentation
A si"nal is a function &hich #epen#s on some physical variable. !he ima"e as a si"nal can be
2C#imensional or <#. +unctions may be cate"oriDe# as continuous$ #iscrete or #i"ital. A continuous
function has a continuous #omain an# ran"e. ,iscrete C if the #omain is #iscrete$ if the ran"e is also
#iscrete then &e have a #i"ital function.
An ima"e is a picture 8"enerally a 2, proection of a <, scene9 capture# by a sensor. 2t is a
continuous function of t&o coor#inates in the ima"e plane% usually e)presse# as f8i$9 or 8column$ ro&9
or some&hat confusin"ly f8)$y9. 0o that the #omain of the ima"e function is a re"ion in the plane.
!he ran"e of ima"e function values is also limite#. 2n monochromatic 8binary9 ima"es the
lo&est values correspon#s to blac* an# the hi"hest to &hite. (ri"htness values boun#e# by these limits
are "reyClevels. 2ma"es can be treate# as #eterministic functions or as realiDation of stochastic process.
"amera
#$%ect(s
Scene
2D image
Digital image
Regions &!ges P#' (e)ture
Image with
Features
(e)ture *.
+i". 1. 2ma"e representation.
19 formation$ 29 #i"itiDation$ <9
feature e)traction$
49 obect #etection
2ma"e ,i"itiDation
!o process such an ima"e on a #i"ital computer it must be both
0ample# into a matri) 8M ro&s an# E columns9$ an#
AuantiDe# so that each element in the matri) is "iven an inte"er value. 2n other &or#s$ the
continuous ran"e is split into some number of intervals 8*9 &here most commonly * O 25;.
0amplin"
,i"ital ima"es are create# by samplin" continuous ima"e into #iscrete elements. ,i"ital ima"e
sensors consist of a 2, array of photosensitive elements$ an# each element 8pi)el=ima"epoint=samplin"
point9 in the array has some fi)e# area over &hich is it photosensitive. !he bi""er issue$ ho&ever$ &ith
samplin" is that pi)els represent the avera"e value 8luminance=chrominance9 over a #iscrete area
&hich in the real &orl# coul# be proecte# from a sin"le obect$ but e>ually 8particularly at the
boun#aries of obects9 coul# be a summation of li"ht reflecte# from multiple obects.
Fig. 2. Four !ifferent samplings of the same image+ top left
2,-)1.2, top right 12/).-, $ottom left -0)0/ an! $ottom
right 12)20 2 3a4son.
!he number of samples in an ima"e limits the obects that can be #istin"uishe# in the ima"e.
/ence$ it is essential that the resolution 8the number of pi)els9 is sufficient for our purpose 8&hatever
that may be9. At the same time$ too hi"h a resolution &ill have more #etail than &e nee# &hich may
ma*e processin" har#er an# &ill #efinitely ma*e it slo&er.
A continuous ima"e is #i"itiDe# at samplin" points. !hese samplin" points are or#ere# in the
plane an# their "eometric relation is calle# the "ri#. !he raster is the "ri# on &hich a nei"hborhoo#
relation bet&een points is #efine#.
One infinitely small samplin" point in the "ri# correspon#s to one picture element calle# a
pi)el. in <# vo)el 8volume element9.
AuantiDation
6ach pi)el in a #i"ital ima"e f8i$9 is a function of scene bri"htness. !he bri"htness values are
continuous$ but &e nee# to represent them #iscretely usin" #i"ital values.
!ypically$ the number of bri"htness levels per channel is * O 2 b &here b is the number of bits
8often 49. !he >uestion &hich must be as*e# is: ho& many bits are really necessary to represent the
pi)elsS !he more bits that are use# the more memory the ima"e &ill re>uire% but as pro"ressively
fe&er bits are use# it is clear that information is bein" lost. 'onsi#erin" +i"ure 4$ you shoul# perceive
little #ifferent bet&een the 4Cbit an# the ;Cbit ima"es althou"h the latter re>uires 25T fe&er bits. !here
are clearly issues &ith the 4Cbit an# 2Cbit ima"es$ althou"h it is still possible for us to reco"niDe many
obects in these ima"es. 0o it appears that the number of bits re>uire# actually #epen#s on the purpose
to &hich the ima"e is to be put. 2f a machine is to automatically interpret a scene$ thou"h$ &e typically
re>uire more >uantiDation levels than mi"ht be e)pecte#$ as other&ise false contours an# incorrect
se"mentation occurs.
Fig. 1. Four !ifferent 5uanti6ations of the same grey7scale
image+ top left / $its, top right - $its,$ottom left 0 $its an!
$ottom right 2 $its 2 3a4son.
Eoise
2ma"es are normally affecte# by noise 8anythin" that #e"ra#es the i#eal ima"e9 to some #e"ree$
an# this noise can have a serious impact on processin". Eoise is cause# by the environment$ the
ima"in" #evice$ electrical interference$ the #i"itiDation process$ an# so on. :e nee# to be able to both
measure noise an# someho& correct it. !he most common measure of noise is the si"nal to noise ratio.
+or an ima"e f8i$ 9 the si"nal to noise ratio is #efine# as follo&:
&here v8i$ 9 is the noise. 2n this section &e &ill consi#er t&o types of noise$ ho& noise affects ima"es
an# ho& &e can simulate noise 8so that &e can evaluate ho& &ell our techni>ues remove it9.
!ypes of Eoise
!he t&o most commonly encountere# types of noise are ?aussian noise an# salt an# pepper noise.
?aussian Eoise
?aussian noise is a "oo# appro)imation to much real noise. Eoise v8i$ 9 is mo#ele# as havin" a
?aussian #istribution aroun# some mean 8S9$ &hich is usually 0$ &ith some stan#ar# #eviation 8S9. +or
an e)ample$ see
s/ Nratio=

ij
f
2
(i , j )/

ij

2
(i , j ) ,
Fig. 0. "olor images 4ith Gaussian noise a!!e! 4ith a
mean of 8 an! a
stan!ar! !eviation of 28 (right. (he signal to noise ratios
of the noisy images are 01.1 2 3a4son.
0alt an# -epper Eoise
2mpulse noise is corruption &ith in#ivi#ual noisy pi)els &hose bri"htness #iffers si"nificantly
from that of the nei"hborhoo# 0alt an# pepper noise is a type of impulse noise &here saturate# impulse
noise affects the ima"e 8i.e. it is corrupte# &ith pure &hite an# blac* pi)els9.
Eoise Mo#els
A##itive Eoise
2n the case of #ata in#epen#ent noise 8i.e. noise &here the amount of noise is not relate# to the
ima"e #ata itself9$ an a##itive noise mo#el is appropriate:
f 8i$ 9 O "8i$ 9 P v8i$ 9
&here "8i$ 9 is the i#eal ima"e$ v8i$ 9 is the noise an# f8i$ 9 is the actual ima"e.
Multiplicative Eoise
2n the case of #ata #epen#ent noise 8i.e. noise &here the amount of noise is relate# to the ima"e
#ata itself9$ a multiplicative noise mo#el is appropriate:
f 8i$ 9 O "8i$ 9 P "8i$ 9 Uv8i$ 9
&here "8i$ 9 is the i#eal ima"e$ v8i$ 9 is the noise an# f8i$ 9 is the actual ima"e.
Fig. ,. "olor images 4ith 189 Salt an! pepper noise
(right. (he signal to
noise ratio of the noisy images are :., 2 3a4son.
/isto"rams
An ima"e histo"ram is an abstraction of an ima"e &here the fre>uency of each ima"e
8bri"htness=intensity9 value is #etermine#.
1, /isto"rams
2n the case of a "rayscale ima"e in &hich there are 25; "rayscale intensities 80$ 2559$ 25;
counts are compute# in#icatin" ho& many pi)els each of the "rayscales in the ima"e have. !he
histo"ram contains "lobal information about the ima"e an# that information is completely in#epen#ent
of the position an# orientation of obects in the scene. 2n some cases$ the histo"ram or information
#erive# from it 8such as the avera"e intensity an# its stan#ar# #eviation9 can be use# to perform
classification. /o&ever$ care must be ta*en as ima"e histo"rams are not uni>ue an# hence many
very #ifferent ima"es may have similar 8or even the same9 histo"ram.
'olor /isto"rams
Another issue that arises is &hat to #o &ith color ima"es. Often histo"rams are #etermine# for
each channel in#epen#ently. !he choice of color mo#el can have a hu"e effect on the usefulness of the
color histo"ram. 1?($ 'MF$ FQV.
/isto"ram=2ma"e 6>ualiDation
Often ima"es may be #ifficult for a human observer to #ecipher #ue to the picture bein"$ for
e)ample$ too #ar*. 2t has been #etermine# that humans can #istin"uish bet&een 300 an# 700 sha#es of
"rey un#er optimal vie&in" con#itions$ althou"h in very #ar* or bri"ht sections of a ima"e the ust
noticeable #ifference re#uces si"nificantly. /o&ever$ it is also clear that it is easier for humans to
#istin"uish lar"er #ifferences$ so if the #istribution of "rayscales in an ima"e is improve#. One
techni>ue for improvin" the #istribution of "rayscales in an ima"e is histo"ram e>ualiDation !his
Fig. -. (he histogram !erive! from a grayscale image 2
3a4son.
techni>ue attempts to #istribute the "rayscales in an ima"e evenly$ so that the resultin" histo"ram is
flat 8i.e. all "rayscales have e)actly the same number of points9.
Eote that &hen e>ualiDin" a color ima"e 8such as that in +i"ure 39 &e "enerally only e>ualiDe
the luminance channel as other&ise the colors can become #istorte#.
(inary 2ma"es.!hreshol#in"
?rayscale ima"es "enerally have 4 bits per pi)el. :hile processin" these ima"es is easier in
some &ays than processin" color ima"es$ there is a simpler form of ima"e$ the binary ima"e$ in &hich
processin" is even more strai"htfor&ar#. 2n fact$ a si"nificant portion of practical applications of
computer vision have been #evelope# usin" binary vision.
A binary ima"e is one in &hich there is only a sin"le bit per pi)el 8e.". blac* or &hite9. !hese
ima"es are create# by threshol#in" &here the threshol#s use# are #etermine# in a variety of &ays.
!hreshol#in"
A binary ima"e is create# from a "rayscale ima"e by threshol#in". !he binary threshol#in"
al"orithm is simply:
+or all pi)els 8i$ 9
f8i$ 9 O 1 &here f 8i$ 9 GO !
O 0 &here f 8i$ 9 J !
Fig. :. ; color image an! its luminance histogram
(left, together 4ith a histogram7e5uali6e! image
an! luminance histogram (right 2 3a4son.
Often "ray level 255 is use# instea# of binary 1 8so that the resultin" ima"e can be represente#
usin" a 4Cbit format an# #isplaye# in the same manner as the ori"inal "rayscale ima"e9.
!he threshol#in" operation is "enerally use# in or#er to separate some obects of interest from
the bac*"roun#. Most typically the obect8s9 of interest are represente# by 1 8or 2559$ but sometimes
the binary ima"e &oul# have to be inverte# for this to be the case.
!hreshol#in" -roblems
!he most important thin" to note about binary ima"in" is that the fore"roun# an# the
bac*"roun# that are bein" separate# nee# to be #istinct in the ima"e bein" threshol#e#. 2f they are not
#istinct then it &ill be #ifficult 8or even impossible9 to accurately se"ment them usin" threshol#in".
/o&ever$ there are a number of techni>ues 8e.". a#aptive threshol#in"9 &hich are use# to try to #eal
&ith situations &here the #istinction bet&een fore"roun# an# bac*"roun# is not clear.
A#aptive !hreshol#in"
2n all of the e)amples sho&n so far$ "lobal threshol#in" has been use# 8i.e. a sin"le threshol#
has been applie# to all points in the ima"e9. 2n some situations it is possible to si"nificantly improve
threshol#in" results by usin" multiple threshol#s. 'onsi#er the threshol#in" in +i" 7$ &here an optimal
"lobal threshol# hi#es most of the &ritten #etails of the #ia"ram. Qsin" a#aptive threshol#in" 8&ith
;4 threshol#s=ima"e bloc*s9 most of the #etails appear correctly.
Fig. /. "orrect threshol!ing (top, too lo4 ($ottom left
an! too high (right 2 3a4son.
!he a#aptive threshol#in" al"orithm is:
1. ,ivi#e the ima"e into subCima"es 8e.". ;4 subCima"es &ere use# an 4V4 "ri#9.
2. +or each of the subCima"es compute a threshol#.
<. +or each point in the ima"e #etermine a threshol# by interpolatin" a threshol# value from the four
nearest threshol#s usin" bilinear interpolation.
Key points
A si"nal is a function &hich #epen#s on some physical variable. !he ima"e as a si"nal can be
2C#imensional or <#. +unctions may be cate"oriDe# as continuous$ #iscrete or #i"ital. A continuous
function has a continuous #omain an# ran"e. ,iscrete C if the #omain is #iscrete$ if the ran"e is also
#iscrete then &e have a #i"ital function. (ri"htness values boun#e# by these limits are "rayClevels. A
continuous ima"e is #i"itiDe# at samplin" points.
!hese samplin" points are or#ere# in the plane an# their "eometric relation is calle# the "ri#. !he
raster is the "ri# on &hich a nei"hborhoo# relation bet&een points is #efine#.
One infinitely small samplin" point in the "ri# correspon#s to one picture element calle# a pi)el. 2n <#
vo)el 8volume element9.
An ima"e histo"ram is an abstraction of an ima"e &here the fre>uency of each ima"e
8bri"htness=intensity9 value is #etermine#.
A binary ima"e is one in &hich there is only a sin"le bit per pi)el.
2ma"e +ormation
1. 'amera Mo#el. Lens !ransformation
2. 'alibration
+i". 1. !he pinhole camera mo#el
!he ima"e is forme# by li"ht rays issue# from the scene facin" the hole. 2f the pinhole &ere
re#uce# to a point$ e)actly one li"ht ray &oul# pass throu"h each point in the plane of the plate 8or
ima"e plane9$ the pinhole$ an# some scene point. 2n reality$ the pinhole &ill have a finite 8albeit small9
siDe$ an# each point in the ima"e plane &ill collect li"ht from a cone of rays. 0o this i#ealiDe# an#
e)tremely simple mo#el of the ima"in" "eometry &ill not strictly apply. 8(runelleschi$ be". @V cent.9
Most cameras use either '', 8char"eCcouple# #evice9 or 'MO0 8complementary metal o)i#e
semicon#uctor9 photosensitive elements both usin" photovoltaic principles. !hey capture bri"htness of
a selecte# color channel or a monochromatic ima"e.
2n a '', sensor$ every pi)elMs char"e is transferre# throu"h ust one output no#e to be
converte# to volta"e$ buffere#$ an# sent offCchip as an analo" si"nal. !he basic sensor element inclu#es
a 0chott*y photo#io#e.
'MO0 technolo"y starte# bein" technolo"ically mastere# from about the 1770s. 2n a 'MO0
sensor$ each pi)el has its o&n char"eCtoCvolta"e conversion$ an# the sensor often inclu#es amplifiers$
noiseCcorrection$ an# #i"itiDation circuits$ so that the chip outputs bits.
!he a#vanta"e of 'MO0 cameras compare# to '', is a hi"her ran"e of sense# intensities 84
times9$ hi"h spee# of rea#Cout 8about 100 ns9 an# ran#om access to in#ivi#ual pi)els. !he #isa#vanta"e
is a hi"her level of noise.
+i". 2. !ra#itional color cameras
Lenses
Most real cameras are e>uippe# &ith lenses. !here are t&o reasons for this. !he first reason is
to "ather li"ht. !he secon# reason is *eepin" the picture in sharp focus &hile "atherin" li"ht from a
lar"e area. 2"norin" #iffraction$ interferences$ an# other physical optics phenomena$ the behavior of
lenses is #ictate# by the la&s of "eometric optics$ see +i".<:
819 li"ht travels in strai"ht lines 8li"ht rays9 in homo"eneous me#ia%
829 &hen a ray is reflecte# from a surface$ this ray$ its reflection$ an# the surface normal are coplanar$
an# the an"les bet&een the normal an# the t&o rays are complementary% an# 8<9 &hen a ray passes
from one me#ium to another$ it is refracte#$ i.e. its #irection chan"es.
Accor#in" to 0nellMs la& the an"les S 1 an# S2 bet&een the normal an# the t&o rays are relate#
by
n
1
sin W
1
O n
2
sin W
2
.
!racin" li"ht rays as they travel throu"h a lens is simpler &hen the an"les bet&een these rays
an# the refractin" surfaces of the lens are assume# to be small$ &hich is the #omain of para)ial 8or
firstCor#er9 "eometric optics$ an# 0nellMs la& becomes n1 S1 S n2 S2. Let us also assume that the lens is
rotationally symmetric about a strai"ht line$ calle# its optical a)is$ an# that all refractive surfaces are
spherical.
Fig. 1. Reflection an! refraction at the interface
$et4een t4o homogeneous me!ia 4ith
in!ices of refraction n1 an! n2 .
Fig. 0. ; thin lens. Rays passing through # are not
refracte!. Rays parallel to the optical a)is are focuse!
on the focal point F<.
'onsi#er a point - locate# at 8ne"ative9 #epth N off the optical a)is$ an# #enote by 8- O9 the
ray passin" throu"h this point an# the center O of the lens 8+i". 49. 2t easily follo&s from the para)ial
form of 0nellMs la& that 8- O9 is not refracte#$ an# that all the other rays passin" throu"h - are focuse#
by the thin lens on the point p &ith #epth D alon" 8- O9 such that$ so calle# the thin lens e>uation$
1=D C 1=N O 1=f$
f O 1 = H28nC19I is the focal len"th of the lens.
!he t&o points + an# + locate# at #istance f from the lens center on the optical a)is are calle#
the focal points of the lens. 2n practice$ obects &ithin some ran"e of #istances 8calle# #epth of fiel# or
#epth of focus9 &ill be in acceptable focus. As sho&n in the problems at the en# of the lecture$ the
#epth of fiel# increases &ith the fCnumber of the lens$ e.". the ratio bet&een the focal len"th of the lens
an# its #iameter.
+i". 5. !he fiel# of vie& of a camera.
Eote that the fiel# of vie& of a camera$ e.". the portion of scene space that actually proects
onto the retina of the camera$ is not #efine# by the focal len"th alone but also #epen#s on the effective
area of the retina 8e.". the area of the sensor in a #i"ital camera9.
A more realistic mo#el of simple optical systems is the thic* lens. !he e>uations #escribin" its
behavior are easily #erive# from the para)ial refraction e>uation$ an# they are the same as thin lens
proection e>uations$ e)cept for an offset.
+i". ;. A simple thic* lens &ith t&o spherical surfaces.
0imple lenses suffer from a number of aberrations. !o un#erstan# &hy$ let us remember that
the para)ial refraction mo#el is only an appro)imation$ vali# &hen the an"le S bet&een each ray alon"
the optical path an# the optical a)is of the len"th is small an# sin .
+i". 3. 0pherical aberration.
!he "rey re"ion is the para)ial Done &here the rays issue# from - intersect at its para)ial ima"e
p. 2f an ima"e plane S &ere erecte# in p$ the ima"e of p in that plane &oul# form a circle of confusion
of #iameter e. !he focus plane yiel#in" the circle of least confusion is in#icate# by a #ashe# line. !he
siDe of that circle &ill chan"e if &e move S alon" the optical a)is. !he circle &ith minimum #iameter is
calle# the circle of least confusion$ an# its center #oes not coinci#e 8in "eneral9 &ith p.
'amera 'alibration
(esi#es spherical aberration$ there is a chromatic aberration: !he in#e) of refraction of a
transparent me#ium #epen#s on the &avelen"th 8or color9 of the inci#ent li"ht rays.
!here are other types C coma$ asti"matism$ fiel# curvature$ an# #istortion.
,istortion chan"es the shape of the ima"e as a &hole. !his effect is #ue to the fact that #ifferent areas
of a lens have sli"htly #ifferent focal len"ths.
Let us estimate the intrinsic an# e)trinsic parameters of a camera from the *no&n ima"e
positions of scene in some fi)e# coor#inate system. 2n this conte)t$ camera calibration can be mo#ele#
as an optimiDation process$ &here the #iscrepancy bet&een the observe# ima"e features an# their
theoretical positions is minimiDe# &ith respect to the cameraMs intrinsic an# e)trinsic parameters.
2n this e)ample$ the calibration ri" is forme# by three "ri#s #ra&n in ortho"onal planes. Other
patterns coul# be use# as &ell an# they may involve lines or other "eometric fi"ures. A precise
#efinition of the camera calibration proce#ure is beyon# the scope of our course.
2n a linear approach to calibration &e #ecompose the calibration process into 819 the
computation of the perspective proection matri) M associate# &ith the camera$ follo&e# by 829 the
estimation of the intrinsic an# e)trinsic parameters of the camera.
!here is no an e)act solution an# an appro)imate one has to be foun# as the solution of a leastC
s>uares minimiDation problem.
A Eonlinear Approach
A nonlinear approach to camera calibration ta*es into account all the relevant constraints. :e
assume once a"ain that &e observe n fi#ucial points -i 8i O 1$ . . . $ n9 &hose positions in some fi)e#
coor#inate system are *no&n$ an# minimiDe the meanCs>uare# #istance bet&een the measure#
positions of their ima"es an# those pre#icte# by the perspective proection e>uation &ith respect to a
vector of camera parameters S$ &hich mi"ht inclu#e various #istortion coefficients in a##ition to the
usual intrinsic an# e)trinsic parameters.