Regularization Methods

An Applied Mathematicians Perspective

Curt Vogel Montana State University Department of Mathematical Sciences Bozeman, MT 59717-2400

SAMSI Opening Workshop p.1/33

Outline
Well (and Ill-) Posedness Regularization Optimization approach (Tikhonov) Bayesian connection (MAP) Filtering approach Iterative approach Regularization Parameter Selection Applied Math Wish List Focus on linear problems.

SAMSI Opening Workshop p.2/33

Well-Posedness
Denition due to Hadamard, 1915: Given mapping , equation

is well-posed provided (Existence) For each (Uniqueness) (Stability)

  

such that

; and

is continuous.

Equation is ill-posed if it is not well-posed.

SAMSI Opening Workshop p.3/33

Linear, Finite-Dimensional Case

( matrix).
               

exists
    

det

         

well-posed

. . .

Existence imposed by considering least squares solutions

! ' & ( %$ #" . ) * +, / - / 5 0 30 1! 2 ' & ( %$ #" ! -4 -

Uniqueness imposed by taking the min norm least squares solution

SAMSI Opening Workshop p.4/33

Innite-Dimensional Example
(Compact) diagonal operator on (Hilbert) space

 6 9 8 7 8  ; <  : 8  : 8 6 @ / A = : / / / 8 8 8 B: : / / / : :  / / / 9 C 7  C B C : : : / / / : / / / : ; / >= ?    : /

Dene

 6

by

Formal (unbounded) inverse is

 

so we have uniqueness (and existence of solutions for certain ).

SAMSI Opening Workshop p.5/33

Dont have stability!

Take
7 E   D I :   : / / / : : : / A /  GF K H / / ;

Then

, but
  --  

Also dont have existence of solns to E.g., .

7 E ; 7 D DB D EL D D : : : / / / : : : D : : : / / /
 6 D D D ;

J

for all

, but

SAMSI Opening Workshop p.6/33

Does this matter?

Example was contrived. Practical computations are discrete, nite dimensional. Can replace (nite dimensional) .
5 

by pseudo-inverse

But ... Discrete problems approximate underlying innite dimensional problems (Discrete problems become increasingly ill-conditioned as they become more accurate). In Inverse Problems applications is often compact, and it acts like the diagonal operator in the above example (Compact operators can be diagonalized using the SVD; diagonal entries decay to zero).

SAMSI Opening Workshop p.7/33

Regularization
Remedy for ill-posedness (or ill-conditioning, in discrete case). Informal Denition: Imposes stability on an ill-posed problem in a manner that yields accurate approximate solutions, often by incorporating prior information. More Formal Denition: Parametric family of approximate inverse operators with the following property. If , and , we can pick parameters such that
NOM P Q VU W V   RS T   Y TZ N M P , , /T N  RS  X

SAMSI Opening Workshop p.8/33

Tikhonov Regularization
Math Interpretation. In simplest case, assume are Hilbert spaces. To obtain regularized soln to , choose to t data in least-squares sense, but penalize solutions of large norm. Solve minimization problem
: - . \ X ' & ( %$ #" U --  -N ) * [ [ ^ ; ] 7 U ] F G

is called the regularization parameter.

` _ a

SAMSI Opening Workshop p.9/33

Geometry of Linear Least Squares

k e e k e k k def d e d e d e d e d f d e d e d d def d e d e d e d e d f d e d e d d
1.8 2

def d e d e d e d e d f d e d e d d def d e d e d e d e d f d e d e d d F G D 

. h hee e h h

c 7

U / GF I GF / D H i
Contours for ||Ax(y)||2 0.2 0.4 0.6 0.8 1 x 1.2

8

/ j H
1.4

g H

Contours for ||Axy|| 2

1.8

1.8

1.6

1.6

1.4

1.4

1.2

1.2

0.8

0.8

0.6

0.6

0.4

0.4

0.2

0.2

0.2

0.4

0.6

0.8

1 x

1.2

1.4

1.6

1.8

1.6

SAMSI Opening Workshop p.10/33

Geometry of Tikhonov Regularization

;-  c 7 VU . F / G
Contours for ||Axy||2 + *||x||2 2 2 1.8 1.8 1.6 1.6 1.4 1.4 1.2 1.2 1 1

0.8

0.8

0.6

0.6

0.4

0.4

0.2

0.2

0.2

0.4

0.6

0.8

1 x

1.2

1.4

1.6

1.8

0.2

0.4

Contours for ||Ax(y+)||2 + *||x||2

0.6

H
0.8 1 x

I
1.2

1.4

-- 
1.6

--

1.8

SAMSI Opening Workshop p.11/33

Bayesian Interpretation (MAP)

Bayes Law: Assume random variables.
7 ; l GF I H

are jointly distributed continuous

; 7 ; E 7 ;

GF

u = H v I m rs

GF

pr GF

l H rs = r = = = pq J Q nJ q mno nq m tn  m nq mn s = q m nq I v G v H G nz v p{ = n =K r ; 7 ; 7 x w y x w y ; x w y $ $ l l $ . . F I F . )

Maximum a posteriori (MAP) extimator is max of posterior pdf. Equivalently, minimize w.r.t.
H l 7

First term on rhs is t-to-data term; second is regularization term.

nz v

SAMSI Opening Workshop p.12/33

If
| } l 7 VU ; l 7 ; B7 | D ; j .

If

c 7 ; --

Normal
 : ~ ^ ; D 7

and

. - j  ~ U X --  : X  ~  j ~  
SAMSI Opening Workshop p.13/33

B7 l 

Tikhonov cost functional is

V 

, then prior is
 l  ~ ; 

Normal

7  :  ~ ^ ;

. ~  . - BE  j / BE  ~

--  

Illustrative Example

, conditional pdf is

SNR

Singular Value Decomposition

Important tool for analysis and computation. Gives bi-orthogonal diagonalization of linear operator,
 / ]

9 7

In and

matrix case,
 : : / / / 

diag

: : / ] / /

with
 / / /  : 

: 0 =

: =

: =

SAMSI Opening Workshop p.14/33

Tikhonov Filtering
In the case of Tikhonov regularization, using the SVD (and assuming matrix with for simplicity),
]    ^ ; ] 7 N M U X ] = b 

] ;

] X / 

^ ;

diag

GF

97 

If

, then
N M

= ;

, so
Y TZ ; ] 5

diag

D E 9

as

SAMSI Opening Workshop p.15/33

Tikhonov Filtering, Continued

7

Plot of Tikhonov lter function shows that Tikhonov regularization lters out singular components that are small (relative to ) while retaining components that are large.
N o
1

0.8

w (s2)

0.6

0.4

0.2

0 5 10

10

Filter Functions for TSVD and Tikhonov Regularization

10

10 s2

10

10

10

SAMSI Opening Workshop p.16/33

Truncated SVD (TSVD) Regularization

TSVD ltering function is
  b / X = : : X = = ; :  D N 7 !

Has sharp cut-off behavior instead of smooth roll-off behavior of Tikhonov lter.

SAMSI Opening Workshop p.17/33

Iterative Regularization
Certain iterative methods, e.g., steepest descent, conjugate gradients, and Richardson-Lucy (EM), have regularizing effects with the regularization parameter equal to the number of iterations. These are useful in applications, like 3-D imaging, with many unknowns. An example is Landweber iteration, a variant of steepest descent. Minimize the least squares t-to-data functional
D B . - c 7 ; -

using gradient descent iteration, initial guess xed step length parameter .
 0@ @ D -  E-

, and

SAMSI Opening Workshop p.18/33

Landweber Iteration
{ { c 7

grad
7 ] ] ;

;  : : : : { { . U = ;{ I ;  ] ; 7 D G s H pq qQ p J= p v
10 s2
1

DB /  s
10
0

diag

Landweber Filter Functions for k=10() and k=100()

0.8

w (s )

0.6

0.4

0.2

0 4 10

10

10

/
10
1

SAMSI Opening Workshop p.19/33

Effect of Regularization Parameter

Illustrative Example: 1-D deconvolution with Gaussian kernel and discrete data
.  7 ; 7 BE   7 P8 ; 7 ; ;

noise

True Solution and Discrete Noisy Data 1.2

0.8

0.6

0.4

0.2

0.2

0.1

0.2

0.3

0.4

0.5 x axis

0.6

0.7

0.8

0.9

SAMSI Opening Workshop p.20/33

/  : / / / :

.=

RS

Graphical Representation of SVD

Singular Values
10
0

Singular Vectors
Singular Vector v1 0.2 0.2 v4

Singular Values of K

10

0.15

0.1

10

0.1

10

0.05

0.1

10

0.2

0.4 0.6 x axis v

10

0.8

0.2

0.2

0.4 0.6 x axis v

20

0.8

10

10

0.2

0.3 0.2

10

12

0.1

0.1 0

10

14

0 0.1 0.2 0.3

10

16

0.1

10

18

10

20

30

40 index i

50

60

70

80

0.2

0.2

0.4 0.6 x axis

0.8

0.4

0.2

0.4 0.6 x axis

0.8

SAMSI Opening Workshop p.21/33

Tikhonov Solutions vs
Tikhonov regularized solution is Solution error is .
N 8 .P 8 N RS T
Norm of Tikhonov Solution Error 4 3.5 3 1.2 1 0.8

^  ;

= 0.0014384

||e||

x(t)

2.5

0.6 0.4 0.2 0

1.5 1 0.5 6 10 10
4

10

10

0.2

0.2

0.4 t axis =1

0.6

0.8

]
0.8

= 1e06 2 1.5 1 1.2 1 0.8

x(t)

x(t)
0 0.2 0.4 t axis 0.6 0.8 1

0.5

0.6 0.4 0.2 0

0.5 1 1.5

0.2

0.2

0.4 t axis

0.6

SAMSI Opening Workshop p.22/33

N Y TZ N Y TZ N .Q P N M O F N M . ^ P ; Q RS . ^ ; RS T U NO V M
SAMSI Opening Workshop p.23/33

Solution Error:

N . P RS T N 7 N M O uo H = G 7 T RS N

Predictive Error:
P Q

Regularized Solution:

Linear, Additive Noise Data Model:

diag
N = ; E 9 7

RS T 7 VU

Error Indicators

I T U uq u tp

P Q

= G = H I F NOM V ]

Unbiased Predictive Risk Estimator

The Inuence Matrix is
Y TZ 7 ; N M O : X 7 7 ;

so we can write the predictive error as

; 7 P Q ^ N X U RS T XV . ;

Residual is
Y TZ . P N . N RS T F . G qQ pJ H p P V 7 ; RS 7 X ; V  T I V ; 7 7 ; ; 7 ^ 7 ; X U X ^

Let denote expected value operator. Assume deterministic (or independent of ), assume note that is symmetric. Then ...

is , and

SAMSI Opening Workshop p.24/33

N --  B F V ] X XV ; 7 ; . ^ 7 7 --

So up to const
7 ; --  V -.

X N ---  UB N --  UB j ~  V ] XV X ; 7 ; 7 -Y TZ

Last equality follows if

 ~ : : U V / - ` ( G RS H T --  ; P

V =  : j 

UPRE, Continued
U V ] N ---  I 7 X  V ;

, an unbiased estimator for

trace

is

SAMSI Opening Workshop p.25/33

Comments about UPRE

UPRE regularization parameter selection method, also known as Mallows method, is to pick to mimimize .
7 X ; -X

Predictive error norm and solution error norm need not have the same minimizer, but the mins are often quite close. There is a variant of UPRE, called generalized cross validation (GCV), which requires minimization of
Y TZ 7 X ; N --  -; 7 7 N -N

--

--

trace

This does not require prior knowledge of variance

SAMSI Opening Workshop p.26/33

Illustrative Example of Indicators

:  ~
0

2-D image reconstruction problem, noise N , Tikhonov regularization. o-o indicates soln error norm; indicates GCV; indicates ; and indicates predictive error norm.
V | 7 . . X ; .
10
7

U() is solid line; P() is dashed line; GCV() is dotdashed line.

10

10

10

10

10 6 10

10

10

10 10 Regularization Parameter

10

;
10
1

10

SAMSI Opening Workshop p.27/33

Mathematical Summary
There exists a well-developed mathematical theory of regularization. There are a number of different approaches to regularization. optimization-based (equivalent to MAP) ltering-based iteration-based There are robust schemes for choosing regularization parameters. These techniques often work well in practical applications.

SAMSI Opening Workshop p.28/33

But Things Can Get Ugly ...

Astronomical Imaging Application. Light intensity
; ^ 97 ; 7 9 7 9 ; / I F : G n . : : . 8

This is measured by a ccd array (digital camera), giving data

^ 7 9 = ;

noise

: = 

For high contrast imaging (dim object near very bright object), accurate modeling of noise is critical. With ordinary (and even weighted) least squares, dim object is missed.

tJ p

SAMSI Opening Workshop p.29/33

Model for Data from CCD Array

7 ; U =W VU D = = : = : / / / :  ^  97 = ;

Photon count for signal

7 | ; ;

Poisson
7

: =

: = 

/ / / =

Background photon count

7

Poisson

xed, known

Instrument read noise

 7

xed, known

SAMSI Opening Workshop p.30/33

Imaging Example, Continued

7 -

Log likelihood (
7 ; .

x w y

 g

<

) is messy
 = U 8 ; 7  r  g  

x w y

>=

Light source (object) intensity is nonnegative. Constraint .

7 ;

With pixel discretization, dimension is very large, e.g., size size or more.
7 ; 7 ; B

Problem is ill-posed. Need regularization (prior), e.g.,

X  - : X b / 

Regularization parameter (strength of prior) is unknown.

>

SAMSI Opening Workshop p.31/33

Applied Mathematicians Wish List

Optimization-based regularization methods (Tikhonov, MAP) require soln of minimization problems. Need fast, robust, large-scale, nonlinear constrained numerical optimization techniques. When the parameter-to-observation map is nonlinear, regularization functionals may be non-convex. Need optimization methods which yield the global minimizer (not just a local min) and are fast, robust, .... Need indicators of reliability (e.g., condence intervals) for regularized solutions. Need good priors. Need fast, robust schemes for choosing regularization parameters.

SAMSI Opening Workshop p.32/33

Challenge for the Statistics Community

Can MCMC techniques provide fast, robust alternatives to optimization-based regularization methods? Relevant Reference: J. Kaipio, et al, "Statistical inversion and Monte Carlo sampling methods in electrical impedance tomography", Inverse Problems, vol 16 (2000), pp. 1487-1522. Relevant Caveat: There is no free lunch.

SAMSI Opening Workshop p.33/33