2015 International Conference on Intelligent Networking and Collaborative Systems

A Novel Exemplar-based Image Inpainting Algorithm

LIU Ying, LIU Chan-juan*, ZOU Hai-lin, ZHOU Shu-sen, SHEN Qian, CHEN Tong-tong
School of Information and Electrical Engineering
Ludong University
Yantai, China
luckycj80@sina.com

Abstract—A novel exemplar-based image inpainting

algorithm is proposed for solving the deficiencies of the For large damaged image regions, methods based on
classical Criminisi method, such as the error repair texture synthesis based on image block can be divided into
accumulation, high time complexity caused by the two classes: One is based on the decomposition method.
unreasonable design of the patch priority, inaccuracy criterion The other one is exemplar based image inpainting algorithm
and its global search strategy. Thus, we construct a local that have been widely used in recent years, Criminisi
structure measurement function by introducing the structure
algorithm [5],that keeps better repair effects in structure and
tensor theory, and optimize the means of patch priority. On
that basis, we design a matching criterion. Experiments show
texture by calculating priority and selecting suitable sample
that the improved algorithm has greater advantages on the matching blocks. While, due to its unreasonable setup of
fidelity of image structure compared with the Criminisi priority, update of confidence coefficient, as well as the
method. Besides, the improved algorithm makes progresses in matching criterion of the optimal matching patches, are
both subjective visual effect and objective indexes, such as likely to cause ‘unreasonable’ repair effects or even an error
peak signal-to-noise ratio (PSNR), repair error and the repair.
running time compared with some of the typical image
restoration algorithms proposed recent years. A. Related work
A number of improved algorithms have been proposed
Keywords: image inpainting; priority; confidence level;
in recent years. By putting forward more reasonable priority
matching criterion
calculation formula, Cheng H.W [6] ensures the accurate
filling of the structure and texture information. Similar to
I. INTRODUCTION [6] Lin Y.L [7] is also focus on the setup of the priority by
As one of the main contents of the image processing, introducing compensation factor α and
image inpainting is a hot issue and the main content in the weighting factor γ . Huang A.B [8] enhances the repair
filed of computer vision. It’s a preprocessing part that effect on the edge of the damaged area by increasing the
belongs to the machine learning [1]. In order to maintain an boundary elements. Ren S [9] presents the mean vector to
integrated image, image inpainting estimates the damaged improve texture extension effect. By introducing the spatial
area using the neighborhood information of the known. Its distribution difference Yang S [10] fills the damaged area
main purpose is to repair the damaged image according to with the best reference blocks. For color image restoration,
certain rules, and make no repair trace to the observers. Nie H [11] introduces the KL transform into the TV model.
Image inpainting has been used in many fields, such as These improved algorithms though promoting visual
cultural relic repair, image matching. Besides it can also effects, because of the invariance of the search mode or
make contribution to the heritage image retrieval. The diffusion mode of, are really time consuming.
inpainted images can obviously increase the retrieval
efficiency and accuracy. The problem has been addressed B. Our approach
by two classes of algorithm: inpainting techniques for In this paper, our work includes:
filling in small image gaps and texture synthesis algorithm
• Set the multiplication form of the priority referred in
for generating large image region from sample textures. the Criminisi algorithm into the sum form to avoid
For small image gaps, method based on partial error repair in the inpainting process and use
differential equation (PDE) is the chief approach that structure tensor to construct a structure control
mainly using anisotropic diffusion to spread information function to depict the image block information. Add
from the known area to the damaged. Typical models the structure control function into the priority
include: the BSCB model [2] , the TV model [3]. Then on formula to increase the structural information affect
the basis of the TV model the CDD model [4] ensures the and ensure appropriate transmission of the structure
visual connectivity by changing the conduction coefficient information, thus reduces the error matching rate.
into the curvature functions.
• Define a new search matching relation and decrease
the calculating number for the distance measure,
*Corresponding author.** which greatly reduces the time complexity by

978-1-4673-7695-2/15 $31.00 © 2015 IEEE 86

DOI 10.1109/INCoS.2015.15
Authorized licensed use limited to: J.R.D. Tata Memorial Library Indian Institute of Science Bengaluru. Downloaded on July 09,2024 at 05:40:31 UTC from IEEE Xplore. Restrictions apply.
 converting the global search matching algorithm into
local search.
¦ C (q)
q∈Ψp Ω | ∇I p⊥ ⋅ n p |
C ( p) = , D( p) = (2)
• Introduce eigenvalues of the structure tensor to the | Ψp | α
matching criterion and reduce the chance of a
random matching block, thus improve the matching Where | Ψ p | is the area of Ψp , α is a normalization
accuracy.
factor (e.g., α =255 for a typical gray-level image), n p is a
C. Paper Outline unit direction vector orthogonal to the front δΩ in the pixel
In the next section, we will briefly introduce Criminisi
p and ⊥ denotes the orthogonal operator. The
algorithm and analysis the generation of its deficiencies. priority P( p ) is computed for every border patch, with
Section 3 presents the definition of structure tensor and its distinct patches for each pixel on the boundary of the target
application in feature description. And then we expound our region. C ( p ) is initialized as follows:
method in detail, including the construction of the local
structure feature measurement function, the priority
optimization, and the optimal matching criterion. The C ( p ) = 0, ∀p ∈ Ω, C ( p ) = 1, ∀p ∈ Φ (3)
experimental study and its analysis are given in section 4.
We conclude this paper and discuss the future work in the Step 2: Determine the optimal matching patch
last section.
Choose the highest priority template block in turns
II. CRIMINISI ALGORITHM according to the above formula, and then find out the most
similar sample patch compared with the target one
The main thought of the Criminsi algorithm is: select throughout the whole known area. The matching criterion
pixels from the boundary of the damaged area, and make it is:
the center to an square template block, then traverse the
known area search for optimal matching patch from the
highest priority calculated in advance and copy to the target Ψqˆ = arg min d ( Ψ pˆ , Ψq ) (4)
Ψq ∈Φ
one finally. Fig. 1 shows the principle of the algorithm.
I Where d ( Ψ pˆ , Ψq ) is the sum of squared differences
 (SSD) between all pixels in patch Ψp̂ and Ψq .

p Step 3: Update the confidence
p
Copy the target piece of pixel value to the matching one
np after finding the optimal matching patch, and then update
the original unknown confidence coefficients.
∇I p⊥ Ω
Constantly repeat the above steps until the whole image
is fully restored.
Fig. 1. Notation diagram. Given the patch Ψ p , n p is the normal to the
contour δΩ of the target region and ∇I p⊥ is the isophote (direction Aimed at solving the deficiencies the Criminisi
algorithm generates, in this paper we propose some
and intensity) at point p . The entire image is denoted with I while improvements in priority setting, searching way and
Φ to be the undamaged area and Ω to be the damaged. matching criterion separately.
The algorithm can be divided into the following three
steps: III. A NOVEL EXEMPLAR-BASED INPAINTING ALGORITHM

Step 1: Calculate the priority A. Priority setting

To determine the repair order, we compare the priority Firstly, Criminisi algorithm adopts multiplication form
P ( p ) of each pixel on the damaged boundary. We define of the confidence level item and the data item. Along with
the inpainting process, the number of known pixels will
the priority P ( p ) as the product of two terms:
increase and the confidence term will become bigger. The
image patch with a lot of known information is still likely to
P( p) = C ( p) D( p) (1) get latter repair with smaller data item, and this may cause
inpainting error. However, image patch with less data term
information and more known pixels can still get priority
We call C ( p ) the confidence term and D( p ) the data repair theoretically, while taking the sum form instead of
term, and they are defined as follows: multiplication.

87

Authorized licensed use limited to: J.R.D. Tata Memorial Library Indian Institute of Science Bengaluru. Downloaded on July 09,2024 at 05:40:31 UTC from IEEE Xplore. Restrictions apply.
 Secondly, the data term is only defined by isophote, From what has been discussed above, we define the
which can not fully describe image structure. While, the structure control function constructed by the local structure
structure tensor can effectively describe the local structure of measurement function as follows:
information, and the relationship of its eigenvalues can
represent different image partitions. Based on this, we
introduce the structure tensor into our proposed model to H ( p ) = K ⋅ h + exp( − h ) (8)
optimal the priority.
Where K is an constant, and after experimental
Structure tensor: verification we can able to recover structure characteristics
of the damaged image accurately and promote the visual
effect obviously when K = 0.8 . The introduction of
§ ∂ 2u ∂u ∂u ·
¨ 2 ⋅ ¸ H ( p ) , firstly, increases the structure factors in the repair
§j j12 · ¨ ∂ x ∂x ∂y ¸ process and ensures the spread of priority for the local
J ρ = ¨¨ 11 ¸¸ = G ρ * ¨ (5)
∂ 2u ¸
2
© j21 j22 ¹ ∂u ∂u structure. Secondly, it also leads to more accurate
¨¨ ⋅ ¸
© ∂x ∂y ∂ 2 y ¸¹ judgments to image structure for the algorithm, thus ensures
the repair effect. The improved priority is:
Where Gρ is a Gaussian function with standard
∂u ∂u P ( p ) = a1C ( p ) + a 2 D ( p ) + a 3 H ( p ) (9)
deviation ρ , u presents the image, and and are
∂x ∂y
horizontal and vertical components of the gradient vector at Where a1 , a 2 , a3 are weights, and a1 + a 2 + a3 = 1 .
each pixel respectively. Since matrix J ρ is symmetric and We can enlarge the proportion of a1 appropriately, when
positive semi-definite, it has two eigenvalues as follows: the damaged image has rich texture feature. Experimental
parameter can be set to a1 : a 2 : a3 = 3 : 1 : 1 . And when the
local structure information is more abundant, we can
1
λ1,2 = ( j11 + j22 ± ( j11 + j22 ) 2 + 4 j122 ) (6) enlarge the proportion of a2 . Experiment parameter can be
2
set to a1 : a 2 : a3 = 1 : 3 : 1 , and similarly set to
This paper introduces the local measure function a1 : a 2 : a3 = 1 : 1 : 3 when the linear structure is rich.
defined in [12] :
B. Search for the optimal patch
h =| λ1 − λ2 | 2
(7) Criminisi algorithm searches the optimal match patch
among the global scope after finding image patch with
highest priority. No doubt, this will increase the time
Equation (9) characterizes the constant area of the complexity and reduce the repair efficiency. In view of this,
image when meet the condition h → 0 , while denotes area we reset the search way for optimal matching block. That is
of rich structure information, such as the edges and corners.
only when the known pixel values in the match patch Ψp
Our structure control function should satisfy the corresponding to the target one, meet the following
following conditions: increases with the growth of h as formula:
well as the improvement of the image structure. In order to
reduce the computation complexity as far as possible, here
we adapt the linear increment form. It indicates that the (1 − δ ) ⋅ sum( Ψ p ) ≤ sum( Ψ′q ) ≤ (1 + δ ) ⋅ sum( Ψ p ) (10)
image tends to constant area when h → 0 .To ensure the
structure control function non-vanishing and the effect to where Ψq' denotes patches with zero pixel values
describe the local structure, we plus a small value exp( −h )
corresponding to the match ones and δ ∈ [0,1] .Demonstrated
on the basis of the linear follow function. When the two
eigenvalues have little value difference, it can ensure the from experimental verification that we can not only obtain
structure change control function also non-vanishing and good visual repair effect but also reduce the time
play a role in describing the influence of structure strength. complexity effectively when we set δ = 0.2
When the difference is enhancing, however its effect is
reducing. As this result, the structure control function can C. Matching criterion
ensure the linear following performance of the structure The matching criterion that the second step of the
control function, and describe the structure changes of the Criminisi algorithm based to search the optimal matching
image accurately. patch is:

88

Authorized licensed use limited to: J.R.D. Tata Memorial Library Indian Institute of Science Bengaluru. Downloaded on July 09,2024 at 05:40:31 UTC from IEEE Xplore. Restrictions apply.
 Ψq~ = arg min d ( Ψ p , Ψq ) (11) use cartoon image ( a1 ), rich structure ( b1 ), rich texture
Ψq ⊂ Φ
details ( c1 ) as the input images respectively to verify the
However, it tends to have more than one target patches validity of our algorithm. Taking peak signal-to-noise
that consistent with the ready matching one according to the ratio(PSNR),the mean absolute error(MAE) and the running
color matching rule. Then the Criminisi algorithm selects time(s) as the objective indexes to evaluate the inpainted
the target block randomly. However the randomness will images treated by CDD model based on partial differential
seriously affect the effect of inpainting if the target patch equation, Criminsi algorithm and [8] model.
selected unsuitably. Here, using the proposed structure Peak signal-to-noise ratio (PSNR) is defined as:
tensor eigenvalues aforementioned, we define a new
matching criterion:
2552
PSNR = 10 lg m n
(13)
n 1
Ψq~ = arg min d ( Ψ p , Ψq ) + ¦ [(λ1 − λ1′) 2 + (λ 2 − λ 2′ ) 2 ]
Ψq ⊂ Φ
(12) ¦¦ (uˆ − u) 2
m × n i =1 j =1
i =1

It ensures higher similarity of the target patch and the Mean absolute error (MAE) is defined as:
matching one to a large extent, thereby reduces the number
of the multiple target patch significantly and the error
matching rate accordingly.
Fig. 2 to Fig. 4 show the image visual effect of the
IV. RESULTS AND ANALYSIS tested images using four approaches separately. TABLEĉ
In order to verify the effectiveness of the algorithm to TABLE ċ show the value of PSNR, MAE and the
proposed in this paper, we perform sets of simulation running time using four different approaches .
experiments make use of MATLAB 㧔 2011b 㧕 . Now we

( a1 ) Original image ( a 2 ) Damaged image ( a 3 ) Criminsi algorithm ( b4 ) CDD model ( b5 ) Reference [8] model ( b6 ) Our algorithm

Fig. 3. Inpainted image 2 using four different approaches.

( a 4 ) CDD model ( a 5 ) Reference [8] model ( a 6 ) Our algorithm

Fig. 2. Inpainted image 1 using four different approaches.
( c1 ) Original image ( c 2 ) Damaged image ( c3 ) Criminsi algorithm

( c 4 ) CDD model ( c5 ) Reference [8] model ( c6 ) Our algorithm

( b1 ) Original image ( b2 ) Damaged image ( b3 ) Criminsi algorithm
Fig. 4. Inpainted image 3 using four different approaches.

89

Authorized licensed use limited to: J.R.D. Tata Memorial Library Indian Institute of Science Bengaluru. Downloaded on July 09,2024 at 05:40:31 UTC from IEEE Xplore. Restrictions apply.
 TABLE I. PSNR(DB) WITH DIFFERENT APPROACHES and search method for optimal matching patch.
Test images Experimental results prove the validity of the improved
Approaches
Image 1 Image 2 Image 3 algorithm in the quality of the restored image. But the
CDD model 14.0762 18.9257 27.1975 threshold set in the proposed algorithm can not change with
Criminisi algorithm 17.9319 25.7220 30.8456 different image structure correspondingly, and this may
Reference[8]model 30.0245 25.4812 30.9023
Our algorithm 35.2745 26.8389 30.9677 limit the applicability of it to some extent. With the
development of the GPU and cloud computing
TABLE II. MAE WITH DIFFERENT APPROACHES technologies, it makes sense to use these resources to
Test images
improve the time efficiency of image inpainting algorithms.
Approaches These are also the next research contents in our future work.
Image 1 Image 2 Image 3
CDD model 15.1962 10.3911 6.4777
Criminisi algorithm 5.4809 5.6140 5.6792 ACKNOWLEDGMENT
Reference[8]model 1.1560 5.4563 5.5326
This work is supported by the National Science
Our algorithm 0.1297 5.3267 5.5629
Foundation of China (No.61170161, No.61300155),
TABLE III. RUNNING TIME(S)WITH DIFFERENT APPROACHES
outstanding Young Scientists Foundation Grant of
Shandong Province (No.BS2014DX016), Ph.D. Programs
Test images Foundation of Ludong University (No.LY2014033,
Approaches
Image 1 Image 2 Image 3
LY2015033).
CDD model 40.0328 38.3720 176.9678
Criminisi algorithm 35.1640 35.9750 160.9770
Reference[8]model 15.9602 22.1341 85.6324
Our algorithm 16.4930 19.5470 70.6720 REFERENCES
As shown in Fig.3, there is an obvious fracture of the roof
[1] S.M Valiollahzadeh, Nazari, M.Babaie-zadeh, “A new approach in
structure in Criminisi algorithm, and appears blur decomposition over multiple overcomplete dictionaries with applicati
phenomena in CDD model and incomplete inpainting in [8] on to image inpainting,” MLSP,IEEE International Workshop on,
model. TABLE ċ .shows that our algorithm has obvious pp.1-6, 2009.
advantage in time efficiency. Especially for image 3, our [2] M. Bertalmio, G. Sapiro, C. Ballester and Caselles,
“Image inpainti-ng,” ACM New York. pp. 417-424, 2000.
method respectively reduces the running time by 60%, 56%,
[3] T.F. Chan, J.H. Shen, “Mathematical models for local non-texture
18% compared with the CDD model, the Criminisi inpainting,” SIAM ,vol.62,pp.1019-1043, 2001.
algorithm and the reference [8] model. [4] T.F. Chan, J.H. Shen, “Non-texture inpainting by curvature-driven
In conclusion, our algorithm performs better than the diffusions,”Visual Communication and Image Representation, vol.1
other three in both visual effects and objective evaluation 2, pp.436-449, 2001.
indexes (PSNR, MAE and running time), because: [5] A. Criminisi, P. Perez, K. Toyama, “Region filling and object
removal by exemplar-based image inpainting,” .IEEE Transactions
Firstly, it ensures the spreading of the image structure on Image Processing, vol.13, pp.1200-1212. 2004.
information by introducing the structure control function [6] H.W. Cheng, C.W. Hsich, S.K. Lin, “Robust algorithm for
H ( p ) and taking the sum form instead of multiplication, exemplar-based image inpainting,” Beijing international conference
on computer graphics, imaging and vision, pp.214-224, 2005.
thereby promotes the repair effects. Secondly, the
[7] Y.L. Lin, J.H. Zhao, X.F. Zhu, “An improved algorithm for image
introduction of the structure tensor makes more precise inpaiting based on texture synthesis,” .Computer Application and
matching to repair piece with the target patches, thus Software, vol.27, pp.:11-12, 2010.
reduces the error matching rate and error accumulation rate [8] A.B. Huang, X.L. Zhu, Y.Y. Xu, “An improved image inpainting
of the original algorithm, and receives obviously algorithm based on texture synthesis,” Journal of Hefei University
improvement of the visual effect. Thirdly, convert of the of Technology, vol.34, pp.313-320, 2011.
searching way reduces the redundancy of the matching and [9] S. Ren, X.H. Tang, G.L. Kang, “An image restoration algorithm
the time complexity. combined texture and edge character,” Journal of Computer-Aided
Design & Computer Graphics, vol.25, pp.1682-1693, 2013.
[10] S. Yang, Z.Z. Yang, “Image restoration based on reference of texture
V. SUMMARY and own color,” Journal of computer applications, vol.34, pp. 1724-
1726, 1734, 2014.
In this paper we introduce the structure of information
[11] H. Nie, J. Peng, “An image restoration algorithm based on KL
measure H ( p ) to priority setting on the basis of analyzing transform and TV model,” Silicon Valley , vol. 169, pp.59-66, 2015.
the deficiencies of the large area of image restoration [12] U. Kothe, “Edge and junction detection with an improved structure
algorithm the Criminisi algorithm. Besides, we have also tensor,” Lecture Notes in Computer Science. Berlin, Springer-Verlag,
made corresponding adjustments in the matching criterion pp.25-32, 2003.

90

Authorized licensed use limited to: J.R.D. Tata Memorial Library Indian Institute of Science Bengaluru. Downloaded on July 09,2024 at 05:40:31 UTC from IEEE Xplore. Restrictions apply.