Automation in Construction 22 (2012) 567–576

Contents lists available at SciVerse ScienceDirect

Automation in Construction
journal homepage: www.elsevier.com/locate/autcon

Adaptive vision-based crack detection using 3D scene reconstruction for condition

assessment of structures
Mohammad R. Jahanshahi ⁎, Sami F. Masri
Sonny Astani Department of Civil and Environmental Engineering, University of Southern California, Los Angeles, CA, 90089, USA

a r t i c l e i n f o a b s t r a c t

Article history: Current inspection standards require an inspector to travel to a target structure site and visually assess the
Accepted 29 November 2011 structure's condition. This approach is labor-intensive, yet highly qualitative. A less time-consuming and
Available online 2 January 2012 inexpensive alternative to current monitoring methods is to use a robotic system that could inspect struc-
tures more frequently, and perform autonomous damage detection. In this paper, a vision-based crack
Keywords:
detection methodology is introduced. The proposed approach processes 2D digital images (image proces-
Crack detection
Computer vision
sing) by considering the geometry of the scene (computer vision). The crack segmentation parameters
Image processing are adjusted automatically based on depth parameters. The depth perception is obtained using 3D scene
Pattern classification reconstruction. This system extracts the whole crack from its background, where the regular edge-based
3D scene reconstruction approaches just segment the crack edges. This characteristic is appropriate for the development of a crack
Morphological operation thickness quantification system. Experimental tests have been carried out to evaluate the performance of
the proposed system.
© 2011 Elsevier B.V. All rights reserved.

1. Introduction initial step towards an autonomous crack quantification approach.

The field implementation which is the main focus of this study,
1.1. Motivation makes the current study different from other proposed crack detec-
tion techniques where the camera-object distance and the image con-
Civil infrastructure system assets represent a significant fraction trast can be controlled. Unlike previous studies (e.g., finding cracks in
of the global assets and in the United States are estimated to be paintings), the specimen under inspection cannot be investigated in a
worth $20 trillion. These systems are subject to deterioration due laboratory environment, and the 3D depth perception of the scene is
to excessive usage, overloading, and aging materials, as well as used to adaptively detect cracks.
insufficient maintenance and inspection deficiencies. Bridges
constitute one of the major civil infrastructure systems in the
U.S. According to the National Bridge Inventory (NBI), more than 1.2. Background
10,400 bridges are categorized as structurally deficient [1]. There
is an urgent need to develop effective approaches for the An automatic crack detection procedure in welds based on mag-
inspection and evaluation of these bridges. In addition, periodical netic particle testing [3] was introduced by Ho et al. in 1990 [4].
inspections and maintenance of bridges will prolong their service This method can only be used on ferromagnetic materials. First, the
life [2]. testing surface is sprayed with white paint to reduce the initial
Visual inspection is the predominant method used for the inspec- noise of subsequently captured images. Next, a magnetic field is ap-
tion of almost all infrastructure systems. It is a subjective process that plied to the weld. Then magnetic ink made of small magnetic particles
relies on an inspector's experience and mental focus, making it highly suspended in oil is sprayed over the testing surface. The change of
prone to human error. The development of automated inspection flux density at the crack causes the magnetic particles to trace out
technology can overcome these shortcomings. the shape of the crack on the weld surface. Lastly, an image of the pre-
In this study, a novel image-based crack detection approach is in- pared surface is captured and cracks are detected by means of the
troduced that includes autoadaptive features that have not been used Sobel edge detection operator [5,6] and by implementing a boundary
in previous crack detection systems. This approach can be used as tracing algorithm. The results were satisfactory as reported by Ho et
al. [4], but clearly this technique has drawbacks since a preprocessing
step is required.
⁎ Corresponding author. Tsao et al. [7] composed image analysis and expert system modu-
E-mail addresses: jahansha@usc.edu (M.R. Jahanshahi), masri@usc.edu (S.F. Masri). lus to detect spalling and transverse cracks in pavements.

0926-5805/$ – see front matter © 2011 Elsevier B.V. All rights reserved.
doi:10.1016/j.autcon.2011.11.018
568 M.R. Jahanshahi, S.F. Masri / Automation in Construction 22 (2012) 567–576

Kaseko et al. [8] and Wang et al. [9] used the image processing and of exposed reinforcement, concrete columns, air pockets, and
neural network techniques to detect defects in pavements. potholes based on their visual attributes. Zhu et al. [34] developed
Siegel and Gunatilake [10] developed a remote visual inspection a system to retrieve the concrete crack properties, such as length,
system of aircraft surfaces. To detect cracks, their proposed algorithm orientation and width, for automated post-earthquake structural
detects rivets, since cracks propagate on rivet edges. Multi-scale edge condition assessment.
detection is used to detect the edges of small defects at small scales
and the edges of large defects at large scales. By tracing edges from 1.3. Contribution
high scale to low scale, it is possible to define the propagation depth
of edges. Using other features based on wavelet transformation In all of the above studies, many important parameters (e.g.,
[11,12] and a trained back-propagation neural network [13] cracks camera-object distance) are not considered or assumed to be
can be classified from other defects such as scratches. Corroded re- constant. In practical circumstances, the image acquisition system
gions can also be detected by defining features based on 2D discrete often cannot maintain a constant focal length, resolution, or distance
wavelet transformation of the captured images and using a neural to the object under inspection. In the case of nuclear power plants,
network classifier [10]. for instance, the image acquisition system needs to be located a
Nieniewski et al. [14] developed a visual system that could detect significant distance from the reactor site. To detect cracks of a specif-
cracks in ferrites. A morphological detector based on top-hat trans- ic thickness, many of the parameters in these algorithms need to be
form [15] detects irregular changes of brightness, which could lead adaptive to the 3D structure of a scene and the attributes of the
to crack detection. k-Nearest Neighbors [13] is used as a classifier to image acquisition system; however, no such study has been reported
classify cracks from grooves. The outcome of this study is very prom- in the open literature. The proposed approach in this study gives a
ising, and this technique is quite robust despite the presence of noise, robotic inspection system the ability to detect cracks in images
unlike other edge detection operators used for crack extraction. captured from any distance to the object, with any focal length or
Moselhi and Shehab-Eldeen [16] used the image analysis tech- resolution. The proposed crack detection system extracts the whole
niques and the neural network to automatically detect and classify crack as opposed to edge-based approaches where just the edges
the defects in sewer pipes. Chae [17] proposed a system consisting are segmented. This characteristic makes the current study appropri-
of image processing techniques along with the neural networks and ate for crack thickness quantification which is under development by
fuzzy logic systems for automatic defect (including cracks) detection the authors.
of sewer pipelines. In human vision, depth perception allows a person to estimate the
Benning et al. [18] used photogrammetry to measure the deforma- size of an object based on the distance to the object. In this study, a
tions of reinforced concrete structures. A grid of circular targets is contact-less crack detection approach based on depth perception is
established on the testing surface. Up to three cameras capture im- introduced to segment crack-like patterns. First, several pictures of a
ages of the surface simultaneously. The relative distances between scene are captured from different views. By solving the Structure
the centers of adjacent targets make it possible to monitor the evolu- from Motion (SfM) problem [35], the sparse structure of a scene as
tion of cracks. well as the camera's position, orientation, and internal parameters
Abdel-Qader et al. [19] analyzed the efficacy of different edge de- for each view are determined. By scaling the reconstructed sparse
tection techniques in identifying cracks in concrete pavements of 3D model of a scene, the depth perception is obtained. Subsequently,
bridges. They concluded that the Fast Harr Transform (FHT), which a morphological crack segmentation operator is introduced. The
is a wavelet transform with mother wavelet of Harr, has the most ac- structuring element parameter for this operator is automatically ad-
curate crack detection capability in contrast with Fast Fourier trans- justed based on the camera focal length, object-camera distance,
form, Sobel, and Canny edge detection operators [20,21]. camera resolution, camera sensor size, and the desired crack thick-
A study on using computer vision techniques for automatic struc- ness. Appropriate features are extracted and selected for each seg-
tural assessment of underground pipes has been done by Sinha et al. mented pattern using the Linear Discriminant Analysis (LDA) [36]
in 2003 [22]. The algorithm proposed by Sinha et al. [22] consists of approach. A trained Neural Network (NN), a Support Vector Machine
image processing, segmentation, feature extraction, pattern recogni- (SVM), and a nearest-neighbor classifier are used to classify real
tion, and a proposed neuro-fuzzy network for classification. cracks. The performance of these classifiers in the problem of interest
Giakoumis et al. [23] detected the cracks in digitized paintings by is discussed. Finally, a multi-scale approach is introduced to obtain a
thresholding the output of the morphological top-hat transform. crack map. The proposed methodology is also effective for other pat-
Sinha and Fieguth [24] detected the defects in underground pipe im- tern analysis purposes (e.g., texture analysis) which are not discussed
ages by thresholding the morphological opening of the pipe images in this paper.
using different structuring elements. Abdel-Qader et al. [25] proposed
algorithms based on Principal Component Analysis (PCA) to extract 1.4. Scope
cracks in concrete bridge decks.
Yu et al. [26] introduced an image-based semi-autonomous ap- Section 2 discusses the proposed adaptive crack detection. The
proach to detect cracks in concrete tunnels. Yamaguchi and Hashimo- components of the proposed system, including interaction of differ-
to [27] proposed a crack detection approach based on a percolation ent image acquisition parameters, the 3D scene reconstruction, seg-
model and edge information. Chen et al. [28] introduced a semi- mentation, feature extraction and pattern classification are
automatic measuring system for concrete cracks using multi- discussed in Sections 2.1, 2.2, 2.3, 2.4 and 2.5, respectively. The con-
temporal images. struction of the multi-scale crackmap is explained in Section 3. Exper-
Fujita and Hamamoto [29] proposed a crack detection method in imental results and discussion are presented in Section 4. Section 5
noisy concrete surfaces using probabilistic relaxation and a locally includes the summary of the paper.
adaptive thresholding. Jahanshahi et al. [30] surveyed and evaluated
several crack detection techniques in conjunction with realistic infra- 2. Adaptive crack detection
structure components.
Guo et al. [31,32] developed an automatic defect detection An adaptive crack detection procedure is proposed in this study.
approach based on interpreting the images or videos that were This system is adaptive because based on the image acquisition
captured for sewer pipeline inspection and condition assessment. specifications, camera-object distance, focal length and image reso-
Brilakis et al. [33] outlined a framework for automatic recognition lution, it automatically adjusts its parameters to detect cracks of
 M.R. Jahanshahi, S.F. Masri / Automation in Construction 22 (2012) 567–576 569

interest. Fig. 1 shows the overview scheme of the proposed system.

The main elements of the proposed crack detection procedure are
data sensing, 3D scene reconstruction, segmentation, feature
extraction, and decision making. Note that before processing any
image, preprocessing approaches can be used to enhance the image
[30].

2.1. Data sensing

Using a simple pinhole camera model, the relation between differ-

ent image acquisition parameters is shown in (1):

WD SS Fig. 2. The geometric relation between image acquisition parameters of a simple pinhole
SF ¼   n; ð1Þ camera model.
FL SR

where SF (mm) is the size of a pattern (e.g., crack thickness) repre- By retrieving the equation of the fitted plane, one can find the intersec-
sented by n pixels in an image, WD (mm) is the working distance tion between the camera orientation line passing through the camera
(camera-object distance), FL (mm) is the camera focal length, SS center and the fitted plane. The distance between this intersection
(mm) is the camera sensor size, and SR (pixels) is the camera sensor point and the camera center is computed as the working distance.
resolution. The geometric relation between some of these parameters Furthermore, the estimated focal lengths from the SfM problem
is shown in Fig. 2. are in pixels. To scale these quantities, the estimated focal length for
The camera sensor size can be obtained from the manufacturer each view is scaled by the ratio of the sensor size to the sensor reso-
and the camera sensor resolution is known from the image size. The lution. This means that Eq. (1) can be simplified to:
measurements for the working distance, and the camera focal length
are needed to quantify an n-pixels feature. These two parameters can WD
be estimated as described below. SF ¼  n; ð2Þ
FL
In this study, the above equation is used to optimize the acquisi-
tion and processing parameters to detect cracks reliably.
where FL is in pixels.Note that if scene reconstruction is impossible
(e.g., not enough views are available), the approximate focal length
2.2. 3D scene reconstruction
can be extracted from the image Exchangeable Image File Format
(EXIF) file. In this case, one can use Eq. (1) to estimate the interaction
To create depth perception, the 3D structure of a scene has to be
of the image acquisition parameters provided that the working dis-
recovered. First, several overlapping images of the object are captured
tance is given.
from different views. The SfM approach aims to optimize a 3D sparse
point cloud and viewing parameters simultaneously from a set of
2.3. Segmentation
geometrically matched keypoints taken from multiple views. Fig. 3
shows the schematic overview of the SfM problem.
Segmentation is a set of steps that isolate the patterns that can be
There are recent studies for generating dense depth map of an in-
potentially classified as a defined defect. The aim of segmentation is
frastructure using 2-D stereo images [37,38]; however, the SfM sys-
to reduce extraneous data about patterns whose classes are not de-
tem developed by Snavely et al. [39], which is not based on stereo
sired to be known. Several segmentation techniques have been eval-
images, is used in this study. In this system, SIFT keypoints [40] are
uated by the authors previously [30], and it has been concluded that a
detected in each image and then matched between all pair of images.
proposed morphological operation by Salembier [15] works best for
The RANSAC algorithm [41] is used to exclude outliers. These matches
crack detection purposes in components that are typically encoun-
are used to recover focal length, camera center and orientation, and
tered in civil infrastructure systems.
radial lens distortion parameters (two parameters corresponding to
a 4 th order radial distortion model [42] are estimated) for each
2.3.1. Morphological operation
view, as well as the 3D structure of a scene. This huge optimization
Morphological image processing, which is based on mathematical
process is called bundle adjustment. Fig. 4 shows an example of the
morphology, is used to extract useful information about the scene ob-
SfM problem and the 3D scene reconstruction from sixteen images.
jects. The foundation of morphological image processing is based on
Since measuring the camera-object distance is not always an easy or
previous studies by Minkowski [43] and Metheron [44] on set algebra
practical task, the reconstructed 3D cloud and camera locations from
and topology, respectively.
the SfM problem are used to estimate the working distance; however,
The proposed morphological operation by Salembier [15] is slightly
the SfM problem estimates the relative 3D point coordinates and cam-
modified here to enhance its capability for crack extraction in different
era locations. By knowing how much the camera center has moved be-
orientations. The proposed operation is shown in Eq. (3):
tween just two of the views, the reconstructed 3D points and camera
locations can be scaled. To obtain the absolute camera-object distance,   
a plane is fitted to the 3Dpoints seen in the view of interest. This can T ¼ max I∘S •S ; I −I; ð3Þ
be done by using the RANSAC algorithm to exclude the outlier points. f0∘ ;45∘ ;90∘ ;135∘ g f0∘ ;45∘ ;90∘ ;135∘ g

Fig. 1. The overview scheme of the proposed crack detection approach.

570 M.R. Jahanshahi, S.F. Masri / Automation in Construction 22 (2012) 567–576

Fig. 3. Schematic overview and components of the SfM problem.

where I is the grayscale image, S is the structuring element that defines nonlinear filter extracts the whole crack as opposed to edge detection
which neighboring pixels are included in the operation, ‘∘’ is the mor- approaches where just the edges are segmented.
phological opening, and ‘•’ is the morphological closing. The output Furthermore, small extracted patterns are eliminated as potential
image T is then binarized using Otsu's thresholding method [45] to seg- noise. For this purpose, if the length of a segmented pattern is less
ment potential crack-like dark regions from the rest of the image. This than a minimum length, which is specified by the user, that pattern

Fig. 4. 3D scene reconstruction: (a) sixteen images of a scene taken from different locations, (b) the 3D reconstructed point cloud and camera poses. Each red cone represents a
camera that corresponds to one of the images in (a).
 M.R. Jahanshahi, S.F. Masri / Automation in Construction 22 (2012) 567–576 571

is eliminated. In order to convert minimum length of interest in unit 0.35, respectively, by trial and error. For post processing, objects less
length to minimum length in pixels, Eq. (4) is used: than 15 mm in length are excluded. It is seen from this figure that
all of these three methods are capable of extracting the dark cracks;
FL however, the bottom-hat and Canny methods have poor performance
lp ¼  l; ð4Þ
WD when they are applied to surfaces with no crack. While several
patterns are falsely extracted as potential cracks using these two
where l is the defined length by the user in unit length, FL and WD methods, Salembier method barely extracts any crack pattern.
(obtained from SfM and scaling, as described in Section 2.2) are in Furthermore, the Canny method is incapable of extracting the whole
pixels and unit length, respectively, and lp is the length in pixels. thickness of a crack which is useful to quantify the crack thicknesses
as a future complement to the current study.
2.3.2. Structuring element
By choosing the size and shape of the structuring element (i.e., 2.4. Feature extraction
neighborhood), a filter that is sensitive to a specific shape can be con-
structed. When the structuring element has a line format, it can seg- After segmenting the patterns of interest, it is time to assign them
ment cracks that are perpendicular to it. If the length of the a set of finite values representing quantitative attributes or properties
structuring element (in pixels) exceeds the thickness of a dark object called features. These features should represent the important charac-
in an image, then this object can be segmented by the operation in Eq. teristics that help identify similar patterns. To determine discrimina-
(3).Consequently, linear structuring elements are defined in 0 ∘, 45 ∘, tive features useful for classification purposes, this study initially
90 ∘, and 135 ∘ orientations. defined and analyzed twenty nine features. Eleven of these features
The challenge is to find the appropriate size for the structuring el- were selected as potentially appropriate features for further analysis.
ement. By having the scaled working distance obtained from Finally, using the LDA [36] approach, the following five features were
Section 2.2, the derived formula in Eq. (1) is used to compute the ap- found to be discriminately appropriate (i.e., preserving 99.4% of the
propriate structuring element. Using this equation, the size of the ap- cumulative feature ranking criteria) for classification: (1) eccentricity
propriate structuring element is computed based on the crack size of (a scalar that specifies the eccentricity of the ellipse that has the same
interest (n is the structuring element size). Fig. 5 shows the relation- second-moments as the segmented object), (2) area of the segment-
ship between these parameters and can be used to determine appro- ed object divided by the area of the above ellipse, (3) solidity (a scalar
priate image acquisition system parameters. specifying the proportion of pixels in the convex hull that also belong
to the segmented object), (4) absolute value of the correlation coeffi-
2.3.3. Comparison of segmentation approaches cient (here, correlation is defined as the relationship between the
Salembier has compared several morphological approaches for horizontal and vertical pixel coordinates), and (5) compactness (the
crack extraction [15]. Jahanshahi et al. evaluated several morpholog- ratio between the square root of the extracted area and its perime-
ical and edge-based crack detection approaches [30]. For completion, ter). The convex hull for a segmented object is defined as the smallest
the segmentation approach used in this study is compared with the convex polygon that can contain the object. The above features are
bottom-hat morphological operation and Canny edge detection ap- computed for each segmented pattern.
proaches. Fig. 6 shows the result of applying these methods on con- Note that LDA approach was used.
crete surfaces. In this figure, column (a) shows the original concrete
surfaces. Columns (b), (c) and (d) show the binary images obtained 2.5. Classification
from processing the original images by the bottom-hat morphological
operation, Canny edge detection and Salembier operators, respective- In this study, a feature set consisting of 1910 non-crack feature
ly. The black pixels represent potential crack patterns. The structuring vectors and 3961 synthetic crack feature vectors was generated to
element used for the bottom-hat and Salembier operator is computed train and evaluate the classifiers. About 60% of this set was used for
as described above. Otsu's thresholding method is used to binarize training, while the remaining feature vectors were used for validation
the images processed by the morphological operations. For Canny and testing. Note that due to the lack of access to a large number of
method, the low- and high-threshold values are set to be 0.14 and real cracks, randomized synthetic cracks were generated to augment
the training database. For this reason, real cracks were manually seg-
mented and an algorithm was developed to randomly generate cracks
Sensor Resolution / Sensor Size (pixel/mm)
from them. The non-crack feature vectors were extracted from actual
Sructuring Element Size / Crack Size (pixel/mm)

30 scenes and 382 non-crack feature vectors are used. The latter testing
set includes some features belonging to non-crack objects that are
very similar to cracks including some edges. This testing set helps in
25
evaluating the classifier for a real application rather than evaluating
the filtered data. The performance of several SVM and NN classifiers
20 was evaluated. Eventually, a SVM with a 3 rd order polynomial kernel
and a 3-layer feedforward NN with 10 neurons in the hidden layer
15 and 2 output neurons were used for classification. A nearest-
70 00

neighbor classifier was used to evaluate the performance of the

0
6 0
50 0

10 above classifiers.
40 0

Table 1 summarizes the performances of these three classifiers. In

30

20
10
0 this table, ‘accuracy’ is the proportion of true classifications in the test
5 0 set, ‘precision’ is the proportion of true positive classifications against
all positive classifications, ‘sensitivity’ is the proportion of actual pos-
0
0 15 30 45 60 75 90 105 120 135 150 itives that were correctly classified, and ‘specificity’ is the proportion
of negatives that were correctly classified. Since the latter two quan-
Working Distance / Focal Length (mm/mm)
tities are insensitive to changes in the class distribution, they were
Fig. 5. Relationship between structuring element size, camera focal length, working used to evaluate the classifier performances in this study. This table
distance, crack size, camera sensor size, and camera sensor resolution. shows that the proposed SVM and NN approaches have very close
572 M.R. Jahanshahi, S.F. Masri / Automation in Construction 22 (2012) 567–576

Fig. 6. Comparison of different crack segmentation approaches: column (a) shows the original concrete images, columns (b), (c) and (d) are the binarized images obtained from
processing the original images by the bottom-hat, Canny and Salembier methods, respectively.

Table 1
performances. Their performance is better than a nearest-neighbor
The performance of different classifiers on synthetic data.
classifier. Note that the SVM method is a discrete classifier, whereas
Classifier Accuracy Precision Sensitivity Specificity the proposed NN approach needs a threshold to act as a discrete clas-
(%) (%) (%) (%)
sifier. In this study, if the value of the crack output neuron was found
Neural network 95.57 95.91 97.60 91.36 to be greater than the value of the non-crack neuron, the pattern was
Support vector machine 95.06 94.98 97.85 89.27 classified as a crack, otherwise, as a non-crack. This is similar to set
Nearest-neighbor 88.76 91.15 92.30 81.41
the threshold equal to 0.5.
 M.R. Jahanshahi, S.F. Masri / Automation in Construction 22 (2012) 567–576 573

Fig. 7 shows the effect of changing the decision making threshold Table 2
on different performance indices for the specific NN used in this The overall performance of the proposed system using real data.

study. In this figure, ‘positive predictive value’ is the proportion of Classifier Accuracy Precision Sensitivity Specificity
the correctly classified positives (i.e., cracks), and ‘negative predictive (%) (%) (%) (%)
value’ is the proportion of the correctly classified negatives (i.e., non- Neural network 79.5 78.4 84.1 74.5
cracks). For applications where it is expensive to miss a crack (e.g., in- Support vector machine 78.3 76.8 84.1 72.0
spection purposes), it is recommended to select a more conservative
threshold (i.e., a threshold less than 0.5). As a threshold moves to-
ward one, specificity and positive predictive rates increase while sen-
sitivity and negative predictive rates decrease. This means there will Algorithm: Adaptive crack detection
be more false negatives and less false positives. For less sensitive ap-
Input: n images of a scene and the camera distance between
plications, one may select a threshold greater than 0.5. Moreover,
two of the views
Fig. 7 helps decide about the appropriate threshold for a specific ap-
For each view:
plication by considering the performance indices. It is worth noting
that if the training set size is infinite, the outputs of the above back 1) Establish the working distance and camera parameters by
propagation NN can converge to the true a posteriori probabilities solving the SfM problem and scaling the reconstructed scene
[13]. 2) Establish the appropriate structuring element based on the
working distance and the focal length of the view, as well,
as the crack thickness of interest
3. Multi-scale crack map 3) Segment the potential crack patterns by applying the de-
scribed morphological operation in (3) on the image
In order to obtain a crack map, the crack detection procedure de- 4) Compute and assign appropriate features to each segmented
scribed above was repeated using different structuring elements pattern
(i.e., different scales). Note that the extracted multi-scale binary 5) Classify cracks from non-crack patterns using a trained clas-
crack map is the union of the detected cracks using different structur- sifier (NN or SVM)
ing elements. The proposed crack map can be formulated as: 6) Repeat steps 2 through 5 for different crack thicknesses of
interest and generate the multi-scale crack map as the
 union of all extracted crack pixels
m
J ðu; vÞ ¼ 1 ∃k ∈ ½Smin ; m; C k ðu; vÞ ¼ 1; ð5Þ
0 Otherwise; Output: the multi-scale crack map

where J m is the crack map at scale (i.e., structuring element) m, Smin is 4. Experimental results and discussion
the minimum structuring element size, C k is the binary crack image
obtained by using k as the structuring element, and u and v are the In order to evaluate the overall performance of the proposed crack
pixel coordinates of the crack map image. detection algorithm, a test set consisted of 220 real concrete crack and
In this study, the structuring elements of ⌈nmin⌉ + 2 to ⌈nmax⌉ + 10 200 non-crack images was used. Table 2 summarizes the performance
were used for generating the crack map where ⌈⌉ is the ceiling func- of the detection system for real patterns. The performance of the system
tion, and nmin and nmax are the structuring element sizes correspond- based on NN is slightly better than the one based on SVM. So, the former
ing to the minimum and maximum crack size of interest, respectively. system is used for the rest of the experiments in this study. The mini-
The crack map can be used for crack localization as well as mum length of the detected cracks was set to 10 mm.
quantification. Fig. 8 shows detected cracks using the proposed crack detection
approach. In this example, a single image was captured from a con-
crete structure. No scene reconstruction took place. The working dis-
tance was manually measured as 20 m. A Canon EOS 7D along with an
100 EF 600 mm f/4 L IS USM super telephoto lens (i.e., 600 mm focal

90
Performance index (%)

80

70

60
Correct rate
Positive predictive value
50 Negative predictive value
Sensitivity
Specificity
40
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Fig. 8. Detected cracks from a far distance: the detected cracks are shown in red. Each
Classifier threshold black box illustrates the boundaries of a correctly detected crack. False negative alarms
are surrounded by dashed lines. A Canon EOS 7D along with an EF 600 mm f/4 L IS USM
Fig. 7. Effect of decision making threshold on different performance indices for the pro- super telephoto lens were used to capture this image. The minimum thickness of the
posed NN. detected cracks in this figure was 0.1 mm and the working distance was 20,000 mm.
574 M.R. Jahanshahi, S.F. Masri / Automation in Construction 22 (2012) 567–576

Fig. 9. Concrete crack: (a), (b), (c), (d), and (e) are crack images on a concrete surface taken from different angles. (f) is the sparse 3D scene reconstruction and recovery of the
camera poses.

length) were used to capture the image. The image resolution was tiny cracks from a far distance with the presence of several edges,
5184 × 3456 pixels. The minimum thickness of the detected cracks rusted regions, and background objects.
in this figure was 0.1 mm. The image acquisition system was selected In order to illustrate the capabilities, as well as the limitations, of the
based on the guidelines introduced in Section 2.1. The minimum proposed system, a real crack detection experiment was performed as
length of the detected cracks was set to 15 mm. This figure includes follows. Five images were taken from a crack on a concrete surface. The
some background objects, as well as some annexed structural sys- image acquisition system was identical to the one that was used in the
tems, which make it hard to identify cracks; however, the perfor- first experiment. These images are shown in Fig. 9(a), (b), (c), (d), and
mance of the proposed system is remarkable where almost all (e). Fig. 9(f) shows the reconstructed scene and recovered camera
cracks were detected and few false negative alarms took place. Note poses. The camera distance between the two side views (i.e., Fig. 9(a)
that there are several edges and rusted regions in this figure which and (e)) was 1600 mm.
correctly have not been detected as cracks. This example shows the Here, Fig. 9(c) is an example used to detect cracks. The retrieved
capability of the proposed crack detection approach for detecting working distance and focal length for this view were 966 mm and

Fig. 10. Crack detection: from left to right, the images in each column correspond to the structural element sizes 9, 11, 15, and 22 pixels, respectively. (a), (b), (c), and (d) are the
extracted patterns in Fig. 9 (c) using (3). (e), (f), (g), and (h) are the binarized images using Otsu's threshold. (i), (j), (k), and (l) are the multi-scale crack maps.
 M.R. Jahanshahi, S.F. Masri / Automation in Construction 22 (2012) 567–576 575

5. Summary and future work

Current visual inspection of civil structures, which is the predom-

inant inspection method, is highly qualitative. An inspector has to vi-
sually assess the condition of a structure. If a region is inaccessible, an
inspector uses binoculars to detect and characterize defects. There is
an urgent need for developing autonomous quantitative approaches
in this field. In this study, a novel adaptive crack detection procedure
is introduced. First, images of a scene are captured from different
views. By solving the SfM problem, the sparse structure of the scene
as well as the camera position, orientation, and internal parameters
for each view are determined. By scaling the reconstructed sparse
3D model of the scene, the depth perception of the scene is obtained.
A morphological crack segmentation operator is introduced to extract
crack-like patterns. The structuring element parameter for this oper-
ator is automatically adjusted based on the camera focal length,
object-camera distance, camera resolution, camera sensor size, and
Fig. 11. Detected crack: the detected cracks are shown in red. Each blue box illustrates
the desired crack thickness. Appropriate features are extracted and
the boundaries of a detected crack.
selected for each segmented pattern using the LDA approach. The
performances of a NN, a SVM, and a nearest-neighbor classifier are
evaluated to classify cracks from non-crack patterns. A multi-scale
15,759 pixels, respectively. The working distance varied from crack map is obtained to represent the detected cracks. The authors
800 mm to 1400 mm. The minimum and maximum crack thicknesses are developing an autonomous crack quantification approach based
of interest were 0.4 mm and 1.0 mm, respectively. The corresponding on the obtained crack map from this approach.
minimum and maximum structuring element sizes were 7 and Validation tests were performed to evaluate the capabilities, as
17 pixels. The minimum length of the detected cracks was set to well as the limitations, of the methodology discussed in this paper.
15 mm. An example of real concrete cracks was also presented to illustrate
Fig. 10(a), (b), (c), and (d) are the extracted patterns using Eq. (3). the performance of the proposed system in the presence of noise.
The corresponding structural element sizes were 9, 11, 15, and 22 This system is appropriate for incorporation with autonomous or
pixels, respectively. Fig. 10(e), (f), (g), and (h) show the correspond- semi-autonomous robotic systems. Since this system extracts the en-
ing binarized images using the Otsu's thresholding method [45]. tire crack, as opposed to edge-base techniques where just the crack
These images have less noise in comparison with their corresponding edges are segmented, the obtained crack map from this study can
original ones (Fig. 10(a), (b), (c), and (d)). Fig. 10(i), (j), (k), and (l) be used for crack thickness quantification.
show the evolution of the multi-scale crack map at different scales As part of future work, there is a need to develop systems that can
(i.e., the structural element sizes of 9, 11, 15, and 22 pixels). To obtain quantify, locate, and classify different crack types and manage the
the crack maps, patterns less than 15 mm in length were removed au- huge amount of data obtained by such autonomous systems. Robust
tonomously. To do this, corresponding segmented patterns with less decision making tools need to be developed to allow fast analysis of
than 244 pixels in a thinned binarized images were removed from the risk of every crack to fill the gap between detecting cracks and
the original binarized image. Then, the classification approach helping asset managers to take the appropriate action accordingly.
explained in Section 2 was used to get rid of the rest of non-crack pat- This study is the first step towards such a system.
terns. A NN was used as the classifier.
Detected cracks are shown in Fig. 11 in red. Each blue box illus- Acknowledgment
trates the boundary of a detected crack. As can be seen, a tiny portion
of the crack, where it splits, is not detected. In fact, this pattern was This study was supported in part by grants from the U.S. National
initially extracted, but later removed as its length was less than Science Foundation. The authors would like to thank Dr. Zahra Tehrani
15 mm. Moreover, there are several grooves in this image, but only for her critical reading of the manuscript.
one was detected as a crack (near the top crack). This example
shows the robustness of the proposed crack detection system in the
References
presence of noise and crack-like patterns.
On an AMD Athlon II X4 (2.6 GHz) processor, it took 206 s to open [1] K.P. Chong, N.J. Carino, G. Washer, Health monitoring of civil infrastructures,
the five overlapping images, extract the keypoints, match them, and Smart Materials and Structures 12 (3) (2003) 483–493.
[2] A. McCrea, D. Chamberlain, R. Navon, Automated inspection and restoration of
solve the SfM problem. More than 85% of this time was dedicated to steel bridges — a critical review of methods and enabling technologies, Automa-
extracting the keypoints. These components were all implemented tion in Construction 11 (4) (2002) 351–373.
in C and C++. No parallel processing was used. For more details re- [3] J.M. Coffey, Non-destructive testing—the technology of measuring defects, CEGB
Research 21 (1988) 36–47.
garding the computational complexity of the 3D scene reconstruction [4] S.K. Ho, R.M. White, J. Lucas, A vision system for automated crack detection in
algorithm, the reader is referred to [35]. Failure modes of the SfM welds, Measurement Science and Technology 1 (3) (1990) 287–294.
problem include insufficient overlap or texture, ambiguous or repeat- [5] R.C. Gonzalez, R.E. Woods, S.L. Eddins, Digital Image Processing Using MATLAB,
Prentice Hall, Upper Saddle River, NJ0-130-08519-7, 2004.
ing textures, bad initialization, and cascading errors (i.e., error propa-
[6] R.O. Duda, P.E. Hart, Pattern Classification and Scene Analysis, John Wiley & Sons,
gation due to the algorithm's mistake in the placement of a camera) New York0-471-22361-1, 1973.
[35]. On the same machine, it took 134 s to process one of the images [7] S. Tsao, N. Kehtarnavaz, P. Chan, R. Lytton, Image-based expert-system approach
to distress detection on CRC pavement, Journal of Transportation Engineering 120
for detecting cracks. The processing time highly depends on the max-
(1) (1994) 62–64.
imum and minimum crack sizes of interest, and the resolution and the [8] M.S. Kaseko, Z.-P. Lo, S.G. Ritchie, Comparison of traditional and neural classifiers
texture of the image under inspection. The crack detection algorithm for pavement-crack detection, Journal of Transportation Engineering 120 (4)
used in this study was implemented in MATLAB. The implementation (1994) 552–569.
[9] K.C. Wang, S. Nallamothu, R.P. Elliott, Classification of pavement surface distress
of these algorithms in C or C++ will highly enhance the computation- with an embedded neural net chip, Artificial Neural Networks for Civil Engineers:
al efficiency. Advanced Features and Applications, ASCE, 1998, pp. 131–161.
576 M.R. Jahanshahi, S.F. Masri / Automation in Construction 22 (2012) 567–576

[10] M. Siegel, P. Gunatilake, Remote enhanced visual inspection of aircraft by a mo- [29] Y. Fujita, Y. Hamamoto, A robust method for automatically detecting cracks on
bile robot, IEEE Workshop on Emerging Technologies, Intelligent Measurement noisy concrete surfaces, Next-Generation Applied Intelligence, Twenty-second
and Virtual Systems for Instrumentation and Measurement, ETIMVIS'98, st. International Conference on Industrial, Engineering and Other Applications of Ap-
Paul, MN, USA, May 1998. plied Intelligent Systems IEA/AIE 2009, 2009, pp. 76–85, Tainan, Taiwan.
[11] L. Prasad, S. Iyenger, Wavelet Analysis with Applications to Image Procesing, CRC [30] M.R. Jahanshahi, J.S. Kelly, S.F. Masri, G.S. Sukhatme, A survey and evaluation of
Press LLC, Boca Raton, FL0-849-33169-2, 1997. promising approaches for automatic image-based defect detection of bridge
[12] J.-P. Abtoine, R. Murenzi, P. Vandergheynst, S.T. Ali, Two-Dimensional Wavelets structures, Structure and Infrastructure Engineering 5 (6) (2009) 455–486, doi:
and Their Relatives, Cambridge University Press, United Kingdom0-521-62406- 10.1080/15732470801945930.
1, 2004. [31] W. Guo, L. Soibelman, J.H. Garrett JR, Automated defect detection for sewer pipe-
[13] R.O. Duda, P.E. Hart, D.G. Stork, Pattern Classification, 2nd Edition Wiley, New line inspection and condition assessment, Automation in Construction 18 (5)
York, 2001. (2009) 587–596. http://dx.doi.org/10.1016/j.autcon.2008.12.003.
[14] M. Nieniewski, L. Chmielewski, Sklodowski Jozwik, Morphological detection and [32] W. Guo, L. Soibelman, J.H. Garrett JR, Visual pattern recognition supporting defect
feature-based classification of cracked regions in ferrites, Machine GRAPHICS reporting and condition assessment of wastewater collection systems, Journal of
and VISION 8 (4) (1999) 699–712. Computing in Civil Engineering 23 (3) (2009) 160–169.
[15] P. Salembier, Comparison of some morphological segmentation algorithms based [33] I. Brilakis, S. German, Z. Zhu, Visual pattern recognition models for remote sens-
on contrast enhancement, application to automatic defect detection, Proceedings ing of civil infrastructure, Journal of Computing in Civil Engineering 25 (5).
of the EUSIPCO-90 — Fifth European Signal Processing Conference, 1990, [34] Z. Zhu, S. German, I. Brilakis, http://www.sciencedirect.com/science/article/-
pp. 833–836. pii/S0926580511000318Vi sual retrieval of concrete crack properties for auto-
[16] O. Moselhi, T. Shehab-Eldeen, Classification of defects in sewer pipes using neural mated post-earthquake structural safety evaluation, Automation in Construction
networks, Journal of Infrastructure Systems 6 (3) (2000) 97–104. 20 (7) (2011) 874–883. http://dx.doi.org/10.1016/j.autcon.2011.03.004 doi:
[17] M. J. Chae, Automated interpretation and assessment of sewer pipeline, Ph.D. the- 10.1016/j.autcon.2011.03.004. URL http://www.sciencedirect.com/science/arti-
sis, Purdue University (December 2001). cle/pii/S09265 80511000318
[18] W. Benning, S. Görtz, J. Lange, R. Schwermann, R. Chudoba, Development of an al- [35] K. N. Snavely, Scene reconstruction and visualization from internet photo collec-
gorithm for automatic analysis of deformation of reinforced concrete structures tions, Ph.D. thesis, University of Washington, Seattle, Washington, USA (2008).
using photogrammetry, VDI Berichte 1757 (2003) 411–418. [36] R.A. Fisher, The use of multiple measurements in taxonomic problems, Annals of
[19] I. Abdel-Qader, O. Abudayyeh, M.E. Kelly, Analysis of edge-detection techniques Eugenics 7 (1936) 179–188.
for crack identification in bridges, Journal of Computing in Civil Engineering 17 [37] I. Brilakis, H. Fathi, A. Rashidi, http://www.sciencedirect.com/science/article/pii/
(4) (2003) 255–263. S092658051100032X Progressive 3d reconstruction of infrastructure with video-
[20] K. Alageel, I. Abdel-Qader, Harr Transform Use in Image Processing, Tech. rep, De- grammetry, Automation in Construction 20 (7) (2011) 884–895. http://dx.doi.
partment of Electrical and Computer Engineering, Western Michigan University, org/10.1016/j.autcon.2011.03.005 doi:10.1016/j.autcon.2011.03.005.URL http://
Kalamazoo, Michigan, 2002. www.sciencedirect.com/science/article/pii/S09265 8051100032X
[21] G. Bachmann, L. Narici, E. Beckenstein, Fourier and Wavelet Analysis, Springer, [38] A. Rashidi, H. Fathi, I. Brilakis, Innovative stereo vision-based approach to gener-
New York, NY, 2000. ate dense depth map of transportation infrastructure, transportation research re-
[22] S.K. Sinha, P.W. Fieguth, M.A. Polak, Computer vision techniques for automatic cord, Journal of the Transportation Research Board 2215 (2011) 93–99.
structural assessment of underground pipes, Computer-Aided Civil and Infra- [39] N. Snavely, S.M. Seitz, R. Szeliski, Photo tourism: exploring photo collections in
structure Engineering 18 (2) (2003) 95–112. 3d, SIGGRAPH Conference Proceedings, ACM Press, New York, NY, USA, 2006,
[23] I. Giakoumis, N. Nikolaidis, I. Pitas, Digital image processing techniques for the de- pp. 835–846.
tection and removal of cracks in digitized paintings, IEEE Transactions on Image [40] D.G. Lowe, Distinctive image features from scale-invariant keypoints, Internation-
Processing 15 (1) (2006) 178–188. al Journal of Computer Vision 60 (2) (2004) 91–110.
[24] S.K. Sinha, P.W. Fieguth, Morphological segmentation and classification of under- [41] M. A. Fischler, R. C. Bolles, Random sample consensus: a paradigm for model fit-
ground pipe images, Machine Vision and Applications 17 (1) (2006) 21–31. ting with applications to image analysis and automated cartography, Commun.
[25] I. Abdel-Qader, S. Pashaie-Rad, O. Abudayyeh, S. Yehia, PCA-based algorithm for ACM 24 (6) (1981) 381–395. http://dx.doi.org/http://doi.acm.org/10.1145/
unsupervised bridge crack detection, Advances in Engineering Software 37 (12) 358669.358692 doi:http://doi.acm.org/10.1145/358669.358692.
(2006) 771–778. [42] J.Y. Bouguet, Camera calibration toolbox for matlab, 2008. http://www.vision.
[26] S.-N. Yu, J.-H. Jang, C.-S. Han, Auto inspection system using a mobile robot for caltech.edu/bouguetj/calibdoc/index.html.
detecting concrete cracks in a tunnel, Automation in Construction 16 (3) [43] H. Minkowski, Volumen und oberfläche, Mathematische Annalen 57 (4) (1903)
(2007) 255–261. 447–495.
[27] T. Yamaguchi, S. Hashimoto, Automated crack detection for concrete surface [44] G. Matheron, Random Sets and Integral Geometry, Wiley, New York, 1975.
image using percolation model and edge information, 32nd Annual Conference [45] N. Otsu, A threshold selection method from gray-level histogrmas, IEEE Transac-
on IEEE Industrial Electronics, 2006, pp. 3355–3360. tions on Systems, Man, and Cybernetics 9 (1) (1979) 62–66.
[28] L.-C. Chen, Y.-C. Shao, H.-H. Jan, C.-W. Huang, Y.-M. Tien, Measuring system for
cracks in concrete using multitemporal images, Journal of Surveying Engineering
132 (2) (2006) 77–82.