applied

sciences
Article
A Generic Automated Surface Defect Detection Based
on a Bilinear Model
Fei Zhou, Guihua Liu *, Feng Xu and Hao Deng
School of Information Engineering, Southwest University of Science and Technology, Mianyang 621010, China
* Correspondence: liughua_swit@163.com




Received: 4 June 2019; Accepted: 31 July 2019; Published: 3 August 2019


Abstract: Aiming at the problems of complex texture, variable interference factors and large sample
acquisition in surface defect detection, a generic method of automated surface defect detection based
on a bilinear model was proposed. To realize the automatic classification and localization of surface
defects, a new Double-Visual Geometry Group16 (D-VGG16) is firstly designed as feature functions of
the bilinear model. The global and local features fully extracted from the bilinear model by D-VGG16
are output to the soft-max function to realize the automatic classification of surface defects. Then the
heat map of the original image is obtained by applying Gradient-weighted Class Activation Mapping
(Grad-CAM) to the output features of D-VGG16. Finally, the defects in the original input image
can be located automatically after processing the heat map with a threshold segmentation method.
The training process of the proposed method is characterized by a small sample, end-to-end, and is
weakly-supervised. Furthermore, experiments are performed on two public and two industrial
datasets, which have different defective features in texture, shape and color. The results show that
the proposed method can simultaneously realize the classification and localization of defects with
different defective features. The average precision of the proposed method is above 99% on the four
datasets, and is higher than the known latest algorithms.

Keywords: automated surface inspection; D-VGG16; bilinear model; Grad-CAM;

classification; localization

1. Introduction
Surface defect detection is an important part of industrial production, and has significant impact
upon the quality of industrial products on the market. The traditional manual detection method is
time-consuming, and its detection accuracy is easily affected by the subjectivity, energy and experience
of the inspector. To overcome the shortcomings of manual inspection, automatic surface defect
detection based on machine vision comes into being.
With the rapid development of computer technology, machine vision has been widely applied
in industrial production, especially for defect detection in industrial products. Over the last decade,
a large number of surface defect detection algorithms have emerged. These algorithms can be
roughly classified into three categories: Traditional methods based on image structure features,
methods combining statistical features with machine learning, and deep learning methods based on
the Convolutional Neural Network (CNN). The traditional defect detection algorithm based upon
image structure features mainly detects the surface defects by analyzing the texture, skeleton, edge
and spectrum of the image. Shafarenko et al. [1] proposed a color similarity measurement for an
automatic detection and segmentation of random texture surface defects, which was realized by using
watershed transform for color images of random textures, and extracting the color and texture features
of the images.

Appl. Sci. 2019, 9, 3159; doi:10.3390/app9153159 www.mdpi.com/journal/applsci

Appl. Sci. 2019, 9, 3159 2 of 17

Ojala et al. [2] utilized histogram analysis to threshold the texture image and then map it into a
special data structure of the skeleton representation, achieving the extraction of texture image defects.
Wen et al. [3] used the image edge intensity and the distribution of the gray values of pixels in the
edge domain to model the surface defects. Zhou et al. [4] realized the defect of the metal surface by
wavelet analysis. Although the detection and segmentation of defects can be realized by analyzing the
structural features of the surface of the object, the parameters of the algorithm need to be set manually
for most of these methods, making them easily affected by interference factors, such as illumination in
the environment, thereby affecting the detection effect.
The methods of combining statistical features with machine learning are mainly to extract statistical
features from the defect surface, and then use machine learning algorithms to learn these features
in order to realize surface defect detection. Ghorai et al. [5] used a combination of discrete wavelet
transforms and a Support Vector Machine (SVM) to detect surface defects in steel. Xiao et al. [6]
realized the detection of the surface defects of steel strips by constructing a series of SVMs with a
random subspace of the features, and an evolutionary separator with a Bayesian kernel to train the
results from the sub-SVM to form an integrated classifier. The combination of statistical features and
machine learning can obtain higher accuracy and robustness than traditional structure-based methods.
However, in image feature modeling, the accuracy of detection may be altered due to the different
selections of feature types, and is closely linked to the extracted features, so it is necessary to find a
suitable feature descriptor for a specific detection object.
Recently, because of the rapid development of deep learning, especially in terms of its strong
feature extraction ability, it has been widely used in image-related tasks, such as graphic analysis [7],
semantic segmentation [8] and target tracking [9]. Many researchers have also applied deep learning
to surface defect detection. Lin et al. [10] proposed a CNN-based LEDNet network for light-emitting
diode (LED) defect detection, and used Class Activation Mapping (CAM) [11] to achieve an automatic
location of defects. Tao et al. [12] used a novel cascade auto-encoder to segment and locate metal
surface defects automatically. Di et al. [13] used a combination of the Convolutional Auto Encoder
(CAE) and Semi-supervised Generative Adversarial Networks (SGAN) to detect surface defects in
steel, where CAE was used to extract the fine-grained features of the steel surface, and SGAN was used
to further improve the generalization ability of the network. The authors tested the steel defect dataset
to verify the effectiveness of the proposed method. Compared with the traditional methods based on
the image structure and statistical features, combined with machine learning, the advantage of using
CNN-based deep learning for surface defect detection is that CNN can simultaneously realize the
automatic extraction and recognition of features in a network, and get rid of the trouble of manually
extracting features.
Defect localization can make the observer find and understand the location of surface defects
more intuitively. In essence, defect localization belongs to the category of object detection. Therefore,
some researchers regarded surface defect detection as the problem of defect detection. Lin et al. [14]
used a Faster-Region Convolutional Neural Network (Faster-RCNN) [15] and a Single Shot MultiBox
Detector (SSD) [16] object detection algorithm to detect steel surface defects, and achieved a higher
accuracy and recall rate. Cha et al. [17] proposed a defect detection method based upon Faster-RCNN,
and verified the effectiveness of the proposed defect detection method on concrete cracks, steel corrosion,
bolt corrosion and steel delamination. The advantage of using an object detection algorithm to detect
and locate surface defects, is that it can directly draw lessons from the successful and excellent
algorithms in object detection tasks, but these algorithms require a large number of pixel-level labeled
training samples, which is difficult to achieve in actual industrial production.
Aiming at the problem of sample labeling difficulty for defect detection in actual industrial
production, Lin et al. [10] and Ren et al. [18] used Class Activation Mapping (CAM), which is a
class-discriminative localization technique that generates visual explanations from the CNN-based
network to automatically locate surface defects. The CAM replaced the last full connection layer of the
Appl. Sci. 2019, 9, 3159 3 of 17

CNN network with Global Average Pooling (GAP) [19] to calculate the spatial average of each feature
mapping in the last convolution layer, serving as input features to the fully-connected layer.
In this way, the importance of the image region can be recognized by projecting the weights of the
output layer back to the convolutional feature map. However, the network with CAM needs to change
the original design structure of the network, resulting in the need to retrain the network, therefore its
usage scenarios are limited. To overcome the shortcomings of CAM, Selvaraju et al. [20] proposed
Gradient-based Class Activation Mapping (Grad-CAM), but calculated the weights by using the global
average of the gradient, which is the generalization of CAM, and is suitable for any CNN-based
network without modifying any architecture of the network or re-training.
Therefore, to solve the problems above, a generic method of automated surface defect detection
based upon a bilinear model is presented in this paper. Firstly, the Double-VGG16 (D-VGG16) that
consists of two completely symmetric sub-networks based on VGG16 [21] is proposed as the feature
extraction network of the bilinear model [22]. The output of the bilinear model uses the soft-max
function to predict the corresponding type of the input image, which is realized as the automatic
detection of surface defects. Then the heat map of the original image is obtained by applying Grad-CAM
to one of the output features of D-VGG16. Finally, the defects in the original input image can be located
automatically after processing the heat map with a threshold segmentation method. For the problem of
insufficient training samples in actual industrial production, the D-VGG16 is initialized by loading the
VGG16 pre-training weights on ImageNet [23] with 1000 classes, and adopt the transfer learning [24]
to train the whole network, attaining the target of small samples training. The training of the entire
network only uses image-level annotation, and is carried out in an end-to-end manner. The main
contributions of this paper are as follows:

(1) The bilinear model for the detect detection tasks was proposed. To the best of our knowledge,
this is the first paper that uses the bilinear model for surface defect detection. Moreover,
the proposed method has a generalization capability, and can be successfully applied to defective
features with texture, shape and color.
(2) A D-VGG16 network based upon VGG16 for the feature function of the bilinear model was
designed. The Experimental results show that such a network structure for defects detection
applications has a higher average precision than that network using VGG16 as the feature function,
and is also higher than the known latest methods.
(3) The training process of the whole network proposed in this paper has the characteristics of a small
sample, end-to-end, and is weakly-supervised. In the training stage, only a few training images
of image-level labeled are needed to locate the defects of input images in the prediction stage.

The rest of the paper is organized as follows: Section 2 describes in great detail the specific
method of the paper, mainly about describing the overall structure of the proposed method. Section 3
presents the details and the results of performing experiments on the datasets, which is followed by
the conclusions drawn in Section 4.

2. Methodology
There are two phases in the proposed method. The first phase is the automatic classification of
defects, during which the features of the original input image are firstly handled by the bilinear model
consisting of two fully symmetrical Double-Visual Geometry Group16 (D-VGG16) networks, and then
the extracted features are sent to the soft-max function to achieve the automatic classification of these
defects. The second phase is the automatic localization of the defects, during which Gradient-weighted
Class Activation Mapping (Grad-CAM) is used to get the heat map of the original input image,
and then the corresponding defects are located by employed threshold segmentation to the heat map.
The overall structure of the automated surface defect detection, based on the bilinear model proposed
in this paper, can be demonstrated in Figure 1. The whole network is a typical bilinear model structure.
Appl. Sci. 2019, 9, 3159 4 of 17
Appl. Sci. 2019, 9, x FOR PEER REVIEW 4 of 17
Appl. Sci. 2019, 9, x FOR PEER REVIEW 4 of 17

Figure 1.
Figure1.
Figure Network
1. Network overallstructure.
Network overall
overall structure.

2.1.2.1.
Defect
DefectClassification
Classification
2.1. Defect Classification
The
Thewhole
The whole
whole process
process
process ofofdefect classification
of defect
defect is as
classification
classification is follows:
is as follows:
as TwoTwo
follows: features
Two that that
features
features function fromfrom
that function
function D-VGG16
from D-
D-
are concatenated
VGG16 are to get
concatenated the
to bilinear
get the vector,
bilinear which
vector, is
whichfed isinto
fed the
into soft-max
the soft-max
VGG16 are concatenated to get the bilinear vector, which is fed into the soft-max function to obtain function
function to obtain
to obtainthe
probability
the of
probabilitytheofcorresponding
the corresponding defects in
defects the
in theinput
input image
image and realize
realize
the probability of the corresponding defects in the input image and realize the defect classification. the
the defect
defect classification.
classification.
TheThe
The whole
whole
whole process
processisisisaaatypical
process typicalbilinear
typical model
bilinear model
bilinear structure,
model structure,
structure, and andits
and itscore
its coreis
core isisD-VGG16
D-VGG16that
D-VGG16 that
that isisis used
used
used asas
as aa a
feature
feature
feature function.
function.
function.

2.1.1.
2.1.1.D-VGG16
2.1.1. D-VGG16
D-VGG16
Feature
Feature
Feature function,
function,as
function, asaaafunction
as functionextraction
function extraction network
extraction network
network of ofaaabilinear
of bilinearmodel,
bilinear model,plays
model, playsan
plays anan important
important
important role
role
role
whatever
whatever for for locating
locating and and classifying
classifying in thein the
whole whole
network. network.
In this In this
paper
whatever for locating and classifying in the whole network. In this paper we used two fullywe paper
used we
two used
fully two fully
symmetrical
symmetrical
D-VGG16
symmetrical D-VGG16
thatD-VGG16
were based that
that onwere basedas
VGG16,
were based onaVGG16,
on VGG16,
feature as as aa feature
featurenetwork
extraction extractionofnetwork
extraction of
ofamodel
a bilinear
network abilinear model
were
bilinear used,
model
were
where used, where
the structure
were used, where the the structure
of the network
structure of the network
is shown
of the network is shown
in isFigure
shown in Figure
2.in Figure 2. 2.
64
64

448 128
448 128
224 256
224 256 512
112 56 512 28 512 28 256
112 56 28 512
28 28
28 256
56 1×1
224 112 28 28
56 3×3 3×3 3×3
224 112 1×1
448 3×3 3×3 3×3 3×3
448
448 3×3 3×3
448
3×3

3×3 28 512
448 28
3×3 64 28 512
448 28
64

448 128
448 128
224 256
224 512
112256 56 28 512 28 256
512 28
112 56 28
224 112 56 28 512 28 256
3×3 1×1
28
56 3×3 28
224 1123×3 3×3 1×1
448 3×3
3×3 3×3
448
3×3

3×3

3×3

Figure 2. Double-Visual Geometry Group16 (D-VGG16) network structure. Feature maps with the
Figure
Figure
same2. 2.Double-Visual
Double-Visual
shape Geometry
Geometry
have the same Group16
Group16
width, height, (D-VGG16)
(D-VGG16)
number network
network
of channels structure.
structure.Feature
and convolutionalFeaturemaps
kernel. mapswith
withthe
the
sameshape
same shapehave
havethe
thesame
same width,
width, height, number of
of channels
channelsand
andconvolutional
convolutionalkernel.
kernel.
Appl. Sci. 2019, 9, 3159 5 of 17

For the classification task using the Convolutional Neural Network (CNN), the simplest way to
improve the accuracy of small sample training and avoid over-fitting is to reduce the feature map of the
last layer of CNN without decreasing the receptive field of the network. However, this will inevitably
influence the output features of the network, thereby limiting the expressive capability of the network.
Given the considerations above, D-VGG16 is designed, as shown in Figure 2. As a 1 × 1 convolutional
kernel with 256 channels, and this is used after the last convolution layer of the VGG16 network,
and then the outputs of two such networks are concatenated to form D-VGG16. On the one hand,
it can not only reduce the risk of an over-fitting of complex CNN for small samples training, but also
maintain the diversity of the network output features, and the output features of two sub-networks can
be conditioned on each other. The feature extraction network consists of two symmetrical D-VGG16,
i.e., the two D-VGG16 are identical in architectures, so the entire network is composed of four VGG16
with exactly the same structure. The advantage of this design is that the global and local features of the
image can be adequately extracted, making the network more easily able to detect the subtle features
in the image. In training, each sub-network loaded the pre-training weights of VGG16 on ImageNet
directly, and used transfer learning, which achieves the goal of small samples training.

2.1.2. Bilinear Model

The bilinear model is composed of two-factors, and is mathematically separable, i.e., when one
factor remains constant, its output is linear in any factor. A bilinear model B for defect classification
consists of a quaternion function, as shown in Equation (1).

B = ( fA , fB , P, F) (1)

where fA and fB are feature functions, D-VGG16 is used in this paper, P represents the pooling function,
and F represents the classification function, which here refers to the soft-max classifier.
The output of the feature function, fA and fB , are combined at each position of the image I using
the matrix inner product, as shown in Equation (2).

bilinear(i, fA , fB ) = fA (i)T fB (i) (2)

where i ∈ I. The feature dimensions of fA and fB must be equal, and the value should be greater than 1
to represent various descriptors that can be written as bilinear models.
To obtain the descriptor of the image, the pooling function P aggregates the bilinear features
across all of the locations in the image. The pooling function can use the weighted sum of all bilinear
features of the image, i.e., the sum of all bilinear features, which was calculated as follows.
X
Φ (I ) = bilinear(i, fA , fB ) (3)
i∈I

If the feature sizes of the fA and fB output are C × M and C × N respectively, then the size of the
bilinear vector Φ(I ) is M × N, and its corresponding class probability can be obtained by inputting the
Φ(I ) reshaped size MN × 1 into the classification function F. The data stream of this bilinear model is
shown in the Figure 3.
From an intuitive point of view, the structure of the bilinear can make the output features of the
feature extraction function, fA and fB , to be fine tuned on each other by considering all of their pairwise
interactions similar to quadratic kernel expansion. Because the entire network is a directed acyclic
graph, and parameters of the network can be trained by the gradient of back-propagating loss.
 Φ ( I ) =  i∈ I bilin ear( i , f A , f B ) (3)

If the feature sizes of the f A and f B output are C × M and C × N respectively, then the
size of the bilinear vector Φ(I) is M×N , and its corresponding class probability can be obtained
by inputting
Appl. Φ(I) reshaped size MN ×1 into the classification function
Sci. 2019, 9, the
3159 F . The data stream6 of
of17
this bilinear model is shown in the Figure 3.

Figure 3. Data stream

Figure stream of
of the
thebilinear
bilinearmodel.
model.

2.2. Defect Localization

Defect localization of the input image enables the inspector to find and understand the specific
location of the defect intuitively, and the implementation process is as follows: Firstly, the heat map of
the original image is obtained by applying Gradient-weighted Class Activation Mapping (Grad-CAM)
to one of the output features of D-VGG16, and then the corresponding defect location can be determined
in the input image by a threshold segmentation to the heat map.

2.2.1. Grad-CAM
Although CNN has significant effects on image processing tasks for a long time, it has been a
controversial method due to the poor interpretability of the CNN internal feature extraction, thus a
new field, which is called the interpretable research of deep learning, appeared. Apart from that,
Grad-CAM is a visualized method of the convolutional neural network, which can be used to visualize
network category location results in the last level of the network’s convolutional layer.
∂yn
In order to obtain a class activation map LnGrad−CAM , the score gradient ∂Ak
of the class n is firstly
calculated, in which Ak
represents the weight of the class n of the first k feature map, and yk represents
the score of the category before the soft-max. Then the gradient of the global average pooling layer is
used to obtain the importance αnk of the first k feature map for the category n.

1 X X ∂yn
αnk = (4)
Z ∂Ak
i j ij

where Z represents the size of the feature map and Akij represents the activation value of the position in
(i, j) the k first feature map. Finally, the weighted sum of the forward activation features is performed
according to Formula (4), and a Grad-CAM of a given class can be obtained using a rectified linear
unit (ReLU). X X
LnGrad−CAM = ReLU ( αnk Ak ) = max(0, αnk Ak ) (5)
k k

Grad-CAM can explain the feature extraction results of the network and enhance the trust of
the network performance, which is particularly important to the training network of small samples,
because the insufficient number of training samples may lead to an inadequately trained network,
thus causing a problem that the judgments of the network for a particular class may not be based on the
real discriminant region in the image, and this results in serious over-fitting. In addition, Grad-CAM is
used in the defect detection network, which can automatically locate the defects of input images in the
prediction stage only by image-level annotation in the training stage.
In this paper, the Grad-CAM of defective images are generated. As shown in Figure 1,
the Grad-CAM highlights the defect regions.
Appl. Sci. 2019, 9, 3159 7 of 17

2.2.2. Segmentation
The threshold segmentation is performed after the heat map of the input image obtained from
Grad-CAM to locate the defect regions. Let f (m, n) represent the binarized image for the heat map,
and f (m, n) is as shown in
255, if fhm (m, n) ≥ σ
(
f (m, n) = (6)
0, otherwise
where fhm (m, n) indicates the heat map after graying, and σ is the threshold, respectively. In f (m, n),
pixels whose gray value is 255 indicate the defect region, and pixels whose gray value is 0 present the
non-defective area. In order to get better localization results, it is significantly important to choose the
threshold segmentation method for σ. Experiments show that different types of defects and defects
distribution in the entire image can result in different methods of threshold segmentation. For images
with defects of limited distribution and signal type, a simple fixed threshold segmentation can be
used to obtain a better result. For images with defects of scattered distributions and variable types,
the adaptive Otsu [25] algorithm can obtain satisfactory results.

3. Experiments
This section evaluates the performance of the surface defect detection method proposed in this
paper on two public and two collected defect datasets in real industrial scenes. Firstly, the experimental
hardware environment and training details are briefly explained. Secondly, the datasets used
will be expounded. Then, the number of images for training and testing is interpreted. Finally,
the proposed method is compared with the latest experimental method of each data set in four datasets,
which highlight the effectiveness and universality of the proposed method on the task of surface
defect detection.

3.1. Hardware Platform and Training Details

Experiments in this paper are implemented on a workstation with 64 GB memory, and we also used
TITAN XP for acceleration. Similar to most deep convolutional neural networks, the back-propagation
algorithm was used as the training rule, and we then minimized the loss function with respect to
the network parameters using Adam [26]. The training of the whole network is carried out in an
end-to-end way. The training and testing images of each dataset are labeled only with the image-level.
Input images were resized to 448 × 448, with no preprocessing of the images except for normalization.
The training process of the whole network was in the form of transfer learning. Specific
implementation steps were as follows: Firstly, the pre-training weights of VGG16 on ImageNet was
loaded to initialize two D-VGG16, and only parameters other than VGG16 were trained. At this time,
the learning rate was 0.001, the momentum was 0.9, and the batch size was 64, and a model with
relatively low loss was trained. Then we load the weights of the last step to continue training the entire
network. At this time, the learning rate was 0.00001, the momentum was 0.9, and the batch size was 16.
In this schedule, a model with lower loss will be obtained by several iterations.

3.2. Datasets Description

The open datasets are DAGM_2007 [27] and hot-rolled strip [28], respectively. The collected
datasets are the diode glass bulb surface defect dataset and the fluorescent magnetic powder surface
defect dataset. These datasets cover texture defects, shape defects and color defects on the actual
industrial product.

3.2.1. DAGM_2007 Defect Dataset

The first open dataset is the DAGM_2007 surface defect dataset, which is manually generated and
can be used for surface defect detection. The dataset contains six types of surface defects with different
textures, where in each of these defects has 1000 defect-free and 150 defective grayscale images, the size
Appl. Sci. 2019, 9, 3159 8 of 17

of the image is 512 × 512 pixels and the pixel precision is 8 bits. The ground truths of all defective
Appl. Sci. 2019, 9, x FOR PEER REVIEW 8 of 17
images are provided in the dataset. Examples of defect images are shown in Figure 4.

(a) (b) (c) (d) (e) (f)

Figure
Figure 4.4. Examples
Examplesof ofthe
theDAGM_2007
DAGM_2007defect dataset.
defect Each
dataset. Eachcolumn
column represents a type
represents of defect,
a type andand
of defect, the
defect areasareas
the defect are labeled by theby
are labeled redthe
bounding boxes. boxes.
red bounding (a) classes1; (b) classes2;
(a) classes1; (c) classes3;
(b) classes2; (d) classes4;
(c) classes3; (d)
(e) classses5; (f) classes6.
classes4; (e) classses5; (f) classes6.

3.2.2. NEU Defect Dataset

3.2.2. NEU Defect Dataset
NEU [24] is a surface defect dataset of hot-rolled steel strips. There are six types of defects,
NEU [24] is a surface defect dataset of hot-rolled steel strips. There are six types of defects,
including crazing, inclusion, patches, pitted-surface, rolled-in scale and scratches. Examples of the
including crazing, inclusion, patches, pitted-surface, rolled-in scale and scratches. Examples of the
defect images are shown in Figure 5. Each class of defect has 300 grayscale images, and the size of the
defect images are shown in Figure 5. Each class of defect has 300 grayscale images, and the size of the
image is 200 × 200 pixels, and the pixel precision is 24 bits. The labels of all of the images are provided
image is 200 × 200 pixels, and the pixel precision is 24 bits. The labels of all of the images are provided
in the dataset, but the ground-truth of the defective images is not provided.
in the dataset, but the ground-truth of the defective images is not provided.

(a) (b) (c) (d) (e) (f)

Figure 5.
Figure 5. Examples
Examples ofof the
the NEU
NEU defect
defect dataset.
dataset. Each
Each column represents aa type
column represents type of
of defect,
defect, and
and the
the defect
defect
areas are
areas are labeled
labeled by
by the
the red
red bounding
bounding boxes.
boxes. (a)
(a) crazing;
crazing; (b)
(b) inclusion;
inclusion; (c)
(c) patches;
patches; (d)
(d) pitted-surface;
pitted-surface;
(e) rolled-in-scale; (f) scratches.
(e) rolled-in-scale; (f) scratches.

3.2.3. Diode Glass Bulb Surface Defect Dataset

3.2.3.
The glass
The glass bulbs
bulbs have
have been
been widely
widely used
used asas packaging
packaging material
material for
for diodes
diodes because
because of of their
their heat
heat
resistance,damp-proof
resistance, damp-proofability
abilityand
and high
high reliability.
reliability. TheyThey
playplay an important
an important role inrole in protecting
protecting the
the diode.
diode.
In orderIntoorder
obtainto surface
obtain surface
images images of the surface
of the surface of theglass
of the diode diode glass
bulb, anbulb,
image anacquisition
image acquisition
system
system consisting
consisting of an camera
of an industrial industrial
withcamera
DAHENG withIMAGING
DAHENGMER-131-75
IMAGINGGM/C-P,MER-131-75 GM/C-P,
a telecentric lensa
telecentric
with lens with 1×and
1× magnification magnification
a dark-fieldand
LED a dark-field
light source LED
waslight
usedsource
in thewas used in the
experiment. experiment.
A total of 1730
A total
color of 1,730
images werecolor imagesofwere
collected, whichcollected,
1020 were of defective
which 1,020 wereincluding
images, defective 390
images, including
of shell 390 of
wall damage,
shellbreaks,
360 wall damage, 360and
270 stains, breaks, 270 stains,
the size and theissize
of the image 661 of the pixels
× 601 image and
is 661
the× 601
pixelpixels and the
precision is 8pixel
bits.
precision is
Examples of8these
bits. defect
Examples of these
images defect in
are shown images
Figureare
6. shown in Figure 6.
Appl. Sci. 2019, 9, x FOR PEER REVIEW 9 of 17
Appl.
Appl.Sci. 2019,9,9,3159
Sci.2019, x FOR PEER REVIEW 99ofof1717

(a) (b) (c) (d)

Figure 6. Examples of the (a)diode glass bulb(b) (c) and the defect
surface defect dataset, (d) areas are labeled by
the red bounding
Figure boxes. (a)diode
break;glass
(b) shell wall damage;
defect(c) stain; (d)
andgood.
Figure 6.6.Examples
Examplesofofthe
thediode glass bulbsurface
bulb surfacedefect dataset,
dataset, and thedefect
the defectareas
areasare
arelabeled
labeledby
by
thered
the redbounding
boundingboxes.
boxes.(a)(a)break;
break;(b)
(b)shell
shellwall
walldamage;
damage;(c)(c)stain;
stain;(d)
(d)good.
good.
3.2.4. Fluorescent Magnetic Powder Surface Defect Dataset
3.2.4.
3.2.4. Fluorescent
Fluorescent Magnetic
Magnetic Powder
Powder Surface
SurfaceDefect
DefectDataset
Dataset
Fluorescent magnetic powder nondestructive testing is a common method for the detection of
Fluorescent
any surface and magnetic
near-surface powder nondestructive
defects of testing
ferromagnetic
Fluorescent magnetic powder nondestructive testing is is a
materialscommonsuchmethod
a common for the
as aero-turbines
method fordetection
[29],
the of anyof
turbines
detection
surface
[30],
anyand and near-surface
trainand
surface bearings defects
[31] in the
near-surface of ferromagnetic
aerospace,
defects materials
military and
of ferromagnetic such as
civil industry.
materials aero-turbines
such asItsaero-turbines [29],
working principleturbines is [30],
[29], turbines that
and
after train
the bearings
ferromagnetic[31] in the aerospace,
material military
work-piece is and civil
magnetized, industry.
the Its
magneticworking force
[30], and train bearings [31] in the aerospace, military and civil industry. Its working principle is that principle
line will isbethat after
locally
the ferromagnetic
distorted
after the when material
there
ferromagnetic work-piece
arematerial on isthemagnetized,
defects work-piece surface and thenear-surface
is magnetized, magnetic force line force
of the
the magnetic will be locally
work-piece.
line willIt distorted
leads
be to
locally
when
magnetic there are
distortedleakage, defects
when there on
absorbingthe surface and
fluorescent
are defects on thenear-surface
magnetic of
surface particles the work-piece.
suspendedofon
and near-surface It leads to magnetic
thethework-piece.
surface of the leakage,
work-to
It leads
absorbing
piece, and fluorescent
the forming magnetic
of visible particles
magnetic suspended
marks on
under the surface
ultraviolet of the
light.
magnetic leakage, absorbing fluorescent magnetic particles suspended on the surface of the work- work-piece,
In the and
experiment, the theforming
image
of visible
acquisition
piece, and magnetic
system
the forming marks
consisted under ultraviolet
of anmagnetic
of visible industrial light.
camera
marks Inwith
undertheultraviolet
experiment,
XIMEA MQ042CG-CM, theInimage
light. acquisition
a fixed focus
the experiment, thesystemlens
image
consisted
with a focal
acquisition of length
an industrial
system camera
ofconsisted
6 mm and with
of an
an XIMEAcamera
ultraviolet
industrial MQ042CG-CM,
light. Thewithsystem
XIMEA awas
fixed focus
used to lens
MQ042CG-CM, detectwith a focal
surface
a fixed length
cracks
focus of
lens
of
the 6 mm and
ferromagnetic an ultraviolet
cylindrical light. The
work-pieces system
withwas
the used
height toofdetect
100 mm surface
and cracks
diameter
with a focal length of 6 mm and an ultraviolet light. The system was used to detect surface cracks of of the
of 45 ferromagnetic
mm, in which
cylindrical
the
thewidth work-pieces
and
ferromagnetic height with
of thethe
cylindrical height
crack
work-pieces of 100
range from
with mm and
the0.3 mm
height diameter
to100
of 1.0mmof
mm 45 mm,
and
and in which
from
diameter 7ofmmthe
45 width
to
mm, 90 and
mm,
in which
height of the and
respectively.
the width crack range from
The height
experimentof the0.3 mm torange
collected
crack 1.0 mm
800 andand
defects
from from 7 mm
0.3 1,000
mm toto1.0
90 mm,
mm respectively.
defects-free color
and from images,7 The
andexperiment
mm the
to 90 sizemm,of
collected
the image 800
is defects
468 × 1324and 1000
pixels, defects-free
and the pixelcolor images,
precision is and
24 the
bits. size
Examplesof the of
respectively. The experiment collected 800 defects and 1,000 defects-free color images, and the size ofimage
defect is 468
images × 1324
are pixels,
shown
and
inthe the
Figure
imagepixel
7. is precision
468 × 1324ispixels,
24 bits.andExamples
the pixelofprecision
defect images are shown
is 24 bits. Examples in Figure
of defect 7. images are shown
in Figure 7.

(a) (b)
Figure 7. Examples of the fluorescent magnetic(a) powder(b)
surface defects dataset. The defect areas are
labeled
Figureby
7. the red bounding
Examples boxes. (a) bad;
boxes.
of the fluorescent (b) good.
magnetic powder surface defects dataset. The defect areas are
3.3. labeled Experiments
Contrast by the red bounding boxes. (a) bad; (b) good.
3.3. Contrast Experiments
3.3. In order to
Contrast
In order to test the
the performance
Experiments
test performance of of the
the proposed
proposed method
method in in work-pieces
work-pieces surface
surface defect
defect detection,
detection,
the proposed
the proposed method is evaluated on two published and two collected work-pieces surface defect
In order method is performance
to test the evaluated onoftwothe published and twoincollected
proposed method work-pieceswork-pieces surface
surface defect defect
detection,
datasets.
datasets. At present,
At present, most
most of the defect detection algorithms only aim at a specific category of defects;
the proposed method is of the defect
evaluated ondetection algorithms
two published and only
two aim at a specific
collected category
work-pieces of defects;
surface defect
however,
however, the
the surface
surface defect
defect detection
detection method proposed
method proposed inin this
this paper
paper is
is aa kind
kind of
of defect
defect that
that can
can be
be
datasets. At present, most of the defect detection algorithms only aim at a specific category of defects;
applied
applied to
to different
different types
types of
of work-pieces.
work-pieces. It
It is
is unreasonable
unreasonable to
to apply
apply a
a defect
defect detection
detection algorithm
algorithm
however, the surface defect detection method proposed in this paper is a kind of defect that can be
suitable
suitable for aa specific
specific category to other categories
categories of defects and
and compare
compare it it with the
the method
method proposed
proposed
appliedfor to differentcategory
types of to other
work-pieces. It is of defects
unreasonable to apply a with
defect detection algorithm
in this
this paper.
insuitablepaper.
for a specific category to other categories of defects and compare it with the method proposed
in this paper.
Appl. Sci. 2019, 9, 3159 10 of 17
Appl. Sci. 2019, 9, x FOR PEER REVIEW 10 of 17

Therefore,
Therefore,inineach
eachdefect
defectdata
dataset,
set, not
not only GLCM ++ MLP
only GLCM [17], gcForest
MLP [17], gcForest [32]
[32] and
and Bilinear
Bilinear
Convolutional Neural Network (BCNN) are used to perform four kinds of generic
Convolutional Neural Network (BCNN) are used to perform four kinds of generic surface defect surface defect
detection
detectionalgorithms,
algorithms,but
butalso
also the
the open
open datasets
datasets will
will also
also be
be compared with the
compared with the known
known latest
latest
experimental results on this dataset.
experimental results on this dataset.

3.3.1. Open Datasets

3.3.1. Open Datasets
Since
Sincethe
thevast
vastmajority of the
majority of theevaluations
evaluationsusing
using thethetwotwo datasets
datasets for performance
for performance evaluation
evaluation had
had only the experimental results of average precision, and average precision
only the experimental results of average precision, and average precision is the main and is the main andmost
most
important
important performance
performanceindicator
indicatorforforthe
themulti-category
multi-category task, task, therefore only the
therefore only the average
averageprecision
precisionofof
thethe
methods is compared on two open datasets.
methods is compared on two open datasets.
(A) Localization
(A) andand
Localization Classification Results
Classification of the DAGM_2007
Results of the DAGM_2007 Defect Dataset:
DefectFor the DAGM_2007
Dataset: For the
dataset, the ratio
DAGM_2007 of the the
dataset, training setthe
ratio of to the test set
training setisto1:1.
the Some
test setexperimental localization results
is 1:1. Some experimental of the
localization
proposed
results ofmethod running
the proposed in the running
method dataset are shown
in the in Figure
dataset 8. in Figure 8.
are shown

(a) (b) (c) (d) (e) (f)

Figure
Figure 8. 8.Examples
Examplesofoflocalization
localizationon onDAGM_2007
DAGM_2007defect
defect dataset.
dataset. From top to to bottom
bottomare
arethetheoriginal
original
image,
image, thethe combinationofofthe
combination theoriginal
originalimage
imageand
andthe
the heat
heat map, and thethe location
locationresults
resultsofofthe
thedefects.
defects.
The Ground-Truth of the defect is marked with
The Ground-Truth of the defect is marked with the red boundingthe red bounding boxes, while the localization results
the localization results
of of
thethe proposed
proposed methodisismarked
method markedwith withthe
theblue
bluebounding
bounding boxes.
boxes. (a)
(a) classes1;
classes1;(b)
(b)classes2;
classes2;(c)
(c)classes3;
classes3;
(d)(d) classes4;
classes4; (e)(e) classses5;(f)(f)classes6.
classses5; classes6.

InIn the
the combinationimage
combination imageofofthetheoriginal
originalimage
image and
and Grad-CAM,
Grad-CAM, the the red
red region
region represents
representsthethe
confidence
confidence level
level ofofthe
thepixels
pixelsthat
thatthe
thenetwork
networkdiscriminates
discriminates against.
against. The
The deeper
deeperthe
thecolor,
color,the
thehigher
higher
thethe confidencelevel
confidence levelofofthe
thepixels
pixelsininthe
theimage.
image. The
The dataset
dataset is compared
compared with
with the
the results
resultsofofsurface
surface
defect detection algorithms proposed by Yu [33] and Zhao [34]. The experimental
defect detection algorithms proposed by Yu [33] and Zhao [34]. The experimental classification resultsclassification
areresults
shown are
inshown
Table 1.in Table 1.
Appl. Sci. 2019, 9, 3159 11 of 17
Appl. Sci. 2019, 9, x FOR PEER REVIEW 11 of 17

Table 1. Comparison of results on DAGM_2007 surface defect dataset.

Table 1. Comparison of results on DAGM_2007 surface defect dataset.
Method
Method AveragePrecision
Average Precision
GLCM++MLP
GLCM MLP 81.68%
81.68%
gcForest
gcForest 86.67%
86.67%
BCNN 95.57%
BCNN 95.57%
FCN [33] 98.35%
FCN [33]
Zhao [34]
98.35%
98.53%
Zhao
Ours[34] 98.53%
99.49%
Ours 99.49%

As Ascan
canbebeseen
seenfrom
fromTable
Table1,1,although
although high
high classification accuracy has
classification accuracy has been
beenachieved
achievedon onthe
the
DAGM_2007 surface defect data set at present, the proposed method can still
DAGM_2007 surface defect data set at present, the proposed method can still further improve thefurther improve the
classification
classificationaccuracy
accuracyononthe
thedata
dataset
setand
andachieve
achieve the
the automatic location of
automatic location of defects
defectsatatthe
thesame
sametime.
time.
(B) Localization and Classification Results of the NEU Defect Dataset: For the NEU
(B) Localization and Classification Results of the NEU Defect Dataset: For the NEU surface defectsurface defect
dataset,
dataset,a anumber
numberofof150
150images
imagesarearerandomly
randomly selected
selected as
as the test set
the test set in
in each
eachclass
classofofdefects,
defects,and
andthe
the
remaining
remainingimages
imagesare
areused
usedasasthe
thetraining
trainingset.
set. Some experimental localization
Some experimental localizationresults
resultsofofthe
theproposed
proposed
method running in the dataset are shown in Figure
method running in the dataset are shown in Figure 9. 9.

(a) (b) (c) (d) (e) (f)

Figure9.9.Localization
Figure Localizationresults
resultsof
ofthe
the proposed
proposed method on the the NEU
NEU defect dataset.
dataset. From
Fromtop
toptotobottom
bottom
arethe
are theoriginal
originalimage,
image,the
thecombination
combination ofof the
the original
original image
image and
and the
the heat
heat map,
map, and
andthe
thelocation
location
results
results ofof the
the defects.The
defects. The Ground-Truth
Ground-Truth ofof
thethe defect
defect is is marked
marked with
with thethe
redred bounding
bounding boxes,
boxes, and
and the
the localization
localization resultsresults of proposed
of the the proposed method
method is marked
is marked withthe
with theblue
bluebounding
bounding boxes. (a) (a)crazing;
crazing;
(b)(b) inclusion;(c)(c)patches;
inclusion; patches;(d)
(d)pitted-surface;
pitted-surface;(e)
(e)rolled-in-scale;
rolled-in-scale; (f) scratches.

Most
Mostimagesimagesofofthe
theNEU
NEUdefect
defect dataset
dataset have
have multiple defects,
defects, and
and the
thetexture
textureofofeach
eachtype
typeofof
defective
defective imageisisdifferent,
image different,which
whichbrings
bringsmore
morechallenges
challengestotoautomatic
automaticlocation.
location.As
Asshown
shownin inFigure
Figure9,
9, although
although the the proposed
proposed method
method does
does not
not performwell
perform wellinindefect
defectlocalization
localization when
when applied
appliedtotoNEUNEU
datasets, it can extract specific pixel regions to identify a certain class of images. Using
datasets, it can extract specific pixel regions to identify certain class of images. Using this dataset, this dataset,
theproposed
the proposedmethod methodwaswascompared
comparedwith
with the
the algorithms
algorithms proposed
proposed byby BYEC
BYEC[6],
[6],Song
Songetetal.
al.[35]
[35]and
and
Renetetal.al.[18].
Ren [18].ToToensure
ensurethethevalidity
validityof
of the
the comparison
comparison results,
results, the
the same
same training
trainingdata
datageneration
generation
methodasasthe
method thepapers
papersmentioned
mentionedabove
aboveisisused.
used. The
The experimental
experimental classification
classificationresults
resultsare
areshown
showninin
Table
Table 2. 2.
Appl. Sci. 2019, 9, 3159 12 of 17
Appl. Sci. 2019, 9, x FOR PEER REVIEW 12 of 17

Table
Table 2.
2. Comparison
Comparisonof
ofresults
resultson
on NEU
NEU surface
surface defect
defect dataset.
dataset.
Method Average Precision
Method Average Precision
GLCM + MLP 98.61%
GLCM + MLP 98.61%
gcForest
gcForest
61.56%
61.56%
BCNN
BCNN 98.56%
98.56%
BYEC
BYEC 96.30%
96.30%
Song [35]
Song [35] 98.60%
98.60%
Ren [18]
Ren [18] 99.21%
99.21%
Ours 99.44%
Ours 99.44%

As can
As canbe
beseen
seenfrom
fromTable
Table2, 2, compared
compared with
with thethe latest
latest methods
methods proposed
proposed by Sun
by Sun and and
Ren, Ren,
the
the proposed
proposed method
method has ahas a higher
higher detection
detection accuracy
accuracy in NEU
in the the NEU defect
defect detection
detection dataset.
dataset.

3.3.2.Real
3.3.2. RealCollected
CollectedDatasets
Datasets
Thetwo
The twokinds
kindsofofdefect
defectdatasets
datasetscollected
collectedcontains
containsdefective
defectiveand
anddefect-free
defect-freeimages,
images,sosothey
theycancan
be regarded as multi-classification or binary classification
be regarded as multi-classification or binary classification tasks. tasks.
(A) Localization
(A) Localization and
and Classification
ClassificationResults
Resultsofofthe
theDiode
DiodeGlass
GlassBulb Surface
Bulb SurfaceDefect Dataset:
Defect ForFor
Dataset: the
diode
the glass
diode bulbbulb
glass surface defect
surface dataset,
defect the ratio
dataset, of the
the ratio oftraining set and
the training settesting set images
and testing is 7:3.isSome
set images 7:3.
Some experimental localization results of defect detection on this dataset by the proposedare
experimental localization results of defect detection on this dataset by the proposed method shown
method
in Figure
are shown10. in Figure 10.

(a) (b) (c)

Figure10.
Figure Examplesof
10.Examples oflocalization
localizationon
onthe
thediode
diodeglass
glassbulb
bulbsurface
surfacedefect
defectdataset.
dataset.From
Fromtoptopto
tobottom
bottom
are the original image, the combination of the original image and the heat map, and the
are the original image, the combination of the original image and the heat map, and the locationlocation results
of these
results ofdefects. The Ground-Truth
these defects. of theofdefect
The Ground-Truth is marked
the defect withwith
is marked the red bounding
the red boundingboxes, while
boxes, the
while
localization results of the proposed method is marked with the blue bounding boxes. (a) break; (b) shell
the localization results of the proposed method is marked with the blue bounding boxes. (a) break;
wall damage; (c) stain.
(b) shell wall damage; (c) stain.

There is no significant texture difference around different defect types in the diode glass bulb
There is no significant texture difference around different defect types in the diode glass bulb
surface defect dataset, and shell wall damage is a typical shape defect. However, it can be found
surface defect dataset, and shell wall damage is a typical shape defect. However, it can be found that
that the proposed method can accurately extract the key pixel regions that discriminate each type of
the proposed method can accurately extract the key pixel regions that discriminate each type of
defect, which can not only explain the reason why it can achieve a higher precision than other methods,
defect, which can not only explain the reason why it can achieve a higher precision than other
but also obtain the better effect of localization. The comparative experiments on this dataset are shown
methods, but also obtain the better effect of localization. The comparative experiments on this dataset
in Table 3.
are shown in Table 3.
Appl.
Appl. Sci. 2019, 99,, x3159
Sci. 2019, FOR PEER REVIEW 1313of
of 17
17

Table 3. Comparison of results on the diode glass bulb surface defect dataset.
Table 3. Comparison of results on the diode glass bulb surface defect dataset.
Method Average Precision
Method
GLCM + MLP Average Precision
91.32%
GLCM + MLP
gcForest 85.25%
91.32%
gcForest
BCNN 85.25%
91.80%
BCNN 91.80%
Ours 99.87%
Ours 99.87%

It can be seen from Table 3 that even in the work-piece surface defect detection task with few
It can
texture be seenthe
features, from Table 3 that
proposed evenhas
method in the
an work-piece
advantage surface defectaccuracy
in detection detectioncompared
task with few
withtexture
other
features, the proposed method has an advantage in detection accuracy compared with other algorithms.
algorithms.
(B) Localization
(B) Localization and
and Classification
Classification Results
Results of
of the
the Fluorescent
Fluorescent Magnetic
Magnetic Powder
Powder Surface
Surface Defect
Defect
Dataset: For this fluorescent magnetic powder surface defect dataset, the ratio of the training
Dataset: For this fluorescent magnetic powder surface defect dataset, the ratio of the training set and set and
the testing
the testing set
set images
images is 7:3. The
The experimental
experimental localization
localization results
results of
of the
the proposed
proposed method
method onon this
this
dataset are
dataset are shown in Figure 11.

Localization results
Figure 11. Localization
Figure results of
of the
the proposed
proposed method
method on on the
the fluorescent
fluorescent magnetic
magnetic powder
powder surface
surface
defect dataset.
defect dataset. From
Fromtoptop to
to bottom
bottom areare the
the original
original image,
image, the
the combination
combination ofof the
the original
original image
image and
and
the heat
the heat map,
map, and
and the
the location
location results
results of
of these
these defects.
defects. The
TheGround-Truth
Ground-Truth ofof the
the defect
defect is
is marked
marked with
with
the red
the red bounding
bounding boxes,
boxes, and
and the
the localization
localization result
result of
of the
the proposed
proposed method
method is is marked
marked with
with the
the blue
blue
bounding boxes.
bounding boxes.

When the
When the ultraviolet
ultraviolet light
light is
is irradiated
irradiated onon the
the smooth
smooth iron
iron work-piece,
work-piece, the the surface
surface of
of the
the
magnetized work-piece will reflect the violet light emitted by the ultraviolet light due
magnetized work-piece will reflect the violet light emitted by the ultraviolet light due to the principleto the principle
of light
of lightreflection.
reflection.This phenomenon
This phenomenon is particularly prominent
is particularly on the cylindrical
prominent work-piece.
on the cylindrical Therefore,
work-piece.
Therefore, the defect image of the fluorescent magnetic powder obtained in the experimentpurple
the defect image of the fluorescent magnetic powder obtained in the experiment has a bright has a
reflective
bright areareflective
purple in the center
areaof inthe
thework-piece,
center of thewhich will cause
work-piece, whicha great interference
will cause a greattointerference
the detectionto
of any
the defects.
detection In the
of any experiment,
defects. the originalthe
In the experiment, image is zoomed
original image isinto a sizeinto
zoomed of 448 × 448,
a size of 448with no
× 448,
pre-processing
with having been
no pre-processing havingperformed on the images
been performed on theexcept
imagesfor normalization,
except and thenand
for normalization, the then
imagetheis
sent to the network for training and testing. As shown in Figure 11, it can be seen that
image is sent to the network for training and testing. As shown in Figure 11, it can be seen that the the network can
network can effectively eliminate interference in the reflective area and extract the defective area. The
classification results of the comparative experiments on this dataset are shown in Table 4.
Appl. Sci. 2019, 9, 3159 14 of 17

effectively eliminate interference in the reflective area and extract the defective area. The classification
results of the comparative experiments on this dataset are shown in Table 4.

Table 4. Comparison of results on fluorescent magnetic powder surface defect dataset.

Method Average Precision

GLCM + MLP 90.56%
gcForest 92.59%
BCNN 93.33%
Ours 99.13%

It can be seen from Table 4 that even if there is a task of defect detection with strong interference
factors, the detection accuracy of the proposed method is still nearly 6% higher than that of BCNN.
(C) Evaluation of Binary Classification Performance: The above experiments have shown that
the average precision of the proposed method on four datasets is higher than that of other methods.
However, the detection rate of defects and the precision of non-defects are often emphasized in defect
detection, and at this time, only the dataset is divided into defects and non-defects. TP and TN denote
the number of true positives and true negatives respectively, FP and FN denote the number of false
positives and false negatives, respectively. Then the definitions of the Precision Rate (PR), True Positive
Rate (TPR), False Positive Rate (FPR) and False Negative Rate (FNR) are as follows.

TP
PR = (7)
TP + FP

TP
TPR = (8)
TP + FN
FP
FPR = (9)
FP + TN
FN
FNR = (10)
FN + TP
Results of the four methods PR, TPR, FPR and FNR on the diode glass bulb and fluorescent
magnetic powder surface defect dataset are shown in Table 5.

Table 5. Results of the four methods PR, TPR, FPR and FNR on the diode glass bulb and fluorescent
magnetic powder surface defect datasets.

Dataset Diode Glass Bulb Surface Defect Dataset Fluorescent Magnetic Powder Surface Defect Dataset
Method PR TPR FPR FNR PR TPR FPR FNR
GLCM + MLP 93.86% 88.21% 6.14% 11.79% 85.71% 89.55% 14.28% 10.45%
grForest 83.19% 79.84% 16.81% 20.16% 95% 91.94% 5% 8.06%
BCNN 81.25% 100% 18.75% 0% 99.65% 94% 0.35% 6%
Ours 100% 100% 0% 0% 98.36% 99.67% 1.64% 0.33%

Precision Rate and True Positive Rate are often a pair of contradiction measure, and generally
speaking, when the Precision Rate is high, the True Positive Rate tends to be low, and the higher True
Positive Rate, the lower the Precision Rate. Therefore, the Precision Rate and the True Positive Rate
cannot accurately reflect the effectiveness of the detection method, but usually F1 is used, which is
defined as follows.
2 × PR × TPR
F1 = (11)
PR + TPR
F1 value of GLCM + MLP, gcForest, BCNN and the proposed method on the diode glass bulb
surface defect dataset and fluorescent magnetic powder surface defect dataset are shown in Figure 12.
Appl. Sci. 2019, 9, 3159 15 of 17
Appl. Sci. 2019, 9, x FOR PEER REVIEW 15 of 17

(a) (b)
Figure 12. Comparison of FF11 curve
Figure 12. curve obtained
obtained from
from four
four methods.
methods. (a) Diode glass bulb surface
surface defect
defect
dataset; (b) Magnetic
dataset; (b) Magnetic powder surface defect dataset.

The results
The results are
are shown
shown inin Figure
Figure 12. The proposed
12. The proposed surface
surface defect
defect detection
detection method
method achieves
achieves aa
higher F among all of the methods. It outperforms both methods combining statistical features
higher F11 among all of the methods. It outperforms both methods combining statistical features with with
machine learning (GLCM + MLP) and the generic deep learning method based on a Convolutional
machine learning (GLCM + MLP) and the generic deep learning method based on a Convolutional
Neural Network (BCNN).
Neural Network (BCNN).
There are many kinds of defects in actual industrial production, and one method which works
There are many kinds of defects in actual industrial production, and one method which works
well in a specific category is usually not applicable to the other types of defects. Experimental
well in a specific category is usually not applicable to the other types of defects. Experimental results
results show that the surface defect detection method proposed in this paper demonstrates excellent
show that the surface defect detection method proposed in this paper demonstrates excellent
detection performance in surface defects with features of texture, shape and color. Furthermore, it can
detection performance in surface defects with features of texture, shape and color. Furthermore, it
simultaneously realize an automatic localization and classification of defects. In the prediction phase,
can simultaneously realize an automatic localization and classification of defects. In the prediction
it takes an average of 0.292 s to a localization and classification of defects for an image at the same time.
phase, it takes an average of 0.292 s to a localization and classification of defects for an image at the
same time.
4. Conclusions
The conclusions from the work are presented as follows.
4. Conclusions

• The conclusions
A generic method from the work are
of automated presented
surface defectasdetection
follows. based on a bilinear model is proposed.
• Firstly,
A as method
generic a featureofextraction
automated network
surface of the bilinear
defect detection model,
basedD-VGG16,
on a bilinearwhich
modelconsists of two
is proposed.
completely
Firstly, as asymmetric VGG16, network
feature extraction is designed, andbilinear
of the the features extracted
model, D-VGG16,fromwhich
the bilinear model
consists are
of two
output to the soft-max function to realize the automatic classification
completely symmetric VGG16, is designed, and the features extracted from the bilinear modelof defects. Then the heat
mapoutput
are of the original image isfunction
to the soft-max obtainedtothrough
realize applying Grad-CAM
the automatic to one ofofthe
classification outputThen
defects. features
the
in D-VGG16. Finally, the defects in the input image can be located automatically
heat map of the original image is obtained through applying Grad-CAM to one of the output after processing
the heat in
features map with a threshold
D-VGG16. segmentation
Finally, the defects in thealgorithm.
input image can be located automatically after
• processing
The training theofheat
themap
proposed
with amethod
threshold is segmentation
carried out inalgorithm.
a small sample, end-to-end, and in a
• weakly-supervised way. Even though the number
The training of the proposed method is carried out in a small of training images used inend-to-end,
sample, the experimentsand were
in a
no more than 1300, over-fitting
weakly-supervised did not the
way. Even though occur duringofthe
number training
training process
images of all
used in the
the datasets,
experiments and
the surface defects can be automatically located using only training images labeled
were no more than 1,300, over-fitting did not occur during the training process of all the datasets, at image-level.
• andThe the
experiments has been
surface defects canperformed on four located
be automatically datasetsusingwith only
different defective
training features.
images labeledThis shows
at image-
that the proposed method can be effectively applied to surface defect detection scenarios with
level.
• texture,
The color andhas
experiments shapebeenfeatures,
performedevenon a diode glass bulb
four datasets surface
with defect
different dataset features.
defective with complexThis
texturethat
shows and the
the proposed
fluorescentmethod
magnetic canpowder surface applied
be effectively defect dataset withdefect
to surface strongdetection
interference factors.
scenarios
The overall
with performance
texture, of the proposed
color and shape features, evenmethod is superior
a diode glasstobulbother methods.
surface defect dataset with
complex texture and the fluorescent magnetic powder surface defect dataset with strong
The proposed method has certain limitations for automatic localization in the datasets with
interference factors. The overall performance of the proposed method is superior to other
complex textures. Since the whole network is composed of four VGG16, and the Grad-CAM used
methods.
in automatic localization is time-consuming, it takes a long time to detect and locate defect in the
Thestage.
testing proposed
Futuremethod
work willhasfocus
certain
on limitations
solving the for aboveautomatic
effect oflocalization in the datasets
automatic location with
and real-time
complex textures.
performance of theSince the whole
method in this network
paper. is composed of four VGG16, and the Grad-CAM used in
automatic localization is time-consuming, it takes a long time to detect and locate defect in the testing
stage. Future work will focus on solving the above effect of automatic location and real-time
performance of the method in this paper.
Appl. Sci. 2019, 9, 3159 16 of 17

Author Contributions: F.Z. designed the algorithm, performed the experiments and wrote the paper. G.L. modified
the paper. F.X. and. H.D. supervised the research.
Funding: This work was supported by National Natural Science Foundation of China (Grant Nos. 11602292,
61701421, 61601381).
Conflicts of Interest: The authors declare no conflict of interest.

References
1. Shafarenko, L.; Petrou, M.; Kittler, J. Automatic watershed segmentation of randomly textured color images.
IEEE Trans. Image Process. 1997, 6, 1530–1544. [CrossRef] [PubMed]
2. Ojala, T.; Pietikäinen, M.; Mäenpää, T. Multiresolution gray-scale and rotation invariant texture classification
with local binary patterns. IEEE Trans. Pattern Anal. Mach. Intell. 2002, 7, 971–987. [CrossRef]
3. Wen, W.; Xia, A. Verifying edges for visual inspection purposes. Pattern Recognit. Lett. 1999, 20, 315–328.
[CrossRef]
4. Zhou, P.; Xu, K.; Liu, S. Surface defect recognition for metals based on feature fusion of shearlets and wavelets.
Chin. J. Mech. Eng. 2015, 51, 98–103. [CrossRef]
5. Ghorai, S.; Mukherjee, A.; Gangadaran, M.; Dutta, P.K. Automatic defect detection on hot-rolled flat steel
products. IEEE Trans. Instrum. Meas. 2012, 62, 612–621. [CrossRef]
6. Xiao, M.; Jiang, M.; Li, G.; Xie, L.; Yi, L. An evolutionary classifier for steel surface defects with small sample
set. EURASIP J. Image Video Process. 2017, 2017, 48. [CrossRef]
7. Santoyo, E.A.R.; Lopez, A.V.; Serrato, R.B.; Garcia, J.A.J.; Esquivias, M.T.; Fernandez, V.F. Reconocimiento de
patrones y evaluación del daño generado en aceros de baja aleación a partir del procesamiento digital de
imágenes e inteligencia artificial. DYNA Ing. Ind. 2019, 94, 357.
8. Li, Y.; Chen, X.; Zhu, Z.; Xie, L.; Huang, G.; Du, D.; Wang, X. Attention-guided unified network for panoptic
segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
Long Beach, CA, USA, 16–20 June 2019.
9. Feng, W.; Hu, Z.; Wu, W.; Yan, J.; Ouyang, W. Multi-Object Tracking with Multiple Cues and Switcher-Aware
Classification. arXiv 2019, arXiv:1901.06129.
10. Lin, H.; Li, B.; Wang, X.; Shu, Y.; Niu, S. Automated defect inspection of LED chip using deep convolutional
neural network. J. Intell. Manuf. 2018, 30, 2525–2534. [CrossRef]
11. Zhou, B.; Khosla, A.; Lapedriza, A.; Oliva, A.; Torralba, A. Learning deep features for discriminative
localization. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, Las Vegas,
NV, USA, 27–30 June 2016.
12. Tao, X.; Zhang, D.; Ma, W.; Liu, X.; Xu, D. Automatic metallic surface defect detection and recognition with
convolutional neural networks. Appl. Sci.-Basel 2018, 8, 1575. [CrossRef]
13. Di, H.; Ke, X.; Peng, Z.; Dongdong, Z. Surface defect classification of steels with a new semi-supervised
learning method. Opt. Lasers Eng. 2019, 117, 40–48. [CrossRef]
14. Lin, W.-Y.; Lin, C.-Y.; Chen, G.-S.; Hsu, C.-Y. Steel Surface Defects Detection Based on Deep Learning.
In Proceedings of the International Conference on Applied Human Factors and Ergonomics (AHFE), Orlando,
FL, USA, 22–26 July 2018.
15. Ren, S.; He, K.; Girshick, R.; Sun, J. Faster r-cnn: Towards real-time object detection with region proposal
networks. In Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada,
7–12 December 2015; pp. 91–99.
16. Liu, W.; Anguelov, D.; Erhan, D.; Szegedy, C.; Reed, S.; Fu, C.-Y.; Berg, A.C. Ssd: Single shot multibox detector.
In Proceedings of the European Conference on Computer Vision (ECCV), Amsterdam, The Netherlands,
8–16 October 2016.
17. Cha, Y.J.; Choi, W.; Suh, G.; Mahmoudkhani, S.; Büyüköztürk, O. Autonomous structural visual inspection
using region-based deep learning for detecting multiple damage types. Comput. Aided Civ. Infrastruct. Eng.
2018, 33, 731–747. [CrossRef]
18. Ren, R.; Hung, T.; Tan, K.C. A generic deep-learning-based approach for automated surface inspection.
IEEE Trans. Cybern. 2017, 48, 929–940. [CrossRef] [PubMed]
19. Lin, M.; Chen, Q.; Yan, S. Network in network. In Proceedings of the International Conference on Learning
Representations (ICLR), Scottsdale, AZ, USA, 2–4 May 2013.
Appl. Sci. 2019, 9, 3159 17 of 17

20. Selvaraju, R.R.; Cogswell, M.; Das, A.; Vedantam, R.; Parikh, D.; Batra, D. Grad-cam: Visual explanations
from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference on
Computer Vision (ICCV), Venice, Italy, 22–29 October 2017.
21. Simonyan, K.; Zisserman, A. Very deep convolutional networks for large-scale image recognition.
In Proceedings of the International Conference on Learning Representations, Banff, AB, Canada,
14–16 April 2014.
22. Lin, T.-Y.; RoyChowdhury, A.; Maji, S. Bilinear cnn models for fine-grained visual recognition. In Proceedings
of the IEEE International Conference on Computer Vision, Santiago, Chile, 7–13 December 2015.
23. Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; Li, F.-F. Imagenet: A large-scale hierarchical image database.
In Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA,
20–25 June 2009.
24. Yosinski, J.; Clune, J.; Bengio, Y.; Lipson, H. How transferable are features in deep neural networks?
In Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada,
8–13 December 2014; pp. 3320–3328.
25. Ostu, N. A threshold selection method from gray-histogram. IEEE Trans. Syst. Man Cybern. 1975, 9, 62–66.
26. Kingma, D.P.; Ba, J. Adam: A method for stochastic optimization. In Proceedings of the International
Conference on Learning Representations, Banff, AB, Canada, 14–16 April 2014.
27. DAGM 2007 Datasets. Available online: https://hci.iwr.uni-heidelberg.de/node/3616 (accessed on
29 July 2019).
28. Song, K.; Yan, Y. A noise robust method based on completed local binary patterns for hot-rolled steel strip
surface defects. Appl. Surf. Sci. 2013, 285, 858–864. [CrossRef]
29. Coro, A.; Abasolo, M.; Aguirrebeitia, J.; López de Lacalle, L. Inspection scheduling based on reliability
updating of gas turbine welded structures. Adv. Mech. Eng. 2019, 11, 1687814018819285. [CrossRef]
30. Artetxe, E.; Olvera, D.; de Lacalle, L.N.L.; Campa, F.J.; Olvera, D.; Lamikiz, A. Solid subtraction model for
the surface topography prediction in flank milling of thin-walled integral blade rotors (IBRs). Int. J. Adv.
Manuf. Technol. 2017, 90, 741–752. [CrossRef]
31. Zhao, M.; Lin, J.; Miao, Y.; Xu, X.J.M. Detection and recovery of fault impulses via improved harmonic
product spectrum and its application in defect size estimation of train bearings. Measurement 2016, 91,
421–439. [CrossRef]
32. Zhou, Z.-H.; Feng, J. Deep forest: Towards an alternative to deep neural networks. In Proceedings of the
International Joint Conference on Artificial Intelligence (IJCAI), Melbourne, Australia, 19–25 August 2017.
33. Yu, Z.; Wu, X.; Gu, X. Fully convolutional networks for surface defect inspection in industrial environment. In
Proceedings of the International Conference on Computer Vision Systems (ICVS), Venice, Italy, 22–29 October
2017.
34. Zhao, Z.; Li, B.; Dong, R.; Zhao, P. A Surface Defect Detection Method Based on Positive Samples.
In Proceedings of the Pacific Rim International Conference on Artificial Intelligence (PRICAI), Nanjing,
China, 28–31 August 2018.
35. Song, K.; Hu, S.; Yan, Y. Automatic recognition of surface defects on hot-rolled steel strip using scattering
convolution network. J. Comput. Inf. Syst. 2014, 10, 3049–3055.

© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access
article distributed under the terms and conditions of the Creative Commons Attribution
(CC BY) license (http://creativecommons.org/licenses/by/4.0/).