applied

sciences
Article
Feature-Learning-Based Printed Circuit Board
Inspection via Speeded-Up Robust Features and
Random Forest
Eun Hye Yuk 1 , Seung Hwan Park 2 , Cheong-Sool Park 1 and Jun-Geol Baek 1, * ID

1 Department of Industrial Management Engineering, Korea University, Seoul 02841, Korea;

eunhyeyuk@korea.ac.kr (E.H.Y.); dumm97@gmail.com (C.-S.P.)
2 Data Science, SK Hynix Semiconductor, Icheon 17336, Korea; seunghwan1.park@sk.com
* Correspondence: jungeol@korea.ac.kr; Tel.: +82-2-3290-3396




Received: 14 May 2018; Accepted: 31 May 2018; Published: 5 June 2018


Featured Application: The main contribution of this work is to propose an inspection method
using image data generated at the actual manufacturing process. This proposed method can help
printed circuit board (PCB) manufacturers more effectively detect defects, such as scratches and
improper etching, in an automated optical inspection (AOI). Moreover, the proposed method of
this work can be also applied to the field of dermatology, where it has to detect skin diseases, as
well as in PCB inspection.

Abstract: With the coming of the 4th industrial revolution era, manufacturers produce high-tech
products. As the production process is refined, inspection technologies become more important.
Specifically, the inspection of a printed circuit board (PCB), which is an indispensable part of electronic
products, is an essential step to improve the quality of the process and yield. Image processing
techniques are utilized for inspection, but there are limitations because the backgrounds of images
are different and the kinds of defects increase. In order to overcome these limitations, methods based
on machine learning have been used recently. These methods can inspect without a normal image by
learning fault patterns. Therefore, this paper proposes a method can detect various types of defects
using machine learning. The proposed method first extracts features through speeded-up robust
features (SURF), then learns the fault pattern and calculates probabilities. After that, we generate
a weighted kernel density estimation (WKDE) map weighted by the probabilities to consider the
density of the features. Because the probability of the WKDE map can detect an area where the defects
are concentrated, it improves the performance of the inspection. To verify the proposed method,
we apply the method to PCB images and confirm the performance of the method.

Keywords: image inspection; non-referential method; feature extraction; fault pattern learning;
weighted kernel density estimation (WKDE)

1. Introduction
Because the era of the Internet of Things (IoT) has been accompanied by the rapid development of
the semiconductor industry and communication technologies, the use of high-tech products, such as
mobile phones and wearable devices, has been spreading widely in our daily lives. The printed circuit
board (PCB) is one of the key components of such electronic devices. A PCB is a thin plate made
by printing an electrically conductive circuit on an insulator. Figure 1 shows an image of a generic
PCB. As shown in Figure 1, a PCB connects to different parts electrically through a point-to-point
wiring process.

Appl. Sci. 2018, 8, 932; doi:10.3390/app8060932 www.mdpi.com/journal/applsci

Appl. Sci. 2018, 8, 932 2 of 14

In the past, handwork wiring led to frequent failures of the wire junctions and a short circuit
as the wire insulation began to age. Furthermore, this type of work was conducted manually for
inner-connecting components within the board. Such wiring, therefore, required a significant amount
of time and effort. With advancements in technology, PCBs were developed for efficient and automated
production. Because PCBs allow for the use of mass production, they allow devices to be smaller and
lighter. In addition, they allow a high level of reliability at low production costs. Because of these
advantages,
Appl. Sci. 2018, 8,most
x FORelectronic devices use PCBs.
PEER REVIEW 2 of 15

Figure 1. An illustrative image of a printed circuit board (PCB).

Figure 1. An illustrative image of a printed circuit board (PCB).

In the past, handwork wiring led to frequent failures of the wire junctions and a short circuit as
According
the wire insulation to thebegan recent trend
to age. of miniaturized
Furthermore, and ofhigh-performance
this type work was conducted electronic
manually devices,
for
the demand for PCBs has increased substantially. The current
inner-connecting components within the board. Such wiring, therefore, required a significant PCB market has shown an average
annual growth
amount of timerate andofeffort.
4%. Based on increasing demand,
With advancements electronics
in technology, PCBsmanufacturers
were developed require perfect
for efficient
quality and a high level of accuracy from their PCB inspections
and automated production. Because PCBs allow for the use of mass production, they allow devices to assure their competitive edge.
To be
to meet theseand
smaller quality requirements,
lighter. In addition, PCBtheymanufacturers
allow a high conduct an inspection
level of reliability at lowbefore proceeding
production to
costs.
the main process. The PCB manufacturing
Because of these advantages, most electronic devices use PCBs. process consists of many steps: cutting, inner layer etching,
an automatic
According optical
to theinspection
recent trend (AOI), of lay-up, lamination,
miniaturized etching, drilling, solder
and high-performance masking,
electronic devices,routing,
the
a bare board test (BBT), quality control, packing, and shipping,
demand for PCBs has increased substantially. The current PCB market has shown an average in that order. Before proceeding to
the main processes, such as lamination and etching, faults in a
annual growth rate of 4%. Based on increasing demand, electronics manufacturers require perfect PCB are detected during the AOI
stage. An
quality andAOIa high is an automated
level of accuracy visual
from inspection
their PCBusing an image
inspections comparison
to assure method. Most
their competitive edge.PCBTo
manufacturers
meet these quality inspect their PCBsPCB
requirements, through the AOI process.
manufacturers conductThe an types of defects
inspection beforedetected
proceeding during
to
the AOI
the mainprocess
process. include
The PCB scratches, improper process
manufacturing etching, consists
and openofcircuits. In particular,
many steps: cutting,scratches
inner layerare
fatal defects
etching, because they
an automatic optical have the potential
inspection (AOI),tolay-up,
changelamination,
the electrical properties
etching, andsolder
drilling, can result in a
masking,
malfunction of the completed product.
routing, a bare board test (BBT), quality control, packing, and shipping, in that order. Before
There are
proceeding to typically
the mainthree methods
processes, applied
such during theand
as lamination AOIetching,
process:faults
a comparison
in a PCBreference
are detected(CR),
non-reference
during the AOI verification
stage. An (NV),
AOIand is aanhybrid approach
automated (HA).
visual The CR method
inspection using ancompares
image an inspected
comparison
image with a reference image. It measures any existing dissimilarities
method. Most PCB manufacturers inspect their PCBs through the AOI process. The types of defects between the reference image and
the inspected
detected during image. theThus,
AOI this method
process requires
include a reference
scratches, image, and
improper inspection
etching, and is difficult
open without
circuits. In
such an image. The NV approach tests the design rule of the PCB
particular, scratches are fatal defects because they have the potential to change the electrical for detecting faults. It essentially
verifies theand
properties widths of thein
can result insulators
a malfunction and conductors. However,
of the completed this approach makes it difficult for
product.
usersThere
to design
are the rules as
typically restrictions
three methods onapplied
the image features,
during the andAOIitprocess:
cannot detect faults without
a comparison such
reference
rules [1]. The HA approach combines various types of CR and
(CR), non-reference verification (NV), and a hybrid approach (HA). The CR method compares an NV methods. That is, the approach
utilizes a reference
inspected image with image or design image.
a reference rules toItinspect
measuresa PCB any image.
existingHowever, it still does
dissimilarities not solve
between the
the limitation of being unable to conduct an inspection without
reference image and the inspected image. Thus, this method requires a reference image, and reference information. Because the
circuits used in a PCB are diverse and complex, the reference information
inspection is difficult without such an image. The NV approach tests the design rule of the PCB for increases significantly,
thereby decreasing
detecting the inspection’s
faults. It essentially verifies efficiency.
the widths Therefore, additional
of the insulators studies
and are required
conductors. However, to detect
this
faults without reference information.
approach makes it difficult for users to design the rules as restrictions on the image features, and it
cannot Numerous
detect faults algorithms
without such haverulesbeen[1]. proposed
The HA to improve
approach the accuracy
combines variousoftypesa PCBof CR inspection.
and NV
Such algorithms can be categorized into two approaches: referential
methods. That is, the approach utilizes a reference image or design rules to inspect a PCB image. and non-referential methods.
However, it still does not solve the limitation of being unable to conduct an inspection without
reference information. Because the circuits used in a PCB are diverse and complex, the reference
information increases significantly, thereby decreasing the inspection’s efficiency. Therefore,
additional studies are required to detect faults without reference information.
Numerous algorithms have been proposed to improve the accuracy of a PCB inspection. Such
Appl. Sci. 2018, 8, 932 3 of 14

A referential method is based on a comparison between the inspected image and a reference
image. To measure the dissimilarity between these two images, image subtraction and template
matching techniques are used. Wu et al. proposed an inspection method based on a subtraction
method [2]. The image subtraction method compares both images using an XOR logic operator [3].
The resulting image, which is obtained after this operation, contains only portions of a fault [4].
Therefore, the reference image and the inspected image should be placed in a fixed position to compare
both images. In addition, a reference image of the same size is required. Template matching is a
technique for identifying the parts of the image that match the reference image. This method extracts
the features of both the reference and inspected images. It then calculates the similarities of these
features. One of the major disadvantages of template matching is that a large amount of information
regarding the reference image must be used. Therefore, this method requires a mass storage device
that can store all of the information. Moreover, the inspected images also have to be precisely matched
for a comparison with the reference image [5]. Acciani et al. suggested an inspection algorithm that
extracts the wavelet and geometric features, and then detects a defect after learning the fault pattern
using a neural network and k-nearest neighbors [6]. When extracting the features of a PCB image,
this method uses the maximum value of the correlation coefficients between the features of the reference
image and the inspected image. This approach of applying machine-learning algorithms is called
the learning-based model. The aim of this approach is to automatically detect faults through pattern
recognition [7]. In [8], the authors proposed a defect detection method using feature matching for
non-repetitive patterned images. The method first extracts features of both the reference image and the
inspected image using a modified corner detector. It then detects a fault by finding a correspondence
between two feature sets. Thus, the methods [6,8] still require a reference image to detect a fault.
A non-referential method, on the other hand, does not require a reference image. However,
this method uses design rules. If the inspected image does not conform to the design specification
standards, it is considered defective. Ye and Danielson suggested a verifying algorithm for minimum
conductive and insulator trace widths [9]. This algorithm uses morphological techniques, which are
methods for processing binary and grayscale images based on shape. Morphological techniques do not
require a predefined model of a perfect pattern because we can construct specific shapes in an image
by choosing an appropriate neighborhood shape [10]. However, this method has a disadvantage in
that we should apply different pre-processing algorithms to check for faults in a PCB. In addition,
it automatically increases the inspection time [5]. Tsai et al. proposed a non-referential defect detection
approach for bond pads [11]. This approach restores the shape of the bond pads using Fourier image
reconstruction. The method then evaluates the similarities and differences in the pad shape. However,
the method has difficulty in reconstructing an original image of a PCB in the absence of a reference
image because the shape of a circuit is quite varied and complex. Thus, the method is not suitable for
detecting faults on a PCB. As mentioned earlier, a non-referential image analysis has certain limitations.
Rosten and Drummon proposed a very fast and high-quality corner detector using
machine-learning techniques [12]. This algorithm uses a decision tree to learn the properties of
the corner points and determine which point is a corner point instead of directly counting the
number of consecutive points of the same type. Pernkopf proposed an approach for the detection of
three-dimensional faults on scale-covered steel surfaces [13]. After extracting features, the approach
uses a Bayesian network to learn the feature property and classify the faults. Classification research,
which is learning the patterns of the extracted features through image processing, is actively carried
out to diagnose skin cancer not only in manufacturing but also in dermatology [14]. It is indispensable
to extract meaningful features that can explain the properties of shape and color in order to effectively
detect a melanoma [15]. Roberta suggested a method to find meaningful features by combining three
characteristics of shape, texture, and color [16].
Based on the concept of a learning property, we propose a new non-referential method for
fault detection by extracting features based on an image-processing method and learning the
fault information using random forests. The proposed method utilizes features obtained through
Appl. Sci. 2018, 8, 932 4 of 14

speeded-up robust features (SURF) to describe the fault information. Therefore, the method can detect
a fault quickly and accurately without a reference image and without being affected by environmental
changes, such as the size, rotation, and location of the PCB. The proposed method first extracts
robust features using an image-processing technique and learns the fault pattern using an efficient
classification technique utilizing high-dimensional data. We then calculate the probability and draw a
weighted kernel density estimation (WKDE) map weighted by the probability and identify whether a
fault has occurred.
The remainder of this paper is organized as follows: In Section 2, the background of the proposed
method is briefly reviewed. Section 3 describes the procedure of the proposed inspection algorithm.
Section 4 presents the experimental results through a comparison of the receiver operating characteristic
(ROC) curves. Finally, some concluding remarks are given in Section 5.

2. Background of the Proposed Method

This section describes the two algorithms used in the proposed method. We first introduce SURF,
which is used for extracting the features of the PCB. These features include important information
in the image, such as the size, angles, coordinates, and color. By digitizing the image, they enable
calculations related to the pattern recognition to be applied during image processing. We then describe
the random forests used to learn the fault pattern and calculate the probability.

2.1. Speeded-Up Robust Features (SURF)

SURF first digitizes a PCB image into a vector to enable computational calculations through image
processing for learning the fault pattern. Among the various types of image processing techniques,
including scale invariant feature transform (SIFT) and orientation by intensity centroid (ORB), we use
SURF, which is the most robust algorithm with regard to environmental factors such as size, shape,
and color. The performance of SIFT does not differ significantly compared to that of SURF. However,
SIFT is not able to extract the features in a small defective area. ORB is specialized in extracting the
features of a rounded or curved edge, rather than a straight line, in a circuit. Because a PCB is mainly
composed in a straight formation, ORB is inappropriate for extracting PCB features. On the other hand,
SURF is robust and useful for feature detection. It also improves the computational speed compared
to SIFT [17]. In addition, SURF provides robust features and distinctive descriptors to the size and
rotation transform. Thus, it results in a strong performance regardless of the fault size and shape of the
circuit. For this reason, the proposed method applies SURF for the feature extraction. The following
subsections describe the SURF algorithm, which has two phases. First, a point of interest is detected in
an image using a Hessian matrix. In the second phase, SURF generates descriptors with 64 dimensions,
which describe the characteristics of the features.

2.1.1. Interest Point Detection Based on a Hessian Detector

SURF uses a Hessian matrix for detecting interest points, which represent the characteristics of
the image and include useful information for identification, including corner points. An integral image
is used in an approximated Hessian matrix to improve the calculation speed of box-type convolution
filters. The integral image J ( x, y) at coordinate x = ( x, y) x = ( x, y) is the sum of all pixels within
a rectangular area formed by the origin and coordinate x in input image I, the equation of which is
as follows:
i ≤ x j≤y
J ( x, y) = ∑ ∑ I (i, j). (1)
i =0 j =0

Given point x = ( x, y) in input image I, a Hessian matrix H ( x, σ ) at a scale of σ is defined through

Equation (2). " #
L xx ( x, σ ) L xy ( x, σ )
H ( x, σ ) = , (2)
L xy ( x, σ ) Lyy ( x, σ )
Appl. Sci. 2018, 8, 932 5 of 14

where L xx ( x, σ), L xy ( x, σ ), and Lyy ( x, σ ) represent the convolution of the Gaussian second-order
derivative space with the image at coordinate x = ( x, y).
Bay et al. proposed the use of 9 × 9 box filters to approximate the second-order Gaussian partial
derivatives and rapidly compute the image convolutions using integral images [17]. An approximation
of the second-order derivatives is denoted by Dxx ( x, σ ), Dxy ( x, σ ), and Dyy ( x, σ. Bay et al. suggested
using a σ of 1.2, which is the lowest scale [17]. Therefore, the determinant of an approximated Hessian
matrix is calculated using Equation (3).


2
det Happrox ( x, σ ) = Dxx ( x, σ ) Dyy ( x, σ ) − 0.9Dxy ( x, σ ) . (3)

To obtain the robust features to scale, SURF forms a pyramid scale space with various scales on
the original image. The size of the box filter expands along with the pyramid scale space while the size
of the original image remains fixed. Consequently, we can have an effect of enlarging or reducing the
size of the image without scaling the original. Moreover, we can apply box filters of any size at the
same speed directly to the original image, which can improve the computational speed.
After the determinant of the approximated Hessian matrix is calculated at each scale,
non-maximum suppression in a 3 × 3 × 3 neighborhood is applied to find the maxima, which describe
the edge of the features best. The maxima are then interpolated in terms of both the scale and image
space [18]. Finally, we can detect the stable location of the feature through one of the maxima.

2.1.2. Descriptor Generation

After the features are detected as mentioned above, SURF generates a descriptor, which describes
the characteristics of the features, such as the shape and color, using the sum of the Haar wavelet
responses [19]. The Haar wavelets enable SURF to increase the robustness and decrease the
computational time. To generate the descriptor, the first step is constructing a square region around the
points of interest and assigning a reproducible orientation based on the orientation-selection method
introduced in [17]. The region is then split equally into 4 × 4 sub-regions to retain some of the spatial
information as shown in Figure 2. In each sub-region, we compute the Haar wavelet responses at
regularly 5 × 5 spaced sample points. Bay et al. suggested that the level of performance is best at
the each of the above sizes [17]. The wavelet responses are calculated in the x- and y-axis directions
(d x and dy ) and summed over each sub-region. The wavelet responses are then weighted with a
Gaussian centered on the point of interest to increase the robustness toward geometric deformations
and localization errors. Moreover, to obtain information regarding the polarity of the intensity changes,
 
the absolute values of the responses |d x | and dy  are also summed. Thus, each sub-region has a
four-dimensional descriptor vector as shown in Equation (4).

v =  ∑ d x , ∑ d y , ∑ | d x |, ∑  d y  .
  
(4)

As a result, the overall descriptor has a 64-dimension vector because 4 vectors are created for each
4 × 4 sub-region. Figure 2 shows a descriptor vector obtained by summing the wavelet response in
a sub-region.
Finally, the descriptor vectors for each sub-region are normalized to reduce the impact on the
environment, such as an external light or illumination. Eventually, we can obtain the robust features
and descriptors of the PCB image using SURF. The descriptor for a single feature is generated, as shown
in Figure 3, where m is the number of extracted features.
Appl. Sci. 2018, 8, 932 6 of 14
Appl. Sci. 2018, 8, x FOR PEER REVIEW 6 of 15

[interest point]
Appl. Sci. 2018, 8, x FOR PEER REVIEW 6 of 15
[sub-region]

[interest point] Calculate the Haar wavelet

[sub-region] responses for each of 25 points
Calculate the Haar wavelet
dx dy responses for each of 25 points
Sum of all the Haar wavelet responses
dx dy
Sum of all the Haar wavelet responses

 d  d  | d | | d
x y x y |

Construct 4 4 square sub-regions

 d  d  | d | | d
x y x y |
around the detected
Construct interest
4 4 square point
sub-regions
around the detected interest point
Figure 2. Diagram
Figure 2. Diagram forfor generating
generating the
the descriptors.
descriptors.
Figure 2. Diagram for generating the descriptors.

Figure 3. Features and descriptors obtained in a PCB image using speeded-up robust features
Figure 3.3.Features
Figure Featuresand descriptors
and obtained
descriptors in a in
obtained PCBa image using speeded-up
PCB image robust features
using speeded-up (SURF).
robust features
(SURF).
(SURF).
2.2. Random Forests
2.2. Random Forests
2.2. Random Forests
Random Randomforests is a classification
forests is a classification method methodproposed
proposedby Breiman
by Breiman and andis a type of ensemble
is a type of ensemble learning
Random
learning forests
method is
that a classification
constructs a method
multitude of proposed
decision by
trees
method that constructs a multitude of decision trees and combines the predictions from them [20]. Breiman
and combines and is
the a type
predictionsof ensemble
from
learning
The themmethod
random [20]. The
forests thatrandom
method forests
constructs amethod
multitude
is relatively is relatively
accurate and accurate
of decision fast trees and
andfast for high-dimensional
combines
for high-dimensional the predictions
data. data.from
Furthermore,
them
it Furthermore,
[20]. overfitting
prevents The random it prevents
forests by
problems overfitting
method problems
votingisonrelatively by voting
multiple accurate on
trees and and multiple
shows fastgood trees and
for predictive shows
high-dimensional good
performance data.
predictive itperformance
Furthermore, prevents for noisy data.
overfitting Becausebythevoting
problems descriptoron vector obtained
multiple trees from shows
and SURF isgood
for noisy data. Because the descriptor vector obtained from SURF is composed of high-dimensional
composed of high-dimensional data, the algorithm is efficient for feature classification. Therefore,
predictive
data, performance
the algorithm for noisy
is efficient data.classification.
for feature Because theTherefore,descriptorwevector use randomobtained forestsfrom SURFthe
to learn is
we use random forests to learn the fault pattern and calculate the probability of a fault feature being
composed
fault pattern of high-dimensional data, the algorithm is
and calculate the probability of a fault feature being classified. efficient for feature classification. Therefore,
classified.
we useThis random
algorithm forests
This algorithm to
begins learn
beginsby by the
drawingfaultmany
drawing pattern and calculate
manybootstrap
bootstrap samples
samples the probability
from
from thetraining
the of set.
training a fault
set. feature being
Classification
Classification
classified.
trees are then built for each bootstrap sample using a decision tree
trees are then built for each bootstrap sample using a decision tree learning method. After learning method. After thetheforests
This
are formed, algorithm
forests the
are newformed,begins
datathe by
arenew drawing
putdata
intoare many
each put bootstrap
classification samples from the
tree for classification.
into each classification training set.
Each treeEach
tree for classification. Classification
votestreeon the
trees votes
result are then
of the ontree’sbuilt
the resultforofeach
decision the bootstrap
tree’s
regarding decisionsample
the class regarding using
of data. the
Theaclass
decision
of data.
forests tree
then The learning
forests
predict method.
thethen
classpredict After
by taking the
the the
forests class
are by taking
formed, the majority
new data vote
are from
put the
into classification
each trees
classification
majority vote from the classification trees [21]. Because the random forests method generates many [21].
treeBecause
for the random
classification. forests
Each tree
votes method
on the generates
classification resultusing
trees ofmany
the classification
tree’s
bootstrapping decision trees using
than bootstrapping
regarding
rather the class
pruning, weof rather
candata. than
obtain pruning,
Thelow forests we
then
variation cantrees.
obtain
predict the
It can
low
class by avoidvariation
takingan thetrees. It can
majority and therefore
votethereby avoid
from the an overfitting
classification and thereby improve
trees [21].However,
Because the the performance.
theperformance
random forests
therefore overfitting improve the performance. can
However, the performance can decrease when the data are extremely imbalanced. The class of PCB
method generates many classification trees using bootstrapping
decrease when the data are extremely imbalanced. The class of PCB features is mostly normal, and the rather than pruning, we can obtain
features is mostly normal, and the number of fault features is low. Thus, the performance inevitably
low variation
number of fault trees. It can therefore avoid an overfitting and thereby improve the performance.
decreases. Tofeatures
complement is low. this Thus, the performance
disadvantage, we suggest inevitably
utilizing thedecreases.
probability of Tobeing
complement
classified this
However,
disadvantage, the performance
we suggest can
utilizing decrease
the when
probability the data
of beingare extremely
classified
as a fault rather than predicting the class using the random forests and by considering the density as imbalanced.
a fault rather The
than class of PCB
predicting
features
the class is mostly
using
of the normal,
the random
features. and
By considering the
foreststhenumber
anddensity, of fault
by considering features is low.
the density
the probability Thus,
of beingof the performance
the features.
a defective feature inevitably
Byincreases,
considering
decreases.
the density,
whereasTo thecomplement
probabilitythis
theprobability disadvantage,
ofofbeing normalwe
being aadefective suggest
feature
feature utilizing
increases,
decreases. the probability
whereas
Therefore, theproposed
the of being
probability classified
of being
method
a normal feature decreases. Therefore, the proposed method detects faults more effectivelydensity
as a fault rather
detects faults than
more predicting
effectively the
thanclass
a using
method the random
using only forests
random and
forests.by considering
We next the
describe the
than a
of the features. By considering the density, the probability of
method using only random forests. We next describe the process for calculating the probability of thebeing a defective feature increases,
whereas the probability of being a normal feature decreases. Therefore, the proposed method
detects faults more effectively than a method using only random forests. We next describe the
Appl. Sci. 2018, 8, 932 7 of 14
Appl. Sci. 2018, 8, x FOR PEER REVIEW 7 of 15

test datafor
process being classifiedthe
calculating as probability
a fault whenofusing random
the test forests.
data being Then, inasSection
classified a fault4,when
we describe how to
using random
use the calculated probability to detect a fault.
forests. Then, in Section 4, we describe how to use the calculated probability to detect a fault.
probability p((y =
The probability
The = k)) ofofbeing classifiedasask is pis(y =
beingclassified ( k=) =) π
=k , where 0 ≤0 π
, where ≤k ≤ ≤ 1. 1.Here, n
Here,
is the number of classes. The density function
is the number of classes. The density function f k ( x ) for the features in class k is
( ) for the features in class f k ( x ) = f ( x | y
is ( ) =.= k )
Thus, the density function f ( x ) for all features is calculated through Equation (5)
( | = ). Thus, the density function ( ) for all features is calculated through Equation (5) [22]. [22].

( ) = ∑n ( ). (5)
f ( x ) = ∑ π k f k ( x ). (5)
Finally, we can estimate the probability thatk=the 1 class of new data will be predicted as
using a Bayesian probability. The equation of this probability is denoted through Equation (6).
Finally, we can estimate the probability that the class of new data xnew will be predicted as k using
a Bayesian probability. The equation (of this |probability ( )
= = ) =is denoted through Equation (6). (6)
( )
πk f k ( xnew )
Thus, the probability = k| xfeature
thatp(ay new = xnew ) =will be classified as a fault or as normal (6)
is
f ( xnew )
calculated using Equation (7).
Thus, the probability pi that a new feature( ) xi will be classified as a fault or as normal is calculated
using Equation (7). , ( = | = )
( )
= 

π F f ( xi ) (7)
f ( x(i ) )
, p(y = Fault| x = xi )
pi = 1− , ( = | = ) (7)
 1 − π F( f ( x)i ) , p(y = Normal | x = xi )
f (x ) i

3. Proposed
3. Proposed Method
Method
Based on
Based on the
the algorithms
algorithms described
described in in Section
Section 2,
2, we
we propose
propose aa scratch
scratch fault
fault detection
detection method,
method,
which isisdepicted
which depictedininFigure 4. First,
Figure SURF
4. First, extracts
SURF the features
extracts of the of
the features PCB theimage. The extracted
PCB image. features
The extracted
are composed of a multivariate vector representing the properties found in the PCB image,
features are composed of a multivariate vector representing the properties found in the PCB image, such as an
edge, blob, or curve. Because SURF also extracts the features in a scratch-defective edge,
such as an edge, blob, or curve. Because SURF also extracts the features in a scratch-defective edge, it is difficult
toisdistinguish
it difficult tobetween a normal
distinguish betweenedge and a fault
a normal edgeedge
andusing
a faultonly
edgethese
using features.
only these features.

SURF based feature extraction and class assignment

Learning the fault pattern and

calculating the probability based on random forests

Generation of WKDE map by weighting the probability

Classifying the fault area with the threshold

Figure
Figure 4.
4. Flowchart of the
Flowchart proposed
of the PCB PCB
proposed inspection method.
inspection WKDE, WKDE,
method. weighted kernel density
weighted kernel
estimation.
density estimation.

Hence, we assign a normal or fault class to the features using the coordinates of the features
Hence, we assign a normal or fault class to the features using the coordinates of the features within
within a defective or normal area. We then learn the normal and fault properties using the
a defective or normal area. We then learn the normal and fault properties using the multivariate vector
multivariate vector and feature class and calculate the probability of each feature being classified as
and feature class and calculate the probability of each feature being classified as a fault class. We then
a fault class. We then generate a new weighted kernel density estimation (WKDE) map. Kernel
generate a new weighted kernel density estimation (WKDE) map. Kernel density estimation (KDE)
density estimation (KDE) is a technique for estimating the density based on the coordinates. That is,
is a technique for estimating the density based on the coordinates. That is, KDE is a technique that
KDE is a technique that considers the density of the features. To maximize the effectiveness of the
considers the density of the features. To maximize the effectiveness of the features, the probability of
features, the probability of affecting the spatial density is given as a weight to the KDE. Thus, we
affecting the spatial density is given as a weight to the KDE. Thus, we consider not only the properties
consider not only the properties but also the densities of the features. After generating a WKDE
map, we can predict the fault area based on the WKDE value. The proposed method is described in
more detail in the following subsection.
Appl. Sci. 2018, 8, 932 8 of 14

but also the densities of the features. After generating a WKDE map, we can predict the fault area based
Appl.
on theSci.WKDE
2018, 8, xvalue.
FOR PEER
TheREVIEW
proposed 8 of 15
method is described in more detail in the following subsection.

3.1. SURF-Based Feature Extraction and Class Assignment

As the first step of the proposed method, SURF digitizes a PCB image as a vector. vector. SURF then
Appl. Sci. 2018, 8, x FOR PEER REVIEW 8 of 15
detects the features and extracts the descriptors in the form of a vector as mentioned in the previous
section. Figure
3.1.Figure
SURF-Based 5ashows
5a shows anan
Feature example
example
Extraction of of
andan an original
original
Class PCBPCB
Assignment image.
image. The circuits
The circuits in thein theinside
area area inside the
the dotted
dotted
line haveline have a defective
a defective edge, the
edge, whereas whereas
other the other
circuits circuits
have a normalhaveedge.
a normal edge. The
The extracted extracted
features are
As the first step of the proposed method, SURF digitizes a PCB image as a vector. SURF then
features are shown in the circles in Figure 5b. As Figure 5b illustrates, the features
shown in the circles in Figure 5b. As Figure 5b illustrates, the features are obtained identically without are obtained
detects the features and extracts the descriptors in the form of a vector as mentioned in the previous
identically
discriminating without
between discriminating
a defective edgebetween
and aanormal
defective edge
edge. and awords,
In other normal edge.
SURF In other
cannot words,
distinguish
section. Figure 5a shows an example of an original PCB image. The circuits in the area inside the
SURF dotted line have a defective edge, whereas the other circuits have a normal edge. The extracted fault
between cannot
a distinguish
defective and abetween
normal a defective
edge. Thus, and
we a normal
should edge.
learn theThus,
fault we should
pattern of learn
the the
features by
pattern
setting of the
the
features features
normal
are andby
shown in setting
fault
theareas the
and
circles normal
in Figureand
assigning fault areas
5b. classes
As and
to each
Figure 5b assigning
feature.
illustrates, theclasses
featurestoare
each feature.
obtained
identically without discriminating between a defective edge and a normal edge. In other words,
SURF cannot distinguish between a defective and a normal edge. Thus, we should learn the fault
pattern of the features by setting the normal and fault areas and assigning classes to each feature.

(a) (b)
Figure 5. An
An example
exampleofofSURF-based
SURF-basedfeature
feature extraction
extraction (a) (a) Original
Original image; (b) A resultant image
Figure 5. (a) (b) image; (b) A resultant image after
after extracting the features of the
extracting the features of the PCB. PCB.
Figure 5. An example of SURF-based feature extraction (a) Original image; (b) A resultant image
As after extracting
shown in Figurethe features
6a, we ofdefine
the PCB.
a lozenge area with four points so that the fault area can be
As shown in Figure 6a, we define a lozenge area with four points so that the fault area can be
visually represented in the clearest manner. We can then consider the fault occurring in the lozenge
visually represented
As shown in in the clearest
Figure manner.
6a, we define We can
a lozenge areathenwith consider theso
four points fault
thatoccurring
the fault areain the
canlozenge
be
area. visually
We assign the features
represented in the within
clearest the lozenge
manner. We areathen
can in Figure
consider 6bthetofault
the occurring
fault class.in Thelozenge
the remaining
area. We assign the features within the lozenge area in Figure 6b to the fault class. The remaining
features
area.from areas the
We assign other than within
features the lozenge area are
the lozenge designated
in Figure 6bas
areadesignated to being of class.
the fault a normal class through
The remaining
features from areas other than the lozenge area are as being of a normal class through
Equation (8).from areas other than the lozenge area are designated as being of a normal class through
features
Equation (8).
Equation (8). (
,i f x ∈∈X
( ) = Fault class,
c ( xi ) = , , ℎ∈
i L
,, (8)
(8)
( ) = Normal class, otherwise , (8)
, ℎ
where indicates each feature composed using a multivariate vector and is a set of features
xi indicates
wherewhere each each
indicates feature composed
feature composed using a multivariate
using a multivariate vector
vectorand X L is aisset
and of of
a set features within
features
within the lozenge area.
the lozenge area.
within the lozenge area.

(a) (b)
(a) (b)
Figure 6. An area selection for a class assignment: (a) A fault occurring in the lozenge area; (b) The
Figure 6. An area selection for a class assignment: (a) A fault occurring in the lozenge area;
Figure 6. Anwithin
features area selection forarea
the lozenge a class assignment:
are the (a) the
fault class and A fault occurring
remaining in are
features thethe
lozenge
normalarea;
class.(b) The
(b) The features within the lozenge area are the fault class and the remaining features are the
features within the lozenge area are the fault class and the remaining features are the normal class.
normal class.the Fault Pattern and Calculating the Probability Based on Random Forests
3.2. Learning
3.2. Learning the the
Given Faultclass-assigned
Pattern and Calculating thefollowing
features, the Probability Based
step on handle
is to Randomfault
Forests
patterns using a
machine-learning algorithm. This method is based on the random forests method, which is a
Given the class-assigned features, the following step is to handle fault patterns using a
well-known ensemble learning model for classification. We first learn the fault pattern of the
machine-learning algorithm. This method is based on the random forests method, which is a
well-known ensemble learning model for classification. We first learn the fault pattern of the
Appl. Sci. 2018, 8, 932 9 of 14

3.2. Learning the Fault Pattern and Calculating the Probability Based on Random Forests
Given the class-assigned features, the following step is to handle fault patterns using a
machine-learning algorithm. This method is based on the random forests method, which is a
well-known ensemble learning model for classification. We first learn the fault pattern of the features.
We use a 136-dimension input vector, Xi , composed using extended descriptors and the information of
each feature, such as its location and class, for a more precise description. The extended descriptors
 
are computed by summing d x and |d x | separately for dy < 0 and dy ≥ 0. Similarly, ∑ dy and ∑dy 
are split for d x < 0 and d x ≥ 0. Thus, the number of vector dimensions of the descriptors doubles
to 128. After building the learning model by training the properties of the features, we can find the
probability pi that each feature of an inspected image will be classified as a fault, which is calculated
using Equation (7). Because the classes of the features are extremely imbalanced, most features are
classified as normal. That is, the probability that the features will be estimated as a fault is considerably
less than the probability that they will be predicted as normal. Consequently, because it is difficult
to detect a fault area using only the results of the random forests, we should consider the density in
addition to the map of the probability of being predicted as a fault.

3.3. Generation of WKDE Map by Weighting the Probability

This section describes how to apply the probability calculated in the previous section to generate a
weighted kernel density estimation (WKDE) map, a technique that reflects the kernel density estimation
(KDE). KDE is a probability density estimation method using a kernel function. In addition, it is a
nonparametric method that is even applicable to high-dimensional data [23]. The representative kernel
functions include Gaussian, uniform, and Epachenikov functions [24]. This paper uses a Gaussian
kernel function to estimate the density.
After calculating the probability, we generate a WKDE map by giving the weight, wi , to the kernel
function in order to complement any disadvantages as mentioned earlier. Because the PCB features
have x- and y-axis coordinates, this method uses the weighted multivariate kernel density estimation.
Let { x1 , y1 }, { x2 , y2 }, · · · , { xm , ym } be the coordinates of the PCB features. The WKDE value is
denoted through Equation (9) [25].
m
( x, y) − ( xi , yi )
 
1
fˆ( x, y) =
mh ∑ wi K h
, (9)
i =1

where m is the total number of PCB features, h is a scaled kernel that controls the smoothness of
the estimate, and wi is the probability of being classified as a fault. In addition, K is a Gaussian
kernel function.
KDE and WKDE maps were drawn using the features extracted from the PCB image in Figure 5
as shown in Figure 7. The x and y axes in the KDE and WKDE maps represent the coordinates of
the features. The z axis of the KDE map indicates the KDE value, which is calculated based on the
coordinates of each feature, whereas the z axis of the WKDE map is the WKDE value obtained by
weighting the probability to the KDE. Because the KDE considers only the density, it has a high KDE
value in areas where many features are extracted regardless of the defective area as shown in Figure 7a.
Consequently, it is difficult to distinguish between a normal area and a fault area on the KDE map.
On the other hand, the WKDE value, which reflects the probability, increases when the probability of
each feature being classified as a fault is higher than the probability of the other features. Otherwise,
the WKDE value decreases because there is a relative difference between the probabilities. The area
marked with a circle in Figure 7b has a relatively high WKDE value owing to the weight. That is,
because the marked area is densely concentrated with fault features, the WKDE value of this area
is higher than the KDE value. In addition, the KDE values in the remaining areas are randomly
distributed, but decrease significantly after being weighted. Thus, we can predict the marked area as a
fault area.
Appl. Sci. 2018, 8, 932 10 of 14

Appl. Sci. 2018, 8, x FOR PEER REVIEW 10 of 15

Figure 8 shows a three-dimensional (2-D) WKDE map overlapping the original image (Figure 5a).
In the 3-D WKDE map, it is difficult to intuitively recognize the coordinates of a feature. Thus, we
convert the 3-D image into a two-dimensional (2-D) image to prove that the marked area in Figure 7b
is consistent with the real fault area. The area within the dotted line in Figure 8 indicates where the
actual faults occur, whereas the shaded area is an area predicted as a fault because the WKDE value
exceeds the threshold. As Figure 8 shows, we can confirm that the shaded area matches the dotted
area. Therefore, we monitor the WKDE value and predict the features as being a fault if the WKDE
value exceeds the threshold. In this paper, we set a WKDE value of higher than 1% as the threshold.
However, the threshold
Appl. Sci. 2018, changes
8, x FOR PEER REVIEW according to the process yield or to past experience. 10 of 15

(a) (b)
Figure 7. The kernel density estimation (KDE) and WKDE map in three-dimensions (3-D) for a PCB
image: (a) KDE map; (b) Proposed WKDE map.

Figure 8 shows a three-dimensional (2-D) WKDE map overlapping the original image (Figure
5a). In the 3-D WKDE map, it is difficult to intuitively recognize the coordinates of a feature. Thus,
we convert the 3-D image into a two-dimensional (2-D) image to prove that the marked area in
Figure 7b is consistent with the real fault area. The area within the dotted line in Figure 8 indicates
where the actual faults occur, whereas the shaded area is an area predicted as a fault because the
WKDE value exceeds the(a) threshold. As Figure 8 shows, we can confirm(b) that the shaded area
matches the dotted area. Therefore, we monitor the WKDE value and predict the features as being a
Figure 7. The kernel density estimation (KDE) and WKDE map in three-dimensions (3-D) for a PCB
fault if the7. WKDE
Figure value
The kernel exceeds
density the threshold.
estimation In this
(KDE) and WKDEpaper,
map we
inset a WKDE value(3-D)
three-dimensions of higher than 1%
for a PCB
image: (a) KDE map; (b) Proposed WKDE map.
as image:
the threshold. However,
(a) KDE map; the threshold
(b) Proposed WKDE map. changes according to the process yield or to past
experience.
Figure 8 shows a three-dimensional (2-D) WKDE map overlapping the original image (Figure
5a). In the 3-D WKDE map, it is difficult to intuitively recognize the coordinates of a feature. Thus,
we convert the 3-D image into a two-dimensional (2-D) image to prove that the marked area in
Figure 7b is consistent with the real fault area. The area within the dotted line in Figure 8 indicates
where the actual faults occur, whereas the shaded area is an area predicted as a fault because the
WKDE value exceeds the threshold. As Figure 8 shows, we can confirm that the shaded area
matches the dotted area. Therefore, we monitor the WKDE value and predict the features as being a
fault if the WKDE value exceeds the threshold. In this paper, we set a WKDE value of higher than 1%
as the threshold. However, the threshold changes according to the process yield or to past
experience.
Figure 8. Two-dimensional (2-D) WKDE map on an original image.
Figure 8. Two-dimensional (2-D) WKDE map on an original image.

4. Experimental Results
4. Experimental Results
The present experiment used PCB image data to verify the proposed method. PCB images
The present experiment used PCB image data to verify the proposed method. PCB images were
were collected from a battery manufacturer in Korea. We obtained 10 PCB images by selecting a
collected from with
fault image a battery manufacturer
a particular scratchinproperty.
Korea. We Asobtained
shown in 10 Figure
PCB images
9, the by selecting
scratch a fault
fault, whichimage is
with
indicated by the dotted line, has a fine and thin form, indicating that the circuit may be broken.the
a particular scratch property. As shown in Figure 9, the scratch fault, which is indicated by
dotted line,ahas
Because a fine
scratch and thin
defect mayform, indicating
prevent a currentthat
flowthe circuitthe
within may be broken.
circuit, it has aBecause a scratch
significant impactdefect
on
may prevent a current flow within the circuit, it has a significant
the PCB quality. Therefore, this paper analyses PCB images with a scratch defect. impact on the PCB quality. Therefore,
this paper
We analyses PCB
first extractedFigureimages with in
the8.features a scratch
Two-dimensional defect.
the 10(2-D)
PCB images
WKDE using
map on anSURF. Weimage.
original then assigned a class to
Wefeature
each first extracted the features
by designating theindefective
the 10 PCB areaimages using SURF.
and organized theWe then assignedinput
136-dimension a classvector
to each
feature by designating
4. Experimental
including Results
information the
suchdefective area and and
as the coordinates organized thedata
class. The 136-dimension
were then dividedinput into
vectortheincluding
test set
and training
information suchsetasfor
thelearning purposes.
coordinates This experiment
and class. The data were applied
thenadivided
10-fold into
crossthevalidation
test set andusing the
training
The present experiment used PCB image data to verify the proposed method. PCB images
set10
forimages.
learning Forpurposes.
example, nine This images are used
experiment appliedfor training,
a 10-fold and thevalidation
cross
were collected from a battery manufacturer in Korea. We obtained 10 PCB images by selecting a
remainingusing imagethe is used
10 for
images.
testing.
Forfault
example,Next, we obtained the WKDE value for each of the 10 PCB images as mentioned in Section
image nine
with images
a particularare used for property.
scratch training, and the remaining
As shown in Figureimage
9, the isscratch
used for testing.
fault, whichNext, is
we3.obtained the WKDE value for each of the 10 PCB images as mentioned
indicated by the dotted line, has a fine and thin form, indicating that the circuit may be broken. in Section 3.
ToTo
Because
diagnose
diagnose
a scratch
a faultarea,
a fault area, statisticalprocess
defect maystatistical
process control
prevent a current flow
control (SPC)
(SPC)
within
can be
thecan beapplied
circuit, applied totomonitor
monitor
it has a significant
the WKDE
the
impact WKDEon
value.
value. SPC is
SPCquality. a method
is a method of quality
of quality control
control that uses
thatPCB statistical
usesimages
statisticalmethods based on the distribution of
the PCB Therefore, this paper analyses with methods based on the distribution
a scratch defect.
We first extracted the features in the 10 PCB images using SURF. We then assigned a class to
each feature by designating the defective area and organized the 136-dimension input vector
including information such as the coordinates and class. The data were then divided into the test set
and training set for learning purposes. This experiment applied a 10-fold cross validation using the
10 images. For example, nine images are used for training, and the remaining image is used for
Appl. Sci. 2018, 8, 932 11 of 14

of data
Appl. [26]. However,
Sci. 2018, 8, x FOR PEER there
REVIEWis no assumption regarding the distribution of the features. 11 of For
15 this

reason, this research cannot utilize SPC. Hence, a binary classification was applied based on a single
data [26]. However, there is no assumption regarding the distribution of the features. For this
continuous WKDE value to examine whether a feature is a fault. The features were classified by
reason, this research cannot utilize SPC. Hence, a binary classification was applied based on a single
comparing the WKDE value with a threshold x ∗ , which is called the cut-off value. A feature was
continuous WKDE value to examine whether a feature is a fault. The features were classified by
classified
comparingas a fault if fˆ( x,value
the WKDE x ∗ and
y) ≥ with was classified
a threshold ∗ as normal
, which otherwise.
is called the cut-off value. A feature was
The appropriate
classified as a fault ifthreshold
( , ) ≥can ∗ be chosen based on various criteria [27]. Thus, we conducted the
and was classified as normal otherwise.
experiments using a variety
The appropriate of thresholds,
threshold can be chosenand adopted
based on an appropriate
various threshold
criteria [27]. Thus, based on a receiver
we conducted
operating characteristic
the experiments using(ROC) curve,
a variety which is a and
of thresholds, tool adopted
for demonstrating
an appropriatethe performance
threshold based of aonclassifier.
a
receiver
An ROC operating
curve characteristic (ROC) curve,
is a threshold-independent which isfor
technique a tool for demonstrating
evaluating the performance
the performance of a modelof and
a classifier.
represents An ROC curve
the relationship is a threshold-independent
between the model’s sensitivitytechnique for evaluating
and specificity [27].the
Inperformance
this study, the of ROC
a model and represents the relationship between the model’s sensitivity and
curve was drawn based on the true- and false-negative rates. The ROC curve for a good model achieves specificity [27]. In this
study, the ROC curve was drawn based on the true- and false-negative rates. The ROC curve for a
a high true-negative rate whereas the false-negative rate is relatively small. Therefore, the performance
good model achieves a high true-negative rate whereas the false-negative rate is relatively small.
of a model can be evaluated by comparing the area under the ROC curve (AUROC). Robust models
Therefore, the performance of a model can be evaluated by comparing the area under the ROC
havecurve
an AUROC
(AUROC). ofRobust
close to 1.0, whereas
models poorer models
have an AUROC of close have
to 1.0,an AUROC
whereas near
poorer 0.5, and
models haveworthless
an
models have a value of less than 0.5 [28].
AUROC near 0.5, and worthless models have a value of less than 0.5 [28].

(a) (b) (c)

(d) (e) (f)

(g) (h) (i) (j)

Figure 9. Example images of the 10 PCB images used: (a)–(j) The area marked with a circle or an
Figure 9. Example images of the 10 PCB images used: (a)–(j) The area marked with a circle or an
ellipsoid on the dotted line is where the scratch-fault occurred.
ellipsoid on the dotted line is where the scratch-fault occurred.
To demonstrate that both the probability and density of the features have to be considered for
detecting a fault, this
To demonstrate that paper
bothcompares the performances
the probability of three
and density of thedifferent methods:
features have to(1)bemonitoring
considered for
only the
detecting probability
a fault, (Method
this paper 1), (2) using
compares a KDE map (Method
the performances of three2), and (3) methods:
different using the (1)
proposed
monitoring
only the probability (Method 1), (2) using a KDE map (Method 2), and (3) using the proposed 1WKDE
WKDE map (Method 3). Each method applies binary classification to detect a fault. Method
mapdetects
(Methoda fault area by monitoring only the probability obtained by the random forests. An area is
3). Each method applies binary classification to detect a fault. Method 1 detects a fault
determined as a fault area when the probability of the feature being a fault exceeds a certain
area by monitoring only the probability obtained by the random forests. An area is determined as
a fault area when the probability of the feature being a fault exceeds a certain threshold. That is,
Appl. Sci. 2018, 8, 932 12 of 14

Appl. Sci. 2018, 8, x FOR PEER REVIEW 12 of 15

this method does not consider the density of the features. Method 2 identifies a defect by considering
threshold. That is, this method does not consider the density of the features. Method 2 identifies a
only the density of the features
defect by considering only thewithout
density taking into account
of the features withoutthe probability.
taking into accountThus,
the it monitors the
probability.
KDE value.
Thus, itFinally,
monitorsMethod
the KDE3, the proposed
value. Finally,method,
Methoddetects a fault bymethod,
3, the proposed monitoring theaWKDE
detects fault byvalue.
This method
monitoringconsiders
the WKDEbothvalue.
the properties
This method of the features
considers and
both thetheir density.
properties of Finally, we draw
the features an ROC
and their
curve density. Finally,
to compare the we draw an ROC
performances of curve
the threeto compare
methods. theFigure
performances
10 showsof the
the three
ROC methods.
curves for Figure
the three
methods. A method in which the AUROC is close to 1.0 detects a PCB fault more precisely. 1.0
10 shows the ROC curves for the three methods. A method in which the AUROC is close to As can
detects
be seen, the aAUROC
PCB faultofmore precisely.
Method As canthan
1 is larger be seen,
thatthe AUROC of
of Method Method
2 for 1 is 10
6 of the larger than that
images. of
However,
Method 2 for 6 of the 10 images. However, the AUROC of Method 1 is less than that of Method 3 for
the AUROC of Method 1 is less than that of Method 3 for all 10 images. Thus, the performance of
all 10 images. Thus, the performance of Method 1 is not proper for detecting faults. Although
Method 1 is not proper for detecting faults. Although Method 2 outperforms Method 1 for certain
Method 2 outperforms Method 1 for certain images, it has a poorer level of performance than
images, it has3.a On
Method poorer level hand,
the other of performance than Method
Method 3 outperforms the3.other
On the other hand,
methods, and itsMethod
AUROC3isoutperforms
close to
the other methods,
1.0 for andTherefore,
all images. its AUROC is close
Method 3 is to 1.0 for allfor
appropriate images.
detectingTherefore,
a scratchMethod
fault on 3a is appropriate for
PCB.
detecting a scratch fault on a PCB.
1.0

1.0
0.8

0.8
True negative rate

True negative rate

0.6

0.6
0.4

0.4
0.2

case1 0.2 case1

case2 case2
case3 case3
0.0

0.0

0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0
False negative rate False negative rate

(a) (b)
1.0

1.0
0.8

0.8
True negative rate

True negative rate

0.6

0.6
0.4

0.4
0.2

0.2

case1 case1
case2 case2
case3 case3
0.0

0.0

0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0
False negative rate False negative rate

(c) (d)
1.0

1.0
0.8

0.8
True negative rate

True negative rate

0.6

0.6
0.4

0.4
0.2

0.2

case1 case1
case2 case2
case3 case3
0.0

0.0

0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0
False negative rate False negative rate

(e) (f)
1.0

1.0
0.8

0.8
True negative rate

True negative rate

0.6

0.6
0.4

0.4
0.2

0.2

case1 case1
case2 case2
case3 case3
0.0

0.0

0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0
False negative rate False negative rate

(g) (h)

Figure 10. Cont.

 Appl.
Appl. Sci. Sci.8,
2018, 2018,
9328, x FOR PEER REVIEW 13 of 13
15 of 14

1.0

1.0
0.8

0.8
True negative rate

True negative rate

0.6

0.6
0.4

0.4
0.2

0.2
case1 case1
case2 case2
0.0 case3 case3

0.0
0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0
False negative rate False negative rate

(i) (j)
Figure 10. Performance comparison of three different methods for 10 scratch-fault images: (a)–(j)
Figure 10. Performance comparison of three different methods for 10 scratch-fault images: (a)–(j) For
For the 10 PCB images, the mean AUROC of the Method 1 is 0.70, the Method 2 is 0.78, and the
the 10Method
PCB images, the mean AUROC of the Method 1 is 0.70, the Method 2 is 0.78, and the Method 3
3 is 0.91. The Method 3 considering both probability and density has better detection
is 0.91.performance
The Methodin 3allconsidering both
10 images than theprobability and
other method density has
considering better
only detection performance in all
one property.
10 images than the other method considering only one property.
5. Conclusions
5. Conclusions
In this paper, we proposed a new non-referential method by learning the fault pattern and
generating
In this paper,a WKDE map. Thisa method
we proposed can learn various
new non-referential fault patterns
method regardless
by learning of thepattern
the fault type of and
defect. Thus, the proposed method allows for a flexible PCB inspection without limitations
generating a WKDE map. This method can learn various fault patterns regardless of the type of
regarding the specific type of fault. Furthermore, it can be extended to deal with unknown faults for
defect. Thus, the proposed method allows for a flexible PCB inspection without limitations regarding
PCB inspection. The performance of the proposed method is demonstrated by comparing the ROC
the specific
curve fortype
theof fault.
three Furthermore,
methods presented itabove.
can beIn extended
addition, itto wasdeal with
found unknown
that consideringfaults
bothfor
thePCB
inspection. The performance of the proposed method is demonstrated by comparing
probability and density of the features is effective for detecting a scratch fault. Thus far, our methodthe ROC curve
for thehas
three methods presented above. In addition, it was found that considering both
dealt only with the 10 scratch faults due to security issues, and it needs to be applied to other the probability
and density of thewith
PCB images features is effective
more data for detecting
and various a scratch
fault patterns. fault. Thus
In addition, using far, our method
a clustering has dealt
technique,
further
only with thework needs faults
10 scratch to be conducted to redefine
due to security issues,the
andfault type when
it needs it is not to
to be applied specified,
other PCBand images
an
appropriate
with more data anddetection
variousmethod
faultfor each type
patterns. Inofaddition,
defect should
usingbeastudied. Although
clustering this paper
technique, used
further work
SURF to detect robust features, other image processing techniques can be applied.
needs to be conducted to redefine the fault type when it is not specified, and an appropriate detection In other words,
the classification accuracy can be further enhanced using another algorithm that includes more
method for each type of defect should be studied. Although this paper used SURF to detect robust
detailed edge information. The classification performance of the proposed algorithm is proved by
features, other image processing techniques can be applied. In other words, the classification accuracy
the ROC curve. In the future, it is necessary to study the threshold selection method and its
can beperformance
further enhanced
evaluationusing
methodanother algorithm
for practical that in
application includes more detailed edge information.
the future.
The classification performance of the proposed algorithm is proved by the ROC curve. In the future,
it is necessary to study the
Author Contributions: E.Y.threshold selection
designed and method
implemented and itstoperformance
the algorithm evaluation
solve the defined method
problem. S.H.P. and for
C.-S.P. performed the experiments
practical application in the future. and data analyses. J.-G.B. validated the proposed algorithm and guided the
whole research. All authors read and approved the final manuscript.
Contributions: E.Y.
AuthorAcknowledgments: designed
This work wasand implemented
supported by thethe algorithm
National to solve
Research the defined
Foundation problem.
of Korea (NRF)S.H.P.
grant and
C.-S.P.funded
performed the
by the experiments
Korea and(MSIP)
government data analyses. J.-G.B. validatedThis
(NRF-2016R1A2B4013678). thework
proposed algorithm
was also andbyguided
supported BK21 the
whole Plus
research. All in
(Big Data authors read andand
Manufacturing approved
Logisticsthe final manuscript.
Systems, Korea University).
Acknowledgments: This work
Conflicts of Interest: was supported
The authors declare noby the National
conflict Research Foundation of Korea (NRF) grant funded
of interest.
by the Korea government (MSIP) (NRF-2016R1A2B4013678). This work was also supported by BK21 Plus (Big Data
in Manufacturing and Logistics Systems, Korea University).
References
Conflicts of Interest: The authors declare no conflict of interest.
1.Malge, P.S.; Nadaf, R.S. A survey: Automated visual PCB inspection algorithm. Int. J. Eng. Res. Technol.
2014, 3, 1. Available online: https://www.ijert.org/browse/volume-3-2014/january-2014-edition?start=20
References
(accessed on 2 June 2018).
1. 2. Wu,
Malge, W.Y.;
P.S.; Wang,
Nadaf, M.J.J.;
R.S. Liu, C.M.
A survey: Automated visual
Automated inspected
PCBof inspection
printed circuit boards through
algorithm. Int. J.machine vision.
Eng. Res. Technol.
Comput.
2014, 3, Ind. 1996,
1. Available 28, 103–111,
online: doi:10.1016/0166-3615(95)00063-1.
https://www.ijert.org/browse/volume-3-2014/january-2014-edition?start=20
3. Sundaraj,
(accessed K. PCB
on 2 June inspection for missing or misaligned components using background subtraction.
2018).
2. WSEAS Trans. Inf. Sci.
Wu, W.Y.; Wang, M.J.J.; Liu, Appl.
C.M.2009, 6, 778–787.inspected
Automated Available online: https://dl.acm.org/citation.cfm?id=1558809
of printed circuit boards through machine vision.
(accessed on 2 June 2018).
Comput. Ind. 1996, 28, 103–111. [CrossRef]
3. Sundaraj, K. PCB inspection for missing or misaligned components using background subtraction.
WSEAS Trans. Inf. Sci. Appl. 2009, 6, 778–787. Available online: https://dl.acm.org/citation.cfm?id=1558809
(accessed on 2 June 2018).
4. Chauhan, A.P.S.; Bhardwaj, S.C. Detection of bare PCB defects by image subtraction method using machine
vision. In Proceedings of the World Congress on Engineering, London, UK, 6–8 July 2011; Volume 2,
pp. 103–111.
Appl. Sci. 2018, 8, 932 14 of 14

5. Moganti, M.; Ercal, F.; Dagli, C.H.; Tsunekawa, S. Automatic PCB inspection algorithms: A survey.
Comput. Vis. Image Understand. 1996, 63, 287–313. [CrossRef]
6. Acciani, G.; Brunetti, G.; Fornarelli, G. Application of neural networks in optical inspection and classification
of solder joints in surface mount technology. Int. IEEE Trans. Ind. Inform. 2006, 2, 200–209. [CrossRef]
7. Huang, S.H.; Pan, Y.C. Automated visual inspection in the semiconductor industry: A survey. Comput. Ind.
2015, 66, 1–10. [CrossRef]
8. Kim, H.W.; Yoo, S.I. Defect detection using feature point matching for non-repetitive patterned images.
Pattern Anal. Appl. 2014, 17, 415–429. [CrossRef]
9. Ye, Q.-Z.; Danielsson, P.E. Inspection of printed circuit boards by connectivity preserving shrinking.
IEEE Trans. Pattern Anal. Mach. Intell. 1988, 10, 737–742. [CrossRef]
10. Elbehiery, H.; Hefnawy, A.; Elewa, M. Surface defects detection for ceramic tiles using image processing and
morphological techniques. WEC 2007, 1, 1488–1492. Available online: https://scholar.waset.org/1307-6892/
15176 (accessed on 2 June 2018).
11. Tsai, D.M.; Su, Y.J. Non-referential, self-compared shape defect inspection for bond pads with deformed
shapes. Int. J. Prod. Res. 2009, 47, 1225–1244. [CrossRef]
12. Rosten, E.; Drummond, T. Machine learning for high-speed corner detection. In Proceedings of the 9th
European Conference on Computer Vision-ECCV, Graz, Austria, 7–13 May 2006; pp. 430–443.
13. Pernkopf, F. Detection of surface defects on raw steel blocks using Bayesian network classifiers. Pattern Anal.
Appl. 2004, 7, 333–342. [CrossRef]
14. Oliveira, R.B.; Papa, J.P.; Pereira, A.S.; Tavares, J.M.R. Computational methods for pigmented skin lesion
classification in images: Review and future trends. Neural Comput. Appl. 2018, 29, 613–636. [CrossRef]
15. Ma, Z.; Tavares, J.M.R. Effective features to classify skin lesions in dermoscopic images. Expert Syst. Appl.
2017, 84, 92–101. [CrossRef]
16. Oliveira, R.B.; Pereira, A.S.; Tavares, J.M.R. Computational diagnosis of skin lesions from dermoscopic
images using combined features. Neural Comput. Appl. 2018. [CrossRef]
17. Bay, H.; Ess, A.; Tuytelaars, T.; Van Gool, L. Speeded-up robust features (SURF). Comput. Vis. Image
Understand. 2008, 110, 346–359. [CrossRef]
18. Su, J.; Xu, Q.; Zhu, J. A scene matching algorithm based on SURF feature. In Proceedings of the
International Conference on Image Analysis and Signal Processing (IASP), Huangzhou, China, 9–11 April
2010; pp. 434–437.
19. Strang, G.; Nguyen, T. Wavelets and Filter Banks, 2nd ed.; Wellesely-Cambridge Press: Wellesley, MA, USA, 1996.
20. Breiman, L. Random forests. Mach. Learn. 2001, 45, 5–32. [CrossRef]
21. Ohn, S.Y.; Chi, S.D.; Han, M.Y. Feature Selection for Classification of Mass Spectrometric Proteomic Data
Using Random Forest. J. Korea Soc. Simul. 2013, 22, 139–147. [CrossRef]
22. Li, C. Probability Estimation in Random Forests. Master’s Thesis, Utah State University, Logan, UT, USA, 2013.
23. Thurstain-Goodwin, M.; Unwin, D. Defining and delineating the central areas of towns for statistical
monitoring using continuous surface representations. Trans. GIS 2000, 4, 305–317. [CrossRef]
24. Kurata, E.; Mori, H. Short-term load forecasting using informative vector machine. Electr. Eng. Jpn. 2009,
166, 23–31. [CrossRef]
25. Anderson, T.K. Kernel density estimation and K-means clustering to profile road accident hotspots.
Accid. Anal. Prev. 2009, 41, 359–364. [CrossRef] [PubMed]
26. Fadel, H.K.; Holloway, L.E. Using SPC and template monitoring method for fault detection and prediction
in discrete event manufacturing systems. In Proceedings of the IEEE International Symposium on Intelligent
Control/Intelligent Systems and Semiotics, Cambridge, MA, USA, 17 September 1999.
27. Kang, S.; Cho, S.; An, D.; Rim, J. Using wafer map features to better predict die-level failures in final test.
IEEE Trans. Semicond. Manuf. 2015, 28, 431–437. [CrossRef]
28. Freeman, E.A.; Moisen, G.G. A comparison of the performance of threshold criteria for binary classification
in terms of predicted prevalence and kappa. Ecol. Model. 2008, 217, 48–58. [CrossRef]

© 2018 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access
article distributed under the terms and conditions of the Creative Commons Attribution
(CC BY) license (http://creativecommons.org/licenses/by/4.0/).