Volume 5, Issue 7, July – 2020 International Journal of Innovative Science and Research Technology

ISSN No:-2456-2165

Steel Surface Defect Detection using Deep Learning

Vira Fitriza Fadli Iwa Ovyawan Herlistiono
Department of Informatics Department of Informatics
Widyatama University Widyatama University
Bandung, Indonesia Bandung, Indonesia

Abstract:- Steel defects are a frequent problem in steel II. RELATED WORKS
companies. Proper quality control can reduce quality
problems arising from steel defects. Nowadays, steel There are many previous algorithms applied to detect
defects can detect by automation methods that utilize steel defects. The algorithms used are various, namely SIFT
certain algorithms. Deep learning can help the steel and Voting Strategy [3], YOLO network [4], ANN, and
defect detection algorithm become more sophisticated. DAN algorithms [5], etc. S. Tian and K. Xu [6] detected
In this study, we use deep learning CNN with Xception steel defects with genetic algorithms and extreme machine
architecture to detect steel defects from images taken learning. This ELM is combined with genetic algorithms to
from high-frequency and high-resolution cameras. improve the results obtained. The results of experiments
There are two techniques used, and both produce with steel defect samples show that the G-ELM algorithm
respectively 0.94% and 0.85% accuracy. The Xception effectively increases the ELM algorithm's identification
architecture used in this case shows optimal and stable accuracy.
performance in the process and its results.
M. S. Kim et al. [7] applied deep learning to classify
Keywords:- Defect Detection, Steel Defect, Deep Learning, steel defects using the CNN model combined with Siamese
Xception. neural network types. In their research, CNN used in feature
extraction and Siamese used as a network structure with a
I. INTRODUCTION scheme using a convolutional layer with a small filter size.
Of all the algorithms used, it produces good accuracy in
In the steel industry, quality control during the classifying steel defects. S. Y. Lee et al. [8] and J. L. Greece
production process becomes very important. Steel quality et al. [9] have also used deep learning and CNN in their
control performed with detecting defects on the steel research to detect steel defects.
surface. Quality degradation will occur if steel defects are
not detected correctly. Detection of steel defects at the right In previous studies that applied deep learning with the
time can help in handling quality problems of steel to be CNN model, we reviewed the primary ability of CNN can
produced. Directly the quality of steel will determine the be affected by the use of architecture (pre-trained models).
durability and lifetime of the steel itself. CNN has a variety of architectures, and generally, these
architectures have several different layers. Three CNN
Nowadays, automation has utilized in carrying out architectures are the most known and commonly used,
quality control. One method in detecting defects can help by namely VGG16 [10], Inception V3 [11], and Xception [12].
the presence of images from high-frequency and high- Based on ImageNet Keras [13], Table 1 describes the
resolution cameras with the application of algorithms in performances of each architecture.
them [1]. This detection requires a sophisticated learning
algorithm that can improve the quality of detecting over Models Size Top-1 Top-5 Acc Depth
time. Deep Learning technology supports the technique Acc
because of its ability to learn its computing. VGG-16 528 MB 0.713 0.901 23
InceptionV3 92 MB 0.779 0.937 159
Deep learning has revolutionized various industries Xception 88 MB 0.790 0.945 126
because of excellent performance in computer vision. Table 1:- Performances pre-trained CNN
Unlike its predecessor machine learning, deep learning can
work without instructions from its creator to produce fast In the table, the top-1 and top-5 accuracy refers to the
and accurate predictions [2] so that it can help the workload performance of the model in the ImageNet validation
of engineers in the steel industry. dataset. Xception architecture has the highest accuracy
results of top-1 and top-5 accuracy and uses smaller
The Deep Learning Model that commonly used for memory compared to VGG-16 and InceptionV3
image recognition is the Convolutional Neural Network architectures.
(CNN). CNN will help find defective objects contained in
steel surface images. Therefore, deep learning with the CNN Some studies discuss the comparison of the three
model will use in this study to detect steel defects. architectures in terms of performance. Comparison of
Xception architecture and Inception V3 in F. Chollet's
research [12] proves that compared to InceptionV3,
Xception shows a small advantage in its classification

IJISRT20JUL240 www.ijisrt.com 244

Volume 5, Issue 7, July – 2020 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165
performance in the ImageNet dataset however Xception has of the feature map to zero. The next step of feature
a significant advantage in JFT. E. Ayan and H. M. Unver extraction is the pooling layer, which works by reducing the
[14], who compared between Xception and VGG16, proved resulting convolutional layer's dimensions. The pooling
that Xception outperforms with VGG16 in diagnosing layer consists of max pooling and average pooling, but
pneumonia. On the other hand, VGG16 shows better generally, what used is max pooling. After the feature
performance in diagnosing normal cases. For a separate extraction stage, it will be forward to the layer whose
comparison of InceptionV3 and VGG16, we have not found function is as a classification. The fully connected layer is
research to discuss the topic. But the investigation by C. A. the last layer that predicts output. The FC layer has an
Mamani Diaz et al. [15] compared the architecture of VGG- activation function that used to find the probability value in
16, Inception V3, and Xception in classifying agricultural forecasting the classification results after the feature map
precision using deep learning. Several experiments has finished operating.
conducted to support the selection of architecture with
satisfactory performance. In the testing process, experiments IV. METHOD
using different epochs in all three architectures prove that at
300 cycles, the training behavior of VGG16 and In this study, two techniques will use to detect steel
InceptionV3 stagnated. At the same time, Xception defects. The first technique is a binary classification that
continued to improve because of its active layers and faster recognizes images that have defects or no defects. Besides
classification capabilities than other architectures. This that, the function of this binary classification for filtering
research proves that the performance of the Xception defect images to the next stage. In the second technique,
architecture is more prominent than the others. multilabel classification performs, which detects images that
have 1 class of defects or multi-class of defects.
From previous studies discussing three architectural
comparisons, we conclude that Xception will use as an
architecture or pre-trained model to detect steel defects in
this study. But in the final results, we will explain the
accuracy and classification results of the VGG16 and
InceptionV3 architectures for this case.

III. DEEP LEARNING

Deep learning is a sub-field of machine learning that

emphasizes a multi-layered learning process. Deep learning
can solve complex problems due to features that can extract
automatically. The ability of deep learning proven in image
recognition, audio, video, text classification, etc. Deep
learning related to an algorithm that mimics how the basic
system of the human brain works, which is commonly called
the Artificial Neural Network.

A deep neural network, Convolutional Neural Fig 1:- Techniques of detection

Network, Recurrent Neural Network, is neural network
methods commonly used in deep learning. Convolutional A. Dataset
neural networks are methods specifically designed for image Dataset takes from [1], with a total of 12,568 pictures
recognition and using certain architectures. Unlike other of training data. Dataset consists of a collection of grayscale
types of neural networks, CNN can hold all the information steel surface images measuring 1600 x 256 pixels, which
contained in the image and can produce a more accurate have 1 class defects, multi-class defects, and no defects.
classification of objects. Steel surface image takes from a high-frequency and high-
resolution camera, which consists of four classes, namely 1,
There are two main block structures of CNN, namely 2, 3, and 4 [1]. From the results of Exploratory Data
feature extraction and classification. Feature extraction Analysis (EDA), 6666 images have defects and 5902 images
consists of convolutional layer operations and the pooling without defect. In the images that have defects, there are 897
layer (sub-sampling layer). The classification consists of images defect class 1, 247 images defect class 2, 5150
fully connected layers equipped with activation functions images defect class 3, and 801 images defect class 4. In
(softmax, sigmoid, etc.). In feature extraction, the addition to the number of each class, the number of images
convolutional layer is the first layer whose operation is to with one label is 6293, with two label is 425 and 2 images
take an image from the input layer and then extract the with three labels. From our analysis through information
image features according to the specified filter. When the obtained from Northeastern University (NEU) database
convolutional layer calculation operation performed, the web, each defect label has its name. Type of defects 1
resulting value may be negative; therefore, the RELU (Pitted surface), defects 2 (Inclusion), defects 3 (Scratches),
activation function will use in here. The RELU activation and defects 4 (Patches).
function performs its action by changing the negative value

IJISRT20JUL240 www.ijisrt.com 245

Volume 5, Issue 7, July – 2020 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165

Fig 6:- Result of image augmentation in Multilabel

Classification

In addition to preprocessing images, preprocessing on

data will also be done with a percentage split. The
percentage split ratio for training is 80%: 20%.
Preprocessing images and preprocessing data for training
are applied equally to binary classification and multilabel
classification techniques.

C. Xception
In this study, the CNN architecture used is Xception
Fig 2:- Image with a defect of a single class
[11] created by Francois Chollet. Xception is an extreme
version of the Inception model whose architecture based on
depthwise separable convolution layers. Inception with this
extreme version is a strong hypothesis that illustrates 1 x 1
convolution, which can map the convolution correlations in
n x n whose numbers are likely to be numerous. In deep
Fig 3:- Image with multiple defect classes learning, depthwise separable convolution is called
separable convolution with depthwise convolution, which is
a spatial convolution that can stand alone in each input
channel and is followed by pointwise convolution, which
projects the output channel.

Fig 4:- Image without defect

B. Preprocessing
Deep learning in the process of image recognition
requires image preprocessing. Preprocessing is influential in
extracting features in the image. The method that will use in
this preprocessing is data augmentation. Data augmentation
is a technique of modifying data without removing the core
of the data. Due to a large amount of data, this data
augmentation helps speed up the preprocessing stage. The
augmentation methods used in this study are (a) Rescale
with 1./255, (b) Shear range with 0.1, (c) Zoom range with
0.1, (d) Brightness range with [0.6, 1.0], (e) fill mode Fig 7:- Xception Module [11]
"constant", (f) Cval with 0, (g) horizontal flip and (h)
vertical flip. In the use of architecture, the image size will be Xception consists of 36 layers that form a network for
resized to 256 x 256 pixels. Resize the image functions to feature extraction. The Xception architecture feature map
speed up the training process. consists of 3, namely entry, middle, and exit. Each groove
represents several blocks of layers supporting feature
extraction. After the feature extraction block ends with the
global average pooling layer, there is a fully connected layer
whose usage is optional and ends with a logistic regression
layer. In this case, the logistic regression layer not used, but
what used is a fully connected layer consisting of global
average pooling 2D layers and Dense, which shows the
number of classification classes.

Fig 5:- Result of image augmentation in Binary

Classification

IJISRT20JUL240 www.ijisrt.com 246

Volume 5, Issue 7, July – 2020 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165

Fig 9:- Accuracy and Loss graph in Binary Classification

Fig 8:- Xception Architecture [11]

D. Performance Evaluation
For presenting the analysis in this study, we will use
several metrics, graphs, and a confusion matrix. Then, the
results of the analysis used to determine the ability of the
model in defect detection.

V. EXPERIMENTAL RESULT

As explained in a method, there are two detection

techniques in this case. Binary classification trained with
10054 training data and 2514 validation data, while
multilabel classification trained with 5676 training data
Fig 10:- Accuracy and Loss graph in Multilabel
images and 1419 validation data identified four classes.
Classification
Both techniques are training with the same parameters and
optimizers. The parameters used in this training include a
As shown in Figures 9 and 10, the accuracy produced
batch size of 16, the optimizer set is Adam, a loss function
in binary classification and multilabel classification is
that is binary cross-entropy, learning rate with sizes 1e-4,
increasing. In the beginning epoch, an accuracy estimated to
and 15 epochs. Due to infrastructure limitations, we
yield 0.80% to above 0.90% besides the loss value
conducted this training using Google Colab with the Nvidia
decreasing in each time. This matter shows that the
Tesla K80 GPU. The use of a GPU can help the execution
interpretation is good at the model used. After performing
speed compared to the CPU.
training in both processes, the Xception model tested for the
stability of its performance. The performance stability
The execution time of the training present in Table 2,
measured with the percentage split technique from 80%:
which shows binary classification requires a longer training
20% to 60%: 40%, which will produce accuracy and loss
time than multilabel. The average estimated time needed for
using the multilabel classification technique.
each process is 0.939s / step in binary and 0.617s / step in
multilabel. From this matter proves that the execution
process does not use a long time even though the number of 80:20 75:25 70:30 65:35 60:40
trained images is large. Training 0,9762 0,9773 0,9775 0,9799 0,9823
accuracy
Techniques Inference Time
Binary Classification 162 minutes Training 0,0562 0,0551 0,0490 0,0483 0,0445
Multilabel Classification 52 minutes loss
Table 2:- Training execution time
Table 3:- Results of accuracy and loss in each percentage
split
For more details, following visualization of the
performance of both techniques that show training accuracy
and training loss in each graph.

IJISRT20JUL240 www.ijisrt.com 247

Volume 5, Issue 7, July – 2020 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165
Based on Table 3, the accuracy and loss values of the
training results don't have wide ranges. The higher
percentage split in validation shows the accuracy increases,
and loss decreases. It's a proven stable Xception model used
in the training process.

VI. RESULT

This section will explain the results of the overall

detection of steel defects. The Xception architecture that
used to detect steel evaluated using several metrics. The
metrics are accuracy, recall, precision, and f1 with the
following formula:
TP+TN
Accuracy = TP+FP+TN+FN
(1)
TP
Recall = TP+FN
(2)
TP
Precision = (3)
TP+FP Fig 11:- Confusion Matrix in Binary Classification
2 x (Recall x Precision)
F1 = Recall+Precision (4)
With:
TP = True Positive
TN = True Negative
FP = False Positive
FN = False Negative

Some of these metrics will calculate the success of the

predictions made. The data used to measure predictions are
validation data with each data amounting to 2514 images
and 1419 images. Accuracy results obtained from the binary
classification process are 0.94% and 0.85% in multilabel
classification. A comparison of other metrics describes in
Table 4 and Table 5.

Binary Precision Recall F1

No Defect (0) 0,90% 0,97% 0,93%
Defect (1) 0,97% 0,91% 0,94%
Fig 12:- Confusion Matrix in Multilabel Classification
Table 4:- Results of Precision, Recall, F1 in Binary
Classification In binary classification, it can see that class 1 (defect)
has more classification errors than class 0 (no defect) while
Multilabel Precision Recall F1 the correct classification results are 2354 from 2514 data.
Pitted (1) 0,75% 0,79% 0,77% Multilabel predicts the correct classification of 1203 from
Inclusion (2) 0,50% 0,84% 0,63% 1419 data, and classification errors occur mostly in class 3
(Scratches).
Scratches (3) 0,93% 0,88% 0,90%
Patches (4) 0,67% 0,69% 0,68% In the multilabel case, results can present with a
Table 5:- Results of Precision, Recall, F1 in Multilabel multilabel confusion matrix. This Confusion matrix
Classification identifies whether the image is multilabel in each class or
not. Based on Figure. 13, the classification of non-multi
In addition to several metrics, a confusion matrix also labelled images is marked with label 0, while the correct
be presented in this section. The confusion matrix classification of multilabel images is label 1.
summarizes the results of the complete classification based
on true and false objects.

IJISRT20JUL240 www.ijisrt.com 248

Volume 5, Issue 7, July – 2020 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165
VIII. CONCLUSION

In this study, the detection of steel defects using two

techniques resulted in good classification. The Xception
model that used shows excellent performance in both
methods. Accuracy results obtained to classify the image is
defective or does not reach 0.94% while in the classification
of images that have 1 class or multi-class defects achieve an
accuracy of 0.85%. Our test in comparing the InceptionV3
and VGG16 with Xception architecture proves that
performance Xception more stands out. For further work,
we will consider adding several data augmentation methods
to improve accuracy and reduce the percentage of errors.

REFERENCES

[1]. Severstal, “Severstal: Steel Defect Detection,” 2019.

[Online]. Available:
https://www.kaggle.com/c/severstal-steel-defect-
detection. [Accessed: 27-Apr-2020].
[2]. I. den Bakker, Python Deep Learning Cookbook.
Birmingham: Packt Publishing, 2017.
Fig 13:- Multilabel Confusion Matrix [3]. B. Suvdaa, J. Ahn, and J. Ko, "Steel surface defects
detection and classification using SIFT and voting
VII. DISCUSSION strategy," Int. J. Softw. Eng. its Appl., vol. 6, no. 2,
pp. 161–166, 2012.
Besides, to perform a detailed analysis of Xception, we [4]. J. Li, Z. Su, J. Geng, and Y. Yin, "Real-time Detection
compared the Xception architecture with Inception V3 and of Steel Strip Surface Defects Based on Improved
VGG16 to detect steel defects using both techniques, as YOLO Detection Network," IFAC-PapersOnLine,
previously done in the Xception. From the training process, vol. 51, no. 21, pp. 76–81, 2018, doi:
accuracy in both architectures obtained. InceptionV3 10.1016/j.ifacol.2018.09.412.
obtaining accuracy 0.91% in binary and 0.83% in [5]. E. A. Kholief, S. H. Darwish, and M. N. Fors,
multilabel. Whereas VGG16 accuracy in binary 0.91% in "Detection of steel surface defect based on machine
binary and 0.78% in multilabel. The following Table 6 and learning using deep auto-encoder network," Proc. Int.
Table 7 will present the overall classification results of the Conf. Ind. Eng. Oper. Manag., no. April, pp. 218–229,
three architecture. 2017.
[6]. S. Tian and K. Xu, "An algorithm for surface defect
identification of steel plates based on genetic
algorithm and extreme learning machine," Metals, vol.
7, no. 8. 2017, doi: 10.3390/met7080311.
[7]. M. S. Kim, T. Park, and P. Park, "Classification of
Steel Surface Defect Using Convolutional Neural
Network with Few Images," 2019 12th Asian Control
Conf. ASCC 2019, pp. 1398–1401, 2019.
Table 6:- Results of all classification in Binary [8]. S. Y. Lee, B. A. Tama, S. J. Moon, and S. Lee, "Steel
Classification surface defect diagnostics using deep convolutional
neural network and class activation map," Appl. Sci.,
vol. 9, no. 24, 2019, doi: 10.3390/app9245449.
[9]. J. L. Yunani and M. V. Medvedev, "Steel Defects
Recognition Using Deep Learning," 2019 Int. Multi-
Conference Ind. Eng. Mod. Technol. FarEastCon
2019, pp. 1–4, 2019, doi:
10.1109/FarEastCon.2019.8933868.
[10]. K. Simonyan and A. Zisserman, "Very deep
convolutional networks for large-scale image
Table 7:- Results of all classification in Multilabel
recognition," 3rd Int. Conf. Learn. Represent. ICLR
Classification
2015 - Conf. Track Proc., pp. 1–14, 2015.

IJISRT20JUL240 www.ijisrt.com 249

Volume 5, Issue 7, July – 2020 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165
[11]. C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z.
Wojna, "Rethinking the Inception Architecture for
Computer Vision," Proc. IEEE Comput. Soc. Conf.
Comput. Vis. Pattern Recognit., vol. 2016-Decem, pp.
2818–2826, 2016, doi: 10.1109/CVPR.2016.308.
[12]. F. Chollet, "Xception: Deep learning with depthwise
separable convolutions," Proc. - 30th IEEE Conf.
Comput. Vis. Pattern Recognition, CVPR 2017, vol.
2017-Janua, pp. 1800–1807, 2017, doi:
10.1109/CVPR.2017.195.
[13]. Keras, “Applications.” [Online]. Available:
https://keras.io/applications/. [Accessed: 04-Feb-
2020].
[14]. E. Ayan and H. M. Ünver, "Diagnosis of pneumonia
from chest X-ray images using deep learning," 2019
Sci. Meet. Electr. Biomed. Eng. Comput. Sci. EBBT
2019, pp. 0–4, 2019, doi:
10.1109/EBBT.2019.8741582.
[15]. C. A. Mamani Diaz, E. E. Medina Castaneda, and C.
A. Mugruza Vassallo, "Deep Learning for Plant
Classification in Precision Agriculture," 2019 Int.
Conf. Comput. Control. Informatics its Appl. Emerg.
Trends Big Data Artif. Intell. IC3INA 2019, pp. 9–13,
2019, doi: 10.1109/IC3INA48034.2019.8949612.

IJISRT20JUL240 www.ijisrt.com 250