Semantic Segmentation of Paved Road and Pothole

Image Using U-Net Architecture

Vosco Pereira∗ , Satoshi Tamura† , Satoru Hayamizu† , Hidekazu Fukai†
∗ Informatics Engineering Department, National University of Timor-Leste
vosco.pereira@untl.edu.tl
† Informatics Engineering Department, Gifu University Japan

{tamura, hayamizu, fukai}@gifu-u.ac.jp

Abstract—Research on road monitoring system has been

actively conducted by using both machine learning and deep
learning technique. One of our nearest goal in the framework of
road condition monitoring system is to segment all road related
object and provide a technical report regarding road condition.
Our final objective is to develop a community participant-based
system for road condition monitoring. As one of our task, in this
research, we start with the segmentation of road and pothole. To
conduct this task we proposed a semantic segmentation method
for road and pothole image segmentation by using one of the
famous deep learning technique U-Net.
Various condition of road images were used for training and
Fig. 1. Smartphone setup.
validating the model. The experiment result showed that U-Net
model can achieve 97% of accuracy and 0.86 of mean Intersection
Over Union (mIOU).
autonomous vehicles system. An ability to segment road-
Keywords—Road, Pothole, Deep Learning, U-Net, Semantic related objects makes it possible to further improve the state
Segmentation, Road Condition Monitoring.
of the road segmentation for road condition monitoring system
[11] particularly for least developing country.
I. I NTRODUCTION
In this paper we proposed a semantic segmentation of paved
Road infrastructure is the most important of all public assets road and pothole image by using one of the deep learning
in a nation. It makes a significant contribution to economic techniques for semantic segmentation U-Net.
development and plays a significant rule in providing access to
employment, social, health and education services. For those II. R ELATED W ORK
reasons, a road monitoring task is a very essential step in In response to abovementioned problems, several studies
the framework of road development process. Several problems regarding road condition monitoring systems by using segmen-
caused from poorly maintened roads can be occured such as tation technique in road images have been done by applying
constraining mobility, raising transportation operation costs both conventional and deep learning techniques.
and increasing of accidents rate [1]. By using the segmentation technique, a road segmentation
To acquired an adequate information of road data and to method called as Y-Net was proposed [8]. Y-Net consist of
obtain an effective as well as an efficient result, a good road a two-arm feature extraction module. In addition, there is a
monitoring system is need to be developed. By having such fusion module for combining the extracted feature module for
system, the problems which caused from poorly maintened road segmentation. As a result, the multiscale roads can be
roads can be reduced. well segmented from high-resolution images.
So far, one of the most practical solutions for road condition Another road surface segmentation technique by using dis-
monitoring is by employing a human expert for visual inspec- parity histogram which defined as VLDH (Vertically Local
tion but there are several disadvantages such as result from Disparity Histogram) also was developed. In this study, the
the human-based inspection are too subjective, making them boundary between road obstacles detected based on VLDH.
difficult to compare and understand as well as this method is The computational time is confirmed based on implemen-
time consumed. tation into a Field-Programmable Gate Array (FPGA). The
In the field of computer vision and image processing, experimental resulted, the accuracy of the proposed method is
semantic segmentation by using deep learning technique can outperformed the conventional method [9].
be used for various purposes [2] [3] . Semantic segmentation On the other hand, a new method to identify cracks on road
is mostly used for road scene segmentation [4] [5] [6] for pavement images was proposed [10]. The proposed algorithm
978-1-7281-3452-9/19/$31.00 ©2019 IEEE. includes a selection of prominent and adopting an efficient

Authorized licensed use limited to: Auckland University of Technology. Downloaded on June 04,2020 at 07:42:54 UTC from IEEE Xplore. Restrictions apply.
 Fig. 3. Sample images of dataset and its annotation.
Fig. 2. Sample images in our dataset.

segmentation procedure after appropriate image smoothing collecting process. The smartphone was set up as shown
and iterative binary pixel classification into the crack or non- in Figure 1. The illumination configuration of smartphone
crack classes. The method classified the identified cracks into camera was set as default. In this data collection task, 700
transversal, longitudinal or miscellaneous. of road images were collected then alocated 560 images for
An evaluation of Fully-CNNs for road segmentation in training set and another 120 images for validation set. The
Synthetic Aperture Radar (SAR) images was also conducted. data collection process was conducted at day time. Figure 2
In the study, the author added spatial tolerance rules to- shows some example of our dataset.
wards thin objects after carefully enhanced it sensitivity. The
proposed model successfully extracted most of the roads in B. Data Preprocessing
the test dataset. The result also reveals that although Fully- The original size of the collected images was 1920 x 1080
CNNs natively lack efficiency for road segmentation task, pixel with RGB channels. Before apply semantic segmentation
good results can be obtained if properly configured [11]. technique, all the collected image we resized to 225 x 225
Another approach is the detection of crack and pothole pixel for reducing the computational time during train the
using spatial features and texture based on deep neural network model.
architecture. In the experiment, a pre-processing method which
ensure real-time performance was proposed. The approach C. Data Normalization
performs well in some viewpoint changes, shadows, occlusion The standardization method is used to normalized all the
and background noise. [12]. collected images. It is done by first subtracting the mean from
Beside using for medical image segmentation [13], There every feature and then diving by standard deviation.
is a modified of U-Net called Wide-Range Attention Unit
(WRAU) which can be used for road segmentation. In this D. Data Annotation
study, a partially branch of dense-connection was added to
the U-Net architecture. In addition, at the cluster of nodes In a semantic segmentation task, it requires a corresponding
of partially dense – connections, the authors also employed ground truth in order to train a model. Therefore, in addition
WRAU. This method resulted a tangible boost to test accuracy. to resizing, the author also annotated the coresponding mask
We have proposed a CNN model for paved and unpaved of original images by using a data annotation tool. There are
road classification [14 ]. In the study, if the input image is three classes annotated: road, sky and pothole. Figure 3 shows
classified as paved road, then we continue to detect whether some examples of images and its coresponding masks for each
the paved road image has only pothole or not [15]. The original classes.
road images are cropped due to the presense of some outliers
before applying a deep learning technique. By applying the E. Description of Model Architecture
cropping process, the proposed model succesfully classsified We used U-Net based architecture as our neural network
whether the input images belong to either paved road or model [7]. It contains of three main paths: contracting, bottle-
pothole. neck and an expansive paths as shown in Figure 4.
In section 3, we explained in detail about our proposed All the descriptions of the model are similar to original U-
method, experimental results explained in section 4 and section Net model. Except for preserving the spatial resolution after
5 provided conclusion and future works. convolution we applied zero padding and we also doubled the
number of feature channels at the contracting path from 32
III. P ROPOSED M ETHOD at the first layer to 512 at the last layer. In addition, to map
each 32 components we also used a 1 x 1 convolution in the
This section consist of data collection, data annotation, expansive path.
description of model architecture and training methodology.
A. Data Collection F. Training Methodology
We utilized ASUS Z01RD smartphone for conducting data The training method of the model is expained in this
collection task. Some urban and rural roads are covered in subsection.

Authorized licensed use limited to: Auckland University of Technology. Downloaded on June 04,2020 at 07:42:54 UTC from IEEE Xplore. Restrictions apply.
 Fig. 4. Graphic depiction of model architecture.

1) Framework and Hardware: We used keras and ten-

sorflow which running on an Intel(R) Core (TM) i7-6700 A ∩B
IoU (A, B) = (2)
CPU @ 3.40 GHz 3.41 GHz, 8.00 GB RAM for network A ∪B
implementation.
Table I reports the result of road and pothole segmentation
2) Training Process and Batching: We trained our model using overall accuracy and mean Intersection over Union
in 500 epochs, the batch size was set to 32. The model was (mIoU). The qualitative results of road and pothole segmen-
trained from scratch without applied any transfer learning. The tation are shown in Figure 5.
resized images with their corresponding mask were used to
train the model. As we have little training data for this task,
V. CONCLUSION AND FUTURE WORK
we applied data augmentation by applying random crops, shift
and vertical flips.
In this paper we presented a semantic segmentation of paved
3) Optimizer: The model was initialized using HeUniform road and pothole images by using one of the deep learning
and then trained using Adam optimizer. The learning rate technique for semantic segmentation U-Net. Our model is
was set at a constant 0.0001. The model was finetuned with trained using images were collected in some urban and rural
learning rate of 0.001. The crossentropy was employed as the roads by using smartphone device. Experimental result showed
cost function. that by using U-Net, paved road and pothole can be well
4) Regularization: In our experiment, a weight decay of segmented and achive high accuracy and obtain good mean
0.0001 and dropout rate of 0.2 were chosen for regularization. Intersection Over Union (mIOU).
We applied the dropout at the last layer of expansive path. This result also reveal that although U-Net was originally
proposed for medical image segmentation but it can be applied
also for segmentation of road and pothole. The approach can
IV. EXPERIMENTAL RESULT
also be used by government authorities to monitor and evaluate
all road condition in over all teritories.
The segmentation results were evaluated by using overall
accuracy and Intersection over Union (IoU). The accuracy is
TABLE I
used to judge how often a model makes the correct prediction. E VALUATION RESULT
Equation (1) indicates the overall accuracy.
Metric Score
Accuracy 97%
TP + TN mIoU 0.85
Accuracy = (1)
TP + FP + TN + FN
An IoU is a method to quantify the percent overlap between However, to prove the performance of this result, this
the target mask and prediction output. It can be define as in experiment needs to be compared with any other state of the art
Equation (2) below. method in the future. In addition, the image dataset used in the

Authorized licensed use limited to: Auckland University of Technology. Downloaded on June 04,2020 at 07:42:54 UTC from IEEE Xplore. Restrictions apply.
 Fig. 5. Model prediction results compared with the ground truth mask.

segmentation task is still small and lack of variety for building [10] Henrique Oliveira, Paulo Lobato Correia, ”Road Surface Crack Detec-
a good and reliable segmentation model which cause the model tion: Improved Segmentation with Pixel-based Refinement.” 2017 25th
European Signal Processing Conference (EUSIPCO).
failed to segment some unseen data. To solve this issue, more [11] Corentin Henry, Seyed Majid Azimi and Nina Merkle, ”Road Segmen-
annotation of corresponding mask need to be conduct in the tation in SAR Satellite Images with Deep Fully-Convolutional Neural
future. Networks.”Arxiv preprint arXiv:1802.01445v2, 2018.
[12] Sukhad Anand, Saksham Gupta, Vaibhav Darbari and Shivam Kohli,
”Crack-pot: Autonomous Road Crack and Pothole Detection.”Arxiv
VI. ACKNOWLEDGMENT preprint arXiv:1810.05107v1, 2018.
[13] Man Yuan, Zhi Liu, Fan Wang (2019) ,”Using the wide-range attention
U-Net for road segmentation”, Remote Sensing Letters, 10:5, 506-515,
The authors would like to thanks to the lecturers and staff DOI: 10.1080/2150704X.2019.1574990
of Faculty of Engineering and Science and Technology of [14] Vosco Pereira, Satoshi Tamura, Satoru Hayamizu, Hidekazu Fukai,
National University of Timor Leste and Gifu University Japan. ”Classification of Paved and Unpaved Road Image Using Convolutional
Neural Network for Road Condition Inspection System.”, International
Special thanks to Japan International Cooperation Agency Conference on Advance Informatics: Concepts, Theory and Applications
(JICA) for funding this research through the project of CADE- (ICAICTA) 2018.
FEST phase 2. [15] Vosco Pereira, Satoshi Tamura, Satoru Hayamizu, Hidekazu Fukai,
”A Deep Learning-Based Approach for Road Pothole Detection in
Timor Leste”. IEEE International Conference on Service Operations and
Logistics, and Informatics (SOLI) 2018.
R EFERENCES
[1] Burningham, Sally; Stankevich, Natalya. 2005. ”Why Road Maintenance
is Important and How to Get it Done. Transport Notes Series”; No. TRN
4. World Bank, Washington, DC. © World Bank.
[2] Jonathan Long, Evan Shelhamer, Trevor Darrell,”Fully Convo-
lutional Networks for Semantic Segmentation”, ArXiv preprint
arXiv:1411.4038v2, 2015
[3] Simon Jegou, Michal Drozdzal, David Vazquez, Adriana Romero,
Yoshua Bengio,”The One Hundred Layers Tiramisu: Fully Con-
volutional DenseNets for Semantic Segmentation”, ArXiv preprint
arXiv:1611.09326v3, 2017
[4] Yecheng Lyu, Lin Bai, Xinming Huang, ”Road Segmentation Using
CNN and Distributed LSTM”, ArXiv preprint arXiv:1808.04450v2,
2019
[5] Yecheng Lyu, Xinming Huang, ”Road Segmentation Using CNN with
GRU”, ArXiv preprint arXiv:1804.05164v1, 2018
[6] Farnoush Zohourian, Borislav Antic, Jan Siegemund, Mirko Meuter,
Josef Pauli, ”Superpixel-based Road Segmentation for Real-time Sys-
tems using CNN”, In Proceedings of the 13th International Joint Con-
ference on Computer Vision, Imaging and Computer Graphics Theory
and Applications (VISIGRAPP 2018) - Volume 5: VISAPP, pages 257-
265
[7] Olaf Ronneberger, Philipp Fischer, and Thomas Brox, ”U-Net: Convo-
lutional Networks for Biomedical Image Segmentation.”Arxiv preprint
arXiv:1801.09454v2, 2018.
[8] Ye Li, Lele Xu, Jun Rao, Lili Guo, Zhen Yan & Shan Jin, ”A Y-
Net deep learning method for road segmentation using high-resolution
visible remote sensing images.” Journal Remote Sensing Letters Volume
10, 2019 - Issue 4, Pages 381-390.
[9] Shinji Kakegawa, Haruki Matono, Hideaki Kido & Takeshi Shima,
”Road Surface Segmentation based on Vertically Local Disparity His-
togram for Stereo Camera.”Int. J. ITS Res. DOI 10.1007/s13177-017-
0140-8.

Authorized licensed use limited to: Auckland University of Technology. Downloaded on June 04,2020 at 07:42:54 UTC from IEEE Xplore. Restrictions apply.