MACHINE PERCEPTION PROJECT REPORT, GROUP 7, FEBRUARY 11, 2020 1

Vehicle Driving Assistant

Akanksha Dwivedi, Anoop Toffy, Athul Suresh, and Tarini Chandrashekhar, International Institute of Information
Technology, Bangalore
(Draft Paper)

Abstract—Autonomous vehicles has been a common term in includes road extraction part and blob detection part. Then the
our day to day life with car manufacturers like Tesla shipping result that we obtained are given briefly in section 5. Finally
cars that are SAE Level 3. While these vehicles include a section 6 concludes the paper. Section 7 gives a light to the
slew of features such as parking assistance and cruise control,
they’ve mostly been tailored to foreign roads. Potholes, and the future enhancements that could be done.
arXiv:2002.03556v1 [cs.CV] 10 Feb 2020

abundance of them, is something that is unique to our Indian

roads. We believe that successful detection of potholes from visual 2. R ELATED W ORK
images can be applied in a variety of scenarios. Moreover, the
In Nienaber et al (2015) [1], a system using basic image
sheer variety in the color, shape and size of potholes makes this
problem an apt candidate to be solved using modern machine processing techniques in a constrained environment without
learning and image processing techniques. relying on any machine learning techniques is used for pothole
Index Terms—Image processing, Machine Learning, Pot hole
detection. It presents a good preliminary method for detecting
detection, Clustering potholes using a single camera within an range of 2 - 20m
from a vehicle moving at a speed of not more than 60km/hr.
The method separates a rectangular area of interest just above
1. I NTRODUCTION
the hood of the vehicle which contains road surface, assuming
The project aims to provide a comprehensive set of assis- that driver maintains a safe distance from the front vehicle.
tance features to aid the driver (or autonomous vehicle) to drive The rectangular area of interest is separated by connecting the
safely. This could includes a number of indicators about the various farthest region of interest using convex hull algorithm.
environment, the major cue being visualisation and detection
of potholes in the road ahead. The work presented by Ajit Danti et al (2012) [2], presents
”A pothole [5] is a structural failure in a road surface, caused a comprehensive approach to address the acute problems of
by failure primarily in asphalt pavement due to the presence Indian roads such as faded lanes, irregular potholes, improper
of water in the underlying soil structure and the presence of and invisible road signs. Instead of using image processing
traffic passing over the affected area”. An example of a road techniques for pothole detection as done by Nienaber et al
with pothole is shown in Figure 1. (2015), Ajith Danti et al (2012) uses K-Means clustering based
algorithm to detect potholes. By addressing the acute problems
above mentioned in the paper it makes automated driving safer
and easier in Indian roads.

3. P ROPOSED W ORK
In this paper we use both image processing techniques as
well as machine learning to study the presence and occurrence
of potholes. High resolution image captured from the hood of a
slow moving vehicle is used for analysis and experimentation.
This paper propose a way to visualize the potholes as well
as identify whether or not the road has a pothole in it. The
visualisation approach is similar to how humans perceive
the pothole images in the brain. The pothole identification
approach will help to signal the driver of a vehicle to take
preventive actions upon pothole infected lanes.

Fig. 1: Pothole example Foliage, incoming vehicles and other background objects
can interfere with the detection of potholes. In all the methods
The remainder of the paper is structured as follows. We described below the area of the image containing the road is
present a brief history of the related work done so far in isolated and extracted. Subsequent processing is then done on
the field of pothole detection and visualisation in Section this extracted area to detect the presence of potholes.
2. Section 3 discusses about the experimental set up we
used in this work.In the subsequent section we describes the The subsequent subsection describes the approaches we
methodology that we follow in the following section 4. It took for visualising the potholes. We used contour detection
MACHINE PERCEPTION PROJECT REPORT, GROUP 7, FEBRUARY 11, 2020 2

after applying morphological transformation as a method I and

then we used contour detection after extracting the road area
by using convex hull algorithm in method II. The methods are
described in detail below.

A. Using morphological transformation

Morphological transformations are some simple operations
based on the image shape. It is normally performed on binary
images. Two basic morphological operators are Erosion and
Dilation. Using which the image is analysed to visualise the
pothole present in the region.

B. Using contours and convex hull

For visualising the pothole we first extract a region of the
interest from the given image which will contain the road
area. This is done by first selecting the largest contour on
the images ROI and then drawing convex hull using convex
hull algorithm. Since extracting the road area will help to
narrow down the area of interest. This way we will be able
to prevent the irrelevant informations from the analysis like Fig. 2: Results of Morphological Transformation
foliage, vehicle hood, other vehicles on the road etc. We then
apply image processing techniques like contour detection, blob
detection and edge detection to draw a bounding box over the 2) Visualisation from isolated roads: The figure 3 show the
area of the road which contains the potholes. image which is collected from the Digital camera. The image
is then converted to its corresponding HSV channel because
HSV channels captures the most variance in the image which
C. Using Machine Learning Methods aids is subsequent processing.
For machine learning technique we used annotated dataset
[4] for training various classifiers and then try various machine
learning algorithms to figure out the presence of a pothole in
the frame given.

4. M ETHODOLOGY
In this section we briefly familiarize the methodology that
we used for the visualisation and detection of potholes in
detail. The subsequent section first discusses about the image
processing techniques that are helpful for pothole visualisation
followed by feature extraction and machine learning learning
techniques used in the study.

A. Pothole Visualisation Techniques Fig. 3: Pothole Image

1) Morphological Transformation Method: The image is
smoothed using Gaussian blur and then thresholding is applied.
Then opening and closing morphological transformation are
applied to clear off the noise in the image and then road area is
extracted from the output image. From the extracted road area
we applied edge detection methods to find potholes present in
the image. This technique works on the Google Images dataset
with partial noise being detected but detected too much noise
in our dataset along with a fraction of potholes.

Fig. 4: Converting to HSV

MACHINE PERCEPTION PROJECT REPORT, GROUP 7, FEBRUARY 11, 2020 3

From the image we select a region of interest (ROI) which

contains the road area. Since we are using a fixed camera
mounted on the window of a moving vehicle we assume that
the road area tend to appear on a fixed region above the hood
of the vehicle. Hence we select a Region of Interest from the
(ROI) image as show in figure 5.

Fig. 7: Mask

The Figure 8 below indicates the convex hull of the largest

contours selected from the threshold images. This largest
contour is used a mask to extract the road area as shown in
the Figure 10.
Fig. 5: Selecting the ROI

Figure 6 shows the enlarged ROI of the image. The mean

and standard deviation per colour channel of this region of
interest is calculated. In order to address the issue of road
colour variation within the image, the road colour is modelled
separately per channel as lying within three standard deviations
of the mean of each channel. These thresholds were used to
binarize the image, and the mask obtained is show in Figure 7.
From the figure, it is clear that the average colour model does
not match the entire road area as there are many areas of the
road that were not included after the thresholding operation.
Contour detection algorithm is then run on the extracted mask. Fig. 8: Convex hull of largest contour
It is assumed that the largest contour in the image will include
the road and therefore only this contour is sent to the convex
hull algorithm. The largest contour is received by the convex
hull algorithm and the output of the algorithm is indicated in
Figure 8. Using a mask made of the computed convex hull,
the road is extracted from the input image as shown in Figure
10.

Fig. 9: Convex hull on Original Image

Fig. 6: Region of Interest

MACHINE PERCEPTION PROJECT REPORT, GROUP 7, FEBRUARY 11, 2020 4

an image is relatively invariant with translation and rota-

tion about the viewing axis, and varies only slowly with
the angle of view. By comparing histograms signatures
of two images and matching the color content of one
image with the other, the color histogram is particularly
well suited for the problem of recognizing an object of
unknown position and rotation within a scene” [13]
2) Machine Learning Algorithms: The following list of
machine learning techniques were used in the study. They are
trained using the above mentioned feature and the results are
analysed.
• KNN [8]
Fig. 10: Extracted Road • GaussianNB
• Logistic Regression [6]
• SVM [10]

Blob Detection Method: The extracted road area is then • Decision Tree [7]

analysed further for pothole visualisation. In computer vision, • Random Forest [9]

blob detection methods are aimed at detecting regions in a • AdaBoost using Decision Tree

digital image that differ in properties, such as brightness or

color, compared to surrounding regions. Since we assume that 5. R ESULTS
pothole appear in the road in different texture than that of A. Pothole visualisation using Image processing Techniques
the road area mainly in dark shadowed regions some time The image processing techniques gave us suboptimal re-
it appear as though it is covered in mud and water. Since sults. Method I using morphological transformation didn’t
there is a differentiating factor that make pothole identifiable work well as there are edges in the roads that created extra
in the road we used blob detection algorithms to highlight the contours which were wrongly highlighted as potholes. Method
potholes. II using road isolation followed by contour detection and edge
detection worked for some scenarios but did not work well for
Contour dilation Method: The extracted road area could most of the cases. Hence pothole visualisation is found difficult
also be analysed using contour detection or edge detection to achieve with the current technology as we were not able to
(Canny edge detection). The detected edges could be dilated find discriminating feature for potholes from our analysis
for highlighting the presence of potholes. But the results
obtained for the potholes were suboptimal. Since potholes B. Pothole detection using Machine Learning Techniques
appear in varied shape and color better discrimination from The classifier are trained using the feature and the results
other object on the road could not be achieved. obtained are shared in the below tables. The table below
show the result obtained while using the downscaled image
B. Pothole detection using machine learning techniques as features.
In this section we describe in details the various machine Methods Accuracy(%)
learning techniques that we used for detecting the presence of Logistic Regression 72.73
the pothole in the roads. As described in the previous section, Multinomial Logistic Regression 68.93
the ROI containing the road is extracted using the previously Decision Tree 86.88
mentioned Convex hull of contour method. This ROI is then AdaBoost using Decision Tree 80.32
subsequently processed by the models to detect the presence GaussianNB 44.26
KNN 73.77
of potholes.
SVM 55.73
1) Feature Modelling: The feature extraction is done in two Random Forest 68.18
ways
TABLE I: Results for Raw downscaled images pixels as
• Downscaled image as features
features
The image is downscaled by a factor of two and then
flattened to be used as a feature vector.
• Color histogram as a feature
A color histogram of an image represents the distribution
of the composition of colors in the image. It shows The table below show the result obtained while using the
different types of colors appeared and the number of color histogram as features.
pixels in each type of the colors appeared.

”The histogram provides a compact summarization of the As we can infer from the tables that the classification
distribution of data in an image. The color histogram of using downscaled images worked well when decision trees and
MACHINE PERCEPTION PROJECT REPORT, GROUP 7, FEBRUARY 11, 2020 5

Methods Accuracy(%)
– Data Set II - https://www.youtube.com/playlist?list=
Logistic Regression 77.27
Decision Tree 77.04
PLzf2LOyk6iN3nNyvu1RI0Np-i8kT1IGE0
AdaBoost using Decision Tree 83.60 – Data Set III [4]
GaussianNB 75.40 • Code - https://github.com/crunchbang/MP Project/blob/
KNN 78.68 master/Code/
SVM 55.73 • Road Extraction Video - https://www.youtube.com/
Random Forest 95.45
watch?v=4vUqzDnZsV8
TABLE II: Results for color histogram as features
R EFERENCES
[1] S. Nienaber, M.J. Booysen, R.S. Kroon Detecting potholes using simple
AdaBoost using decision tree are used. Meanwhile Gaussian image processing techniques and Real-world Footage, 2015. SATC,
Naive Bayes performed the worst using downscaled images. July 2015, Pretoria, South Africa http://scholar.sun.ac.za/handle/10019.
1/97191
[2] Ajit Danti, Jyoti Y. Kulkarni, and P. S. Hiremath, Member, IACSIT An
The color histogram as a feature worked relatively better Image Processing Approach to Detect Lanes, Pot Holes and Recognize
when compared to the downscaled images. We were pretty Road Signs in Indian Roads, December 2012 http://www.ijmo.org/papers/
204-S3015.pdf
surprised to see the performance of color histogram using Ran- [3] S. Nienaber, R.S. Kroon, M.J. Booysen “A Comparison of Low-Cost
dom Forest which gives an accuracy of 95%. SVM didn’t give Monocular Vision Techniques for Pothole Distance Estimation” IEEE
good accuracy in both color histogram as well as downscaled CIVTS, December 2015, Cape Town, South Africa.
[4] The annotated image dataset used in the pothole detection is freely
images. available at https://goo.gl/3QyeMs
[5] https://en.wikipedia.org/wiki/Pothole
[6] https://en.wikipedia.org/wiki/Logistic regression
6. C ONCLUSION [7] https://en.wikipedia.org/wiki/Decision tree
We presented a mix of methods, incorporating both image [8] https://en.wikipedia.org/wiki/K-nearest neighbors algorithm
[9] https://en.wikipedia.org/wiki/Random forest
processing and machine learning, to detect the presence of [10] https://en.wikipedia.org/wiki/Support vector machine
potholes in an image. Given an image, the road was isolated [11] Dataset with negative and positive examples separated https://drive.
and extracted with best results using contours and finding the google.com/drive/folders/0B7LHCitTUdEYZFEwNWo4V2RldjQ?usp=
sharing
convex hull of the largest contour. Through successive testing, [12] Open CV is used for Image processing and Machine Learning http:
it was found that using machine learning models yielded //docs.opencv.org/
better results than conventional image processing methods [13] Color Histogram https://en.wikipedia.org/wiki/Color histogram
as far as detection of potholes were concerned. The models
were trained using features extracted from the isolated road
segment. Comparison between the different methods of feature
extraction and the performance of different models on these
features have also been exhibited.

7. F UTURE W ORK & I MPROVEMENTS

The present work yields suboptimal results in scenes fea-
tures heavy shadows. Better discriminating features could be
extracted which helps to discern potholes even in environments
where the lighting is uneven and interspersed with shadows.
Methods could be developed to accommodate the presence of
incoming traffic and road markings.

ACKNOWLEDGMENT
We would like to thank Prof. Dinesh Babu Jayagopi for
guiding us throughout this venture, Prof. MJ (Thinus) Booy-
sen, Associate Professor at the Electrical & Electronic Engi-
neering Department at Stellenbosch University for prividing
us with the pothole datasets [4].

A PPENDIX A
DATA AND C ODE
• Data
– Data Set I - https://www.youtube.com/watch?v=
hmeBmdZzlLU
MACHINE PERCEPTION PROJECT REPORT, GROUP 7, FEBRUARY 11, 2020 6

Akanksha Dwivedi (MT2016006) is currenlty purs-

ing Master of Technology from International Insti-
tute of Information Technology, Bangalore.

Anoop Toffy (MT2016016) is currenlty pursing

Master of Technology from International Institute of
Information Technology, Bangalore.

Athul Suresh (MT2016030) is currenlty pursing

Master of Technology from International Institute of
Information Technology, Bangalore.

Tarini Chandrashekhar (MT2016144) is currenlty

pursing Master of Technology from International
Institute of Information Technology, Bangalore.