Date of publication xxxx 00, 0000, date of current version xxxx 00, 0000.

Digital Object Identifier 10.1109/ACCESS.2024.Doi Number

Advanced Self-Driving Vehicle Model for

Complex Road Navigation Using Integrated
Image Processing and Sensor Fusion
Kalapraveen Bagadi1, (Senior Member, IEEE), Naveen Kumar Vaegae2, (Senior Member,
IEEE), Visalakshi Annepu3, Khaled Rabie, (Senior Member, IEEE), Shafiq Ahmad5,
Thokozani Shongwe6, (Senior Member, IEEE)
1
School of Electronics Engineering, VIT–AP University, Amaravati 522 237, India
2
School of Electronics Engineering, Vellore Institute of Technology, Vellore 632 014, India
3
School of Computer Science and Engineering, VIT–AP University, Amaravati 522 237, India
4
Department of Engineering at Manchester Metropolitan University (MMU), UK
5
Industrial Engineering Department, College of Engineering, King Saud University, P.O. Box 800, Riyadh 11421, Saudi Arabia
6
Electrical and Electronic Engineering Technology, University of Johannesburg, South Africa
Corresponding authors: Naveen Kumar Vaegae (e-mail: vegenaveen@vit.ac.in); Khaled Rabie (e-mail: k.rabie@mmu.ac.uk).

ABSTRACT This paper presents a sophisticated self-driving vehicle (SDV) model that addresses the
challenges of navigating complex road networks characterized by high traffic and unpredictable
environments. The model integrates state-of-the-art image processing techniques with advanced sensor
fusion, utilizing EfficientDet D0 and Haar Cascade object detection models to accurately identify obstacles,
road signs, and traffic signals. The integration of data from cameras and ultrasonic sensors enables the
creation of a precise 2D map of the vehicle's surroundings, which, combined with a robust decision-making
algorithm, allows for optimal performance in challenging traffic scenarios. The SDV prototype was tested
extensively in a custom-built artificial environment, where it demonstrated its ability to handle various real-
world scenarios, including lane detection, obstacle avoidance, and decision-making in the presence of
stationary obstacles and heavy traffic. The experimental results confirm the model's effectiveness in
enhancing SDV capabilities, paving the way for safer and more efficient autonomous transportation systems.

INDEX TERMS Arduino, efficientdet, image processing, object detection, raspberry Pi, self-driving
vehicles, Sensor fusion.

I. INTRODUCTION challenging real-world scenarios. SDV technology presents

The advent of self-driving vehicle (SDV) technology has both risks and possibilities. Risks include safety concerns
heralded a new era in transportation, promising safer and due to potential errors in autonomous systems, cybersecurity
more efficient mobility solutions. However, the widespread threats from malicious actors, ethical dilemmas in decision-
adoption of autonomous vehicles faces significant making algorithms, and job displacement in driving-
challenges, particularly in navigating complex road networks dependent industries [3]. However, SDVs also offer
characterized by high traffic volumes, diverse road improved safety by eliminating human error, enhanced
conditions, and unpredictable environments [1]. Addressing mobility and accessibility for underserved communities,
these challenges is paramount to realizing the full potential efficiency gains leading to reduced congestion and
of autonomous driving systems and ensuring their seamless emissions, and transformative impacts on urban planning
integration into modern transportation infrastructure [2]. and infrastructure. Balancing these risks and possibilities is
This research endeavors to tackle the complexities associated crucial for realizing the full potential of SDV technology
with autonomous driving on intricate road networks through while addressing its challenges [4].
a comprehensive approach to enhancing performance and In this paper, we present a novel model designed to improve
safety. By leveraging advanced technologies and innovative the performance of self-driving vehicles in navigating
methodologies, we seek to overcome the obstacles that complex road networks. Our approach integrates state-of-the-
hinder the widespread adoption of autonomous vehicles in art techniques from computer vision, sensor fusion, and

VOLUME XX, 2017 1

 Author Name: Preparation of Papers for IEEE Access (February 2017)

decision-making algorithms to enable autonomous vehicles to findings show that the suggested method successfully
operate effectively in diverse and dynamic environments. executes a variety of maneuvers with no parameter
Specifically, we focus on addressing the unique challenges adjustments. It incorporates a novel reference speed
posed by complex road networks, such as congested traffic generation function and a lateral maneuver initiation
patterns, unpredictable road conditions, and varying levels of mechanism. Finally, the research in [11] offers a framework
infrastructure. Through a combination of theoretical analysis, for decision-making that applies linear temporal logic (LTL)
algorithm development, and practical experimentation, we to self-driving car adaptation to various traffic laws.
aim to demonstrate the efficacy of our approach in enhancing Utilizing an LTL-based reinforcement learning framework
the capabilities of autonomous driving systems. By providing improves vehicle adherence to changing traffic laws in
a comprehensive solution that addresses the multifaceted various situations without requiring preset interfaces.
Autonomous driving technology holds the potential to
challenges of navigating complex road networks, we seek to
drastically alter how people drive to reduce accidents and
accelerate the adoption of autonomous vehicles and pave the
congestion [12]. Despite tremendous advancements,
way for a safer, more efficient transportation landscape.
difficulties still exist in human behavior, technical
Overall, this research contributes to the ongoing efforts to complexity, ethics, and legal concerns. A common goal is
realize the vision of autonomous driving on complex road cooperative vehicle behavior, and many approaches and
networks, offering insights and methodologies that can inform models investigating vehicle-to-vehicle (V2V) cooperation
the development of future autonomous vehicle technologies and platooning have been proposed [12]. Once
and shape the future of transportation. technologically possible, legal and ethical issues ranging
from privacy responsibility could prevent broad use. The
A. RELATED WORK design and application of an autonomous electric car for path
The development of SDV technology has been dramatically tracking are covered in a different paper [13]. With visual-
impacted by several research projects that have concentrated inertial odometry and a limited input-output nonlinear model
on essential areas such as obstacle avoidance, object predictive controller, the car can drive safely without relying
detection, and decision-making algorithms. The impending on global navigation satellite system signals. Its performance
deployment of SDVs on city roadways is addressed by the is validated by simulation and real-time experiments. Then,
TrustFSDV framework, which was put forth in [5]. It the increasing attention in artificial intelligence (AI)
emphasizes the critical need of trustworthiness to guarantee techniques, as well as in deep learning (DL) systems, has
societal acceptability. To advance SDV integration, it significantly advanced semantic segmentation, essential for
promotes teamwork, open communication, and proactive scenario comprehension in self-driving systems [14]. Using
steps to address moral and legal issues. The study in [6] the CamVid dataset, a proposed model combines
examines cutting-edge testing techniques critical for autoencoders and convolutional neural networks to produce
assessing autonomous vehicle intelligence, which is essential accurate real-time segmentation. Emerging autonomous
for manufacturers and consumers. Experiments show that it vehicle (AV) technologies are considered while examining
provides a quantifiable way to evaluate autonomous vehicle transportation policies [15]. Connect cars and a conceptual
intelligence by criticizing current methods and proposing a navigation model based on fleet optimization are suggested
new testing methodology. According to the study in [7], to improve system efficiency and traffic circulation.
which emphasizes the need for cooperative behavior in Path planning is essential for the comfort and safety of
networked, automated vehicles, AI capabilities should be autonomous driving in real-world road traffic situations.
integrated into socio-cyber-physical ecosystems. It Trajectory planning is, therefore, a topic of interest for many
highlights how crucial it is for driverless cars to develop researchers. To improve traffic safety, an automated lane-
social-AI capabilities by moving beyond simple sensing change system for self-driving articulated trucks is created in
capabilities. To enhance fuel economy and lower gas basic research [16]. Numerical simulations are used to
emissions, the study in [8] addresses concerns about fuel evaluate a novel safety lane-change path planning technique
consumption and emissions in moving vehicles and suggests and a combined controller, providing strong stability and
a green protocol designed for highway driving. The efficient path tracking. This work adds multi-objective
methodology optimizes acceleration and deceleration decision-making modes and quantitative trajectory models
reflexes and maintains constant speeds through extensive for autonomous vehicles. In [17], trajectory prediction for
experimentation. The paper [9] discusses the expected role self-driving cars is examined, focusing on the interactions
of people in remote supervision capacities for fully between the vehicles and their environment. An attention
autonomous SDVs, using the Internet of Everything (IoE) mechanism represents higher-order interactions and
and recent advances in cyber-physical systems. The produces various possible pathways, drawing inspiration
effectiveness of human-on-the-loop haptic teleoperation from human reasoning. The experimental results show better
with FA-SDVs is examined, focusing on cooperation performance than the current approaches, especially when
between humans and vehicles and encouraging results to using datasets related to highways. A solution for Frenet-
improve the suggested framework. A Model Predictive based path planning is shown and tested by simulated driving
Control method is presented in [10] to create safe routes for situations in [18]. This solution addresses scenarios where
self-driving cars in mixed traffic situations. Simulation

2 VOLUME XX, 2017

 Author Name: Preparation of Papers for IEEE Access (February 2017)

conventional assumptions may not hold. The method's object localization, which is critical for seamlessly
effectiveness is demonstrated by the pathways it creates, navigating diverse terrains.
which follow reference lines and keep the right directions. In  Enhanced Obstacle Detection through Sensor Fusion
[19], adaptive lane change trajectory planning for Technology: Our methodology incorporates sensor
autonomous driving is presented, considering occupant fusion technology, merging sensor data with image
comfort, maneuver safety, vehicle speed, and road friction. processing algorithms to augment obstacle detection
A technique based on polynomial functions guarantees accuracy. This fusion enhances the SDV's ability to
trajectory continuity and flexibility under different perceive and react to its surroundings effectively.
circumstances, improving ride acceptability and comfort.  Comprehensive Methodology and Prototyping: We
Experiments and simulations confirm that the suggested plan present a comprehensive methodology encompassing
works well. The authors in [20] examine expanding vehicle prototype design, dataset utilization, ODM performance
perception using infrastructure-based communication. Real- evaluation, hardware implementation, and
time obstacle mapping for improved perception is possible experimentation. This holistic approach ensures thorough
using a map management architecture that uses vehicle-to- validation and optimization of the SDV system.
infrastructure connectivity. As demonstrated by testing and  Precise Object Positioning and Informed Decision-
implementation, vehicles can optimize trajectory planning Making: Our approach meticulously integrates ODM
based on the received information, enabling automated data, sensor fusion technology, and lane detection
movement. algorithms, enabling precise object positioning relative to
In [21], researchers also created a prototype that combined road lanes. This facilitates informed decision-making,
a support vector machine (SVM) for anomaly identification which is crucial for safe and efficient autonomous
with road lane detection and disparity map methods. navigation.
Although some SDV functionality is addressed in this work, Overall, our contributions aim to advance the
it emphasizes the need for full solutions to navigate capabilities and reliability of SDVs, particularly in complex
complicated road networks properly. Bill and Shahnasser and dynamic driving environments, paving the way for safer
suggested a prototype with cutting-edge hardware and object and more efficient autonomous transportation systems.
detection algorithms that could monitor people [22]. The rest of the paper is organized as follows: Section 2
Although this study showcases improvements in object presents the Proposed System Model, which comprises
detection, it is primarily concerned with watching subsections detailing the Image Processing Block, Lane
pedestrians and could need to be modified to handle various Detection Model, Object Detection Models, Sensor Fusion for
barriers seen on intricate road networks. Additionally, 2D Mapping, and Decision-Making Algorithm. In Section 3,
attempts have been made to improve the efficiency and Datasets and Performance Metrics are discussed. Section 4
accuracy of object detection. One promising framework for
elaborates on the experiments and results subdivided into
accurate object localization—which is essential for SDV
artificial testing environments and hardware implementation.
operations under challenging environments—is provided by
Finally, Section 5 concludes the paper by summarizing key
the EfficientDet D0 object detection model (ODM)
architecture [23]. Furthermore, it has become clear that findings and outlining avenues for future research.
integrating sensor fusion technology is essential to
II. MATH
enhancing SDV capabilities. Researchers have improved the
precision of obstacle detection by integrating data from To design an autonomous driving system for complex traffic
many sensors, including cameras and ultrasonic sensors [24]. networks, numerous problems and scenarios must be
considered, such as potholes, domestic animals, pedestrians
moving on the road, and many others. The prototype of our
B. CONTRIBUTIONS
SDV has been designed to give accurate and efficient results
Building upon prior work, this paper makes several
with minimal hardware resources. The basic architecture of
significant contributions to the field, as discussed as follows:
the whole design is shown in Fig 1. When turned on, the
 Integration of State-of-the-Art Object Detection Models:
vehicle captures the video using a Raspberry pi camera and
Inspired by recent hardware and algorithmic sends the data to the Raspberry Pi frame by frame via a
advancements, we integrate state-of-the-art object camera serial interface (CSI) bus. Then, image processing
detection models into our SDV prototype. Leveraging the techniques and ODMs have been applied to individual
EfficientDet D0 architecture, our approach aims to frames. The overall image processing is done in Raspberry
enhance detection accuracy and efficiency, particularly Pi. The lane detection algorithm monitors the road lanes and
regarding complex road networks. ensures the vehicle moves within them. The ODMs have
 Multi-Model Object Detection Approach: Recognizing been trained and developed to detect local vehicles,
the complexity of real-world environments, we adopt a pedestrians, animals, potholes, traffic lights, and other road
multi-model object detection strategy to ensure signs present on the roads.
robustness and expedited processing. By employing All the data processed in the image processing block in
multiple object detection models, we achieve precise Raspberry pi is then transferred to the Arduino UNO for
further decision-making. The data is transmitted via a 4-bit

2 VOLUME XX, 2017

 Author Name: Preparation of Papers for IEEE Access (February 2017)

data bus. The 2D mapping is done by sensor fusion protocol. The data received is then passed on to the
technology using a camera sensor and an array of ultrasonic sophisticated decision-making algorithm, which provides
sensors, which keeps measuring the distances of obstacles. optimal vehicle control decisions. Then, the master Arduino
All the sensors are connected to the slave Arduino UNO, device sends pulse width modulation (PWM) signals to the
which then transfers the data to the master Arduino device motor driver, which drives the motors making the robot
via an I2C (Inter-Integrated-Circuits) communication move accordingly.

FIGURE 1. Basic design structure of the prototype.

(a) (b) (c)

FIGURE 2. (a) Frame with highlighted ROI (b) Warp perspective transform (c) Lanes detection.

2 VOLUME XX, 2017

 Author Name: Preparation of Papers for IEEE Access (February 2017)

A. IMAGE PROCESSING BLOCK a 

The video frames are captured using a Raspberry Pi camera q  max  H ( x )  , where x    1, a  1 (4)
2 
and sent to Raspberry pi via CSI cable. The Raspberry pi
camera is attached in such a way that it captures the frames x2  H 1 ( q) (5)
in the best possible view. The received video frames are pre-
processed by resizing them into 400 × 240 resolutions and x1  x2
are captured in lower resolutions to reduce the amount of LaneCentre  (6)
2
computational processing. It also results in faster execution
of the ODMs and algorithms. The ODMs use convolutional a 1
FrameCentre  (7)
neural networks (CNNs) to detect objects in an image. The 2
Raspberry pi camera captures the images in BGR format, and
error = LaneCentre – FrameCentre (8)
we convert the frames from BGR to RGB. All the digital
image processing algorithms are executed in Raspberry pi as
C. OBJECT DETECTION MODEL
they require hardware capable of processing large amounts
Identifying and detecting obstacles and road signs are
of data. The image processing block mainly has four sub-
essential tasks for an autonomous vehicle. To detect objects
blocks of algorithms.
in a given frame, we have used state-of-the-art ODMs, which
have high mean average precision (mAP) values and require
B. LANE DETECTION MODEL
fewer computational resources compared to other numerous
To detect the lanes on the road, we first considered a region
existing object detection models. However, for the hardware
of interest (ROI) from the frame, and the warp perspective
SDV prototype experimentation, we trained ODMs using the
transform [25] is applied to it as shown in Fig. 2(a) and 2(b),
Haar Cascade classifier due to the hardware constraints of
respectively. Using perspective transform, we can change the
Raspberry Pi.
perspective of a given image to get better insights into the
required information. The thresholding function and canny 1) EFFICIENTDET OBJECT DETECTION MODEL
edge detection are applied to the current pre-processed frame The EfficientDet D0 ODM [23] has been developed to detect
to distinguish the road lanes from the road and other highly accurate objects using less computational power. This
surroundings. Let us assume I(x, y) is the 2D image matrix model has been developed by scaling single-shot detector
of the warp perspective transform of the ROI frame, and its (SSD) Efficient Net neural networks [26], incorporating a
resolution is a × b pixels. So, x ∈ (0, a) and y ∈ (0, b). Let novel bi-directional feature network (BiFPN). This ODM
the histogram be H(x) where 0 < x < a. uses CNNs as the base network to classify and detect objects
in an image. EfficientDet D0 ODM has broken existing
b records by achieving state-of-the-art accuracy by nine times
H ( x )   I ( x, y ) (1) smaller than the existing state-of-the-art object detectors. It
y 0 also requires less computational power than other object
detectors.
Let x1 and x2 be values of x at which H(x) is extensive and 2) CUSTOM TRAINED OBJECT DETECTION MODELS
represents the position of lanes in the x-direction. From this, To reduce the computational time and the neural network's
we can find the positions of the lane centre. By finding the size, we have trained and built three object detection models
pixel difference between the positions of the lane centre, see instead of detecting different kinds of objects.
(6), with the centre of the frame, see (7), we can calculate the Obstacle detector: This model is trained to detect obstacles
deviation of the vehicle from the centre of the road. This usually found on complex roads. It can detect indigenous
deviation is called the error, see (8). In Fig. 2(c), the center vehicles like cars, motorcycles, scooters, buses, trucks, auto-
blue line represents the frame centre and the green line rickshaws, tractors, etc. It can also detect the presence of
represents the centre of the lanes. This error can be used to obstacles like pedestrians, dogs, cows, and other domestic
steer the vehicle to move within the road lanes. The values animals, which are usually found on complex roads. This
of LaneCenter can be altered based on the position and model classifies the obstacles and gives the rectangular
motion of the obstacles to maneuver the vehicle around the coordinates of their presence.
obstacles within the road lanes. To find the obstacles in the Traffic signs and signals detector: This model is built to
given frame, we used ODMs. detect various traffic signs and signals that SDVs usually
come across on a complex road. For this, we have chosen
 a around 21 symbols that are important and frequently occur.
p  max  H ( x )  , where x   0,  (2) Pot-holes detector: A separate object detector has been
 2
trained to detect potholes on the road. To train this detector,
x1  H 1 ( p ) (3) we have collected data from various sources as the pot-holes
can be of different types and sizes based on climatic
conditions, seasons and surrounding environment.

2 VOLUME XX, 2017

 Author Name: Preparation of Papers for IEEE Access (February 2017)

FIGURE 3. Detection of obstacles, pot-holes, road signs, and traffic signals using EfficientDet ODMs.

3) OBJECT DETECTION USING HAAR CASCADE devastation. Thus, to overcome these problems, we have
CLASSIFIER implemented the 2D mapping feature, which plots a 2D map
Although the EfficientDet ODMs have excellent accuracy of obstacles around the prototype. The 2D mapping uses a
and require less computational power due to the limited camera sensor and an array of ultrasonic sensors around the
hardware resources in Raspberry Pi, the execution of vehicle. Using sensors such as Radars and Lidars can give
EfficientDet ODMs gives fewer frame rates. This can reduce precise and accurate 2D plots, but it also leads to higher
the performance of the SDV prototype. To test the SDV processing power consumption and hardware resources.
prototype in the artificial environment, we built ODMs using These sensors are placed at the critical hotspots of the vehicle
Haar Cascade classifiers based on the Viola-Jones algorithm as shown in Fig. 4(a). Each ultrasonic sensor calculates the
[27]. These ODMs require very little computational power to distance of obstacles in a particular direction as shown in Fig.
execute, and using them results in higher frame rates. But the 4(c). This sensor data is then plotted into a 2D map and
drawback of using these ODMs is they cannot be used to stored as a structured packet, which is then transferred to the
detect universal data, unlike EfficientDet ODMs. We obtain master Arduino via an I2C communication protocol. This
the bounding box coordinates with respective class names data is used for performing simultaneous localization and
and their confidence scores in the images shown in Fig. 3. mapping (SLAM) of the surroundings of the SDV based on
The data from lane detection and object detection blocks are decisions made using a decision-making algorithm.
combined into a structured data packet sent to the master
Arduino via a 4-bit data bus. Only the objects present in ROI E. DECISION MAKING ALGORITHM
with good confidence scores and close to the SDV prototype An autonomous vehicle's essential part is designing the
are considered. This data is used to change the path and control system and decision-making algorithm. It is crucial
motion of the SDV. This data is used to construct a 2D plot to control the speed, acceleration, and direction of the
of the surroundings of the SDV by combining the data with vehicle's motion based on the surrounding obstacles
the sensor data using sensor fusion technology. efficiently and effectively. In [24], the researchers have
discussed the state-of-the-art sensor fusion technologies and
D. 2D MAPPING USING SENSOR FUSION different embedded software methodologies that define the
There are some unique cases when the cameras on the SDV logic between sensor information and actuation decisions. A
cannot detect the obstacles around the vehicle. These cases state-of-the-art algorithm has been designed and developed
involve the inability of object detection models to detect after many experiments and simulations of the prototype in
obstacles or obstacles present in the blind spots of the camera different traffic scenarios using “automated driving system
or the presence of transparent obstacles like glass, which design and simulation” in MATLAB. We developed a robust
might be hard to detect using image processing techniques. control system based on these simulations that can make
These situations might lead to accidents and cause near-optimal decisions.

2 VOLUME XX, 2017

 Author Name: Preparation of Papers for IEEE Access (February 2017)

(a) (b) (c) (d)

FIGURE 4. (a) Position of sensors on the vehicle (b) Obstacles around a prototype in real-time (c) A rough 2D map using ultrasound sensors readings
(d) 2D view (bird eye view) using sensor fusion.

FIGURE 5. Functional diagram of designed self-driving vehicle prototype.

The error data from the lane detection block is used to These metrics demonstrate the effectiveness of the ODMs in
help the vehicle move within the road lanes. Also, it receives detecting various objects and obstacles in complex road
the data of the obstacles' type, position and distance from the conditions.
obstacle detection model and 2D mapping block based on
which it will decide if the SDV has to change its motion. A. DATA SET
SLAM is used to plan the vehicle's path according to To train custom ODMs, we have created datasets by
overcoming the obstacles. The decision-making algorithm combining available datasets, such as the “COCO dataset”
also considers GPS data while making the decision. The [28], with our custom-created data to achieve better
traffic signs and signals object detector gives the position and performance on complex road networks. To train the
type of the signs and signals in the given frame. The obstacle detection model, we have prepared a dataset that
functional diagram of the SDV is shown in Fig. 5. This data contains around 12000 labeled images with 12 different
allows the SDV to make appropriate decisions and control classes of objects such as cars, motorcycles (including
and plan the vehicle's path accordingly using the vehicle scooters), bicycles, pedestrians, trucks, auto-rickshaws,
control system. busses, tractors, dogs, cows, and other domestic animals as
The vehicle control system uses PID controllers to keep shown in Fig. 6(a). To build a traffic signs and signals
the vehicle within the desired road lanes, control the speed, detection model, we have created a dataset of 1500 labeled
and change the vehicle's smooth direction of motion. images with 21 signs frequently seen on complex roads, as
shown in Fig. 6(b). This data has been collected from various
III. DATA SET AND PERFORMANCE METRICS sources online, such as Google images, and labeled
The dataset used for training custom ODMs in the self- manually. A custom dataset of 900 labeled images of pot-
driving vehicle SDV system was created by combining holes of different types and sizes, which are explicitly found
publicly available datasets, such as the COCO dataset, with on complex roads, has been created to train the object
custom data tailored for complex road environments. This detection model. The data is divided into 80% of the data
section also includes performance metrics of various ODMs. used to train the ODMs and 20% for testing.

2 VOLUME XX, 2017

 Author Name: Preparation of Papers for IEEE Access (February 2017)

precision for each object class is shown in Figs. 7(a) and

7(b). In [29], researchers have built a model that can detect
vehicles of 5 different classes on complex roads with a mean
average precision (mAP) of 0.70 by using the YOLO V3
object detection model. We have built a model that can detect
11 different obstacle classes with a mAP of 0.729. Other
metrics, such as average recall and total loss, are calculated
for every ODM. Further, Table I presents the performance
metrics of various ODMs.

(a) TABLE I
PERFORMANCE METRICS OF ODMS.
ODM Obstacles Traffic Signs Potholes
mAP 0.729 0.652 0.745
Average Recall 0.758 0.677 0.793
Total Loss 0.313 0.462 0.218

IV. EXPERIMENTS AND RESULT

This section outlines the testing and evaluation of a SDV
prototype designed for complex road navigation. The
experiments and results provided in this section validate the
effectiveness of the designed SDV prototype in handling a
(b) range of challenging road conditions, paving the way for
FIGURE 6. (a) Obstacles Dataset (b) Traffic signs and signals dataset. further advancements in autonomous vehicle technology.
A. ARTIFICIAL TESTING ENVIRONMENT
B. PERFORMANCE METRICS OF ODMS We built a road track 4.5 meters in length and 20 inches in
width to test the prototype of an SDV. This artificial mini-
road track resembles the road in real time. It also has some
curves and consists of two different lanes. The track also
contains many obstacles, potholes, road signs, and traffic
signals to test the prototype's performance.

(a)

FIGURE 8. Artificial mini road track.

B. HARDWARE IMPLEMENTATION
The outer structure of the prototype was built using 3D-
printed materials. Many on-shelf hardware components have
been used to build this model prototype. Raspberry Pi does
all the tasks that require high computational power, such as
(b) Image processing, execution of ODMs, and running complex
FIGURE 7. (a) Class-wise average precision of obstacle detector, (b) algorithms on the video frames captured by the Raspberry Pi
Class-wise average precision values of Traffic signs and signals
detector.
camera. The Arduino UNO boards collect data from
ultrasound sensors and control the DC motors via the L298N
The ODMs have been trained by using our custom-made motor driver using PWM signals. Lithium-ion batteries with
datasets. After completing the models' training, we were able 3.8V and 10000 mAh power banks power up the entire
to achieve excellent results with good precision and prototype. The overall hardware circuit design is shown in
accuracy. Average precision is a popular and essential metric Fig. 9(a), and the final 3D view of the built prototype looks
for measuring the accuracy of object detectors. The average like it does in Fig. 9(b).

2 VOLUME XX, 2017

 Author Name: Preparation of Papers for IEEE Access (February 2017)

(a)

(b)

FIGURE 9. (a) Hardware circuit design (b) Prototype of SDV.

C. EXPERIMENTS original GPS module has a precision of 4.9 meters that

The prototype has been tested in different scenarios and cannot be used to test a mini prototype. So we have generated
recreated in an artificial testing environment. We have artificial GPS signals to test our prototype. In this
created multiple tests to check the performance of the experiment, we have manually sent a signal to the prototype
prototype of the SDV. These results are compared with the to take the next U-turn. The master device in the prototype
software simulations that were created using the “driving has recognized this signal and sent a sequence of control
scenario designer” application in MATLAB. signals.
1) EXPERIMENT–1 (a) It measures the distance of the next U-turn and
In this experiment, a scenario is created when the road is adjusts the speed accordingly, as shown in Fig.
empty with no vehicles and obstacles, as shown in Fig. 10. 11(a).
The lanes are successfully detected in different lighting (b) It checks if there are any obstacles present to take a
conditions, and the lane detection algorithm has calculated U-turn as it reaches the U-turn.
accurate error values, which are used to move the prototype (c) If an obstacle is present, it waits for clearance and
within the road lanes. The algorithm has taken very little time successfully takes a U-turn, as shown in Fig. 11(b)
to execute for each frame and has achieved excellent results and 11(c).
with an average frame rate of 24 fps. (d) The prototype followed the sequence of steps and
successfully took a U-turn. The test results of the
2) EXPERIMENT–2 (ARTIFICIAL GPS SIGNALS) prototype in the artificial environment have
The GPS signals have been generated artificially as the matched the simulation results in MATLAB.

2 VOLUME XX, 2017

 Author Name: Preparation of Papers for IEEE Access (February 2017)

FIGURE 10. Lane detection and following.

(a) (b) (c)

FIGURE 11. (a) vehicle approaching U-Turn (b) vehicle taking U-Turn (c) vehicle completes taking U-Turn.

3) EXPERIMENT–3 (ROAD SIGNS AND TRAFFIC (a) The image processing techniques are applied to the
SIGNALS) detected traffic signal to determine its state.
In this experiment, we tested the prototype's response when (b) If the red light is detected, then signals are sent to
it came across road signs and traffic signals. A scenario has the vehicle control system to stop and wait until the
been recreated where a stop sign and a traffic signal are red light changes to a green light; when the traffic
present in the middle of the track. The prototype detects the light changes from red to green, the vehicle
stop sign present on the road and calculates the distance of continues moving in its designated path.
the signboard from the current position, as shown in Fig. 12. (c) If the green light is detected, it keeps on moving in
As soon as it reaches close to the stop sign, about 30 cm, it its original path.
stops for 10 seconds and then continues moving in its path. (d) The prototype has successfully passed this test and
After driving a certain distance, the vehicle comes across a followed the designed algorithms without any
traffic signal and calculates the distance from the current errors.
position. As it reaches close to the traffic signal, a sequence
of steps is followed:

2 VOLUME XX, 2017

 Author Name: Preparation of Papers for IEEE Access (February 2017)

FIGURE 12. SDV prototype coming across stop sign.

(a) (b)

(c) (d)

FIGURE 13. (a) SDV moving in its path (b) Detection of stationary obstacle (c) waits for some time and overtakes the stationary obstacle (d) Successfully
changes the lane.

4) EXPERIMENT–4 (PRESENCE OF STATIONARY 5) EXPERIMENT–5 (ROAD WITH HEAVY TRAFFIC)

OBSTACLE IN BETWEEN THE PATH) A scenario has been recreated in which vehicles are
A scenario has been created when a stationary vehicle is simultaneously moving in the current lane and the opposite
present in the middle of the road, as shown in Fig. 13(a). lane, along with pedestrians crossing the road. When the
When the prototype comes across such an obstacle when it prototype has been placed in this situation, the object
is moving in its assigned path, the obstacle detector present detection model and sensor array detect the distance and
in the prototype first detects the obstacle, then calculates its position of all the obstacles in the given frame. The decision-
distance, and finally follows a sequence of steps, see Fig. making algorithm follows a sequence of steps and generates
13(b). Then, as the prototype approaches close to the control signals to maneuver the SDV prototype by applying
obstacle, it stops and waits for 15 seconds. If the obstacle brakes, changing the direction of motion, and changing
does not move yet, the prototype applies the horn and waits speed accordingly. This scenario has been successfully
for another 10 seconds. If the obstacle does not move even tested in the custom-built artificial environment, as shown in
after applying the horn, it overtakes it, as shown in Figs. Fig. 14, and has matched the MATLAB simulation results.
13(c) and 13(d).

2 VOLUME XX, 2017

 Author Name: Preparation of Papers for IEEE Access (February 2017)

(a)

(b)

(c)

FIGURE 14. (a) SDV prototype moving on a road with heavy traffic (b) SDV prototype approaching a pedestrian crossing road and manoeuvring around
to avoid collision (c) SDV prototype approaching the auto-rickshaw.

6) EXPERIMENT–6 (POT-HOLE DETECTION) generates control signals either to reduce the speed of the
An experiment is conducted in such a way that a scenario is SDV prototype or to maneuver around the pot-hole, reducing
recreated where artificial potholes are created on the road. the impact of the pot-hole causing on the vehicle. In this
When the vehicle approaches close to the pot-hole, the pot- experiment, we created pot-holes of different sizes and tested
hole detector detects the positions of pot-holes, as shown in the prototype's performance. The vehicle was successful in
Fig. 15, and using this data the size of the pot-holes are tackling the pot-holes present on the road as per the designed
calculated. Based on this, the decision-making algorithm algorithm.

2 VOLUME XX, 2017

 Author Name: Preparation of Papers for IEEE Access (February 2017)

FIGURE 15. Presence of domestic animals on the path.

(a)

(b)

(c)

FIGURE 16. SDV prototype approaching a cow crossing the road (b) SDV prototype stops when it approaches close to cow and applies horn till it
moves from the vehicle’s path (c) SDV starts moving in its designated path when the cow moves away.

2 VOLUME XX, 2017

 Author Name: Preparation of Papers for IEEE Access (February 2017)

7) EXPERIMENT–7 (PRESENCE OF DOMESTIC Systems Using Potential Fields,” IEEE Trans. on Intel. Transport.
ANIMALS ON THE PATH) Sys., vol. 24, no. 5, pp. 5351-5361, May 2023.
[5] K. Kuru, “TrustFSDV: Framework for Building and Maintaining Trust
A scenario has been created where the vehicle moving on the in Self-Driving Vehicles,” IEEE Access, vol. 10, pp. 82814-82833,
road comes across a cow present in between the roads, as Aug. 2022.
shown in Fig. 16(a). The obstacle detection model in the [6] L. Li, W. -L. Huang, Y. Liu, N. -N. Zheng, and F. -Y. Wang,
prototype detects the cow's position and sends this data to the “Intelligence Testing for Autonomous Vehicles: A New Approach,”
IEEE Trans. on Intel. Vehi., vol. 1, no. 2, pp. 158-166, Jun. 2016.
decision-making algorithm. A sequence of predefined steps [7] S. W. Loke, “Cooperative Automated Vehicles: A Review of
is followed to tackle this kind of situation. As the prototype Opportunities and Challenges in Socially Intelligent Vehicles Beyond
approaches close to the cow, the decision-making algorithm Networking,” IEEE Trans. on Intel. Vehi., vol. 4, no. 4, pp. 509-518,
in Raspberry Pi sends signals to slow down and stop the Dec. 2019.
[8] M. B. Younes, and A. Boukerche, “Towards a Sustainable Highway
vehicle, as shown in Fig. 16(b). Then, it applies the horn and Road-Based Driving Protocol for Connected and Self-Driving
waits until the cow moves from the vehicle’s path. If it has Vehicles,” IEEE Trans. on Sust. Comp., vol. 7, no. 1, pp. 235-247, Mar.
not moved yet, the signals are sent to maneuver around the 2022.
cow slowly and then continue moving in its original [9] K. Kuru, “Conceptualisation of Human-on-the-Loop Haptic
Teleoperation With Fully Autonomous Self-Driving Vehicles in the
designated path, as shown in Fig. 16(c). The prototype has Urban Environment,” IEEE Open J. of Intel. Transp. Sys., vol. 2, pp.
successfully passed this scenario by following the steps of 448-469, Dec. 2021.
the designed algorithm accurately without causing any [10] Z. Li, J. Jiang, W. -H. Chen, and L. Sun, “Autonomous Lateral
errors. Maneuvers for Self-Driving Vehicles in Complex Traffic
Environment,” IEEE Trans. on Intel. Vehi., vol. 8, no. 2, pp. 1900-
1910, Feb. 2023.
V. CONCLUSIONS AND SCOPE OF FUTURE WORK [11] J. Liu et al., “Semantic Traffic Law Adaptive Decision-Making for
The research work given in this paper presents a Self-Driving Vehicles,” IEEE Trans. on Intel. Transp. Sys., vol. 24, no.
comprehensive approach to enhancing autonomous driving 12, pp. 14858-14872, Dec. 2023.
performance on complex road networks. While previous [12] M. Martínez-Díaz, F. Soriguera, “Autonomous vehicles: theoretical
and practical challenges,” Transp. Res. Proc., vol. 33, pp. 275-282,
prototypes, such as those developed by researchers in [30], 2018.
have made strides in detecting vehicles and traffic signals [13] A. Barzegar, O. Doukhi, and D. J. Lee, “Design and Implementation
using YOLO V3 ODMs and Canny edge detection for lane of an Autonomous Electric Vehicle for Self-Driving Control under
identification, they struggle to navigate the myriad GNSS-Denied Environments,” Appl. Scien., vol. 11, pp. 3688, Apr.
2021.
challenges posed by complex road traffic conditions. [14] D. Oliva, et al., “Intelligent Semantic Segmentation for Self-Driving
Addressing these challenges, our prototype excels in Vehicles Using Deep Learning,” Comp. Intel. and Neurosci., Jan 2022.
detecting a wide range of obstacles and road signs commonly [15] S. A. Bagloee, M. Tavana, M. Asadi, and T. Oliver, “Autonomous
encountered in complex road networks. Moreover, our vehicles: challenges, opportunities, and future implications for
transportation policies,” J. of Modern Transp., vol. 24, pp. 284-303,
robust decision-making algorithm and vehicle control Aug. 2016.
system enable smooth navigation on chaotic and uneven [16] T. Peng et al., “Lane-change path planning and control method for
roads. Leveraging sensor fusion technology to plot a 2D map self-driving articulated trucks,” J. of Intel. and Conn. Vehi., vol. 3, no.
significantly improves accuracy compared to existing 2, pp. 49-66, 2020.
[17] K. Messaoud, I. Yahiaoui, A. Verroust-Blondet, and F. Nashashibi,
prototypes [21]–[23], [30]. Moving forward, our future work “Attention Based Vehicle Trajectory Prediction,” IEEE Trans. on Intel.
will focus on optimizing the decision-making algorithm for Vehi., vol. 6, no. 1, pp. 175-185, Mar. 2021.
even wiser decisions and incorporating state-of-the-art safety [18] Y. Sun, D. Ren, S. Lian, S. Fu, X. Teng, and M. Fan, “Robust Path
features for SDVs. In summary, our successful design and Planner for Autonomous Vehicles on Roads With Large Curvature,”
IEEE Robotics and Auto. Let., vol. 7, no. 2, pp. 2503-2510, Apr. 2022.
implementation of a mini hardware prototype demonstrate its [19] J. Hu, Y. Zhang, and S. Rakheja, “Adaptive Lane Change Trajectory
effectiveness in replicating various scenarios without errors, Planning Scheme for Autonomous Vehicles Under Various Road
laying the foundation for further advancements in Frictions and Vehicle Speeds,” IEEE Trans. on Intel. Vehi., vol. 8, no.
autonomous driving technology on complex road networks. 2, pp. 1252-1265, Feb. 2023.
[20] M. Szántó, C. Hidalgo, L. González, J. P. Rastelli, E. Asua, and L.
Additionally, the integration of new sensor technologies Vajta, “Trajectory Planning of Automated Vehicles Using Real-Time
such as Lidar holds promise for further improving our Map Updates,” IEEE Access, vol. 11, pp. 67468-67481, 2023.
prototype's performance. [21] M. Fathy, N. Ashraf, O. Ismail, S. Fouad, L. Shaheen, and A. Hamdy,
“Design and implementation of self-driving car,” Proc. Comp. Sci.,
vol. 175, pp. 165-172, 2020.
REFERENCE [22] L. Bill, and H. Shahnasser, “Development and Implementation of an
[1] A. Chowdhury, G. Karmakar, J. Kamruzzaman, A. Jolfaei, and R. Das, Autonomously Driven Vehicle Prototype,” in IEEE 2019 2nd Int.
“Attacks on Self-Driving Cars and Their Countermeasures: A Survey,” Conf. on Electr. Tech., Chengdu, China, pp. 310-314.
IEEE Access, vol. 8, pp. 207308-207342, Nov. 2020. [23] M. Tan, R. Pang, and Q. V. Le, “EfficientDet: Scalable and Efficient
[2] E. Ohn-Bar, and M. M. Trivedi, “Looking at Humans in the Age of Object Detection,” in 2020 IEEE/CVF Conf. on Comp. Vis. and Patt.
Self-Driving and Highly Automated Vehicles,” IEEE Trans. on Intel. Recog., Seattle, WA, USA, pp. 10778-10787.
Vehi., vol. 1, no. 1, pp. 90-104, Mar. 2016. [24] S. J. Babak, S. A. Hussain, B. Karakas, and S. Cetin, “Control of
[3] W. Liu et al., “A Systematic Survey of Control Techniques and autonomous ground vehicles: a brief technical review,” in 2017 IOP
Applications in Connected and Automated Vehicles,” IEEE Inter. of Conf. Series: Mate. Sci. and Eng. vol. 224.
Things J., vol. 10, no. 24, pp. 21892-21916, Dec. 2023. [25] Warp Perspective Transform, https://theailearner.com/tag/cv2-
[4] N. P. Bhatt, A. Khajepour, and E. Hashemi, “MPC-PF: Socially and warpperspective/
Spatially Aware Object Trajectory Prediction for Autonomous Driving

8 VOLUME XX, 2017

 Author Name: Preparation of Papers for IEEE Access (February 2017)

[26] M. Tan, and Q. V. Le, “EfficientNet: Rethinking Model Scaling for VISALAKSHI ANNEPU received the B. Tech.
Convolutional Neural Networks,” in Proc. of the 36th Int. Conf. on degree in computer science and engineering from
Mach.Lear., Long Beach, pp. 6105-6114, 2019. Jawaharlal Nehru Technical University, Kakinada,
[27] P. Viola, and M. Jones, “Rapid object detection using a boosted India, in 2011, the M. Tech. degree in computer
cascade of simple features,” in Proc. of the 2001 IEEE Comp. Soc. science and Technology from Andhra University,
Conf. on Comp. Vis. and Pat. Recog., Kauai, HI, USA, 2001. Visakhapatnam, India, in 2013, and Ph.D. degree
[28] T. Y. Lin, et al., “Microsoft COCO: Common Objects in Context,” in computer networks from VIT Vellore, India, in
Lect. Notes in Comp. Sci., vol. 8693, pp. 740-755, 2014. 2020. She was an Assistant Professor with the
[29] A. K. Prajapati, A. Gora, A. Agarwal, and I. Ghosh, “Use of Computer Department of Computer Engineering, Sri Sivani
Vision to Automatize Traffic Data Collection under Mixed Traffic Institute of Technology, Srikakulam, Andhra
Condition,” in 2nd ASCE India Conf. on Chal. of Resi. and Sust. Infra. Pradesh, from June 2014 to May 2016. Currently, she is an Assistant
Develop. in Emerg. Econ., 2020. Professor with the School of Computer Science and Engineering, VIT-AP
[30] M. V. Rajasekhar, and A. K. Jaswal, “Autonomous vehicles: The future University, Amaravati, India. Her research interests include wireless
of automobiles,” in 2015 IEEE Int. Transp. Electrifi. Conf., Chennai, networking, artificial intelligence, soft computing techniques, and neural
India, pp. 1-6, 2015. networks.

KALAPRAVEEN BAGADI received a B.E.

degree in electronics and communication Khaled Rabie received the M.Sc. and Ph.D.
engineering from Andhra University, India, in degrees in electrical and electronic engineering
2006 and an M.Tech. degree in electronic systems from the University of Manchester, in 2011 and
and communication and the Ph.D. degree in 2015, respectively. He is currently a Reader with
wireless communication from the Department of the Department of Engineering, Manchester
Electrical Engineering, National Institute of Metropolitan University (MMU), UK. He has
Technology, Rourkela, India, in 2009 and 2014, worked as a part of several largescale industrial
respectively. He was a Full Professor with the School of Electronics projects and has published 200+ journal and
Engineering (SENSE), Vellore Institute of Technology (VIT), Vellore, conference articles (mostly IEEE). His current
India, until April 2023. Currently, he is with the School of Electronics research interests focus on designing and
Engineering (SENSE), VIT-AP University, Amaravati, Andhra Pradesh, developing next-generation wireless communication systems. He serves
India. His research interests include SDMA, MIMO, OFDM, NOMA, D2D regularly on the technical program committee (TPC) for several major IEEE
communication, cognitive radio, UAV communication, and artificial conferences, such as GLOBECOM, ICC, and VTC. He has received many
intelligence. He has completed six Ph.D. theses under his guidance and is awards over the past few years in recognition of his research contributions
currently guiding three scholars. He is an Academic Editor of Wireless including the Best Paper Awards at the 2021 IEEE CITS and the 2015 IEEE
Communication and Mobile Computing and Applied Computational ISPLC, and the IEEE ACCESS Editor of the month award for August 2019.
Intelligence and Soft Computing. He also reviews journals like IEEE He is currently serving as an Editor of IEEE COMMUNICATIONS
Access, Wireless Personal Communications, IET Communications, and LETTERS and IEEE Internet of Things Magazine. Khaled is also a Fellow
Telecommunication Systems. He has published over 50 research articles in of the U.K. Higher Education Academy (FHEA) and a Fellow of the
various refereed international journals, such as IEEE Access, Neural European Alliance for Innovation (EAI).
Computing and Applications, Wireless Personal Communication, IET
Communication, International Journal of Communication Systems, and Shafiq Ahmad received his Ph.D. degree from
Radio Science. RMIT University, Melbourne, Australia, master's
degree from Asian Institute of Technology,
VAEGAE NAVEEN KUMAR received the Bangkok, Thailand & Technical University
B.Tech. degree in instrumentation engineering Hamburg, Germany, bachelor's degree in
from Nagarjuna University, Guntur, India, in mechanical engineering from UET Lahore,
2001, the M.Tech. degree in electronics Pakistan. Dr. Ahmad has nearly thirty years work
engineering from Jawaharlal Nehru Technological experience both in industry and academia. He has
University, Kakinada, India, in 2010, and the research experience in R & D Centre and process
Ph.D. degree in electrical and electronics automation with Deutsche Aerospace AG (DASA)
engineering from Vellore Institute of Technology, Hamburg, Germany and Seagate Technology at
Vellore, India. His interests are in the area of Bangkok, Thailand. He obtained several distinguished performance awards.
sensors and signal conditioning, signal processing, soft computing, Dr. Ahmad is also a certified professional and consultant in deploying Six
intelligent systems, electronic nose, genomic signal processing, and IoT. He Sigma (DMAIC) business improvement model. Before joining King Saud
is currently working as an Associate Professor with the School of University (KSU), Dr. Ahmad was faculty at Al-Yamamah University, and
Electronics Engineering, Vellore Institute of Technology, Vellore, India. He he has achieved Best performance Award. Currently, Dr. Ahmad is
has authored over fifteen SCI-IF publications, and over sixteen Scopus Professor at IE dept. College of Engineering KSU Riyadh, Saudi Arabia.
indexed journals and conferences. His publications are in IEEE Sensors Dr. Ahmad has an excellent research profile and has obtained Excellent
journal, Biocybernetics & Biomedical Engineering, Biomedical Signal Researcher Award in Scientific Research from College of Engineering at
Processing & Control, Sensors & Actuators, A: Physical, Measurement: KSU in 2019. He has published a research book, a book chapter and more
Journal of the International Measurement Confederation and IET Science than 150 refereed scientific research articles. Dr. Ahmad is ranked among 2
Measurement and Technology. Dr. Naveen is a lifetime member of ISTE % of top-cited scientists.
and member of International Association of Engineering. He has been a
Faculty advisor and coordinator for IEEE signal processing society, VIT Thokozani Shongwe received the B.Eng.
student chapter since 2017. degree in electronic engineering from the
University of Swaziland, Swaziland, in 2004, the
M.Eng. degree in telecommunications engineering
from the University of the Witwatersrand, South
Africa, in 2006, and the D.Eng. degree from the
University of Johannesburg, South Africa, in 2014.
He is currently an Associate Professor with the
Department of Electrical and Electronic Engineering Technology,
University of Johannesburg. His research interests include communications,
error-correcting coding, power-line communications, cognitive radio, smart

8 VOLUME XX, 2017

 Author Name: Preparation of Papers for IEEE Access (February 2017)

grids, visible light communications, machine learning, and artificial

intelligence. He was a recipient of the 2014 University of Johannesburg
Global Excellence Stature (GES) Award, which was awarded to him to carry
out his postdoctoral research with the University of Johannesburg. He was
a recipient of the TWAS-DFG Cooperation Visits Program funding to do
research in Germany, in 2016. Other awards that he received in the past are
the Post-Graduate Merit Award Scholarship to pursue the master’s degree
with the University of the Witwatersrand, in 2005, which is awarded on a
merit basis. In the year 2012, he (and his coauthors) received the Best
Student Paper Award from the IEEE ISPLC 2012 (Power Line
Communications Conference), Beijing, China.

8 VOLUME XX, 2017