Realisé par : Imane ELAMRAOUI

Encadré par : pr.Youness ADNANI

Filliére : IISA
Année universitaire :2023/2024

Advanced Driver Assistance Systems (ADAS)

4-Role of Machine Learning & Deep Learning in
ADAS
1-INTRO:
this section of the course focuses on the role of machine learning and deep learning in the
development of automotive technology. It highlights the evolution of these technologies over
the past decade and their increasing adoption in modern automotive systems. The section aims
to provide an awareness of how artificial intelligence is currently utilized in ADAS and
autonomous driving applications, without delving into the intricacies of implementation.
Instead, it outlines the general areas where machine learning and deep learning are employed,
such as sensor fusion and processor development.
2-ML & DL in ADAS –Overview :

ML & DL in ADAS:

Machine learning (ML) and deep learning (DL) play integral roles in Advanced Driver
Assistance Systems (ADAS), revolutionizing vehicle safety and autonomy. Here's how they're
applied:
1. Object Detection and Recognition: ML/DL models are trained to detect and recognize
objects such as vehicles, pedestrians, cyclists, and traffic signs from sensor data like cameras,
LiDAR, and radar. DL models like Convolutional Neural Networks (CNNs) excel in this task
due to their ability to learn hierarchical features.
2. Semantic Segmentation: ML/DL techniques are used for pixel-level classification,
enabling the understanding of the scene's context and spatial relationships. This aids tasks like
lane detection, road boundary identification, and obstacle segmentation.
3. Sensor Fusion: ML/DL algorithms combine data from various sensors to improve
detection accuracy and robustness. By integrating information from cameras, LiDAR, radar,
and ultrasonic sensors, ADAS systems make more informed decisions.
4. Behavior Prediction: ML/DL models predict the future trajectories of objects based on
current behavior and contextual cues. This capability helps anticipate potential collisions and
enables proactive driving decisions.
5. Driver Monitoring: ML/DL algorithms monitor driver behavior in real-time, detecting
signs of fatigue, distraction, or impairment. This information is crucial for issuing alerts or
initiating autonomous interventions to prevent accidents.
6. Anomaly Detection: ML/DL techniques identify abnormal situations or events on the road,
such as sudden obstacles or erratic behavior of other road users. Early detection allows ADAS
systems to respond promptly to ensure safety.
7. Map and Localization: ML/DL methods are used for mapping and localization, accurately
positioning vehicles within their surroundings. This enables features like lane-keeping
assistance and autonomous driving by providing precise spatial awareness.
8. Adaptive Control Systems: ML/DL algorithms develop adaptive control systems that
adjust vehicle dynamics based on road conditions, traffic patterns, and driver behavior. This
enhances vehicle stability and safety while providing smoother driving experiences.

1
9. Continuous Learning: ML/DL models in ADAS systems continuously learn from new
data and experiences, improving performance and adaptability over time. Techniques like
online learning, transfer learning, and reinforcement learning facilitate this continuous
improvement.
High amount of data output from Advance sensors-for ADAS and AD:
The puck LITE provides high definition 3-dimensional information about the surrounding
environment.

High resolution Radar from Astyx provides more than 1000 Detections per scan.

Multiple Camera setup on vehicle-From Mobileye:

2
 Multiple sensors setup on vehicle-From Tesla & Ford:

3-ML & DL in ADAS- Sensors and sensors fusion:

Use of Deep Learning in Automotive Radar sensor:

 Object Classification:
 Traditional approach: Use of object dimensions, speed and other paramters to
estimate the type of object.
 Deep Learning approach: Use of CNN (convolutional neural networks) Models
 and using directly Radar spectrum (RD map) as input.
 Reference Technical Paper: Deep Learning-based Object Classification on
Automotive Radar Spectra

1
 Vehicle Detection:
 Traditional approach for vehicle detection using automotive radar, clustering
algorithms were employed to group detections from one object, followed by manual
feature extraction to detect vehicles.
 in the deep learning-based approach, range, azimuth, and Doppler tensors are
utilized to generate RA (Range-Azimuth), RD (Range-Doppler), and AD (Azimuth-
Doppler) tensors, which are 2D representations used as input to the deep learning
network for detecting vehicles.
 Reference technical paper titled "Vehicle Detection With Automotive Radar Using
Deep Learning on Range-Azimuth-Doppler Tensors" likely details this method
further, explaining how deep learning algorithms are trained on these tensors to
accurately detect vehicles in radar data. This approach offers the advantage of
automated feature extraction and potentially improved accuracy compared to
traditional methods.

Use of Deep Learning in Automotive Lidar:

 Lidar 3D Point Cloud Semantic Segmentation: The paper likely discusses deep
learning approaches for segmenting LiDAR point clouds into semantically meaningful
regions , such as roads, buildings, vehicles, pedestrians, and vegetation. Semantic
segmentation is crucial for scene understanding and environment perception in
autonomous driving applications.
 Object Detection and Classification: The review paper likely explores deep
learning-based methods for detecting and classifying objects in LiDAR point clouds,
including vehicles, pedestrians, cyclists, and other obstacles. Object detection and
classification are essential for collision avoidance, path planning, and decision-making
in autonomous vehicles.
 Reference Paper: Deep Learning for LiDAR Point Cloudsin Autonomous Driving: A
Review

2
 Deep Learning enabled Automotive Thermal Camera-From Adasky (a company
specializing in automotive thermal cameras) :
 ADASKY Thermal Camera is a revolutionary intelligent, high-resolution LWIR
thermal camera, designed for vehicle safety and perception systems to enhance ADAS
and enable all levels of automation. It is the only automotive grade shutterless thermal
camera available that can see and detect what other sensors cannot, day or night,
without being compromised by complete darkness, blinding lights or harsh weather.

 The ADASKY Thermal Camera can detect objects up to 300 meters (984 ft) and
classify living beings at over than 200m (656 ft) in all visibility conditions, while
typical vehicle low-beam headlamps combined with daytime cameras are only able to
detect an object around 80m (262 ft) and cannot determine if the object is a living
being. Current sensors for ADAS haven’t been able to prevent the growing number of
pedestrian fatalities, especially at night. By turning night into day, ADASKY Thermal
Camera can reduce accidents and save countless lives.

 The adasky thermal camera hardware and software are designed and
manufactured in-house for optimal compatibility, quality and scalability. the camera
comes in multiple fields of view to address different scenarios on the road, enveloping
the automated vehicle in a 360° layer of safety. being shutterless, the adasky thermal
camera never “blinks” or pauses to recalibrate, making driving safer than other sensors
in its class. its state-of-the-art computer vision algorithms also include object detection
and classification of multiple types of objects, as well as free-space detection, and
even ttc (time to collision) calculation to support forward collision warning (fcw) and

1
 automatic emergency braking (aeb) features, all made possible by using only a single
thermal camera.

ML & DL enabled Automotive Multi Function Mono Camera-From Continental:

Multi Function Mono Camera – MFC500 The latest camera generation MFC500 from
Continental is a crucial part of our modular, scalable and interlinked camera platform and
provides solutions ranging from advanced driver assistance functions (e.g. NCAP 2020) to
Highly Automated Driving (HAD). It features excellent night vision, high image resolution of
up to eight megapixel and wide field of view of up to 125 degrees, enabling cross-traffic
objects (vehicles, bicycles, pedestrians etc.) to be detected even earlier. In addition, it includes
leading-edge technologies such as machine learning and neural networks to come to a real
scene understanding and is connected with the world with eHorizon, Road Database and over-
the-air updates.
 Features & Benefits
1. Head Light Assist supporting binary high beam control up to glare-free matrix beam
lights.
2. Lane & Road Departure Protection (targeting NCAP's ELK and LSS) incl. consideration
of oncoming vehicles as well as Lane Keeping Assist.
3. Emergency Brake Assist (targeting NCAP's AEB city/VRU/interurban).
4. Camera-based Adaptive Cruise Control with speed limit input (targeting NCAP's
intelligent ACC).
5. Free Space Detection for Highly Automated Driving.

2
6. Traffic Sign Assist recognizing explicit & implicit speed limits incl. supplementary,
warning and give-way signs (targeting NCAP's basic and advanced SLIF).
7. Traffic Light Recognition - even when host vehicle is standing first in row.
8. Unsteady Driving Warning to indicate potential driver distraction or fatigue.
9. eHorizon and Road Database for map creation and self-localization.
10. Cornerstone for Highly Automated Driving.

Use of Deep Learning in Automotive Sensor Data Fusion:

 Object Detection and Classification using Sensor Data fusion

 Automotive Radar detections and camera images are fused together using Deep
Learning Network

 Radar points are mapped on the camera images and then data fusion is carried out.

 Dataset from nuScenes and the dataset from TUM are used for work.

 Reference paper: A Deep Learning-based Radar and Camera Sensor Fusion

Architecture for Object Detection.

1
Use of Deep Learning in Automotive Sensor Data Fusion:
The use of deep learning in automotive sensor data fusion, particularly for 3D object
detection, has become increasingly prevalent in recent years. In the referenced paper, the
authors leverage synchronized data from cameras, lidar, and radar sensors to train and
evaluate their model.
Here's a breakdown of the key components mentioned in the paper:
 Dataset: The dataset consists of 455 frames of synchronized camera, lidar, and radar
data. This dataset serves as the foundation for training and evaluating the deep
learning model.
 Sensor Specifications:
 Radar: Astyx 6455 HiRes sensor is used, generating point clouds with approximately
1000-10000 points per frame. Each point contains x, y, z position, magnitude, and
Doppler information (radial velocity).
 Camera: Point Grey Blackfly camera captures images with a size of 2048×618 pixels.
 Lidar: Velodyne VLP-16 is used to obtain lidar point clouds.
 Objective: The objective is 3D object detection, where the deep learning model
processes the synchronized sensor data to detect and localize objects in 3D space.
 Model Architecture: The paper likely describes the deep learning architecture
designed for this task, which likely involves fusion layers to effectively integrate
information from multiple sensor modalities (camera, lidar, radar).
 Training and Evaluation: The model is trained on the provided dataset and evaluated
to assess its performance in detecting objects accurately across different scenarios.
 Results: The paper would present the results of the experiments, including metrics
such as precision, recall, and accuracy, demonstrating the effectiveness of the
proposed approach compared to existing methods.
 Applications: Such technology has significant applications in autonomous driving,
advanced driver-assistance systems (ADAS), and robotics, where accurate perception
of the environment is crucial for safe and efficient operation.

2
4-ML & DL in ADAS – Processors:
AI enabled image recognition processor from ADAS-from toshiba Visconti:

1
Evaluation of embedded vision processor for Deep Learning Algorithms:
 Vision processors for object detection

Since the driver is already facing forward, a front facing camera may seem unnecessary. So,
to be of value, the front facing camera has to be consistently faster than the driver in detecting
and alerting for obstacles. While an ADAS system can physically react faster than a human
driver, it needs embedded vision to provide real-time analysis of the streaming video and
know what to react to.

Vision processors are based on heterogeneous processing units. That means the programming
tasks are divided into processing units with different strengths (Figure 2). Most of the code
will be written using C or C++ for a traditional 32-bit scalar processor, which provides an
easy-to-program processor. The vector DSP unit will perform most of the computations,
because it’s very large instruction word can handle a lot of parallel computations for pixel
processing of each incoming image.

Detecting a pedestrian in front of a car is part of a broad class of “object detection.”

For each object to be detected, traditional computer vision algorithms were hand-crafted.
Examples of algorithms used for detection include Viola-Jones and more recently Histogram
of Oriented Gradients (HoG). The HOG algorithm looks at the edge directions within an
image to try to describe objects. HOG was considered state-of-the art for pedestrian detection
as late as 2014.
 Emergence of deep learning for object detection
Although the concept of neural networks, which are computer systems modeled after the
brain, have been around for a long time, only recently have semiconductors achieved the
processor performance to make them a practical reality. In 2012, a convolutional neural
network (CNN)-based entry into the annual ImageNet competition showed a significant
improvement in accuracy in the task of image classification over the traditional computer
vision algorithms. Because of the improved accuracy, the use of neural network-based
techniques for image classification, detection and recognition have been gaining momentum
ever since.
The important breakthrough of deep neural networks is that object detection no longer has to
be a hand-crafted coding exercise. Deep neural networks allow features to be learned
automatically from training examples. A neural network is considered to be “deep” if it has an

2
input and output layer and at least one hidden middle layer. Each node is calculated from the
weighted inputs from multiple nodes in the previous layer. CNNs are the current state-of-the
art for efficiently implementing deep neural networks for vision. CNNs are more efficient
because they reuse a lot of weights across the image.

Early CNNs in the embedded space were performed using a GPU or using the vector DSP
portion of a vision processor. However, it’s helpful to look at the task performed in terms of
three different heterogeneous processing units, as shown in Figure 3.

Early implementations of CNNs in hardware had a limited number of Multiply-Accumulator

(MAC) units. For example, Synopsys’ EV5x, the industry’s first programmable and
configurable vision processor IP cores implemented a CNN engine with 64 MACs. Running
at 500 MHz, the EV5x could produce 32 GMACs/s or 64 GOPs/s of performance (a multiply-
accumulator performs two operations in one instruction). That was not enough performance to
process an entire 1MP (1280 x 1024) frame or image. However, it was enough processing
power to perform a CNN on a portion of the image (say a 64×64 pixel patch). To process the
entire image, a two-step process for pedestrian detection was needed. The vector DSP would
perform a computationally intensive Region of Interest (ROI) algorithm on each incoming
image of the video stream. ROI identifies candidates using a sliding window approach that
could be a pedestrian (ruling out, for example, portions of the sky). Those “pedestrian”
patcheswere then processed by the CNN to determine if it was in fact a pedestrian. CNN-
based pedestrian detection solutions have been shown to have better accuracy than algorithms
like HoG and perhaps more importantly it is easier to retrain a CNN to look for a bicycle than
it is to write a new hand-crafted algorithm to detect a bicycle instead of a pedestrian.
Deep Learning Algorithms on Embedded Vision Processor:
 Training and deploying CNNs

As mentioned earlier, a CNN is not programmed. It is trained. A deep learning

framework, like Caffe or TensorFlow, will use large data sets of images to train the CNN
graph – refining coefficients over multiple iterations – to detect specific features in the
image. Figure 5 shows the key components for CNN graph training, where the training
phase uses banks of GPUs in the cloud for the significant amount of processing required.

1
The deployment – or “inference” – phase is executed on the embedded system. Development
tools, such as Synopsys’s MetaWare EV Toolkit, take the 32-bit floating point weights or
coefficients output from the training phase and scale them to a fixed point format. The goal is
to use the smallest bit resolution that still produces equivalent accuracy compared to the 32-
bit floating point output. Fewer bits in a multiply-accumulator means less power required to
calculate the CNN and smaller die area (leading to lower the cost) for the embedded solution.
Based on Synopsys calculations, 10-bit or higher in resolution is needed to assure the same
accuracy of the 32-bit Caffe output without graph retraining.
The MetaWare EV tools take the weights and the graph topology (the structure of the
convolutional, non-linearity, pooling, and fully connected layers that exist in a CNN graph)
and maps them into the hardware for the dedicated CNN engine. Assuming there are no
special graph layers, the CNN is now “programmed” to detect the objects that it’s been
trained to detect.
To keep the size small, the CNN engine is optimized to execute for key CNN features such as
3×3 and 5×5 matrix multiples, but not so optimized that it becomes a hard wired solution. It’s
important to be programmable to maintain flexibility. As CNNs continue to evolve – new
layer techniques or pooling methods for example – the vector DSP can play another important
role in the vision processing. Since the vector DSP and CNN engine are closely coupled in the
Synopsys EV6x, it is easy to dispatch tasks from the CNN to the vector DSP as needed.
OpenVX runtime, incorporated into the MetaWare EV tools, makes sure those tasks are
scheduled with other vector DSP processing requirements. The vector DSP future-proofs the
CNN engine.
Figure 6 shows the inputs and outputs of an embedded vision processor. The streaming
images from the car’s camera are fed into the CNN engine that is preconfigured with the
graph and weights. The output of the CNN is a classification of the contents of the image.

2
5-Outro:
As a summary of this section of the course, we covered topics such as machine learning and
deep learning applications, sensor data fusion, and processors designed for image processing
using deep learning algorithms. We discussed the increasing amount of sensor data available
and the use of different sensors like radar and camera for various applications such as object
classification and vehicle detection. Additionally, we explored how deep learning algorithms
are used for sensor data fusion, combining data from different sensors for improved results.
Finally, we delved into processors specifically designed for image processing using deep
learning algorithms, with dedicated CNN engines for efficient computations.