APPLICATION OF DEEP LEARNING TO DEVELOP

AN AUTONOMOUS VEHICLE
Mohamed Salman, Nitte Meenakshi Institute of Technology,
R Manoj, Nitte Meenakshi Institute of Technology,
Rudresh M, Nitte Meenakshi Institute of Technology,
Mohamed Jaffar Niyaz, Nitte Meenakshi Institute of Technology,
Dr. Nalini N, Professor, Nitte Meenakshi Institute of Technology

Abstract:
Travelling by road is the most common mode of transport for day-to-day travel, but it is also the leading
cause of deaths among all other means of transport. Road accidents most often are a consequence of driver
errors that stem from the casual nature of driving and recklessness. Self-driving cars would effectively
remove the human factors behind road accidents and help bring down the number of fatalities that take place
every day.

Rise in technologies like Artificial Intelligence, Machine Learning and Image Processing has propelled the
development of autonomous vehicles. Self-driving vehicles are those vehicles where human intervention are
never required to take control of the operation of the vehicle, also known as autonomous or “driverless” cars.
Self-driving cars combine sensors and software to control, navigate, and drive the vehicle. The advancements
in the fields of computer vision and artificial intelligence have prompted engineers to use the deep learning
approach and move away from the conventional robotics approach.

This paper is a demonstration of a working model of self-driving car that is capable of driving itself on any
environment that it has been trained on. It is also capable of detecting sign boards and traffic signals, and
making appropriate decisions. The car uses a fusion of hardware and software. A camera module mounted
over the top of the car captures images of the environment and efficient algorithms help the car in making
appropriate decisions which keep it centred on the road lanes while also obeying traffic rules.

Thus, the paper proposes an end-to-end solution for an autonomously driven vehicle.

Keywords - Artificial Intelligence, Deep Learning, Self-driving car, Raspberry Pi, Arduino, OpenCV

1. INTRODUCTION

1.1. BACKGROUND:
The World Health Organization launched its Global Status Report on Road Safety for the year 2018[11] and it
stated that the number of annual road traffic deaths had reached a staggering 1.35 million. According to the
Ministry of Road Transport and Highways [12], close to 150,000 deaths had taken place due to road accidents.
India alone contributes to more than 10% of annual deaths worldwide due to road accidents. It is needless to say
that these accidents take place more often than not due to human negligence. The root cause of a lot of accidents
that take place on the road relate to speeding, drink and driving and using cellular devices while driving. This
has been the motivation behind the development of autonomously controlled cars. If technology can remove the
human factor behind the steering wheel of automobiles, the number of accidents could come down by a huge
number.

1.2. HISTORY OF TECHNOLOGY:

[13] The idea of self-driving vehicles dates back to Futurama, from the New York’s Fair. General Motors (GM)
was the first to implement the autonomous vehicle back in the late 1930s, and since then there have been many
breakthroughs in this field. Main aim of autonomously controlled car is one which promises to take the traveller
to their destination with minimal human control while considering the safety as its first priority. Norman Bel
Geddes created the first self-driving vehicle, which was an electric vehicle. In the late 1970s, cameras were used
for processing of the images and this gave birth to the use of computing power to control cars.
1.3. RESEARCH MOTIVATION:
1. Create an innovative solution to tackle the problems faced in modern transport.
2. To survey and understand the state of autonomous cars.
3. Understand the use of Artificial Neural Networks in the field of mobility and transport.

1.4. PROBLEM STATEMENT:

1. Implement a system for a remote-controlled car that is automated and can make intuitive decisions for
itself.
2. Merge technologies such as Artificial Neural Networks, Deep Learning and Image Processing. in the
system.
a. Play around with Deep Learning and figure out how we can get a better performance out of it.
3. Test the system in real world scenarios.

1.5. OBJECTIVES:
1. Incorporate a thoroughly autonomous driving vehicle.
2. Work with Deep Learning and Artificial Neural Networks to train a self-driving car.
3. Get ideal results in self-driving car performance.

2. RELATED WORK

T. Do, M. Duong, Q. Dang and M. Le [1] describe the process of how they developed a self-driving car that is
powered by raspberry pi. It is a miniscule of the entire model. They were able to achieve high accuracy using a
CNN model. They also stated that with the camera latency, the entire model working slows down, which deals
more into the hardware functionality. As per the work done by K. Bimbraw, [2] he decodes the past
technologies that led to the increase in interest of autonomous vehicle technology and reviews the current
technologies that govern this field. He also observes certain trends in the future technologies. He outlines
features such as ACC introduced by Volvo bring in a new dimension in the technology. According to T.
Okuyama, T. Gonsalves and J. Upadhay [3], the simulation results of autonomous car can learn and be driven in
those simplified environments which resembles to a real-world scenario. Algorithm for learning purpose was
performed by using a Deep Q Network which consists of Q values (rewards) corresponding to the actions
available to a self-driving car.

Many distance determination algorithms have been proposed and developed in the last decade [4- 6]. Active
detection systems are widely implemented commercially in vehicles today because of their immunity to
changing ambient light conditions; however, the cost is usually higher than passive systems because they
involve transmitters and receivers. A complete distance measurement system includes two steps: vehicle
detection and distance calculation. Motion-based and Appearance-based are two main approaches for vehicle
detections [6]. We will first apply the combination of Histogram of Oriented Gradients (HOG) descriptor and
Support Vector Machine (SVM) classification for vehicle detection, because it has shown promise by many
previous works [7].
R. Kulkarni, S. Dhavalikar and S. Bangar [8] have specified applications of AI over detections of traffic lights,
sign boards, unclear lane markings and more. To overcome these obstacles, technologies like Deep Learning are
used. The authors have proposed a model build using a deep neural network which could recognize traffic lights.
It includes use of fast Convolution NN (R-CNN) Inception V2 model in TensorFlow for transfer learning. J.
Kim, Y. Koo and S. Kim [9] have proposed a method to detect moving objects called as MOD (Moving Object
Detection) technology combined with recognition, ID tracking, detection and classification by using sensor
fusion to get information that local & global position estimation, pose estimation, velocity from around objects
in real time over 15 fps. Darknet based DL method and modified detector is used to obtain a local position
estimation. M. V. Smolyakov, A. I. Frolov, V. N. Volkov and I. V. Stelmashchuk [10] have given a possibility
of using images from the emulator for training Deep NN’s for the prediction of steering angle. There exists an
emulator generating the desired number of images of the vehicle movement easily. This approach allows the car
to move in automatic mode. It basically explores various architectures of CNN’s in order to obtain good results
with a minimum number of parameters.

3. LITERATURE SURVEY

2.1. DEEP LEARNING TECHNIQUES ON AUTONOMOUS VEHICLE:

Looking at the advancements being done in the field of Artificial Intelligence, Deep Learning, it gets practically
more sophisticated to incorporate various methods into a single working model and to function it at a faster rate.
Main consideration is the training of the model (Neural Network) depending upon the Datasets present with us.
The efficiency of the trained Datasets will basically predict how well the model will proceed in being complete
autonomous.
Multiple sensors are utilised to detect the obstacles. Sensors detect the obstacle, send the information to the
computer, further to the Raspberry Pi to proceed with the Neural Networks.
Tensor Flow is used for such Neural Network operation.
Multiple data sets are used with respect to lane detection, road sign detection, traffic light detection for the
training purpose. After a successful completion of training, the model will be ready to be put out on the actual
test.
There consists of different types of datasets for different scenarios:
1. Different variations of lanes, being it from straight lane, curvy lane.
2. Different sets of sign boards (Stop sign), in various designs.
3. Detection of traffic lights.
The autonomous vehicle can also be tracked remotely, through the website.
The application has access to the vehicle’s Pi Cam, for its monitoring. Voice Recognition Network is being used
to move and stop the vehicle as per our needs. The instruction being told will get processed by the Neural
Network in its back-end, analyse it and then trigger the vehicle as per the instruction provided.

4. HARDWARE AND SOFTWARE REQUIREMENTS

4.1. HARDWARE REQUIREMENTS:

1. Raspberry Pi:
Raspberry Pi is a miniaturised version of computer consisting of GPIO pins (general purpose
input/output) and interfaces giving us to have control over the electronic components for physical
computing and explore Internet of Things (IoT). In this project, the Pi has been used to control the
camera, send and receive data from the computer, and send instructions to the Arduino board.

2. Pi Camera:
The Pi Camera module is a portable lightweight camera that supports Raspberry Pi. It communicates
with Pi using the MIPI camera serial interface protocol. It is connected to the Pi through the CSI flex
cable. The camera has been used in this project to capture images of the surroundings for further image
processing.

3. Arduino Uno:
Arduino is an open-sourced platform which is used for building electronic projects. It consists of both a
microcontroller and an IDE (Integrated Development Environment), where we can write and upload the
code onto the microcontroller [17]. The board used in this project is the Arduino Uno and it is used to
control the actuators of the car.

4. Ultrasonic Sensor (HC-SR04):

Ultrasonic sensors are those which use sonar to detect the distance between the device and the nearest
object. The sensor uses trigger and echo pins which send and receive ultrasonic signals. The HC-SR04
sensor comes complete with an ultrasonic transmitter and receiver module. It has been used in this
project for the purpose of obstacle detection.
4.2. SOFTWARE REQUIREMENTS:
1. Tensor Flow:
TensorFlow is a Machine Learning Domain consisting of open-source software libraries and flexible
tools. This provides the developer to work on ML applications, build upon it and to deploy these power
applications. It's usually used for working under Neural Network applications [18].

2. Keras:
Keras is a high-level, open-sourced Neural Network library written in Python, and built on the
Tensorflow backend. It consists of multiple back-end API's and engines. Its designed to be more user-
friendly, easy to use, fast experimentation, modular and extensible [19].

3. Natural Language Processing:

Natural language processing (NLP) is the subfield of Artificial Intelligence which gives the ability to
make easy and smooth interactions between human languages and computer. Development of NLP is a
challenging task as the computer has to recognize varied variations of "speaking's" from the humans and
even the human speech is not always precise, its often ambiguous and the linguistic structure can depend
on many complex variables, including slang, regional languages and social context [20].

4. OpenCV:
OpenCV is an open source computer vision library that performs image processing on images/videos
and can perform tasks like facial recognition. It supports languages like C, C++, Python and Java and
operating systems like Windows, Linux, Mac OS, iOS and Android. Written in optimized C/C++, the
library can take advantage of the multi-core processing with optimized code [22].

5. IMPLEMENTATION

Proper functioning of the car is the proper arrangement of the hardware components as displayed in Figure 1
and an efficient algorithm is used for performing the given tasks in the right manner. The car consists of two
main controlling boards, an open-source microcontroller board - Arduino Uno and the latest version of the Pi
family of computers - Raspberry Pi 4. The Arduino board has been used to interface the DC motors with an
L298N bridge which drives the motors. The functions of the Raspberry Pi are to interface the camera and
communicate between the artificial neural network model and the Arduino.

Figure 1: Pictorial view of model

The Pi camera is connected to the CSI Camera connector of the Raspberry Pi. As per the given dataflow model
provided below (Figure 2), the camera captures images and sends it to the system through sockets. The system,
through the given specified algorithm performs its analysis, extracts the desired information and then passes
instructions back to the Raspberry Pi. Raspberry Pi predicts the next paths to be performed and this information
is sent to the Arduino, which further controls the motors of the car.

Figure 2: Dataflow model

The main features of the self-driving car in this project are:
1. Road lane detection,
2. Obstacle detection,
3. Sign board detection, and,
4. Traffic lights detection.
1. For the road lane detection, an artificial neural network is used, where the car is trained first on an actual
scenario. The dataset for training the neural network is obtained by capturing multiple images when the car is
simulated on a track. These images are converted into numpy arrays and compressed into a numpy archive, in
a .npz file. These images are reshaped into the desired resolution, so as to obtain the region of interest, which is
the track. The figure below illustrates the process of collecting the images for the training dataset. An image is
first captured by the came and is cropped to contain the region of interest. Each jpg image is converted to a
numpy array whose dimensions are the resolution of the image containing the region of interest. Each image
also contains a label which is input by the user. The images along with their labels are stored in a npz file. From
this file, analysis is done for decision making. If the Pi encounters both the lane strips from the camera, it
predicts it as 'Straight Path'. If Pi encounters left strip, it predicts as 'Right Turn', for right strip as 'Left Turn' and
if it encounters an obstacle in front, it 'Stops'.

Figure 3: Training Data Collection Process

The neural network used to build the model is a CNN and uses the Sequential model, which stacks sequential
layers from the input layer to the output. Figure 4 shows a plot of the model.

Figure 4: Plot of the Neural Network Model

The input layer takes a numpy array of dimensions 120,360 and the output layer returns one of 3 possible
classes: left, right or forward. There is a total of 12 layers in the sequential model as shown above. The hidden
layers of the network use the relu activation function and the output layer uses the softmax activation function.
A dropout rate of 0.5 was set in two hidden layers to avoid overlearning of the model. 16 was set as the size of
the batch and the number of epochs for training was set to 50. Each epoch took approximately 1 minute to
complete and thus the model was trained in approximately 1 hour.
When working autonomously (Figure 5a, 5b), the camera sends every frame that is captured to the neural
network model that was trained previously. The model accurately predicts the direction that the car is supposed
to move in, based on the data that it has learned previously from. This prediction is then sent back to the
Raspberry Pi, which in turn communicates with the Arduino through the GPIO pins on the former and digital
pins on the latter.

Figure 5a: Straight Path Figure 5b: Curved Path

2. For the distance measurement aspect for obstacle detection, an ultrasonic sensor, HC-SR04 was placed at the
front of the car. This sensor is used for the purpose of obstacle detection. The sensor is connected to the
Raspberry Pi and it determines the distance to an obstacle provides accurate results, as shown in Figure 6. It also
takes the surface condition into consideration. The car functions as normal when there is no obstacle in front of
the sensor, and stops when the distance between itself and the obstacle is less than or equal to a predefined
threshold distance.

Figure 6: Object Detection

Figure 7: System Interface

The above picture (Figure 7) remotely monitors the live feed from the car to the system. The left corner is the
RPi cam interface. The program helps you to capture live feeds by letting IP address of Raspberry Pi. On the
right corner, the terminal shows the predicted outcomes, i.e., in which direction the car is moving in and below
terminal streams data from RPi to the system.
3.In the case of Road Sign Detection, a Haar Cascade Classifier using OpenCV is used, which paves the way for
efficient identification of traffic signs for feature matching and detector methods (Figure 8). Cascading [25] is a
particular case of ensemble learning based on the concatenation of several classifiers, using all information
collected from the output from a given classifier as additional information for the next classifier in the cascade.
A pre-trained classifier has been used to identify road signs. Depending on the sign that the system detects, the
appropriate action is taken.
 Figure 8: Road sign detection and estimation of distance
4. For detecting traffic signals, the ORB feature matching algorithm has been used. A reference image for the
traffic lights has been used, with each image having close to 1500 features. ORB detector and Brute Force
Feature matcher are used to extract the image features by scanning through all those features. The color
extracted is then ascertained using the Brute Force Feature matching which then helps in indicating the correct
color detection. In reference to Figures 9a and 9b, if there exist lesser features around 'Red' color, it's detected as
'Red' and the same applies to 'Green' color. The Classifier also has mechanisms for detection of the colours
through the edges around the traffic light. Furthermore, the entire model is loaded in the system and the
decisions are then streamed to the RPi so as to reduce the latency.

Figure 9a: Detection of 'Red' light Figure 9b: Detection of 'Green' light.
 Figure 10a: Application Login Page Figure 10b: Home Page
The car can also be controlled remotely via an application, named as 'Infernus'. A glimpse of the application is
displayed above (Figure 10a, 10b). It's a Desktop/Android application that consists of a few sets of controls
within it to control the car. The application has been built using the KIVY Framework in Python. KIVY [23] is
an open source Python library for rapid development of applications that make use of innovative User Interface.
Application has a Login Page where only the application owner can access it. After login, the home page opens
up, which shows the camera stream, and a set of controls for manual functioning of the car, including stopping
and starting the car. A voice command feature has also been included. The speech recognition uses Natural
Language Processing. It uses the 'SpeechRecognition' Python Library which helps in converting spoken words
into equivalent textual form. It uses Google's Speech APIs to convert the voice commands into text.

6. CONCLUSION AND FUTURE WORK

6.1. CONCLUSION:
Through this paper, we have presented a novel idea on building a scaled down model of a self-driving car. A
brief overview of the requirements was specified, and the implementation of the model was discussed. The role
of emergent technologies like Machine Learning, Deep Learning, Image Processing and Internet of Things in
the development of a self-driving car was studied. Through the fusion of hardware and software, a successful
model was developed and this model could function as per expectations in the test scenario. Thus, the model
was successfully designed, implemented and tested.
The project successfully completed all the objectives that were discussed in Chapter 1.
The paper aimed to come up with the following results which are going to be important from an algorithmic
point of view:
1. Implement a system for a remote-controlled car that is automated and can make intuitive decisions for
itself.
 2. Combination of different hardware components along with the software and the Neural Network
configurations.
3. Illustrate the use of Deep Learning concepts in the field of mobility and transport.

6.2. FUTURE WORK:

We believe there is a lot to be explored and implemented in the field of Deep Learning in the application of self-
driving cars. Smoothening the decision making of autonomous cars and making it more consistent, identification
of lanes and traffic signs, navigating through deserts, mountainous terrains debris, potholes, etc. is a huge
challenge. Extensive work and research have to be conducted to overcome these challenges.

7. REFERENCES

[1] T. Do, M. Duong, Q. Dang and M. Le, "Real-Time Self-Driving Car Navigation Using Deep Neural
Network," 2018 4th International Conference on Green Technology and Sustainable Development (GTSD), Ho
Chi Minh City, 2018, pp. 7-12.
[2] K. Bimbraw, "Autonomous cars: Past, present and future a review of the developments in the last century,
the present scenario and the expected future of autonomous vehicle technology," 2015 12th International
Conference on Informatics in Control, Automation and Robotics (ICINCO), Colmar, 2015, pp. 191-198.
[3] T. Okuyama, T. Gonsalves and J. Upadhay, "Autonomous Driving System based on Deep Q Learning,"
2018 International Conference on Intelligent Autonomous Systems (ICoIAS), Singapore, 2018, pp. 201-205.
[4] Sun, Zehang, et al., "On-road vehicle detection: A review." Pattern Analysis and Machine Intelligence, IEEE
Transactions on 28.5 (2006): 694-711.
[5] Cualain, Diarmaid O., et al., “Distance detection systems for the automotive environment: a review.” In Irish
Signals and Systems Conf. 2007.
[6] Sivaraman, Sayanan, and Mohan Manubhai Trivedi. “Looking at vehicles on the road: A survey of vision-
based vehicle detection, tracking, and behavior analysis.” Intelligent Transportation Systems, IEEE Transactions
on 14, no. 4 (2013): 1773-1795.
[7] H. Tehrani Niknejad, et al., “On-road multivehicle tracking using deformable object model and particle filter
with improved likelihood estimation,” IEEE Trans. Intell. Transp. Syst., vol. 13, no. 2, pp. 748–758, Jun. 2012.
[8] R. Kulkarni, S. Dhavalikar and S. Bangar, "Traffic Light Detection and Recognition for Self Driving Cars
Using Deep Learning," 2018 Fourth International Conference on Computing Communication Control and
Automation (ICCUBEA), Pune, India, 2018, pp. 1-4.
[9] J. Kim, Y. Koo and S. Kim, "MOD: Multi-camera Based Local Position Estimation for Moving Objects
Detection," 2018 IEEE International Conference on Big Data and Smart Computing (BigComp), Shanghai,
2018, pp. 642-643.
[10] M. V. Smolyakov, A. I. Frolov, V. N. Volkov and I. V. Stelmashchuk, "Self-Driving Car Steering Angle
Prediction Based On Deep Neural Network An Example Of CarND Udacity Simulator," 2018 IEEE 12th
International Conference on Application of Information and Communication Technologies (AICT), Almaty,
Kazakhstan, 2018, pp. 1-5.
[11] https://www.who.int/publications-detail/global-status-report-on-road-safety-2018
[12] https://www.prsindia.org/policy/vital-stats/overview-road-accidents-india
[13] https://www.titlemax.com/resources/history-of-the-autonomous-car/
[15] https://www.hongkiat.com/blog/pi-operating-systems/
[16] https://www.raspberrypi.org/documentation/hardware/camera/
[17] https://en.wikipedia.org/wiki/Arduino
[18] https://en.wikipedia.org/wiki/TensorFlow
[19] https://en.wikipedia.org/wiki/Keras
[20] https://en.wikipedia.org/wiki/Natural_language_processing
[21] https://randomnerdtutorials.com/complete-guide-for-ultrasonic-sensor-hc-sr04/
[22] https://opencv.org/
[23] https://kivy.org/#home
[24]https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_feature2d/py_orb/py_orb.html
[25] https://en.wikipedia.org/wiki/Cascading_classifiers