International Journal of Scientific Research in Science, Engineering and Technology

Print ISSN - 2395-1990 Available Online at :www.ijsrset.com

Online ISSN : 2394-4099 doi :https://doi.org/10.32628/IJSRSET2411269

Self-Driving Car Using Simulator

Prof. Shamal Patil1, Hrushikesh Bhujbal2, Fatru Shaikh2, Priya Lokhande2, Aashika Jain5
1Professor, 2Student
Department of Artificial Intelligence & Data Science, ZEAL College of Engineering & Research, Pune,
Maharashtra, India

ARTICLEINFO ABSTRACT

Due to rapid technological growth in transportation, self-driving cars

Article History:
became the topic of concern. The main purpose of this project is to use the
Accepted: 15 April 2024 CNN and train the neural network in order to drive the car in autonomous
Published: 28 April 2024 mode in a simulator environment. Front camera of a car captures the
images and we use those captured images in order to train the model, in
short we can say we have used the concept of behavioural cloning. In
Publication Issue : behavioural cloning, the system tries to mimic the human driving
Volume 11, Issue 2 behaviour by tracking the steering angle. That means a dataset is generated
March-April-2024 in the simulator by a user driven car in training mode, and the deep neural
Page Number : network model then drives the car in autonomous mode.
508-516 In one track a car is trained and in other tracks the car drives in
autonomous mode. The dataset for Track 1, which was straightforward to
drive and had good road conditions, was utilized as the training set for the
automobile to drive itself on Track 2, which has abrupt curves, barriers,
heights, and shadows are all things to consider. Image processing and other
augmentation techniques were utilized to solve this difficulty, allowing for
the extraction of as many data and features as feasible. In the end, the
vehicle performed admirably on Track 2. In the future, the team hopes to
achieve the same level of accuracy using real-time data.
Keywords: Self-driving car, CNN, Image Processing, Dataset Generation,
Real Time Data, Augmentation Techniques

I. INTRODUCTION The main challenge is to mimic the driving behavior

of the human driver on the simulator provided by
The purpose of this project is to build a better Udacity. Here we train the neural networks so that we
autonomous driver which is capable of driving itself can mimic the behavior which is called behavior
without falling off the track which uses all the cloning technique.
functionalities of the car provided like brake, There are two tracks and two modes in the simulator:
accelerator etc. training mode and autonomous mode. The dataset is

Copyright © 2024 The Author(s): This is an open-access article distributed under the terms of the Creative 508
Commons Attribution 4.0 International License (CC BY-NC 4.0)
 Prof. Shamal Patil et al Int J Sci Res Sci Eng Technol, March-April-2024, 11 (2) : 508-516

created by the user when driving the automobile in pixel values, resizing to ensure uniform image
training mode in the simulator. The "excellent" dimensions, and data augmentation techniques to
driving data is another name for this dataset. The deep increase the diversity of the dataset. Augmentation
learning model is then tested on the track to see how techniques such as random rotation, flipping, and
it performs after being trained with the user data. brightness adjustment were employed to simulate
Another difficulty is transferring the results to various real-world variations in lighting and perspective.
tracks.
That is, the model is trained using the dataset 2.3 Model Architecture
produced on one of the simulator's tracks, and it is The architecture of the convolutional neural network
then tested on the other track. (CNN) used in this research consisted of multiple
layers designed to extract relevant features from the
II. METHODOLOGY input images and make predictions regarding steering
angles. The CNN architecture comprised
2.1 Data Collection convolutional layers for feature extraction, followed
The data collection process involved utilizing a by pooling layers for dimensionality reduction, and
simulator environment to capture images from the fully connected layers for steering angle prediction.
front camera of a car. The simulator provided a Specific configurations, including the number of layers
realistic virtual driving experience, allowing for the and filter sizes, were determined through
generation of a diverse dataset representative of experimentation to optimize model performance.
various driving scenarios. The car's front camera
recorded images as the vehicle traversed through 2.4 Training Procedure
different tracks with varying road conditions, The training procedure involved feeding the
including straight stretches, curves, barriers, heights, preprocessed images into the CNN model to learn the
and shadows. mapping between input images and corresponding
steering angles. The Adam optimization algorithm was
employed to minimize the mean squared error loss
function, which quantified the disparity between
predicted and actual steering angles. A batch size of
[specify batch size] and a predefined number of epochs
[specify number of epochs] were used during training
to iteratively update the model parameters and
improve performance.

Fig - 1

2.2 Preprocessing
Prior to training the neural network model, the
collected images underwent preprocessing steps to
enhance their suitability for training. This
preprocessing included normalization to standardize

International Journal of Scientific Research in Science, Engineering and Technology | www.ijsrset.com | Vol 11 | Issue 2 509
 Prof. Shamal Patil et al Int J Sci Res Sci Eng Technol, March-April-2024, 11 (2) : 508-516

limitations, enabling informed decisions regarding its

suitability for real-world deployment. Additionally, it
facilitated the identification of areas for further
improvement, guiding future research efforts aimed at
enhancing the robustness and efficacy of autonomous
driving systems.

Fig – 2

2.5 Validation and Testing

To comprehensively evaluate the effectiveness of the
trained model, a systematic approach was adopted,
involving the segregation of the dataset into distinct
subsets for validation and testing purposes. This Fig - 3
division facilitated rigorous scrutiny of the model's
performance under various conditions. III. BACKGROUND
The validation dataset played a crucial role in the
iterative training process by serving as a means to CNN is a type of feed-forward neural network
monitor the model's performance and prevent computing system that can be used to learn from input
overfitting. By periodically assessing the model's data. Learning is accomplished by determining a set of
performance on this subset during training, weights or filter values that allow the network to
adjustments could be made to optimize its learning model the behavior according to the training data. The
dynamics and prevent it from memorizing noise in the desired output and the output generated by CNN
training data. This iterative refinement process initialized with random weights will be different. This
ensured that the model's predictive capabilities were difference (generated error) is back propagated
honed to generalize well to unseen data. through the layers of CNN to adjust the weights of the
Subsequently, the testing dataset served as the neurons, which in turn reduces the error and allows
ultimate litmus test for the model's generalization us to produce output closer to the desired one .
ability. By evaluating the model on previously unseen
data, its capacity to make accurate predictions in real-
world scenarios was rigorously assessed. Performance
metrics such as accuracy, mean squared error, and root
mean squared error were meticulously computed to
quantitatively gauge the model's predictive prowess
across various evaluation criteria. Fig – 4
This comprehensive evaluation framework provided CNN is good at capturing hierarchical and spatial data
valuable insights into the model's strengths and from images. It utilizes filters that look at regions of an

International Journal of Scientific Research in Science, Engineering and Technology | www.ijsrset.com | Vol 11 | Issue 2 510
 Prof. Shamal Patil et al Int J Sci Res Sci Eng Technol, March-April-2024, 11 (2) : 508-516

input image with a defined window size and map it to Dropout, Dense, and other layers were ordered in
some output. It then slides the window by some series, and various combinations of Time-Distributed
defined stride to other regions, covering the whole Convolution layers, MaxPooling, Flatten, Dropout,
image. Each convolution filter layer thus captures the Dense, and other layers were utilized. The best-
properties of this input image hierarchically in a series performing ones are highlighted in further detail. The
of subsequent layers, capturing the details like lines in parameters required to create them may be found in
image, then shapes, then whole objects in later layers. the model listings. The architectural figures present a
CNN can be a good fit to feed the images of a dataset high-level view of the layers utilized to construct the
and classify them into their respective classes. models.
Model 1:
IV. SIMULATOR AND DATASET

Udacity Simulator
Udacity has provided the simulator for training the car
in the specific path and the platform to run the car
through our prepared model of autonomous driving .
It is an open source for the software enthusiasts to
work in this field so that it could be applied in a real-
time environment. Unity, a video game creation
platform, is used to create it. The simulator is highly Fig –6
user friendly and has a changeable resolution and Network architecture containing various layer is
controls configuration. shown below:

Fig –5

V. NETWORK ARCHITECTURE

Various Network architectures are used to determine

the steering angle so that the car can be driven in the
autonomous mode smoothly. In architectures, Time-
Distributed Convolution layers, MaxPooling, Flatten, Fig - 7

International Journal of Scientific Research in Science, Engineering and Technology | www.ijsrset.com | Vol 11 | Issue 2 511
 Prof. Shamal Patil et al Int J Sci Res Sci Eng Technol, March-April-2024, 11 (2) : 508-516

VI. AUGMENTATION AND IMAGE PRE- Approximately 30% of the top section of the
PROCESSING image is cropped during training.
3. Brightness Adjustment:
Image Preprocessing • Mimic varying lighting conditions by adjusting
In our preprocessing phase, we focused on enhancing the brightness of images, allowing the model to
the quality of captured images and extracting relevant learn to navigate through different weather
features for lane detection. While these enhancements conditions, such as bright sunny days or low-
may not directly contribute to the training output, light environments.
they play a crucial role in improving the robustness of 4. Flipping:
our model. Here's an overview of the preprocessing • Flip images horizontally by 180 degrees to
steps: augment the dataset with mirrored
representations of road features. This helps the
1. Gray Conversion (COLOR_RGB2GRAY): model generalize better to road configurations
• Convert color images to grayscale to simplify involving left and right turns.
processing and reduce computational
complexity. VII. OBSERVATIONS
2. Blur Images (GaussianBlur):
• Apply Gaussian blur to smooth the images and The result that has been observed during the training
reduce noise, resulting in a more uniform phase from the architectural model we have used is
gradient and fewer abrupt changes. value loss or accuracy.
3. Canny Edge Detection:
• Utilize the Canny edge detection algorithm to Accuracy
highlight changes in gradient pixels, providing a After training the model with some of the dataset and
clear delineation of edges in the image. passing other data for testing we have come up with
the performance measure which reflects the loss over
Augmentation Epoch.
The challenge of generalizing the car's behavior across Epoch:
diverse tracks necessitated the use of augmentation Epoch is an arbitrary cutoff used to divide training
techniques during training. These techniques simulate into different phases, which is important for recording
various real-world conditions and enhance the and periodic assessment. It is commonly described as
model's adaptability. Here are the augmentation "one run over the complete dataset." When the fit
approaches employed: technique of Keras models is used with validation data
or validation split, evaluation is performed at the
1. Zoom: conclusion of each epoch.
• Adjust the image zoom level to simulate
different perspectives and increase the model's
ability to anticipate upcoming road conditions.
2. Panning:
• Crop a portion of the image containing relevant
road features, ignoring extraneous information
Fig – 8
from the surrounding environment.

International Journal of Scientific Research in Science, Engineering and Technology | www.ijsrset.com | Vol 11 | Issue 2 512
 Prof. Shamal Patil et al Int J Sci Res Sci Eng Technol, March-April-2024, 11 (2) : 508-516

Fig – 11

IX. RECEIVING THE TELEMETRY INFORMATION

The automatic communication mechanisms from

Fig – 9
many data sources are referred to as telemetry.
Customer experiences are improved through the
usage of telemetry data, which is used to track
security, application health, quality, and
performance.Areas may be regulated and controlled
remotely via telemetry. Temperature, pressure,
humidity, movement, and illumination, among other
sensors, may all be controlled and monitored with it.
Telemetry allows measurements to be taken without
having to be present, regardless of mobility.

X. DRIVE PERFORMANCE

Fig – 10 We evaluate the performance on the metrics of

generalization , the working of car in track-2. This
VIII. CONNECTION may be characterized as how effectively the models
forecast values when driving on a track for which
There is Configuration for the Connection between they were not trained. The expected steering angle,
the simulator and the model we have trained earlier. brakes, and throttle are the values here. It's not
The simulator side is called server whereas the user something that can be charted, but it can be measured
made a link with python code said to be client. Socket in terms of how far the car can travel on the second
programming is the way where ports listen up and track without tipping over. The pace, twists, and track
exchange the information between the nodes. python- circumstances such as hills, shadows, and so on can all
socketio is the package that we have used to transport have an impact.The models are only given training on
the telemetry information back and forth Track 1 data because it is easier, but they are validated
on Track 2. As a result, working with Track 2 was
quite difficult. During the trial, a few new findings
were made that support this claim.

International Journal of Scientific Research in Science, Engineering and Technology | www.ijsrset.com | Vol 11 | Issue 2 513
 Prof. Shamal Patil et al Int J Sci Res Sci Eng Technol, March-April-2024, 11 (2) : 508-516

• Most architectures were unable to take the first safe and reliable autonomous driving across
turn on Track 2 while performing well on diverse environments. Investigating robust
Track 1 and providing the greatest accuracy optimization techniques and sensor fusion
during training (loss over epochs). strategies can fortify the model against these
• Overfitting for the Track 1 dataset might be the challenges.
cause. Overfitting occurs when a software
attempts to mimic the training data too 3. Dynamic Environment Adaptation:
accurately. The model trains for details and • Developing adaptive algorithms that enable the
even noise for the data it has passed through in model to dynamically respond to changes in the
general. This, in turn, has a detrimental driving environment, such as construction
influence on the capacity to generalize. zones, temporary road closures, and pedestrian
crossings, will bolster its versatility and
XI. CHANGE MADE TO WORK AROUND THE responsiveness. This may involve incorporating
PROBLEM reinforcement learning mechanisms or
advanced planning algorithms capable of
• Gaussian filter and augmentation techniques handling dynamic scenarios.
are utilized to do something about this
challenge. 4. Multi-Agent Interaction:
• Exploring methods for enabling autonomous
XII. FUTURE DIRECTIONS vehicles to interact intelligently with other
agents, including vehicles, pedestrians, and
The current research lays a solid foundation for cyclists, in shared spaces is essential for
advancing autonomous driving technology, yet there enhancing safety and efficiency in urban
are several avenues for future exploration and driving scenarios. Research in cooperative
improvement. Here, we outline comprehensive decision-making, negotiation strategies, and
directions for future research: communication protocols can facilitate
harmonious interactions among multiple agents
1. Real-Time Deployment: on the road.
• Transitioning the developed model from
simulation to real-world application is 5. Continual Learning Frameworks:
paramount. Integrating the trained neural • Implementing continual learning frameworks
network into an actual self-driving vehicle and that enable the model to adapt and improve
evaluating its performance in real-time driving over time through exposure to new data and
scenarios will provide invaluable insights into experiences is crucial for ensuring continual
its practical viability and effectiveness. refinement and optimization of autonomous
driving capabilities. This involves developing
2. Robustness Enhancement: algorithms capable of incremental learning,
• Enhancing the model's robustness to various knowledge transfer, and adaptation to evolving
uncertainties, including adverse weather environments.
conditions, changing road surfaces, and
unexpected obstacles, is critical for ensuring 6. Hardware Optimization:

International Journal of Scientific Research in Science, Engineering and Technology | www.ijsrset.com | Vol 11 | Issue 2 514
 Prof. Shamal Patil et al Int J Sci Res Sci Eng Technol, March-April-2024, 11 (2) : 508-516

• Optimizing the hardware architecture and using a mix of real-world and simulator data. Then I'll
computational efficiency of onboard systems is understand how a model may be taught in a simulator
imperative for meeting the computational and then generalized to the actual world, or vice versa.
demands of real-time processing and decision- There are several experimental implementations in
making in autonomous vehicles. Research in the field of self-driving automobiles, and our project
efficient neural network architectures, contributes to a large portion of them. In a nutshell,
hardware acceleration techniques, and energy- we became successful in predicting the steering angle
efficient computing platforms can facilitate using CNN.
scalable and cost-effective deployment of
autonomous driving systems. XIV. REFERENCES

7. Ethical and Regulatory Considerations: [1]. SAE International, "SAE J3016C (Taxonomy and
• Addressing ethical and regulatory challenges Definitions for Terms Related to Driving
associated with autonomous driving, such as Automation Systems for On-Road Motor
liability assignment, privacy protection, and Vehicles)", 2021.
safety standards compliance, necessitates [2]. Lerner, J., &Tirole, J. (2003). Some Simple
interdisciplinary collaboration and policy Economics of Open Source. The Journal of
development. Research in ethical decision- Industrial Economics, 50(2), 197-234.
making frameworks, transparent accountability [3]. Lerner, J., &Tirole, J. (2005). The Economics of
mechanisms, and regulatory frameworks for Technology Sharing: Open Source and Beyond.
autonomous vehicles can ensure responsible Journal of Economic Perspectives, 19(2), 99-
deployment and societal acceptance. 120.
[4]. von Krogh, G., &Spaeth, S. (2007). The open
By pursuing research in these comprehensive source software phenomenon: Characteristics
directions, we can continue to advance the field of that promote research. Journal of Strategic
autonomous driving and accelerate the realization of Information Systems, 16, 236-253.
safe, efficient, and accessible transportation for all. [5]. Apollo. [Online]. Available:
https://apollo.auto/index.html.
XIII. CONCLUSIONS [6]. CARLA Simulator. [Online]. Available:
https://carla.org/.
This study began with the models being trained and [7]. von Krogh, G., & von Hippel, E. (2006). The
parameters being tweaked to achieve the greatest Promise of Research on Open Source Software.
performance on the tracks, and then attempting to Management Science, 52(7).
generalize that performance across multiple songs. [8]. YOLO: Real-Time Object Detection. [Online].
Because the models that performed well on one track Available: https://pjreddie.com/darknet/yolo/.
performed badly on Track 2, picture augmentation [9]. Krizhevsky, A., Sutskever, I., & Hinton, G. E.
and processing were required to accomplish real-time (2012). Imagenet classification with deep
generalization. Substituting recurrent layers for convolutional neural networks. In: NIPS.
pooling layers might reduce the loss of information [10]. LeCun, Y., Bengio, Y., & Hinton, G. E. (2015).
and would be worth exploring in the future projects Deep learning. Nature.
It's fascinating to see how these models are trained [11]. Zablocki, E., Ben-Younes, H., Perez, P., & Cord,
M. (Year). Explainability of vision-based

International Journal of Scientific Research in Science, Engineering and Technology | www.ijsrset.com | Vol 11 | Issue 2 515
 Prof. Shamal Patil et al Int J Sci Res Sci Eng Technol, March-April-2024, 11 (2) : 508-516

autonomous driving systems: Review and

challenges.
[12]. Ullman, S. (1980). Against direct perception.
Basic Books.
[13]. Redmon, J., &Farhadi, A. (2017). YOLO9000:
better faster stronger. In: CVPR.
[14]. Redmon, J., Divvala, S. K., Girshick, R. B.,
&Farhadi, A. (2016). You only look once:
Unified real-time object detection. In: CVPR.

International Journal of Scientific Research in Science, Engineering and Technology | www.ijsrset.com | Vol 11 | Issue 2 516