Received 4 December 2023, accepted 5 January 2024, date of publication 11 January 2024,

date of current version 19 January 2024.

Digital Object Identifier 10.1109/ACCESS.2024.3352629

DEPHIDES: Deep Learning Based Phishing

Detection System
OZGUR KORAY SAHINGOZ 1, EBUBEKIR BUBER 2, AND EMIN KUGU 3, (Member, IEEE)
1 Computer Engineering Department, Biruni University, 34100 İstanbul, Turkey
2 Computer Engineering Department, Yıldız Technical University, 34469 İstanbul, Turkey
3 Software Engineering Department, TED University, 06420 Ankara, Turkey

Corresponding author: Emin Kugu (emin.kugu@tedu.edu.tr)

ABSTRACT In today’s digital landscape, the increasing prevalence of internet-connected devices, including
smartphones, personal computers, and IoT devices, has enabled users to perform a wide range of daily
activities such as shopping, banking, and communication in the online world. However, cybercriminals are
capitalizing on the Internet’s anonymity and the ease of conducting cyberattacks. Phishing attacks have
become a popular method for acquiring sensitive user information, including passwords, bank account
details, social security numbers and more, often through social engineering and messaging tools. To protect
users from such threats, it is essential to establish sophisticated phishing detection systems on computing
devices. Many of these systems leverage machine learning techniques for accurate classification. In recent
years, deep learning algorithms have gained prominence, especially when dealing with large datasets. This
study presents the development of a phishing detection system based on deep learning, employing five
different algorithms: artificial neural networks, convolutional neural networks, recurrent neural networks,
bidirectional recurrent neural networks, and attention networks. The system primarily focuses on the fast
classification of web pages using URLs. To assess the system’s performance, a relatively extensive dataset
of labeled URLs, comprising approximately five million records, was collected and shared. The experimental
results indicate that convolutional neural networks achieved the highest performance, boasting a detection
accuracy of 98.74% for phishing attacks. This research underscores the effectiveness of deep learning
algorithms, particularly in enhancing cybersecurity in the face of evolving cyber threats.

INDEX TERMS Deep learning, cyber security, phishing attack, classification algorithms, phishing detection.

I. INTRODUCTION not affected by cyber-attacks. Phishing attacks are the most

Today, the internet is used to carry out and manage a well-known cyber-attacks aimed at deceiving customers.
large portion of critical activities. All sectors, such as shop- Phishing is a form of fraud in which the attacker tries to learn
ping and banking, which have intense financial activities, sensitive information, such as login credentials or account
have been serviced over the internet. In addition to increas- information, by sending it as a reputable entity or person via
ing the availability of companies, this situation also brings email or other communication channels.
many security vulnerabilities. Due to the uncontrolled and Typically, a victim receives a message that appears to have
anonymous nature of the Internet, cyber-attackers can easily been sent by a known contact or organization. The message
perform attacks, resulting in severe damage as a result. contains malicious software targeting the user’s computer or
It has been one of the responsibilities of companies to has links to direct victims to malicious websites to trick them
ensure that their employees/customers are not harmed by into divulging personal and financial information, such as
attacks against them. So, companies are working hard to take passwords, account IDs, or credit card details. Attackers use
the necessary precautions to ensure that their customers are some specific digital tricks to hide themselves and deceive
users without being noticed. This makes it quite easy to
The associate editor coordinating the review of this manuscript and deceive users who are not aware of phishing attacks. It is
approving it for publication was Sedat Akleylek . easier to carry out this type of attack that exploits people’s
2024 The Authors. This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.
8052 For more information, see https://creativecommons.org/licenses/by-nc-nd/4.0/ VOLUME 12, 2024
O. K. Sahingoz et al.: DEPHIDES: Deep Learning Based Phishing Detection System

weaknesses than other types of attacks. Because the main phishing web page with a legal appearance, and even people
elements needed for the initiation of the attack are based only who do not have technical knowledge can create a phishing
on the creation of content that the users will believe. web page very quickly. Therefore, this type of attack is quite
Due to their widespread use, phishing attacks are the focus simple and can be preferred even by novice users. Attackers
of many cyber security researchers and academic groups. can perform phishing attacks either by targeting large masses
The Anti-Phishing Working Group (APWG) [1] has had a of users or by targeting a specific user segment.
major impact on the research of phishing attacks, and they As a result of these properties, there are a growing number
emphasize that most of the phishing attacks target the victim of phishing attacks all over the world. Phishing attacks target
to gather their sensitive information by acting as if they were the weakest part of the security chain, end users. Volkamer et.
legal sites to get sensitive information. al. [3] emphasize that computer users may fall into phishing
According to the APWG Phishing Activity Trends Report due to the following reasons:
in the second quarter of 2022, the Unique Phishing attack
• Limited knowledge about the URLs and their structure,
were increased by almost 43% in 12 months and the
• Not knowing about trustable web pages,
most targeted industry sectors by phishing attacks were
• Due to the use of hidden URLs and redirections, original
financial institutions, SAAS/Webmail, social media, and
website address/URL can be hidden from the users and
payments [1]. Additionally, according to Cloudflare’s 2023
not displayed in the message,
Phishing Threats Report these attackers mainly use different
• Due to the workload of the users, they omit to consult
tactics whose distribution is shared in Fig. 1, in which the
the web pages/URLs, or they can accidentally enter to
most popular one is use of Deceptive Links (URL Addresses)
them,
[2].
• Users don’t know anything about phishing and cannot
distinguish legitimate web pages from phishing ones.
Phishing attacks are a type of cyber-attack based on
exploiting the weaknesses of computer users. For this rea-
son, a two-layer security mechanism should be considered to
prevent this attack. The first of these layers includes steps
to increase user awareness. It should be ensured that the
awareness of the users is always high by using factors such as
repeated training and warnings. To prevent phishing attacks,
the other security layer includes the development of the nec-
essary software to detect and prevent these attacks before they
reach the end user.
There are several developed approaches to detection of
phishing web pages such as machine learning based trained
systems, list-based system which use blacklists/whitelists,
image-based systems that take care of visual similarities and
third party-based systems which are connected to DNS or
whois based web services.
FIGURE 1. Phishing statistics in the second quarter of 2022. On the other side, in the literature, it is easily seen that
there is a growing tendency towards the use of machine
Apart from other cyber-attacks, phishing aims to steal learning-based solutions due to their dynamicity and adapt-
identities by exploiting the vulnerabilities of computer users. ability to new attack types. At the same time, in recent years,
Even though many security products have been put in place a specific learning mechanism, named deep learning, has
to keep the network infrastructure safe and the systems are emerged, and it is especially useful for training big data
always being watched, companies can lose a lot of money if systems or systems that do not have some defined features
a user accidentally shares sensitive information. and the tendency has shifted to deep learning mechanisms
Phishing attacks are based on referring the user to a web in recent years [4], [5]. In this paper, we aim to implement
page that looks legal. These web pages seem to belong to a a real-time detection of phishing web pages by investigat-
real company or web application. This information is stolen ing their URL addresses by using a deep learning (DL)
by the attacker when the user enters the credentials in the form based trained system. The developed DL model exhibits a
on this page. An attacker can use a variety of methods to direct notable capability to identify previously unrecognized phish-
users to the web page they want. The most common way is ing attacks. This encompasses scenarios where phishing
to send an email to the user and direct them to the malicious threats have not been shared with any external source before-
web page with the link in the message. Fig. 2 shows what a hand. Notably, even in instances where users initiate requests
phishing email looks like and what it looks like on the web. to URLs classified as phishing, the model enables preemp-
Currently, several sophisticated tools can be used to create a tive measures by blocking access to these URLs, thereby

VOLUME 12, 2024 8053

 O. K. Sahingoz et al.: DEPHIDES: Deep Learning Based Phishing Detection System

FIGURE 2. Sample of a phishing mail and a web page.

preventing potential attacks. This proactive approach ensures • Utilization of Large-Scale Data Analysis for Training:
that a significant proportion of zero-day phishing attacks can The incorporation of huge size data analysis in train-
be effectively detected and blocked. The proposed work does ing the system with a vast dataset signifies a robust
not employ third-party solutions, including Whois Data, Web approach. This utilization of extensive datasets enhances
of Trust, Google Safe Browsing, and URL/IP Blacklists. This the system’s learning and adaptability.
intentional exclusion facilitated faster detections, emphasiz- • Language-Independent Detection Process: The paper
ing a self-contained and independent approach in the study’s introduces a language-independent detection process,
methodology. The results obtained from traditional machine underlining the versatility of the proposed system across
learning methods were analyzed in this article and the results different languages. This adaptability is crucial for
were compared. The experimental and comparative results addressing the diverse nature of phishing attacks.
showed that the proposed solution gives incredibly attractive The rest of the paper is organized as follows: related works
results especially compared with the standard machine learn- about phishing detection are detailed in Section II. Section III
ing based methods. The major contributions of the paper are: focuses on background knowledge about the methods and
• Construction of a Well-Balanced Large Dataset: The tools. The details of the proposed system are depicted in
creation of a sizable dataset with a balanced class dis- Section IV. The experimental results and discussions are
tribution stands out as a substantial and challenging presented in Section V and Section VI respectively. Finally,
achievement, particularly when compared with existing conclusions and future works about the research are pre-
datasets. sented.
• Enhanced Phishing Detection with Reduced False Posi-
tives: The paper demonstrates improved phishing detec- II. RELATED WORK
tion capabilities, showing a reduced false positive rate. Phishing attacks are based on taking advantage of people’s
This refinement is crucial for enhancing the reliability weaknesses. Therefore, the first step to ensuring security
of the detection system. against phishing attacks is to raise awareness of the users
• Fast Phishing Page Detection via URL Analysis: The and to keep their awareness high. Through simulations of
ability to rapidly detect phishing pages by analyzing the sample scenarios, it is necessary to assess the awareness
URL addresses represents a significant advancement. levels of the users at regular intervals and repeat the training.
This quick analysis adds a layer of efficiency to the Users, even experienced ones, can fall victim to this type of
phishing detection process. attack. Although security training decreases the number of
• Highly Accurate Detection of ‘‘Zero-Day Attacks’’: The deceived users, software-based detection systems, which can
paper introduces a method for accurately identifying be categorized mainly into five groups as depicted in Fig. 3,
‘‘Zero-Day Attacks,’’ contributing to an increased level can also warn the user about suspicious web pages. List Based
of security. This capability is pivotal for staying ahead of Detection Systems
emerging threats by distinguishing between normal and List-based phishing detection systems are the ones that can
abnormal requests. be easy to set up and maintain. They mainly use two lists as
• Independence from Third-Party Services: The proposed blacklists and whitelists for classifying URL addresses and/or
system operates independently of third-party services, IP addresses. These addresses can be classified either through
reducing reliance on external sources and enhancing the voting or by using experienced human classifiers who receive
self-sufficiency of the detection mechanism. computerized support for their decision.

8054 VOLUME 12, 2024

O. K. Sahingoz et al.: DEPHIDES: Deep Learning Based Phishing Detection System

Machine algorithm for classification, and it is indepen-

dent of third-party services and languages. They focused
on only Internet Banking pages, using 549 legitimate sites
and 1158 unique e-banking websites, and implemented the
detection system as a Chrome browser extension.
In [12], the authors implemented a rule-based incremental
method that is lightweight and proactive in its detection of
FIGURE 3. Phishing detection model.
previously not encountered phishing URLs. Because of its
computational intelligence and independence from blacklist
signatures, this tool is extremely accurate when it comes to
The Google Safe Browsing API enables the client appli- detecting zero-day and spear phishing exploits.
cations to check the URLs against the constantly updated
blacklists [6]. Similarly, Office 365 provides an Advanced B. VISUAL SIMILARITY BASED DETECTION SYSTEMS
Threat Protection service for enterprise organizations, which The main deceptive aspect of a phishing webpage stems from
checks the URLs when the user clicks on them in the email its high visual resemblance to the legitimate page in terms of
message [7]. The system immediately checks this address texts, fonts (by style, size, and color), used images (by their
before opening it and identifies itself either as safe, malicious features such as height and width, and their positions), page
or blocked. layouts, HTML tags, etc. [13]. Some phishing detection sys-
On the other hand, Jain and Gupta [8] implemented tems use visual similarity-based techniques by utilizing these
an auto-updated whitelist-based phishing detection system, resemblance features to make the classification as reviewed
which aims for a faster access time with a high detection in [13]. After making a feature-based comparison, if the
rate. They tested their proposed system with a limited number similarity is greater than the predefined threshold value, it is
of phishing and legitimate datasets (with 1120 phishing and identified as a fake/phishing webpage.
405 legitimate addresses) and reached an 86.02% true posi- The authors of [14] visually compared a suspicious web
tive rate and a 1.48% false negative rate in their work. page to a legitimate one to see if they were more similar than
In [9] Azeez et al. demonstrate how to detect phishing an acceptable rate. To check web pages, they used three-page
attacks using an automated white-list method. The whitelist features such as text features, image features, and the overall
is developed following an in-depth study of both the visible visual appearance of the page.
and genuine connections. The similarities of the known trust- In [15], to detect the phishing pages, authors used visual
worthy site are computed by juxtaposing the domain name characteristics of the web page by measuring the suspicious
with the whitelist’s contents, comparing it to the IP address pages’ similarity to legitimate ones. They checked the block
prior to making a judgment, and then assessing the real link level similarity, layout similarity, and style similarity for
and visual link and calculating the known trustworthy site’s this detection process. If the system calculates a similarity
similarities. value that is greater than the threshold value, it generates an
The main advantage of these systems is their quick alert report for the user. The system was tested with only a
response time. Therefore, they are suitable for real-time 328-page dataset. According to this dataset, they achieved
environments. However, it is hard to maintain the up-to- satisfactory results.
dateness of the lists. Therefore, these systems are vulnerable As demonstrated by the preceding research, there is no
to zero-day attacks on websites, which have not been used globally accepted dataset for making meaningful compar-
previously. isons. In many projects, authors generate their own dataset
with a limited number of data points. An extended dataset is
A. RULE-BASED DETECTION SYSTEMS used in relatively new work [16], with 1,367 legitimate pages
Depending on the features of the web pages, these types and 2,129 real phishing pages. In this work, they used global
of systems detect phishing attacks according to rules (either and local features of the webpage. Apart from that, this work
static rules or fuzzy rules) either depending on the features of relies on image level comparison by using a snapshot image
the URL or its content. of the legitimate webpage and the suspicious webpage. As a
Abdelhamid et al. [10] use Associative Classification by result, it can effectively deal with tricks that get help from
defining simple ‘‘If-Then’’ rules to achieve a good accuracy the use of flash objects, images, or form of HTML content.
rate for the classification of web pages. The rules are not Depending on the dataset, they reached a true positive rate of
static in the paper; they tested 6 different algorithms in their over 90 % and a true negative rate of over 97 %.
work. Each algorithm produces a different number of rules. The primary advantage of these approaches is that they
However, it depends on third-party services to detect the age can quickly detect embedded objects such as images, Java
of the domain, web site traffic, DNS records etc. Applets, ActiveX and Flash, which are located in HTML
The authors proposed a rule-based web page classifica- text. Additionally, they generally use signatures to identify
tion system by using only 10 rules for detecting zero-day the fake pages by taking into account the common features of
phishing attacks [11]. The system uses a Support Vector the whole website.

VOLUME 12, 2024 8055

 O. K. Sahingoz et al.: DEPHIDES: Deep Learning Based Phishing Detection System

C. THIRD PARTY BASED DETECTION SYSTEMS deployed in their lab, they randomly selected 600,000 URLs,
Due to its simplicity and ability to reach an acceptable accu- 400,000 benign and 200,000 malicious in their dataset. How-
racy rate, some authors developed their system by getting help ever, there are no details about the dataset, and they are also
from third-party services such as search engines/rankings or not shared publicly.
searching the whois/DNS records of the web page [17], [18], Some of the authors concentrated on IoT device security
[19]. With these systems, they can check the legitimacy of by blocking phishing URLs [25]. They used an autoencoder
the webpage and the authenticity of IP addresses and their mechanism to increase the performance of the proposed sys-
associated domain names. tems. They also tested different optimizers and said that the
However, the false positive rate of these methods is rel- Adam optimizer gave the best results for this application area.
atively high because if a legitimate web page is newly Some studies have tried to obtain more successful models
constructed, it does not appear in the search results or returned by combining different deep learning architectures. In [26],
values, which have a negative effect on this detection process. they proposed a customized model consisting of CNN and
Additionally, a legitimate domain name can host phishing LSTM layers. 3 different feature sets are used in this study.
web pages on its host, which is hard to catch for this type These are lexical features, character-level emphases, and
of system. word embeddings. Different models were built on these fea-
ture sets, and the outputs of these models were combined into
D. MACHINE LEARNING BASED DETECTION SYSTEMS one model. Lexical features are obtained because of feature
In recent years, there has been an increasing trend in the use of engineering. 30 different hand-crafted lexical features are
machine learning (ML) techniques, especially for solving lots used. A balanced dataset was used in the study. The total size
of real-world problems. These techniques train the system of the dataset is approximately 2.5 million.
with the help of a classification algorithm that uses a few A phishing URL detection system based on the RNN
features, like the page content, URL, and/or network features, architecture was developed in [27] and tests were done using
to tell the difference between a legitimate website and a only URL information. There are no manually created fea-
phishing website. tures. The feature that distinguishes this study from others
In [20], Karim et al. proposed a hybrid model to improve is its ability to visualize the effects of the URL sections on
the accuracies of the machine learning models. The combina- the outcome. Thanks to the visualization they’ve developed,
tion of linear regression (LR), support vector classifier (SVC) which fields in the URL cause phishing characters can be
and decision tree (DT) is created using soft and hard voting understood by the human eye. LSTM, GRU, BiLSTM, and
methods and named as (LR + SVC + DT). BiGRU were tested in the study and achieved 0.99 accuracy
Prabakaran et al. proposed an enhanced Deep Learning- on the 1.5M dataset.
based phishing detection approach that combines the There is no widely used dataset accepted in the literature
strengths of Variational Autoencoders (VAE) and Deep yet. Some of the researchers collect their datasets from open-
Neural Networks (DNN) to effectively identify malicious source sources, and some of them use the data collected in
URLs [21]. In [22], Alshingiti et al. proposed three different earlier studies. In [28], the authors created a dataset that they
techniques including LSTM, CNN and an LSTM–CNN- collected and performed training and test operations on this
based approach to identify phishing websites. They got the dataset. In addition, they made comparisons using 3 different
accuracies of the suggested techniques, i.e., 99.2%, 97.6%, public datasets in previously published studies in literature.
and 96.8% for CNN, LSTM–CNN, and LSTM, respectively. In the study, which was made with the CNN architecture, both
In the study [23], the authors proposed a URL-based phish- traditional machine learning methods and many deep learning
ing detection system by analyzing the login URLs. As a methods were tested and compared.
discriminative property, they focused on homepages with They proposed an approach in [29] that combines CNN
login forms by emphasizing that the current systems have a and attention based hierarchical RNN architecture. An unbal-
high positive rate when tested with the legitimate login pages. anced dataset was collected and used within the scope of the
They detected a total of six distinct phishing web domains, study. 4.5M legitimate samples and 241K phishing samples
and their findings were contingent on the kind of service used were used. In this study, an accuracy value of 0.97 was
by the attacker. Automatic feature extraction based on ‘‘Term obtained.
Frequency Inverse Document Frequency’’ at the character In [30], a phishing URL detection system was developed
N-gram level paired with the Logistic Regression algorithm using machine learning techniques. In this study, hand-crafted
was one of their methods of choice in the implementation and was used. A dataset of 73,575 URLs with a balanced class dis-
they achieved valuable results. tribution is used with 0.978 accuracy. A CNN-based approach
The used datasets are generally imbalanced due to the large is suggested in the study [31]. The study developed in [32]
number of legitimate web pages. Therefore, He et al. [24] can be used as a real-time system and can be improved
focused on the detection of phishing attacks by using the according to user feedback. A system working in real-time
cost sensitive XGBoost method. To balance the dataset, they has been developed with a Chrome extension. In this study,
preferred the use of the synthetic minority over-sampling besides traditional machine learning approaches, RNN-based
technique (SMOTE) for oversampling. By using a crawler GRU and LSTM approaches were tested. The accuracy rate

8056 VOLUME 12, 2024

O. K. Sahingoz et al.: DEPHIDES: Deep Learning Based Phishing Detection System

of this study was 0.99. The dataset used in this study was A. URL ADDRESSES
collected from open sources. The authors aimed to utilize The address of a web page is named as Uniform Resource
features for developing a phishing detection system and Locator (URL), which uses mainly the HTTP or HTTPS
designing an automated, complete, real-time system. A sys- protocol. A URL is essential for specifying the location of
tem working in real-time has been developed with a Chrome any type of resource, such as an HTML page, file, image,
extension. In this study, besides traditional machine learning directory, program, etc. The URL components, as depicted in
approaches, RNN-based GRU and LSTM approaches were Fig. 4, play a critical role in understanding the used method-
tested. The accuracy rate of this study was 0.99. The dataset ology.
used in this study was collected from open sources. On the
other hand, [33] aims to utilize features for developing a
phishing detection system and designing an automated, com-
plete, real-time system. Pre-processing, clustering, feature
selection, classification, and k-fold cross validation were
applied in the study. This study uses web page informa- FIGURE 4. URL components.
tion, not only URL information, to enhance the accuracy
rate. A dataset consisting of 500 web page records in total A URL may contain some or all of the following compo-
was used in the study and 0.963 accuracy was achieved. nents.
Reference [34] proposes a data-driven approach to detect
• A hostname consists of three parts: A domain name is
phishing websites using various machine learning classifiers,
such as Decision Tree, XGBoost, Random Forest, Support always present in the URL, and it is used for locating the
Vector Machine, and Naive Bayes by implementing vari- computer on the Internet. It is a registered identification
ous numbers/types of features such as URL-based features, ‘‘string’’ and is used instead of numeric IP addresses.
• A top-level domain is also always present in the URL.
hyperlink-based features, and hybrid features. They devel-
oped a dataset with 6000 URLs containing 3000 legitimate It shows the shortened form of an organization type as
URLs and 3000 phishing URLs. They collect web page infor- ‘‘.com’’, ‘‘.org’’, ‘‘.net’’ etc. in the hierarchical structure
mation with a crawler and apply feature extraction to the data of the Domain Name System (DNS). A subdomain is a
collected. Experimental results on the dataset show that the part of a larger domain that has a specific interest area
XGBoost algorithm with hybrid features has a 98.81% true and services.
• The protocol is always present, but not always visible
positive rate and 0.49% false-positive rate.
The spread of phishing attacks over email is quite com- in the URL and it determines how data is transferred
mon. It is also possible to detect phishing via email content. between the server and client side, mainly HTTP and
In [29] and [30] the authors developed a study that detects HTTPS; additionally, SMTP, FTP, DHCP, etc.
• Path/File shows where the page or file is on the website.
phishing attacks using email content. It is a confusing system
to adapt to the real world because e-mail is mostly considered The notation of the path/file depends on how the website
confidential. The study developed in the former one achieved is set up.
• HTML anchors or fragments are used on the webpage to
0.98 accuracy with the dataset consisting of 10K samples.
In [36], only traditional machine learning algorithms were show the internal page navigation.
evaluated while [35] focused on machine learning meth- Parameters are located at the end of the URL; they are
ods and deep learning-based CNN and RNN architectures. represented with key/value pairs, which use ‘?’ and ‘&’ char-
Preprocessing, clustering, feature selection, classification, acters respectively.
and k-fold cross validation were applied in the study. This
study uses web page information, not only URL information, B. DEEP LEARNING
to enhance the accuracy rate. A dataset consisting of 500 web While the capability of computers has increased, Artificial
page records in total was used in the study and 0.963 accu- Intelligence (AI) has become an expanding concern in com-
racy was achieved. The detailed comparison of the literature puter science due to its aim of making computers intelligent
on machine learning/deep learning-based phishing detection like humans. Learning is perhaps the distinguishing factor
systems is shown in Table 1. In this research, it is aimed to use that makes us human.
deep learning-based solutions with the dataset that is known Therefore, researchers focused on how to program this
as the greatest up to now. factor for computers, and a new subfield of AI emerged as
machine learning (ML). An ML algorithm enables forecast-
III. BACKGROUND ing future things without rules and models, building models
The proposed framework processes the URL address of the to explain the execution styles of real-world entities, and
web page and classifies them either as phishing or legitimate identifying patterns according to observed data.
by using a deep learning-based approach. Deep learning is one of the algorithms or subsets of
A background knowledge is required to understand the rest machine learning. It primarily refers to deep artificial neural
of the article clearly. These are explained in this section. networks or deep reinforcement learning, both of which use

VOLUME 12, 2024 8057

 O. K. Sahingoz et al.: DEPHIDES: Deep Learning Based Phishing Detection System

TABLE 1. Comparison of the literature.

8058 VOLUME 12, 2024

O. K. Sahingoz et al.: DEPHIDES: Deep Learning Based Phishing Detection System

FIGURE 5. The flowchart of DEPHIDES.

the term ‘‘deep’’ to refer to the number of layers in the sys- from ‘‘commoncrawl.org’’, in the dataset created. PhishTank
tem [37]. There are different methods (such as convolutional is an open-source community where phishing URLs are
neural networks, recurrent neural networks, long short-term shared and kept up to date. Shared URLs are subject to a
memory, and auto-encoders) to apply DL, and these methods vote. A URL submitted because of voting is assigned to one
depend on the use cases, such as the kind of data that needs of three classes. These classes are Valid Phish, Invalid Phish,
to be processed. and Unknown. A URL is marked as valid phishing if enough
One of the main differences and applicability of DL and votes are collected to show that it is phishing. Otherwise,
ML is their data dependencies. ML algorithms are preferable it is assigned to the invalid phishing class if enough votes
and result in well even if there is a small dataset. However, are collected showing that the URL is not phishing. URLs
DL is a ‘‘Data Hungry’’ approach, and if you have more data, that cannot be identified as valid or invalid are marked as
you will get more accurate results. Due to its structure and unknown. This control mechanism ensures high accuracy of
depth, DL needs more computation. However, with the emer- collected phishing URLs. URLs in the Valid Phish class,
gence of new computation technologies such as the graphical which were shared by PhishTank until August 2018, were
processing unit (GPU), parallel computation is possible with collected. In this way, 2,320,893 phishing URLs are included
the use of thousands of cores compared to a single CPU, in the dataset created in this article. An example of the dataset
which has a small number of cores. is shown in Table 2.
Although DL methods generally use the power of parallel To develop a machine learning application for the detection
computation, the extended use of big data results in increased of phishing attacks, it is also necessary to collect URLs
training time, which ranges from hours to weeks to months. that are not phishing. We got help from the Common Crawl
However, traditional ML algorithms often train fast, which corpus which contains petabytes of data collected over the last
ranges from a few minutes to hours. 7 years. It contains raw web page data, extracted metadata,
and text extractions. The concept that well-known legitimate
IV. DEEP LEARNING BASED PHISHING DETECTION domains receiving a considerable number of backlinks tend
SYSTEM (DEPHIDES) DETAILS to have high rankings is a valid observation. It’s challenging
In the proposed model we made a data mining process accord- for a domain to rank prominently in search engine results
ing to the flowchart in Fig. 5. Initially we collected a relatively if it is being used for malicious purposes, as search engines
big dataset from Internet sources. aim to provide users with the most relevant and trustworthy
The dataset is divided into three parts for making a com- content. Therefore, the inclusion of such domains in the
parable performance measurement. 70 % is used for training, legitimate URL dataset ensures the reliability of the data
while 20 % is for validation and 10 % is used for testing. collected.
By using this flowchart, the system is trained for five dif- To compile the legitimate URL dataset, the first 100,000
ferent deep learning algorithms as Artificial Neural Network, domains from PageRank rankings worldwide, calculated by
Convolutional Neural Networks, Recurrent Neural Networks, Common Crawl using data from web crawling, were uti-
Bidirectional Recurrent Neural Networks, Attention Net- lized. For each of these domains, approximately 30 random
works. Finally, the trained model can be used in real world URLs were selected, resulting in a collection of 2,881,948
implementation for detection of phishing web pages. legitimate URLs. This extensive dataset provides a robust
foundation for the development and testing of phishing detec-
A. DATASET COLLECTION tion systems, as it represents a wide array of legitimate online
Within the scope of this study, a comprehensive dataset resources.
has been created and several experiments have been The developed phishing detection system adopts an
run on this dataset. Phishing URLs are collected from approach that involves collecting URLs from domains that are
‘‘www.phishtank.com’’, and legitimate URLs are collected unquestionably legal. Typically, the URL structure within a

VOLUME 12, 2024 8059

 O. K. Sahingoz et al.: DEPHIDES: Deep Learning Based Phishing Detection System

TABLE 2. Data samples.

domain remains consistent and standard. However, to ensure legitimate URLs. The other version (big_dataset) includes
that the algorithm does not become overly specialized or the whole dataset as given in Table 3. Due to hardware
memorize the specific structure of a particular domain, only constraints, most of the tests in this article were run on
10 or fewer URLs from such domains have been added a small_dataset to compare the results obtained by other
to the dataset. This approach helps maintain a broader and researchers doing research in this area.
more generalizable training dataset, allowing the system to
effectively detect phishing attempts across various legitimate B. VECTORIZATION
domains and their respective URL structures. Attackers employ various tactics in phishing attacks, some of
which pose significant challenges for detection. The attackers
TABLE 3. Dataset details. employ techniques that pose significant challenges for con-
ducting word-based analyses. This is due to the widespread
use of subtle alterations that are hard for individuals to dis-
cern. To perform word-based analysis, one must undergo
training on a corpus forming a substantial volume of nonsen-
sical words. Furthermore, word-based analysis introduces a
level of language dependence. Consequently, in this article,
we employed a character-based embedding approach for the
vectorization process. Since the embedding size for each char
is determined as 50, the vectorized version of the dataset is as
shown in Fig. 6.

In our study, approximately 5.1 million URLs were col-

lected, with 2.3 million being phishing and 2.8 million being
legitimate. As far as we know, the dataset is the largest shared
dataset in literature for this purpose. To ensure the detection
of the latest attacks and address the limitations of supervised
learning, the iterative updating of the model with newly
acquired training data proves highly beneficial. In a com-
mitment to transparency and collaboration, all source code FIGURE 6. An example of dataset vectorization.
and datasets generated within this project have been shared
as open source, which facilitates continuous improvement, During the vectorization phase, it is essential to constrain
allowing the model to be regularly updated with additions to the size of the URL to a specified character limit. To figure
the dataset, thereby enhancing its efficacy and adaptability out the appropriate character limit, statistical metrics are eval-
over time. uated, taking into account the character lengths of the URLs.
There is a need for high hardware resources to process Consequently, for each class, we calculate the minimum,
these amounts of data by machine learning algorithms. For maximum, standard deviation, and median separately. These
this reason, two different versions of the dataset have been statistical findings are presented in Table 4, with the data
created for the tests performed in this article. The first of being derived from the extensive dataset.
these versions (small_dataset) consists of a 10% subsample It becomes clear that a URL length of 200 characters
of the total dataset. This means 232,090 phishing, 288,195 adequately encapsulates the majority of the dataset with

8060 VOLUME 12, 2024

O. K. Sahingoz et al.: DEPHIDES: Deep Learning Based Phishing Detection System

TABLE 4. Character length statistics for dataset.

more than 95% of the dataset. Consequently, this specific employed for training the model consists of sequential data,
value is adopted in the vectorization process. URLs exceed- therefore the proposed model necessities the use of an
ing 200 characters are truncated to this length, while URLs algorithm, which is capable of processing such sequences.
shorter than 200 characters are padded to reach 200-character In addressing the phishing URL detection challenge, the
length. In Fig. 7, a histogram chart is presented, depicting the approach involves treating the URL as a single entity and
URL length distribution in the ‘‘big_dataset.’’ The histogram learning the relationships among its smaller sub-parts.This
is generated by grouping URL lengths into 20-character buck- approach is widely used to make sense of textual data. For
ets. example, studies that perform semantic analysis have sim-
ilarly been able to achieve good success by evaluating the
sub-parts of the text together. In addition, it was thought that
it would be effective to use the most commonly preferred
algorithms for similar problems in literature. Due to the
high complexity of certain algorithms, training them becomes
a challenging task. Recognizing the critical importance of
faster response in detecting cyber-attacks, there is a deliberate
effort to avoid favoring algorithms with high complexity.
A total of five distinct deep learning architectures were exam-
ined. Furthermore, analyses were conducted to enhance the
accuracy rates by exploring various configurations for each
architecture.
Artificial Neural Network (ANN) is characterized by its
acyclic connections between units. In this approach, the archi-
FIGURE 7. Histogram of the URL length ranges.
tecture primarily includes dense layers. Each node performs a
direct linear calculation, and the parameters are then updated
In our dataset, a consistent dimension is imposed during the based on the output of the loss function.
training process, irrespective of the original sizes of URLs. Convolutional Neural Networks (CNNs) belong to the
URLs shorter than the specified length are padded to reach the realm of deep learning algorithms and are tailored to demand
designated dimension, while those exceeding the specified minimal preprocessing. They employ a form of multilayer
length are truncated to fit. This URL length parameter is perceptron and execute convolution operations by envision-
applied uniformly to all records in the dataset. Selecting ing them as sliding window functions applied to a matrix.
the best URL length entails striking a balance between the When it comes to word-based approaches, this window tra-
model’s architectural complexity and its coverage. If the verses through words in the input data. Each convolutional
URL length is excessively long, it substantially prolongs operation yields a response when it detects specific pat-
both training and testing times, thus diminishing the model’s terns. These patterns may encompass expressions such as
practicality and efficiency. Conversely, an overly short URL ‘‘I hate’’ or ‘‘very good,’’ enabling CNNs to identify them
length compromises the model’s training performance. As a within a sentence irrespective of their position. Character-
result, for this study, a URL length of 200 was chosen, as it is based approaches are instrumental in recognizing malicious
deemed well-suited for effectively representing the majority patterns in URLs used in phishing attacks. In Section III-B,
of the dataset. we delved into the challenges associated with identifying
Attackers may attempt to bypass prevention systems by phishing attacks.
utilizing URL shorteners. Even if the DEPHIDES URL Recurrent Neural Network (RNN) constitutes another
checking service fails to successfully classify the short- subset of deep learning techniques, where connections
ened URLs, the attackers’ objectives cannot still be reached between nodes create a directed graph along a sequence. RNN
because the shortened URLs are directed to original web can be envisioned as a sequence of interconnected neural
pages during the attack, and they will be accurately classified network blocks, forming a chain-like structure. It can be
as phishing by the system. likened to multiple replicas of the same network, each passing
information to its successor. This architecture is particularly
C. OVERVIEW OF TESTED ALGORITHMS well-suited for handling sequential data, such as text.
This article presents the development of a phishing URL In the RNN framework, when processing a word, it uses the
detection system based on deep learning. The dataset knowledge of preceding words. Each word takes the output
VOLUME 12, 2024 8061
 O. K. Sahingoz et al.: DEPHIDES: Deep Learning Based Phishing Detection System

TABLE 5. Summary of architectural components of each algorithm used in scenarios.

of the previous words as its input. RNN allows for one-way utilized in subsequent steps. These networks pinpoint and
information flow, while Bidirectional Recurrent Neural Net- select the important or distinctive elements of the data.
works (BRNN) facilitate two-way information exchange. This means that neural networks make informed decisions
This means that not only is information from preceding words regarding which features deserve their attention, thus assign-
used in processing a word, but the subsequent words are also ing more weight to essential elements across the entire
taken into account during the process. dataset.
Bidirectional Recurrent Neural Network (BRNN) Many deep learning approaches eliminate the need for
architecture operates on the principle of splitting the neurons the feature engineering step. Basic keyword searches are a
found in a regular RNN into two directions: one for the method that can be easily avoided. For instance, instead of
positive direction and another for the negative direction. Tra- using amazonsecurelogin.com, a similar domain like ama-
ditional RNNs have unidirectional information flow, meaning zonserucelogn.com could be employed for attacks. Since
they only consider information from preceding words when the words ‘‘secure’’ and ‘‘login’’ are not used directly
processing a word. However, for comprehensive word inter- in this example, keyword checking may not be success-
pretation, it’s often necessary to incorporate both preceding ful. Furthermore, attackers can not only target English but
and subsequent words. BRNN accomplishes this by running also launch regional attacks by creating URLs based on
one RNN in a forward direction from the beginning and the language of that region. This approach will impose
another in reverse from the end. additional tasks on keyword checking systems, such as
In contrast, Attention Networks (ATTs) employ an gathering, using, and updating an extensive word cloud
approach that prioritizes comprehending the entire context covering multiple languages. Having a specific phishing
rather than focusing on isolated components. ‘‘Attention’’ keyword may not always provide useful information for
in this context refers to the conscious directing of the classifying the URL as phishing. For instance, a URL like
mind toward a specific object or aspect. Attention net- amazon.com/customerservices/login may legitimately serve
works discern critical portions of the data and incorporate a legal purpose. Although deep learning algorithms have
this information into their memory, which is subsequently shown success in making such inferences.

8062 VOLUME 12, 2024

O. K. Sahingoz et al.: DEPHIDES: Deep Learning Based Phishing Detection System

URL hijacking, also known as URL redirection or URL At the stage of creating architecture, parameters such as
spoofing, is a type of cyber-attack where an attacker redirects the number of layers and number of neurons are given. The
a website’s traffic from a legitimate URL to a malicious one. default values of the frameworks used in other hyperparame-
However, the proposed system cannot detect URL hijacking ters are used.
attacks due to its structure. This article investigated the performance of five distinct
It is not necessary to retrain the entire dataset when incor- deep learning architectures: ANNs, CNNs, RNNs, BRNNs,
porating new data. The model can be updated iteratively, and ATTs. These architectures exhibit varying memory and
continuing from where it left off in the previous training. processing power requirements. Some may demand less
In this article, in addition to deep learning-based memory while others consume more, and the same principle
approaches, tests were conducted with traditional machine applies to processing power required during training.
learning algorithms and the results were compared. To ensure optimal GPU utilization and an equal accelerator
• Random Forest (Ensemble Based Algorithm) effect across all architectures, the batch size parameter was
• Naïve Bayes (Statistics Based Algorithm) fine-tuned in the tests. In cases where processing the entire
• Logistic Regression (Statistics Based Algorithm dataset at once posed a challenge, the dataset was divided into
smaller portions, each of which was processed sequentially.
Architectural components were used in the Keras tool.
It’s important to note that larger batch sizes consume more
A custom architectural design was made for each scenario
power and memory. In the tests conducted as part of this
using architectural components. Table 5 provides summaries
study, adjustments were made to the batch size parameter to
of the designed architectures. The detailed architectures of
maximize GPU utilization for all test scenarios. Importantly,
all scenarios can be accessed by clicking on the link given in
altering batch size values did not have a discernible impact on
each row at the table.
the accuracy and loss values. Instead, this parameter primarily
affects the volume of data processed within a given time-
V. EXPERIMENTAL RESULTS frame, without influencing the overall quality of the results.
In the context of this study, a deep learning-based system was
developed for the purpose of detecting URLs used in phishing A. DEEP LEARNING APPROACHED WITH BASE SCENARIO
attacks. The effectiveness of this system was assessed by
To facilitate a comprehensive comparison of running times
comparing its performance with traditional machine learning
and architectural performance, all five architectures under-
algorithms through a series of tests. In the implementation of
went testing with a common configuration featuring a single
deep learning architectures, the Keras Framework was used
layer of 128 neurons. This configuration served as the base-
in the front end, with the backend powered by the Tensor-
line, and subsequent performance-enhancing techniques were
flow Framework. Traditional machine learning algorithms
applied to enhance overall success. The tests for this configu-
were implemented and tested using the Scikit-Learn package,
ration spanned 20 epochs, and the results are visually depicted
which was also used for reporting test results.
in Fig. 8, and batch size, train accuracy, validation accuracy
The evaluation of the system’s performance entailed the
and test accuracy values are shown in Table 6.
use of several key metrics, including accuracy, loss, precision,
recall, and F-score. A detailed explanation of these metrics
TABLE 6. Performance metrics for base scenario with 20 epochs.
can be found in Section III.E. In addition to these metrics,
the study also considered the execution time of the tests.
Performance evaluation is a crucial aspect in comparing the
accuracy of different algorithms, and execution times may
vary based on the underlying hardware. For this project,
a server from Floydhub was leased for conducting the tests.
The leased server is equipped with a Nvidia Tesla V100
model GPU. V100 has 5120 cores and 16 GB of VRAM. The
operating frequency of each core is 1370 Mhz.
Although the focus of this study was to develop a deep
learning-based system, the results were compared with those
of traditional machine learning algorithms. Hyperparameters The findings presented in Table 6 revealed that the RNN
should be determined for the use of deep learning algorithms. architecture emerged as the most successful deep learning
Common hyperparameters used in the tests carried out within architecture, boasting an impressive test accuracy of 95.1%.
the scope of this study are: It’s worth noting that accuracy rates among the architectures
• Loss function: Binary Cross Entropy were relatively close, but RNN led the pack. Interestingly,
• Sequence size: 200 both RNN and BRNN architectures exhibited lower initial
• Update function and its parameters: ADAM; Learning validation accuracy rates compared to the other architectures.
rate: 0.01, Beta_1: 0.9, Beta_2: 0.999. Beyond accuracy rates, the evaluation of algorithm perfor-
• Character embedding dimension: 50 mance also considered the runtime durations. In the baseline

VOLUME 12, 2024 8063

 O. K. Sahingoz et al.: DEPHIDES: Deep Learning Based Phishing Detection System

scenario, the tested architectures underwent training over

20 epochs, and the average durations for each epoch can be
found in Table 6.
Based on the obtained results, it is evident that recurrent
algorithms (RNN, BRNN, and ATT) operate at a notably
slower pace compared to the other architectures. Among
these, the BRNN architecture proved to be the slowest, with
an average time of 150 seconds per epoch.

B. DEEP LEARNING APPROACHED WITH COMPLEX

SCENARIO
To enhance the likelihood of accuracy, one approach is to FIGURE 9. Complex scenario validation accuracies.
increase the complexity of deep learning architectures or
extend the number of training epochs. In the subsequent
phase, more intricate architectures were examined, and their executed on GPUs with better performance, architectures still
outcomes were assessed. All architectures subjected to testing required a substantial amount of time for testing. Due to hard-
were designed with seven layers during this phase. Deter- ware limitations, not all architectures could be assessed using
mining the number of neurons in the model architecture is complex configurations. In this stage of the study, tests were
a critical consideration. Augmenting the number of neurons performed on three architectures. For example, RNN archi-
introduces complexity to the model, resulting in a larger tecture took approximately four hours to complete 20 epochs
number of parameters that must be learned during training. of training on the GPU. Based on the runtimes presented
The complexity of a model with increased training and test- in the base scenario (Table 6 ), the BRNN architecture was
ing times is a crucial trade-off. Conversely, setting the number roughly four times slower than the RNN architecture. Mean-
of neurons too low can impede the model’s ability to learn while, the ATT architecture lagged the RNN architecture
effectively. When determining the number of neurons, careful by approximately 3.2 times. In this study, it wasn’t feasible
consideration of these factors is essential to make an informed to provide the BRNN and ATT architectures with sufficient
choice. Given that the number of neurons is a numerical resources for training over 20 epochs, given the imperative
parameter independently adjustable for each layer, a wide need for extended training durations as seen in Table 7.
range of values is possible. Different architectures have their
unique neuron structures, resulting in varying complexities
TABLE 7. Performance metrics for complex scenario with 20 epochs.
between, for instance, RNN and CNN neurons. To ensure a
controlled experiment across all deep learning architectures,
a consistent number of neurons (128) was employed for all
layers. This choice was made to ensure successful testing
while maintaining reasonable training times for all architec-
tures. After thorough research and preliminary testing, it was
determined that using 128 neurons in all layers was a well-
balanced decision. The experiments in this study were also
trained over 20 epochs, and the test results for the architec-
tures can be found in Fig. 9.

The results indicate that both the CNN and RNN archi-
tectures demonstrated superior performance on the test set
compared to the ANN architecture. Furthermore, the CNN
architecture outperformed the RNN architecture, achieving
more successful results in both the test set and the vali-
dation set. Additionally, the CNN architecture exhibited a
substantially shorter total running time, completing 20 epochs
in about eight minutes, whereas the RNN architecture took
approximately 238 minutes, encompassing vectorization and
other preprocessing tasks. In the complex scenario tests,
there was no substantial improvement observed in the ANN
FIGURE 8. Base scenario’s validation accuracies. architecture, but accuracy values increased for the CNN and
RNN architectures. These results suggest that the architec-
Apart from other performance metrics, running times play tural complexity in CNN and RNN approaches contributes
a significant role in architecture selection. While tests were positively to accuracy metrics. Notably, the complex scenario

8064 VOLUME 12, 2024

O. K. Sahingoz et al.: DEPHIDES: Deep Learning Based Phishing Detection System

tests indicated increased running times for all tested architec- TABLE 9. Performance metrics of CNN1 architecture with 20 epochs.
tures, highlighting the effect of architectural complexity on
running time.

C. COMPLEX SCENARIO DIFFERENT CNN

ARCHITECTURES
The study concludes that as one of the Deep Learning
Approach, CNN architecture is more suitable for phishing
URL detection than the other architectures tested. As depicted
in the graphs in Fig. 10, it is apparent that training has the complex CNN scenario. The distinction lies in the train-
not yet reached a convergent state. Consequently, there is ing duration: the complex CNN scenario underwent training
potential to achieve higher performance values with longer for 100 epochs, whereas the other scenario was trained
training sessions. Therefore, the tests conducted in the next for 20 epochs. Upon examining the values in this table,
phase encompassed 100 epochs. Additionally, two more CNN it becomes evident that longer training periods had a positive
architectures were examined in this experiment, and their impact on the CNN1 architecture. In the tests conducted with
outcomes were compared. The performance metrics for the the complex CNN architecture over 100 epochs, the accuracy
tests conducted on three complex CNN architectures with rate further increased to 98.0% in both the test and validation
smaller datasets are presented in Fig. 10. sets. Following the conducted tests, CNN1 was selected as
the most successful architecture. This architecture was tested
with 100 epochs of training on the big dataset, and the results
were evaluated.
Table 5 provides a breakdown of the architectures for
CNN1, CNN2, and CNN3. The differences between CNN1,
CNN2 and CNN3 are the factors that affect the architectural
complexity, such as the number of layers and the num-
ber of neurons as depicted in Table 5. Among these three
architectures, CNN1 distinguishes itself with its lower com-
plexity, resulting in faster training times compared to the
others. While increasing model complexity can enhance per-
FIGURE 10. Complex CNN architectures’ validation accuracies with
formance in deep learning-based problems, it is a judgment
100 epochs. that doesn’t hold universally true and may not apply in all
cases.
In the testing process, the performance metrics are detailed Interestingly, among the tested complex architectures,
as in Table 8, where the batch size was consistently set to CNN1, with its lower complexity, yielded the most success-
7,000 for all tests in this scenario. CNN1 is designed with ful results.In a test conducted on the big_dataset, both the
lower complexity, featuring a 7-layer architecture. validation and test accuracies achieved an impressive value
of 98.74%. This indicates the system’s robust performance
TABLE 8. Performance Metrics for Complex CNN Architectures with and reliability in detecting phishing attacks. Moreover, dur-
20 epochs.
ing the tests conducted on complex CNN architectures with
the smaller dataset, a test accuracy of 98.0% was obtained.
However, when using the big_dataset, the test accuracy rose
to an even higher 98.74%. This demonstrates the significant
positive impact of utilizing a larger dataset on the system’s
overall performance.
Precision, recall, and F1-score are essential metrics for
evaluating performance in classification problems. These
The architecture that emerged as the most successful in metrics are provided in Table 9 and Table 10, differentiating
both the validation and test sets was CNN1. Notably, this between phishing and legitimate classes, offering a compre-
architecture also proved to be the fastest one when compared hensive view of the system’s performance.
to the other architectures. In the complex scenario, the CNN Accuracy values are changed depending on the iteration
architecture was identified as the most successful among the (epoch) numbers and it was shown in Fig. 11. These findings
tests conducted over 20 epochs. The accuracy rates of the underscore the system’s ability to distinguish between the
CNN architecture in the complex scenario were 97.6% in two classes with a high degree of accuracy and minimal
the validation set and 97.5% in the test set. It’s important confusion.
to note that the CNN architecture used in the complex sce- For measuring the performance of the system GPU perfor-
nario is the same as the CNN1 architecture employed in mance and its resource usage is also critical metric. The GPU

VOLUME 12, 2024 8065

 O. K. Sahingoz et al.: DEPHIDES: Deep Learning Based Phishing Detection System

TABLE 10. Performance metrics of CNN algorithms with 20 epochs. deep learning architectures, is increasingly being adopted for
use with traditional machine learning algorithms.

TABLE 11. Resource utilization of GPUs.

In n-gram approaches, working with sparse data is essen-

tial, while embeddings make use of dense vectors with lower
dimensions. In the domain of natural language processing,
word embeddings are widely recognized and employed in
literature as an effective technique. This adaptability allows
for a more comprehensive evaluation of the dataset using both
traditional and deep learning-based methods.
FIGURE 11. Accuracy values of CNN1 architecture for big_dataset with
In this article, character embedding is employed instead
100 epochs. of word embedding because the focus of the study is on
vectorizing URLs using character-based operations. While
pre-trained word embeddings are commonly available and
usage details for the test conducted on the big_dataset can be used in literature, character-based embeddings are less fre-
found in Table 11. quently shared. Bypassing an AI model is straightforward by
The test conducted on the big_dataset is optimized to directly employing a predetermined word list. To safeguard
nearly full capacity, with the total running time for the test the system from such factors, it was designed using character
amounting to 188 minutes, encompassing preprocessing and embedding.
test durations. This demonstrates efficient processing for a This article specifically utilizes an embedding size of 50,
comprehensive dataset. as enlarging the embedding size increases the number of
Speed is indeed a critical factor for real-time protection parameters to be trained. Therefore, the experiments did not
systems. In the test, which involved 515,080 URLs from explore varying embedding sizes, and the article focuses on
the big_dataset, the running time was a mere 3.96 sec- the chosen embedding size of 50.
onds. This signifies that the developed system can clas- In this approach, each character is represented by a vector
sify a single URL in a matter of seconds. In practical of size (1, 50), and character embeddings are learned during
terms, it can classify a remarkable 130,070 URLs in just the training process. These character embeddings are drawn
one second, attesting to its capability for rapid, real-time from the most successful deep learning architecture applied
protection. to the small_dataset, which is CNN1 trained for 100 epochs.
This architecture was selected to utilize character embed-
D. TESTS WITH TRADITIONAL MACHINE LEARNING dings.
ALGORITHMS The deep learning architectures investigated in this arti-
While the primary focus of this study revolved around the cle are designed to work with sequence data. To introduce
development of a deep learning-based phishing URL detec- sequence data to machine learning algorithms using character
tion system, the collected dataset was also subjected to tests embeddings, the embedding vectors for each character are
using traditional algorithms, and the accuracy values were employed. This allows for the effective handling and process-
compared. These tests involving traditional machine learning ing of sequence data within the chosen architectures.
methods were conducted using the small dataset. Traditional algorithms were tested using two different vec-
To ensure a fair and meaningful comparison between the torization methods, with the first one being concatenation.
tests, it’s essential to conduct them under conditions where In this vectorization method, designed to match the max-
only the algorithms are altered as consistently as possible. imum character length used in deep learning architectures
Traditional machine learning algorithms have, until recently, (which was set to 200 characters), 50-dimensional vectors for
relied on vectorization methods like n-grams for processing each character were concatenated. This resulted in generating
textual data. However, embedding, which can be derived from 10,000 (200 × 50) dimension vectors for each data sample.

8066 VOLUME 12, 2024

O. K. Sahingoz et al.: DEPHIDES: Deep Learning Based Phishing Detection System

The generated vectors were then subjected to testing using low-level features. It’s speculated that traditional machine
three different traditional machine learning algorithms. One learning algorithms, which tend to emphasize high-level fea-
of the primary objectives of testing these traditional algo- tures, could potentially yield improved results if they could
rithms was to highlight the differences between the training utilize word-based features. However, such an investigation
processes employed in deep learning approaches and to eval- is beyond the scope of this study.
uate the accuracy rates achieved by deep learning methods
when applied to the same problem. This comparison allowed E. EVALUATION CRITERIA
for an assessment of the necessity of employing deep learning The system developed in this article will analyze whether a
techniques for solving the phishing detection problem, ulti- URL is phishing or legitimate. In the case of binary classi-
mately demonstrating that deep learning architectures were fication problems, four different situations may occur in the
more effective in addressing this issue. test phase. These are the following:
The three algorithms tested were Random Forest, Logistic • False Negative (FN): A condition in which a harmful
Regression, and Naive Bayes, which are statistics-based algo- URL is classified as clean.
rithms. The small dataset used in these experiments contained • False Positive (FP): This is the case where the URL is
364,199 training samples, 104,576 validation samples, and reported as harmful even if it is not harmful. In this
51,510 test samples. The performance values obtained on case, the system does not allow this access when the user
the small dataset using these traditional machine learning wants to access a legitimate domain.
algorithms are presented in Table 12 on the validation set • True Negative (TN): A clean condition of clean URLs.
using the 50 and 10,000-dimensional vectors. The second • True Positive (TP): This is where harmful URLs are
vectorization method is tested with traditional algorithms classified as harmful.
involving character embeddings.
TP
Precision = (1)
TABLE 12. Performance metrics FOR 50/10000 dimensional vectors.
TP + FP
TP
Recall = (2)
TP + FN
Precision × Recall
F − Measure = 2 × (3)
Precision + Recall
TP + TN
Accuracy = (4)
TP + TN + FN + FP
In case of a successful binary classification problem, it is
expected that the number of samples related to TP and TN
status will be higher, and the number of samples related to
FP and FN status will be lower.
The metrics used for the evaluation of the test results are
as follows: precision, recall, f-measure, and accuracy. These
statistics, whose formulation is depicted in Equations (1)–
In this method, each URL is represented by a 50 and (4), are also important for making a comparison between the
10,000-dimensional vectors calculated as the average of all tested machine learning approaches.
character embeddings within the URL.
The average values are taken at the same index number for VI. DISCUSSION
each character embedding. For instance, the first value of the In this research paper, the primary objective is to create
output vector is computed as the average of the first values a high-performing phishing detection system using Deep
of all character embeddings. According to the results, the Learning techniques. To achieve this goal, a substantial
logistic regression algorithm was the most successful with the amount of data is required for training and testing the detec-
vectorization created by concatenation, achieving a validation tion system effectively. Within the context of this study,
accuracy of 0.938. On the other hand, the Random Forest a significant dataset was collected and made publicly avail-
algorithm was the most successful with a validation accuracy able for the specific purpose of detecting phishing attacks.
of 0.877. Notably, this dataset is one of the largest known datasets for
The highest accuracy rate observed in the tests using deep phishing detection in academic literature.
learning was 0.987. The results of traditional machine learn- The dataset comprises a total of 2,320,893 URLs associ-
ing algorithms were somewhat less successful than those of ated with phishing attacks and 2,881,948 URLs correspond-
deep learning approaches. It’s worth noting that this article ing to legitimate websites. Such a large and balanced dataset
primarily employed character-based vectorization, omitting is crucial for developing and evaluating machine learning
word-based features from URLs, and thus, focusing on models capable of accurately distinguishing between phish-
low-level features. Deep learning approaches have proven ing URLs and legitimate ones. This rich dataset is expected to
more successful in detecting phishing URLs when handling contribute to the development of robust and high-performing

VOLUME 12, 2024 8067

 O. K. Sahingoz et al.: DEPHIDES: Deep Learning Based Phishing Detection System

phishing detection algorithms. The evaluation of the dataset B. HUGE SIZE OF PHISHING AND LEGITIMATE DATA
involved the utilization of deep learning algorithms, which Constructing a dataset for an anti-phishing system is often
are known for their capability to handle extensive datasets. a challenging task. To achieve high accuracy rates with
Given the substantial volume of data, significant processing machine learning algorithms, a substantial amount of training
power is required to perform the necessary computations data is crucial.
effectively.
For the convenience of conducting experiments within this C. REAL-TIME EXECUTION
study and to facilitate further research in the field, two distinct As phishing attackers can quickly create fraudulent web
versions of the dataset were made available. The first version pages that are active for short durations, real-time detection
is a sub-sample of the dataset, comprising 10% of the com- is essential for effective prevention. The proposed system can
plete dataset, and is referred to as the ‘‘small_dataset.’’ The classify over 130,070 URLs per second, making it suitable
second version, termed the ‘‘big_dataset,’’ encompasses the for deployment in high-traffic environments. Furthermore,
entire dataset, providing researchers with a comprehensive this processing rate can be further improved by leveraging
and larger dataset to work with. These two dataset versions powerful GPUs, enhancing real-time protection capabilities.
are designed to accommodate different research needs and the
computational resources available to researchers.
D. DETECTION OF NEW WEBSITES
Deep learning algorithms are known for their reliance
The proposed system can identify new phishing websites that
on numerous hyperparameters, which are essential for
haven’t been previously classified as phishing. This capa-
fine-tuning and optimizing the performance of these models.
bility makes the system resilient against zero-day attacks,
In this study, some of the hyperparameters were adjusted
a particularly dangerous type of phishing attack. By detecting
while others were maintained at constant values. The exper-
threats at the initial stages of a URL, the system significantly
imentation process involved the use of an Nvidia Tesla
reduces the potential impact of such attacks.
V100 GPU, which was rented for the purpose of con-
ducting the tests. To identify suitable hyperparameters and
architectural configurations, initial tests were carried out E. INDEPENDENCE FROM THIRD-PARTY SERVICES
on the ‘‘small_dataset.’’ Once the optimal configuration While many works in the literature rely on third-party ser-
was determined for this dataset, it was subsequently eval- vices like ‘‘whois’’ records, web-based blacklists/whitelists,
uated on the more extensive ‘‘big_dataset.’’ Among the and network traffic measures to enhance detection and pre-
various deep learning architectures that were tested, the vention, these services can introduce delays in real-time
Convolutional Neural Network (CNN) demonstrated the execution. The proposed system operates independently of
highest level of success, achieving an impressive accuracy such third-party services, ensuring efficient and rapid phish-
rate of 98.74%. ing detection, especially in high-traffic settings.
The developed system exhibits an impressive ability to
classify over 130,000 URLs per second, and this perfor- VII. CONCLUSION AND FUTURE WORKS
mance can be further improved by utilizing more powerful The widespread use of the Internet has increased the impor-
processing units. In addition to exploring deep learning tance of the security of assets on the Internet. Phishing
architectures, traditional machine learning algorithms, such attacks are one type of cyber-attack that threatens users
as Naïve Bayes, Random Forest, and Logistic Regression, today. This type of attack is intended to exploit people’s
were also evaluated. Two different vectorization methods weaknesses. In this article, a deep learning-based system for
that utilized character embeddings were employed in these detecting fishing attacks was proposed, and five different
assessments. deep learning architectures were tested. These architectures
Among the traditional algorithms tested, the logistic are recurrent neural networks, bi-directional neural networks,
regression algorithm emerged as the most effective, achieving convolutional neural networks, artificial neural networks, and
a notable validation accuracy of 93.8% in our experiments. attention-based networks.
The proposed deep learning model on CNN structure can The study revealed that the results obtained from
reach 98.7% accuracy value system which offers several sig- deep learning approaches outperformed those of traditional
nificant advantages, as outlined below: machine learning algorithms. The primary focus of this
research was the development of a system for phishing
A. LANGUAGE INDEPENDENCE attack detection from URLs using deep learning meth-
Many phishing detection systems are heavily dependent ods. To facilitate transparency and access to the findings,
on language-specific features, making them less versatile. all application codes and datasets pertaining to the deep
In contrast, the proposed system can detect phishing using learning tests conducted in this study have been shared
URL information alone, focusing solely on the characters through Github, IEEE DataPort and Code Ocean [38], [39],
within the URL. This word-independent approach enables the [40]. These shared resources include comprehensive infor-
system to detect phishing attacks regardless of the language mation about all tests, encompassing accuracy and loss
used. plots, confusion matrices, GPU utilization data, model and

8068 VOLUME 12, 2024

O. K. Sahingoz et al.: DEPHIDES: Deep Learning Based Phishing Detection System

weight files, precision, recall, f1-score metrics, and execution [18] P. A. Watters, A. Herps, R. Layton, and S. McCombie, ‘‘ICANN or
times. This commitment to providing detailed information ICANT: Is WHOIS an enabler of cybercrime?’’ in Proc. 4th Cybercrime
Trustworthy Comput. Workshop, Nov. 2013, pp. 44–49, doi: 10.1109/
ensures reproducibility and thorough analysis of the study’s CTC.2013.13.
results. [19] H. Kim and J. H. Huh, ‘‘Detecting DNS-poisoning-based phishing attacks
from their network performance characteristics,’’ Electron. Lett., vol. 47,
no. 11, p. 656, 2011, doi: 10.1049/el.2011.0399.
ACKNOWLEDGMENT [20] A. Karim, M. Shahroz, K. Mustofa, S. B. Belhaouari, and S. R. K. Joga,
The authors would like to thank Roksit for their support ‘‘Phishing detection system through hybrid machine learning based
during the implementation of this work. on URL,’’ IEEE Access, vol. 11, pp. 36805–36822, 2023, doi:
10.1109/ACCESS.2023.3252366.
[21] M. K. Prabakaran, P. Meenakshi Sundaram, and A. D. Chandrasekar,
REFERENCES ‘‘An enhanced deep learning-based phishing detection mechanism to
effectively identify malicious URLs using variational autoencoders,’’
[1] Anti-Phishing Working Group. (Sep. 2022). Phishing Attacks Trends
IET Inf. Secur., vol. 17, no. 3, pp. 423–440, May 2023, doi: 10.1049/
Report-Q2 2022. Accessed: Oct. 15, 2022. [Online]. Available: https://
ise2.12106.
apwg.org/trendsreports/
[22] Z. Alshingiti, R. Alaqel, J. Al-Muhtadi, Q. E. U. Haq,
[2] Cloudflare’s 2023 Phishing Threats Report. Accessed: Oct. 1, 2023.
K. Saleem, and M. H. Faheem, ‘‘A deep learning-based phishing
[Online]. Available: https://www.cloudflare.com/lp/2023-phishing-report/
detection system using CNN, LSTM, and LSTM-CNN,’’
[3] M. Volkamer, K. Renaud, B. Reinheimer, and A. Kunz, ‘‘User expe- Electronics, vol. 12, no. 1, p. 232, Jan. 2023, doi: 10.3390/
riences of TORPEDO: Tooltip-powered phishing email detection,’’ electronics12010232.
Comput. Secur., vol. 71, pp. 100–113, Nov. 2017, doi: 10.1016/j.cose. [23] M. Sánchez-Paniagua, E. F. Fernández, E. Alegre, W. Al-Nabki, and
2017.02.004. V. González-Castro, ‘‘Phishing URL detection: A real-case scenario
[4] N. Q. Do, A. Selamat, O. Krejcar, E. Herrera-Viedma, and H. Fujita, through login URLs,’’ IEEE Access, vol. 10, pp. 42949–42960, 2022, doi:
‘‘Deep learning for phishing detection: Taxonomy, current challenges and 10.1109/ACCESS.2022.3168681.
future directions,’’ IEEE Access, vol. 10, pp. 36429–36463, 2022, doi: [24] S. He, B. Li, H. Peng, J. Xin, and E. Zhang, ‘‘An effective cost-
10.1109/ACCESS.2022.3151903. sensitive XGBoost method for malicious URLs detection in imbal-
[5] T. Mahara, V. L. H. Josephine, R. Srinivasan, P. Prakash, A. D. Algarni, anced dataset,’’ IEEE Access, vol. 9, pp. 93089–93096, 2021, doi:
and O. P. Verma, ‘‘Deep vs. shallow: A comparative study of 10.1109/ACCESS.2021.3093094.
machine learning and deep learning approaches for fake health [25] S. Tiwari, H. Rizvi, and K. Kalaiselvi, ‘‘Malicious website naviga-
news detection,’’ IEEE Access, vol. 11, pp. 79330–79340, 2023, doi: tion prevention using CNNs and URL vectors: A study,’’ in Proc. Int.
10.1109/ACCESS.2023.3298441. Conf. Comput. Commun. Informat. (ICCCI), Jan. 2022, pp. 1–6, doi:
[6] Google Safe Browsing. Accessed: Oct. 1, 2023. [Online]. Available: 10.1109/ICCCI54379.2022.9741056.
https://safebrowsing.google.com/ [26] T. Rasymas and L. Dovydaitis, ‘‘Detection of phishing URLs by
[7] (2019). Office 365 Advanced Threat Protection Safe Links. Accessed: using deep learning approach and multiple features combinations,’’
Jul. 10, 2023. [Online]. Available: https://docs.microsoft.com/en- Baltic J. Modern Comput., vol. 8, no. 3, pp. 471–483, Sep. 2020, doi:
us/office365/securitycompliance/atp-safe-links 10.22364/bjmc.2020.8.3.06.
[8] A. K. Jain and B. B. Gupta, ‘‘A novel approach to protect against phish- [27] T. Feng and C. Yue, ‘‘Visualizing and interpreting RNN models in
ing attacks at client side using auto-updated white-list,’’ EURASIP J. URL-based phishing detection,’’ in Proc. 25th ACM Symp. Access
Inf. Secur., vol. 2016, no. 1, pp. 1–11, Dec. 2016, doi: 10.1186/s13635- Control Models Technol., Jun. 2020, pp. 13–24, doi: 10.1145/3381991.
016-0034-3. 3395602.
[9] N. A. Azeez, S. Misra, I. A. Margaret, L. Fernandez-Sanz, and [28] A. Aljofey, Q. Jiang, Q. Qu, M. Huang, and J.-P. Niyigena, ‘‘An effective
S. M. Abdulhamid, ‘‘Adopting automated whitelist approach for detecting phishing detection model based on character level convolutional neural
phishing attacks,’’ Comput. Secur., vol. 108, Sep. 2021, Art. no. 102328, network from URL,’’ Electronics, vol. 9, no. 9, p. 1514, Sep. 2020, doi:
doi: 10.1016/j.cose.2021.102328. 10.3390/electronics9091514.
[10] N. Abdelhamid, A. Ayesh, and F. Thabtah, ‘‘Phishing detection based [29] Y. Huang, Q. Yang, J. Qin, and W. Wen, ‘‘Phishing URL detec-
associative classification data mining,’’ Expert Syst. Appl., vol. 41, no. 13, tion via CNN and attention-based hierarchical RNN,’’ in Proc. 18th
pp. 5948–5959, Oct. 2014, doi: 10.1016/j.eswa.2014.03.019. IEEE Int. Conf. Trust, Secur. Privacy Comput. Commun./13th IEEE Int.
[11] M. Moghimi and A. Y. Varjani, ‘‘New rule-based phishing detection Conf. Big Data Sci. Eng., Aug. 2019, pp. 112–119, doi: 10.1109/Trust-
method,’’ Expert Syst. Appl., vol. 53, pp. 231–242, Jul. 2016, doi: Com/BIGDATASE.2019.00024.
10.1016/j.eswa.2016.01.028. [30] O. K. Sahingoz, E. Buber, O. Demir, and B. Diri, ‘‘Machine learning based
[12] M. SatheeshKumar, K. G. Srinivasagan, and G. UnniKrishnan, phishing detection from URLs,’’ Expert Syst. Appl., vol. 117, pp. 345–357,
‘‘A lightweight and proactive rule-based incremental construction Mar. 2019, doi: 10.1016/j.eswa.2018.09.029.
approach to detect phishing scam,’’ Inf. Technol. Manage., [31] S. Singh, M. P. Singh, and R. Pandey, ‘‘Phishing detection from
vol. 23, no. 4, pp. 271–298, Dec. 2022, doi: 10.1007/s10799- URLs using deep learning approach,’’ in Proc. 5th Int. Conf.
021-00351-7. Comput., Commun. Secur. (ICCCS), Oct. 2020, pp. 1–4, doi:
[13] A. K. Jain and B. B. Gupta, ‘‘Phishing detection: Analysis of visual 10.1109/ICCCS49678.2020.9277459.
similarity based approaches,’’ Secur. Commun. Netw., vol. 2017, pp. 1–20, [32] L. Tang and Q. H. Mahmoud, ‘‘A deep learning-based framework for
Oct. 2017, doi: 10.1155/2017/5421046. phishing website detection,’’ IEEE Access, vol. 10, pp. 1509–1521, 2022,
[14] E. Medvet, E. Kirda, and C. Kruegel, ‘‘Visual-similarity-based phishing doi: 10.1109/ACCESS.2021.3137636.
detection,’’ in Proc. 4th Int. Conf. Secur. Privacy Commun. Netowrk, [33] J. Anitha and M. Kalaiarasu, ‘‘A new hybrid deep learning-based
Sep. 2008, pp. 1–6, doi: 10.1145/1460877.1460905. phishing detection system using MCS-DNN classifier,’’ Neural Comput.
[15] W. Liu, X. Deng, G. Huang, and A. Y. Fu, ‘‘An antiphishing strategy based Appl., vol. 34, no. 8, pp. 5867–5882, Apr. 2022, doi: 10.1007/s00521-
on visual similarity assessment,’’ IEEE Internet Comput., vol. 10, no. 2, 021-06717-w.
pp. 58–65, Mar. 2006, doi: 10.1109/MIC.2006.23. [34] S. Das Guptta, K. T. Shahriar, H. Alqahtani, D. Alsalman, and
[16] Y. Zhou, Y. Zhang, J. Xiao, Y. Wang, and W. Lin, ‘‘Visual similarity based I. H. Sarker, ‘‘Modeling hybrid feature-based phishing websites detection
anti-phishing with the combination of local and global features,’’ in Proc. using machine learning techniques,’’ Ann. Data Sci., vol. 2022, pp. 1–26,
IEEE 13th Int. Conf. Trust, Secur. Privacy Comput. Commun., Sep. 2014, Mar. 2022, doi: 10.1007/s40745-022-00379-8.
pp. 189–196, doi: 10.1109/TRUSTCOM.2014.28. [35] M. Alshehri, A. Abugabah, A. Algarni, and S. Almotairi, ‘‘Character-
[17] G. Varshney, M. Misra, and P. K. Atrey, ‘‘Improving the accuracy of search level word encoding deep learning model for combating cyber
engine based anti-phishing solutions using lightweight features,’’ in Proc. threats in phishing URL detection,’’ Comput. Electr. Eng.,
11th Int. Conf. Internet Technol. Secured Trans. (ICITST), Dec. 2016, vol. 100, Mar. 2022, Art. no. 107868, doi: 10.1016/j.compeleceng.
pp. 365–370, doi: 10.1109/ICITST.2016.7856731. 2022.107868.

VOLUME 12, 2024 8069

 O. K. Sahingoz et al.: DEPHIDES: Deep Learning Based Phishing Detection System

[36] A. Mughaid, S. AlZu’bi, A. Hnaif, S. Taamneh, A. Alnajjar, and EBUBEKIR BUBER received the bachelor’s, grad-
E. A. Elsoud, ‘‘An intelligent cyber security phishing detection sys- uate, and Ph.D. degrees in artificial intelligence,
tem using deep learning techniques,’’ Cluster Comput., vol. 25, no. 6, and the master’s degree in cyber security.
pp. 3819–3828, Dec. 2022, doi: 10.1007/s10586-022-03604-4. He was the Manager in projects involving arti-
[37] Z.-H. Zhan, J.-Y. Li, and J. Zhang, ‘‘Evolutionary deep learning: ficial intelligence in the field of cyber security
A survey,’’ Neurocomputing, vol. 483, pp. 42–58, Apr. 2022, doi: during his business experience. To develop more
10.1016/j.neucom.2022.01.099. efficient projects in this field, he received the mas-
[38] Github Resources. Accessed: Dec. 3, 2023. [Online]. Available: ter’s degree. For over seven years, he has been
https://github.com/ebubekirbbr/phishing_url_detection
working on artificial intelligence projects in the
[39] Deep Learning Based Phishing Detection System (DEPHIDES).
field of cyber security. He has also created several
Accessed: Dec. 3, 2023. [Online]. Available: https://codeocean.com/
capsule/0874584/tree academic publications on AI and cyber security.
[40] Phishing Attack Dataset. Accessed: Dec. 3, 2023. [Online]. Available:
https://dx.doi.org/10.21227/4098-8c60

EMIN KUGU (Member, IEEE) received the B.S.

degree from the Computer Science Engineer-
ing Department, Istanbul University, the master’s
OZGUR KORAY SAHINGOZ received the B.Sc. degree in software engineering from Air Force
degree from the Computer Engineering Depart- Academy, and the Ph.D. degree in electrical and
ment, Boğaziçi University, in 1993, and the computer engineering program from Old Domin-
M.S. and Ph.D. degrees from the Computer ion University. He was a part-time Lecturer with
Engineering Department, Istanbul Technical Uni- different universities. He was the Project Manager
versity, in 1998 and 2006, respectively. He is in the Air Force Command Information System
currently a Professor with the Computer Engineer- (HvBS) Project and took part in the software
ing Department, Biruni University, İstanbul. His development processes of international defense projects. His research inter-
research interests include artificial intelligence, ests include cyber security, computer networks, artificial intelligence, and
machine/deep learning, data science, and UAV machine/deep learning.
networking.

8070 VOLUME 12, 2024