Vulnerability-oriented Testing for RESTful APIs

Wenlong Du and Jian Li, School of Electronic Information and Electrical Engineering,
Shanghai Jiao Tong University; Yanhao Wang, Independent Researcher; Libo Chen,
Ruijie Zhao, and Junmin Zhu, School of Electronic Information and Electrical Engineering,
Shanghai Jiao Tong University; Zhengguang Han, QI-ANXIN Technology Group;
Yijun Wang and Zhi Xue, School of Electronic Information and Electrical Engineering,
Shanghai Jiao Tong University
https://www.usenix.org/conference/usenixsecurity24/presentation/du

This paper is included in the Proceedings of the

33rd USENIX Security Symposium.
August 14–16, 2024 • Philadelphia, PA, USA
978-1-939133-44-1

Open access to the Proceedings of the

33rd USENIX Security Symposium
is sponsored by USENIX.
 Vulnerability-oriented Testing for RESTful APIs

Wenlong Du*1 , Jian Li*1 , Yanhao Wang2 , Libo Chen B1 , Ruijie Zhao1 ,
Junmin Zhu1 , Zhengguang Han3 , Yijun Wang1 , and Zhi Xue1
1 School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University
2 Independent Researcher 3 QI-ANXIN Technology Group

{beet1e, vfenux483, bob777, ruijiezhao, junmin.zhu, ericwyj, zxue}@sjtu.edu.cn,

wangyanhao136@gmail.com, hanzhengguang@qianxin.com

Abstract API development, REST [9] (REpresentational State Trans-

With the increasing popularity of APIs, ensuring their secu- fer) has emerged as a widely embraced approach, with APIs
rity has become a crucial concern. However, existing security adhering to this style commonly known as RESTful APIs.
testing methods for RESTful APIs usually lack targeted ap- Unfortunately, numerous security issues [10, 11] have
proaches to identify and detect security vulnerabilities. In arisen in APIs in recent years, resulting in significant repercus-
this paper, we propose VOAPI2 , a vulnerability-oriented API sions for providers and users. For instance, a vulnerability [10]
inspection framework designed to directly expose vulnera- in a Facebook API interface exposed millions of users’ private
bilities in RESTful APIs, based on our observation that the information. As we see, API security has become a critical
type of vulnerability hidden in an API interface is strongly concern, especially for service providers. Meanwhile, APIs
associated with its functionality. By leveraging this insight, have increasingly become a focal point for attackers, making
we first track commonly used strings as keywords to iden- them a common target in various services. Hence, proactively
tify APIs’ functionality. Then, we generate a stateful and identifying and addressing API flaws is vital in safeguarding
suitable request sequence to inspect the candidate API func- services and their users. Detecting vulnerabilities early on is
tion within a targeted payload. Finally, we verify whether an essential strategy to ensure the protection of these systems.
vulnerabilities exist or not through feedback-based testing. Currently, API testing has gained significant popularity
Our experiments on real-world APIs demonstrate the effec- as a solution to enhance API security, and a considerable
tiveness of our approach, with significant improvements in amount of research has been conducted in this field. However,
vulnerability detection compared to state-of-the-art methods. existing testing methods for RESTful APIs often encounter
VOAPI2 discovered 7 zero-day and 19 disclosed bugs on limitations when it comes to identifying security vulnerabili-
seven real-world RESTful APIs, and 23 of them have been ties, especially in black-box testing scenarios [12–17]. These
assigned CVE IDs. Our findings highlight the importance of methods heavily rely on the API’s status feedback, which
considering APIs’ functionality when discovering their bugs, means they may not detect certain logical vulnerabilities (e.g.,
and our method provides a practical and efficient solution for SSRF) that do not affect the normal functioning of APIs. On
securing RESTful APIs. the other hand, white-box testing methods [17–21] require
source code security audits, which can be time-consuming,
inefficient, and prone to generating false positives. Gray-box
1 Introduction testing methods [22, 23] require access to the internal state of
API execution, offering a more comprehensive view. Never-
Application Programming Interfaces (APIs), as the contract theless, the diverse code frameworks used in API implemen-
between information providers and information users, are tations present a challenge in developing a unified detection
gaining immense popularity as they effectively facilitate com- approach. Furthermore, testing methods tailored to specific ar-
munication and data exchange between diverse web appli- chitectures [24] have limited scope and are difficult to extend
cations. It also enables businesses to harness their data and beyond their original application scenarios.
services well to support innovative scenarios and broaden In addition, several other methods [12, 15, 16, 25–32] have
their impact. Currently, it has been widely used and spanned been proposed that leverage OpenAPI1 specifications to en-
across various scenarios, such as cloud services [1, 2], content
management systems (CMS) [3–6], and Internet of Things 1 OpenAPI [33] (i.e., Swagger) stands as the most widely used specifica-

(IoT) [7, 8] devices. Among different architectural styles for tion for describing RESTful API interfaces. A Swagger-based specification
outlines the usage guidelines for RESTful APIs, including the types of service
* Co-leading authors. B Corresponding author. requests accepted, the expected response format, and other relevant details.

USENIX Association 33rd USENIX Security Symposium 739

able unified testing of APIs. These methods include functional request sequence that aligns with the execution context of
testing [12, 15, 16, 25–29, 34], compliance testing [30], and the corresponding functions. By utilizing inspection payloads
security testing [31, 32]. Parts of these approaches [26–29] tailored to different vulnerability types, we can effectively
utilize OpenAPI specifications to parse HTTP requests and assess these candidate functions for potential vulnerabilities.
generate fuzzing test cases that adhere to syntax rules. Con- We evaluated our tool on 7 real-world RESTful APIs, which
versely, some methods [12, 15, 16, 25, 34] use OpenAPI spec- resulted in the discovery of 7 zero-day vulnerabilities and 19
ifications to infer the dependencies between requests and disclosed bugs. Out of these vulnerabilities, 23 have been ver-
generate test messages that align with the API interaction ified and acknowledged by the respective vendors, and have
context. This enables effective testing of API functionality, as been assigned CVE IDs. The experiment results demonstrate
well as the identification of exceptions and specific security that VOAPI2 surpasses the state-of-the-art methods by a sig-
vulnerabilities. nificant margin, not only in terms of the number of identified
Despite the emergence of various testing approaches, it bugs but also in terms of detection efficiency.
is noteworthy that most of these methods do not effectively In summary, we make the following contributions:
utilize the specific characteristics of security vulnerabilities
to implement vulnerability-targeted testing strategies. Instead, • We propose and implement a novel method, VOAPI2 , to
they primarily rely on the detection of “500 Internal Server Er- automatically reveal the vulnerability of RESTful APIs. It
rors” as an indication of capturing bugs within API interfaces. can perform vulnerability-oriented testing based on API
Consequently, this approach leads to limited visibility and low specifications to effectively capture defects in RESTful
testing efficiency. For example, when dealing with a large- APIs, and output the corresponding PoC. We will release
scale API project similar in scale to GitLab, RESTler [12], on the source code as well as the experiment data at https:
average, requires more than 5 hours to perform a comprehen- //github.com/NSSL-SJTU/VoAPI2.
sive test. Furthermore, the security vulnerabilities uncovered • We evaluated our tool with several state-of-the-art methods.
by these tools, such as RestTestGen [16], represent only a The experiment results demonstrate that VOAPI2 surpasses
small fraction of these bugs, with the majority being func- all these methods by a significant margin, not only in terms
tional issues that primarily impact the reliability of APIs. of the number of identified bugs but also in terms of detec-
Based on our analysis of publicly disclosed API vulnera- tion efficiency.
bilities in recent years, we observed that there exists a clear
correlation between different API security vulnerabilities and • We conducted an evaluation of VOAPI2 on 7 real-world
the functionality of APIs. For instance, file upload interfaces RESTful APIs, and VOAPI2 identified 26 vulnerabilities,
can pose a risk of malware uploads, while proxy interfaces including 7 previously-unknown and 19 known security
may harbor SSRF vulnerabilities. Building upon this obser- bugs. All zero-day bugs have been reported to vendors and
vation, we propose the core concept of our method, which fixed; four of them have been assigned CVE IDs.
involves identifying API functions associated with vulnera-
bilities within API specifications and conducting targeted 2 Problem and Approach Overview
security testing on those functions. This approach aims to
achieve vulnerability-oriented API inspection, thereby fa- In this section, we first provide the background of vulnerabili-
cilitating the effective discovery of bugs across a wide range ties in RESTful API. Then, we present a motivation example,
of APIs. and our method to detect the hidden vulnerability in the sam-
There are three challenges in realizing vulnerability- ple. At last, we discuss the associated challenges and present
oriented API inspection in practice. The first challenge is effi- an overview of our solution.
ciently differentiating between various functional interfaces
in an API and identifying functions that may have security
vulnerabilities. The second challenge is effectively generating
2.1 Vulnerability of RESTful API
valid test cases based on different function interfaces and cor- REST, short for REpresentational State Transfer, is a widely
responding security vulnerability types. The last challenge is adopted software architecture style for network applications.
generating test sequences that comply with protocol states, in- It is commonly utilized within the context of HTTP, and an
corporating the related test cases, and successfully executing API adhering to this style is referred to as a RESTful API.
the function interfaces. While existing tools are available for testing REST APIs,
To address these challenges, we propose VOAPI2 , a many of them primarily focus on assessing the reliability
Vulnerability-Oriented API Inspection framework designed of these APIs rather than specifically targeting potential se-
to directly expose vulnerabilities in RESTful APIs. Our ap- curity vulnerabilities. However, similar to web applications,
proach begins by identifying semantic keywords associated REST APIs are also vulnerable to various security threats, as
with potentially weak functions within the API specification. demonstrated by the presence of such vulnerabilities in the
It then integrates a suitable corpus and employs a stateful CVE database. These vulnerabilities encompass issues like

740 33rd USENIX Security Symposium USENIX Association

1 /Images/Remote:
2 get: /Images/Ratings
3 operationId: GetRemoteImage
/Images/Remote
4 parameters:
5 description: The image url. /Items
6 in: query
/Items/Filter
7 name: imageUrl
8 required: true …
9 schema:
REST APIs
10 description: The image url. Attacker Internal Service
11 format: uri
12 type: string
13 responses: Figure 2: Exploitation process of CVE-2021-29490.
14 "200":
15 content:
16 image/*: (Line 26) that offers category tags to classify the API.
17 schema:
18 format: binary In Jellyfin (version 10.7.3 and earlier), there exists a
19 type: string Server-Side Request Forgery (SSRF) vulnerability (CVE-
20 description: Remote image returned. 2021-29490 [35]) in the aforementioned API endpoint. Ex-
21 "404": ploiting this vulnerability requires the attacker to send a GET
22 content:
23 # omitted request to the /Images/Remote API with testing payloads (e.g.,
24 description: Remote image not found. http://LAN_IP/ssrf) in the query parameter “imageUrl”.
25 summary: Gets a remote image. The root cause of this vulnerability is the lack of filtering
26 tags:
for the remote resource address specified by the “imageUrl”
27 - RemoteImage
parameter in the backend of the API service. This oversight
allows attackers to freely access internal resources by sending
Figure 1: Specification of an API endpoint.
an HTTP request (i.e., the testing payload) from the Jellyfin
server, as depicted in Figure 2.
Server-Side Request Forgery, Unrestricted File Upload, OS As we see, the vulnerability in the motivation sample is
Command Injection, Path Traversal, and more. closely related to the functionality of the corresponding API
endpoint. Specifically, the SSRF bug typically arises in the
Motivation Example. OpenAPI, formerly known as Swag-
API path responsible for retrieving a remote resource. As
ger, provides a specification for describing RESTful APIs.
demonstrated in this example, the purpose of this API end-
Typically, the specification document is presented in JSON or
point is to fetch a remote image, as indicated in its summary
YAML format and encompasses the resources and operations
(Line 25). The parameter “imageUrl” is utilized to specify the
of the RESTful API. It comprises a collection of Schema Ob-
address of the resource (Line 5). Consequently, if the back-
jects, which encompass various elements such as API service
end of this API interface fails to rigorously validate input
endpoints (commonly referred to as paths), CRUD operations
parameters, it may result in an SSRF vulnerability.
(such as GET, POST, PUT and DELETE) performed on an endpoint,
input parameters, and expected responses. Each schema object
follows a predefined structure with corresponding parame- 2.2 Observation and Our Method
ter types. Users can leverage the specification to generate
valid API operations and communicate with API services by Current security testing methods for APIs suffer from two
presenting them as HTTP requests. primary efficiency issues and most of them cannot detect the
Figure 1 illustrates a simplified fragment of the OpenAPI vulnerability hidden in the motivation sample. Firstly, they
specification for Jellyfin [3], an open-source media man- are time-consuming. For example, RESTler [12] required over
agement system. This fragment specifies an API endpoint, five hours to test four groups of API interfaces in the open-
/Images/Remote, which supports a single CRUD operation, source code management software GitLab, with the longest ef-
GET. It defines the input parameters and expected response fective request sequence length tested remaining at just three.
for this endpoint. Specifically, the input parameter is named This inefficiency can be attributed to testing tools needing
“imageUrl” (Line 7), which is required and used to specify to traverse all interfaces indiscriminately, rather than focus-
the URL of the image (Line 10). The corresponding schema ing on a specific API interface in-depth. Secondly, existing
(Line 9) designates the parameter type as a string (Line 12). testing tools heavily rely on service status feedback from
The response section includes the HTTP status code and API endpoints to gauge testing effectiveness, resulting in the
response message (Line 13). Furthermore, this example in- identification of mostly service-disrupting bugs. For instance,
cludes a “summary” field (Line 25), which typically provides a RESTler discovered 28 bugs causing GitLab [36] service un-
brief description of the API’s functionality (similar to the availability, while RestCT [25] identified eight similar issues
“description” field). Additionally, there is a “tags” field in Bing Maps [37] and GitLab.

USENIX Association 33rd USENIX Security Symposium 741

Table 1: Statistical Data of Known APIs’ Vulnerabilities. Here we list the disclosed API bugs of six common types that we
collected from the CVE database. #Bug indicates the number of disclosed bugs. #Ratio indicates the proportion of common
functionality in all vulnerable functions.
CWE ID Vulnerability Type #Bug Functionality Summary of Vulnerable APIs #Ratio Keywords
CWE-434 Unrestricted Upload of File 42 Most of them focus on uploading. 83%(35/42) “upload”,“submit”,“import”, etc.
Most of them need to request a remote source,
CWE-918 Server-Side Request Forgery 21 81%(17/21) “remote”,“proxy”,“URL”, etc.
like Proxy interfaces.
They are responsible for handling file through
CWE-22 Path Traversal 31 52%(16/31) “path”,“dir”,“file”, etc.
a "Path" variable.
Most of them are used to set configurations
CWE-78 OS Command Injection 55 40%(22/55) “CMD”,“command”,“system”, etc.
through commands executed in OS shell.
CWE-89 SQL Injection 70 They are responsible for handling SQL database. 53%(37/70) “SQL”,“database”,“select”, etc.
CWE-79 Cross-site Scripting 55 They present front-end pages that show text. 35%(19/55) “display”,“content”,“view”, etc.

Furthermore, these testing methods often lack a targeted by a specific type of bug belonged to the same functional-
strategy and approach based on specific security vulnerabil- ity. These findings confirm the validity of our intuition for
ities. Instead, they primarily rely on detecting generic bugs these common vulnerabilities. Additionally, we identified
through feedback such as “500 Internal Server Errors” from a group of commonly used strings in specific functionality
API endpoints. Consequently, this leads to an indiscriminate as keywords. For example, in an upload interface, we may
and inefficient testing approach, with the majority of detected observe frequently used strings such as “upload”, “submit”,
bugs being functional issues rather than genuine security vul- and “import” which can be used as keywords to identify cor-
nerabilities. responding interfaces from API specifications and support
Observation. According to the motivation sample, we ob- subsequent bug detection, as shown in Table 1. It is also im-
serve that we can assess the functionality of APIs by exam- portant to note that these six types of vulnerabilities are the
ining certain feature strings commonly used to define API mainstream of web vulnerabilities, collectively accounting
features in the specification, such as the presence of remote for over 70% of the total [38].
in the current example. Meanwhile, the keyword remote is Our Method. Building upon our observation and verifica-
also strongly associated with the type of vulnerability (SSRF). tion result, we propose the core concept of our method, which
To validate our observation that the category of vulnerability involves identifying API functions associated with vulnera-
present in an API interface is strongly linked to the func- bilities within API specifications and conducting targeted
tionality of that interface via some keywords, we gathered security testing on those functions. This approach aims to
data from the CVE database on the most common types of achieve vulnerability-oriented API inspection, thereby facili-
revealed API vulnerabilities (e.g., Unrestricted Upload and tating the effective discovery of bugs across a wide range of
SSRF) and selected vulnerabilities that had been disclosed in RESTful APIs. Different from the previous method, it con-
CVE records with specific details, such as the exact location siders the API functionality when identifying and testing for
of the bug within the API interface. vulnerabilities.
Based on our collected data2 , we first conduct a word fre-
quency analysis on the API paths and parameters of API
interfaces according to these vulnerabilities’ CVE descrip- 2.3 Challenges and Our Approach
tions. Then, combining with expert knowledge (analyze the There are three challenges in realizing vulnerability-oriented
root cause of these vulnerabilities), we extract high-frequency API inspection in practice.
words that also reflect the functionality of these vulnerable C1: How to efficiently differentiate between various func-
APIs, such as "upload" and "submit," and mapped these for tional interfaces in an API and identify functions that may
the functionalities (e.g., File Upload function) and vulner- have security vulnerabilities? API documentations usually
abilities (e.g., Unrestricted Upload of File). Finally, we try contain a large number of APIs, but only a small portion of
to construct a mapping from the vulnerability type to the them may have security vulnerabilities. Hence, it is crucial to
corresponding function functionality and its feature strings employ a method that can identify candidate APIs, enabling
frequently applied in context. us to enhance testing efficiency and minimize ineffective test
Our analysis results are presented in Table 1, which shows requests.
that, on average, 57% of vulnerable API interfaces affected C2: How to generate test case sequences that comply with pro-
2 All the vulnerability context information and keywords gathered during tocol states and trigger the vulnerable functional interfaces?
this construction process can be found at the link: https://github.com/N API invocations are state-based and often require a series of
SSL-SJTU/VoAPI2/CVE-Information.xlsx. pre-request actions (request sequences) to fulfill the execution

742 33rd USENIX Security Symposium USENIX Association

 /account
Test Sequence Generator Test Case
/account/email PoC
… Generator
/avatars/favicon
/avatars/image CVE & NVD Expert Analysis Reverse Parameter
… Sequence Values
/functions
…
Construction Generation Vulnerability Verifier
Path Param Test Case ② Response
API Specification Keywords Keywords Sender ① Send

③ Verification
④ Feedback
Build API
1. /avatars/favicon
tagged_param: url Application
Specification Function Vulnerability 2. /avatars/image

Bug？
tagged_param: url
Parser Mapper …
Validation
Candidate APIs Server

Step 1: Specification Analysis Step 2: Candidate Interface Extraction Step 3: Test Sequence Generation Step 4: Feedback-based Test

Figure 3: System Architecture of VOAPI2 . VOAPI2 first analyzes the API specification and identifies semantic keywords
associated with potentially weak functions within the API specification. It then integrates a suitable corpus and employs a stateful
request sequence that aligns with the execution context of the corresponding functions. By utilizing inspection payloads tailored
to different vulnerability types, VOAPI2 can effectively assess these candidate functions for potential vulnerabilities.

dependencies of the API, such as providing required input 3 Design

parameters, in order to ensure the proper execution of the API.
Making standalone API calls is usually considered invalid. Overview. As shown in Figure 3, our approach begins by
identifying semantic keywords associated with potentially
C3: How to effectively generate valid test cases based on weak functions within the API specification. It then integrates
different functional interfaces and corresponding security vul- a suitable corpus and employs a stateful request sequence that
nerability types? (1) Different types of vulnerabilities have aligns with the execution context of the corresponding func-
distinct principles and require different detection methods. (2) tions. By utilizing inspection payloads tailored to different
Overly complex test messages can significantly increase the vulnerability types, we can effectively assess these candidate
cost of testing while compromising efficiency and accuracy. functions for potential vulnerabilities.
(3) Verifying the presence of vulnerabilities may necessitate
additional triggering requests. Therefore, the design of test
messages and checkers should strive to encompass a compre- 3.1 Semantic Keyword Collection
hensive range of real-world scenarios.
VOAPI2 utilizes semantic keywords to identify APIs that
To solve these challenges, we design VOAPI2to detect exhibit specific functionalities. As shown in Table 1, APIs
various vulnerabilities in REST APIs effectively. To solve the related to file uploads are often vulnerable to arbitrary file
first challenge, we identify and extract the relevant functional uploads. The functionality of an API is typically indicated
APIs from the API specifications by utilizing semantic key- by its access path, as demonstrated by /Images/Remote in
words (e.g., path keywords and parameter keywords). With a Figure 1. On the other hand, the parameters of an API provide
focus on vulnerability analysis, we identify potential vulnera- specific details about its intended function. For instance, a file
bility types and their possible locations within these functional upload API is likely to include the keyword “upload” in its
APIs. And then, we employ reverse sequence construction access path, while the parameters are expected to involve the
and parameter values generation methods, which is expanding keyword “file”.
on the techniques used in RESTler and RestCT, to generate In this part, we follow a two-step process to extract API
test requests that adhere to the protocol’s state transitions. access path and parameter keywords. First, we collected 544
This ensures the proper context for executing the candidate vulnerabilities related to APIs with publicly available detailed
APIs. At last, to address different vulnerability types, we cre- descriptions from sources such as CVE [39] and NVD [40] to
ate specific test corpora. By combining the test sequences build a dataset. We then clustered these vulnerabilities based
developed for candidate APIs and the marked potential vul- on their CWE (Common Weakness Enumeration) IDs and
nerability locations, we incorporate tailored test data into the ranked the CWE IDs according to the number of vulnerabili-
corresponding parameters, resulting in comprehensive test ties they encompassed. We selected the six clusters of CWE
cases. The presence of potential vulnerabilities in candidate IDs with the highest proportion. Using this dataset as a foun-
APIs is assessed by the feedback received from the backend dation, we analyzed the types of API functions associated
checker, enabling timely adjustments to the testing strategy with these CWE IDs and created mappings between the CWE
when necessary. ID and the corresponding API functionality. Specifically, we

USENIX Association 33rd USENIX Security Symposium 743

 1 # Endpoint : / Images / Remote , method : Get
Table 2: Keywords of API paths and parameters. Here we 2 request = requests . Request ([
list the mappings between Vulnerability Type and API Type 3 primitives . restler_static_string (" GET ") ,
4 primitives . restler_basepath ("") ,
as well as the corresponding number of keywords collected 5 primitives . restler_static_string ("/") ,
6 primitives . restler_static_string (" Images ") ,
in the Semantic Keyword Collection module. 7 primitives . restler_static_string ("/") ,
#Keywords 8 primitives . restler_static_string (" Remote ") ,
Vulnerability Type API Type 9 primitives . restler_static_string ("?") ,
API path API parameter 10 primitives . restler_static_string (" imageUrl =") ,
11 primitives . restler_fuzzable_string (" fuzzstring " , quoted =
SSRF 10 22 Resource Request APIs
False ) ,
Unrestricted Upload 12 8 File Upload APIs 12 primitives . restler_static_string (" HTTP /1.1\ r\n") ,
Path Traversal 12 3 Path Processing APIs 13 primitives . restler_static_string (" Accept : application /
Command Injection 12 7 System Configuration APIs json \r\n") ,
14 primitives . restler_static_string (" Host : localhost \r\n") ,
SQL Injection 11 4 Database Operation APIs
15 primitives . restler_refreshable_authentication_token ("
XSS 15 12 Text Display APIs authentication_token_tag ") ,
16 primitives . restler_static_string ("\r\n") ,
17 ],
18 requestId ="/ Images / Remote "
19 )
analyzed vulnerabilities within these six CWE clusters and
obtained functional descriptions for the vulnerable APIs from Listing 1: Sample of Candidate Interface Extraction.
various sources, including source code, API documentation
comments, and vulnerability descriptions. We conducted word
frequency analysis and leveraged expert experience to identify Listing 1 as an example, we obtained the API access path
the frequently occurring API functionalities within these six /Images/Remote (Line 1), the API request method Get (Line
CWE clusters. Finally, we established the mappings as shown 1) and the unique parameter of this API imageUrl (Line 10).
in Table 2 (i.e., Vulnerability Type and API Type). By applying these patterns, VOAPI2 is able to extract rel-
Next, we conducted a statistical analysis of the words found evant information from the grammar file and help identify
in the API access paths and parameters, and gathered two dis- the API functions that are associated with potential vulner-
tinct clusters of semantic keywords: one for API paths and abilities. VOAPI2 utilizes the semantic keywords generated
another for parameters. In detail, we extracted the paths and by the previous module to check for their presence in the
parameters from each vulnerable API in the dataset. We then paths and parameters of a given API. Based on this analy-
performed word frequency analysis on these paths and param- sis, VOAPI2 categorizes the API functions and assigns them
eters, and sought input from human experts to identify the potential vulnerability types (one or more). Our statistical
keyword collections. By incorporating the additional insights analysis of past API vulnerabilities indicates that all parame-
and expertise from the experts, we refined and expanded the ters could be vulnerable when an API includes path keywords
existing keyword collections. This process allowed us to es- but lacks parameter keywords. As a result, we employ the fol-
tablish meaningful connections between the keywords, the lowing strategy to extract and classify API functions: (1) APIs
corresponding API functionalities, and the vulnerability types. that solely consist of path keywords without any parameter
These strings serve as semantic keywords that reflect the func- keywords are identified as candidate APIs, and all parameters
tion of the respective category. The number of each type’s of these APIs are marked as test parameters; (2) APIs that
keywords is listed in Table 2. contain parameter keywords are also recognized as Candidate
APIs, but only the parameters corresponding to the parameter
keywords are designated as test parameters.
3.2 Candidate Interface Extraction The extracted candidate APIs are categorized into corre-
In this module, we analyze the API specification to identify sponding functional categories based on the keywords, and
interface functions that may potentially have vulnerabilities by potential vulnerability types are mapped, which guides the
searching for semantic keywords in the API documentation. selection of testing corpus and methods. VOAPI2 categorizes
The process involves the following steps. Firstly, VOAPI2 the candidate API functions and their corresponding potential
utilizes the RESTler Compile Module to analyze the API vulnerability types as shown in Table 2.
specification and generate a grammar file (shown in Listing 1).
This grammar file contains information about parameters, 3.3 Test Sequence Generation
responses, and the dependencies between individual requests.
And then, VOAPI2 parses grammar files based on the This module is designed to create a sequence of API requests
following three patterns: (1) the API access path is identi- that accurately reflect the state transitions of the API under
fied by locating the string “Endpoint:”; (2) the API request testing. Since API calls often depend on the current state, a
method is identified by locating the string “method:”; (3) single request is often inadequate to thoroughly test an API.
The API parameters are identified by finding the line above To address this, VOAPI2 employs reverse sequence construc-
the restler_fuzzable_string and checking if they are sur- tion techniques to identify other APIs that one candidate API
rounded by parentheses “()”. Taking the grammar file in depends on, thereby forming a series of requests that precisely

744 33rd USENIX Security Symposium USENIX Association

Algorithm 1 Reverse Sequence Construction Algorithm contains:
1: function G ENERATE _S EQUENCE(candidate_api) P1 =“GET /account” (P represents Producer),
2: S ← candidate_api Pn1 =“GET /database/collections”,
3: i ← -1
4: while True do
Pn2 =“POST /database/collections”,
5: producers ← F IND _P RODUCERS(S[i]) Pn3 =“GET /database/collections/{collectionId}”,
6: for producer ∈ producers do Pn4 =“POST /database/collections/{collectionId}/documents”,
7: if I S _VALID _P RODUCER(producer, S) && not and so on. And according to the strategy of the priority
I S _D UPLICATE(producer, S) then
8: S.I NSERT(0, producer) system, the producer Pn1 will be removed because it shares
9: if S[i] == S[0] then the same path as Pn2 but has a lower priority.
10: Break CRUD Semantics. CRUD is an acronym that stands for
11: else CREATE, READ, UPDATE, and DELETE. According to the CRUD
12: i = i−1
semantics, a resource (and all its sub-resources) should not
13: return S
be accessed before its creation or after its deletion. In reverse
sequence construction, the CRUD semantic constraints only
capture the state transitions of the API being tested. apply to a pair of producers and consumers. For example,
because the consumer’s request method is GET, if there is a
producer with a request method of DELETE, we need to remove
3.3.1 Reverse Sequence Construction it from the producer set.
We list the reverse sequence construction algorithm Resource Hierarchy. In RESTful APIs, the hierarchical
in Algorithm 1 and illustrate the algorithm by using relationships between resources are indicated by the for-
GET /database/collections/{collectionId}/documents ward slash “/” in the URLs of the resources. For example,
/{documentId}, an API of Appwrite [5]. It initializes a /user/{id} is a direct sub-resource of /user, whereas there
sequence set S with the candidate API (candidate_api) and is no direct relationship between /user and /team. We need
proceeds to reverse-traverse the set S, starting from the end to remove producers that do not satisfy the resource hierarchy
(S[-1] is candidate_api). This process involves identifying relationship. For example, the producer P1 that does not have
all producer APIs of the candidate API. For example, for a resource relationship with candidate_api will be removed.
the sample API, it searches the responses field in the API After applying these three constraints to filter out
documentation for potential producers. It uses string fuzzy invalid producers from the producer set, we are left with
matching to match the producers of the parameters (i.e., three valid producers: “POST /database/collections”,
collectionId, documentId) of candidate_api. Among these “GET /database/collections/{collectionId}”, “POST
producer APIs, only those that meet the producer-consumer /database/collections/{collectionId}/documents”.
relationship, CRUD semantic constraints and resource
hierarchy constraints (we explain the three key definitions 3.3.2 Parameter Values Generation
and their corresponding constraints used in this algorithm in In order to create a concrete request, each input-parameter,
the following segments.) are retained and appended to the such as imageUrl (Line 7) in Figure 1, in the sequence needs
front end of the sequence set S (Line 6-8). The algorithm to be assigned a value domain. VOAPI2 utilizes one of the
then continues the reverse-traversal process, identifying the following strategies to determine the value domain for each
producers of the preceding API until it reaches the beginning input-parameter based on RestCT [25]:
of the sequence S (Line 9-10).
Producer-Consumer Relationship. In the context of the • CONSUMER. This strategy uses the resources from the
API testing framework, the Producer-Consumer Relationship producer’s response message (i.e., http response message
refers to a rule that governs the execution order of API for one request). If an input-parameter p corresponds to
requests based on their input parameters. If the input a resource in the producer’s response message, then the
parameter of request A, which can be located in the Path, resource is used as the value domain for p during runtime.
Query, Header, Form, or Body, is derived from the response
• SPECIFICATION. This strategy uses the values described
description of request B using heuristic string matching,
in the API specification. If an input-parameter p has an
then request B must be executed before request A. In other
enum or default field, then the corresponding value is di-
words, request B acts as the producer of the input parameter
rectly used as the value domain for p. Otherwise, the strat-
for request A. In cases where the same input parameter
egy searches for all input parameters with the same name as
corresponds to multiple producers with the same path but
p and selects sample values at random from their example
different request methods, a priority system is used to select
fields to use as the value domain for p.
and add a single producer to the producer set. The priority is
determined as follows: POST takes precedence over PUT, GET, • FORMAT. This strategy uses the values from the preset
and PATCH. For the sample API, its all potential producers formatting dictionary. If an input-parameter p’s name can

USENIX Association 33rd USENIX Security Symposium 745

 cases (e.g., evil.jsp and evil.php) tailored to different pro-
Table 3: Sample Testing Corpus for Different API Types.
gramming languages. For another example, we developed a
API Type Sample in Testing Corpus
comprehensive set of payloads (e.g., "../"*9+"etc/passwd")
Resource Request API http://IP:PORT/ssrf/{0} that attempt to bypass various types of path checks to explore
File Upload API evil files (evil.jsp, evil.asp, evil.php, etc)
Path Processing API /etc/passwd; C://Windows//win.ini
path traversal vulnerabilities in APIs that handle file paths.
System Configuration API curl http://IP:PORT/command/{0} More importantly, we design an extensible interface to scale
Database Operation API 1" or "1"="1; SQLMap the test corpus. For example, in addition to employing con-
Text Display API <img src=‘http://IP:PORT/xss/{0}’> ventional SQL injection test cases (e.g., 1" or "1"="1), we
augment the Test Case Generator by integrating one existing
powerful detection tool (i.e., SQLMap [41]) to help determine
be matched to a preset formatting dictionary, then the cor- the presence of vulnerabilities with sophisticated corpus.
responding value is used as the value domain for p. The
VOAPI2 preset formatting dictionary is mainly designed
for parameters that have specific formatting requirements, 3.4.2 Feedback-based Vulnerability Verifier
such as email (which requires a valid email format) and
The Feedback-based Vulnerability Verifier primarily com-
password (which requires a length of at least 8 characters).
prises two key components: the Test Case Sender and the
• SUCCESS. This strategy uses the values from the previous Validation Server. The Test Case Sender is tasked with trans-
successful requests (identified by a 200 HTTP status code). mitting test cases to the target API application and receiving
If an input-parameter p has the same name as an input- the corresponding responses. It subsequently forwards the
parameter from a previous successful request, then the value received response, candidate API information, and testing cor-
for that input-parameter is used as the value domain for p. pus data to the Validation Server for in-depth analysis and
If the value for the same input-parameter was generated verification. Meanwhile, the Validation Server is responsible
using the RANDOM strategy, then the value is mutated for confirming the presence of vulnerabilities.
to increase the diversity of test inputs and avoid creating For SSRF, XSS, and Command Injection testing corpus,
duplicates. when a vulnerability is present, the vulnerable behavior pro-
duced by the testing corpus prompts the API application to
• RANDOM. This strategy generates a random value to use send a vulnerability verification request to the Validation
as the value domain for an input-parameter p if none of the Server. The Validation Server determines the existence of
above strategies can determine a value domain for p. the vulnerability based on whether it receives this request.
VOAPI2 applies these strategies in decreasing priority until If the request is received, the Validation Server infers that
a value domain is determined for each input-parameter. the vulnerability exists. For the remaining types of testing
corpus, the Validation Server determines the existence of vul-
nerabilities based on the verification information sent by the
3.4 Feedback-based Testing and Verification Test Case Sender. For example, if the response content of
This module consists of two parts: Test Case Generator and an upload API test case contains the term “success” or the
Feedback-based Vulnerability Verifier. For each candidate response status code is in the 2xx range, the Validation Server
API in the test sequence, Test Case Generator generates valid concludes that the API is vulnerable to Unrestricted Upload
test cases with the assistance of the test sequence generation of File with Dangerous Type. Following this, the Validation
module. These test cases are then sent to the API application Server further checks if the returned information includes
by the Feedback-based Vulnerability Verifier to detect the the path of the uploaded file. If it does, the Validation Server
presence of corresponding vulnerabilities. accesses the file and conducts relevant tests to determine if
the malicious file can be successfully executed.
The Validation Server provides feedback on the verification
3.4.1 Test Case Generator
results to the Test Case Generator. When a vulnerability is
The test case generator leverages the output of the Test Se- identified, Test Case Generator proceeds to the next test case.
quence Generator module to generate test cases that target When Validation Server reports that no vulnerability exists,
potential vulnerabilities in the APIs. Specifically, it gener- Test Case Generator makes the following determinations. (1)
ates test cases by inserting the testing corpus into the marked The request method in the test case is POST. (2) The testing
testing parameter positions. For the unmarked parameters, corpus corresponds to one of the following: SSRF, XSS, or
the values assigned by the Test Sequence Generation module Command Injection testing corpus. (3) The testing request
are utilized. As shown in Table 3, we collected some test containing the testing corpus has been executed (determined
cases and built the testing corpus for different categories of by a 2xx response status code).
APIs. For instance, to address variations in server-side archi- When all of these conditions are met, the Test Case Genera-
tectures, we gathered and designed malicious file upload test tor infers that a triggering request is required to further verify

746 33rd USENIX Security Symposium USENIX Association

the existence of the vulnerability. The triggering request is RQ1 (Vulnerability Detection): How well are the vulnera-
sent to further test the existence of the vulnerability. To il- bility identification capability (§5.2) and accuracy (§5.3) of
lustrate this point, consider CVE-2023-27161 as an example. VOAPI2 ? Can VOAPI2 discover vulnerabilities in real-world
The SSRF vulnerability is located in the /Repositories API APIs? (§5.4)
and can be exploited by embedding a payload in the url pa- RQ2 (Efficiency): Can VOAPI2 efficiently generate test
rameter using a POST request. However, merely sending the cases and explore operations, and how does it perform com-
POST request does not trigger the vulnerability. In such cases, pared with the state-of-the-art tools? (§5.5)
an additional triggering request, specifically GET /Packages, RQ3 (Ablation Study): How does the vulnerability-oriented
needs to be sent to activate the vulnerable behavior. This trig- strategy of VOAPI2 affect the testing results? (§5.6)
gers the API application to send a vulnerability verification
request to the Validation Server, which ultimately leads to the 5.1 Experiment Setup
discovery of this vulnerability.
The triggering requests consist of two types: (1) an API Compared Tools. We compared our prototype system with
with the same path and request method as GET, and (2) all several popular vulnerability scanners and state-of-the-art
APIs that do not require parameters and have a request method tools in RESTful API testing. It is important to note that these
of GET. The selection of these two types of triggering requests tools are designed for different purposes. First, vulnerability
is not only aimed at cost savings but also based on historical scanners are primarily used to evaluate web applications. Fur-
vulnerability analysis, which has shown a high correlation thermore, these selected vulnerability scanners must support
between these types of APIs and triggering vulnerabilities. If API testing by sending mutated requests to these endpoints
the vulnerability cannot be triggered through these requests, and reporting potential vulnerabilities based on the response
it is recorded as a type of vulnerability that requires manual results. On the other hand, RESTful API testing tools aim to
triggering. By introducing a triggering mechanism, VOAPI2 generate better API test cases and discover more bugs (i.e.,
can more accurately identify API vulnerabilities that require 500 Internal Server Error). However, they are not designed to
specific triggering and improve the overall effectiveness of reveal in-depth vulnerabilities. We focus on different aspects
vulnerability testing. of VOAPI2 and compare them with corresponding tools to
answer the questions we are concerned about, for example,
whether our method can find more bugs than vulnerability
4 Implementation scanners. Finally, we select the following two vulnerability
scanners and three RESTful API testing tools:
We have developed a VOAPI2 prototype system based on
RESTler, which consists of 2,700 lines of Python code. The • Zed Attack Proxy (ZAP) [44] is an open-source black-box
Semantic Keyword Collection module is implemented based web vulnerability scanner developed by OWASP, primarily
on Scrapy [42] and Beautiful Soup [43]. The Candidate Inter- used for web vulnerability assessment and penetration test-
face Extraction module is extended from the RESTler’s Com- ing. We leverage ZAP’s resolver to parse API specifications
pile module to support parsing API documentation of multiple and inspect API services directly.
formats. In the Test Sequence Generation module, the re-
verse sequence construction is improved based on RESTler’s • Astra [45] is a scanner specifically designed for vulnerabil-
producer search and sequence construction strategy, and the ity scanning of RESTful APIs, testing various vulnerabil-
parameter values generation is extended from RestCT’s input- ities by loading payloads into parameters, HTTP headers,
parameter value rendering module. In the Feedback-based and other locations.
Testing and Verification module, the test case generator is built • RESTler [12] is an open-source black-box RESTful API
by modifying the test corpus of RESTler, while the feedback- testing tool developed by Microsoft. It generates stateful
based vulnerability verifier is implemented using the socket test cases by inferring producer-consumer dependencies
library and by expanding response handling. More impor- between operations and feedback at runtime.
tantly, to support various vulnerabilities (e.g., Unrestricted
Upload), we have extended the multipart_formdata primitive • RestTestGen [16] uses data dependencies between opera-
in RESTler Compile. After all, the current prototype system tions to generate test cases and deploy two distinct oracles
supports multiple API documentations adhering to OpenAPI to test RESTful API, that test cases can reveal implementa-
v2/v3 and can detect six types of API vulnerabilities. tion defects.
• MINER [46] uses length-oriented strategies to generate
5 Evaluation sequence templates and a neural network model to predict
key request parameters and provide appropriate parame-
We conduct experiments to evaluate the performance of ter values to get a long sequence request in RESTful API
VOAPI2 , and our evaluation targets the following questions: testing.

USENIX Association 33rd USENIX Security Symposium 747

 two vulnerability scanning tools discovered fewer vulnerabili-
Table 4: Benchmark Applications. #Endpoint represents
ties. The three RESTful API testing tools primarily uncovered
the number of endpoints in each Application. #Download
bugs that resulted in HTTP 500 errors, with a tiny proportion
represents the number that each application is downloaded in
being security vulnerabilities, as shown in Figure 4. It can be
the Docker hub [47].
observed that VOAPI2 detected vulnerabilities in all seven
Applications #Endpoint Version Description #Download applications, with the highest number and variety of vulner-
GitLab 358 8.17.0 Code Repository 100M+ abilities. In contrast, the vulnerability scanning tools (i.e.,
Jellyfin 405 10.7.1 CMS 100M+ ZAP and Astra) did not detect any vulnerabilities in Gitea,
Appwrite 95 0.9.3 CMS 5M+
Rbaskets and GitLab. The capabilities of these two tools were
Microcks 44 1.17.1 CMS 600K+
Casdoor 121 1.13.0 CMS 20K+ similar, with slight differences observed only in the Appwrite.
Gitea 325 1.16.7 Code Repository 20K+ A comparison reveals that VOAPI2 has the strongest ability
Rbaskets 22 1.2.3 Web Service 10K+ to identify vulnerabilities compared to ZAP and Astra, par-
ticularly in SSRF and XSS vulnerabilities. The other three
RESTful API tools discovered only a small number of security
Evaluation Benchmarks. We use the following three criteria vulnerabilities in four applications (i.e., Appwrite, Jellyfin,
to select real-world RESTful API applications as our bench- Microcks and GitLab), and they required manual payload in-
mark for evaluation. (1) Having a complete API specification sertion to validate the security vulnerabilities associated with
document that can serve as input for all comparative testing the identified HTTP 500 errors.
tools. (2) The application should have APIs covering multiple The three RESTful API testing tools were able to discover
orders of magnitude, ranging from small to large-scale. (3) multiple bugs that resulted in HTTP 500 errors when detect-
The application should be open-source and widely used in ing targets, as shown in Table 5. The average proportion of
different scenarios. Table 4 illustrates the seven applications vulnerabilities among the HTTP 500 errors discovered by the
we have chosen as our testing subjects. Based on the number other three tools is very low. Through manual analysis, we
of API endpoints, Rbaskets [48], and Microcks [6] are cate- determined that the percentage of vulnerabilities triggered by
gorized as small-scale API applications, while Appwrite [5] these errors is only 5.4% (RESTler), 3% (MINER), and 2.5%
and Casdoor [4] belong to medium-scale API applications. (RestTestGen), respectively. Because paths with security vul-
Gitea [49], Jellyfin [3] and GitLab [36], on the other hand, are nerabilities may not necessarily result in internal errors in
considered large-scale API applications. All of these applica- real-world API applications. The mentioned tools randomly
tions are easily deployable and supply API services well. modify parameters’ syntax format and content, leading to
Evaluation Settings. For the target services, we install them back-end parameter parsing issues rather than posing a sig-
with a basic configuration to ensure the proper functioning of nificant security threat. Additionally, incorporating payloads
their API services. Moreover, we employ the default settings for all types of vulnerabilities without distinguishing the API
for all compared tools and run each with a maximum time functionalities would result in low efficiency, as observed
limit (i.e., 5 hours). After each round of testing, we restore the in our subsequent ablation experiments (§5.6). Meanwhile,
service environment to ensure the consistency of the target VOAPI2 found fewer instances of HTTP 500 errors, mainly
service. We conduct all experiments on a server with 4 CPU due to the targeted nature of VOAPI2 testing, with fewer API
cores, 8 GB RAM, and the Ubuntu 20.04 LTS operating sys- paths being tested, which can be observed from the number of
tem, which provides adequate resources for conducting the messages. Moreover, VOAPI2 strives to generate messages
experiments. In papers such as RESTler, a bug is identified by that comply with grammatical rules to trigger API behavior
the reception of a 500 HTTP status code following the exe- and successfully execute the payload that confirms the bug.
cution of a request sequence. However, it’s important to note We collected a summary of all the security vulnerabili-
that 5xx errors typically refer to server errors during request ties discovered by various tools, including the vulnerability
processing, and not all 5xx errors are necessarily related to se- type, vulnerability position, vulnerable endpoint, and vulnera-
curity vulnerabilities, such as Denial of Service (DoS). Hence, ble parameters, as shown in Table 6. It can be observed that
in our paper, we classify a genuine security vulnerability as a vulnerability-oriented testing methods allow VOAPI2 to dis-
bug, distinct from a mere 5xx error. cover corresponding types of security vulnerabilities in APIs
with different functionalities. More importantly, VOAPI2
has discovered the highest number of vulnerabilities and the
5.2 Vulnerability Detection (RQ1) widest range of vulnerability types compared to other tools.
VOAPI2 identified 26 vulnerabilities, including 7 previously Specifically, VOAPI2 has identified four types of vulnerabili-
unknown and 19 known security bugs, across seven API ap- ties, 100% higher than the average of other tools. Meanwhile,
plications, encompassing six different types of vulnerabilities. VOAPI2 found 26 vulnerabilities, which was 420% more than
All zero-day bugs have been reported to vendors and fixed; the average of other tools.
four of them have been assigned CVE IDs. In comparison, the Among these vulnerabilities, our tool uniquely uncovers

748 33rd USENIX Security Symposium USENIX Association

 SSRF XSS Path Traversal SQL Injection Command Injection Unstricted Upload

Appwrite Rbaskets Jellyfin Casdoor Microcks Gitea GitLab

8 8 8 8 8 8 8

7 7 7 7 7 7 7

6 6 6 6 6 6 6

5 5 5 5 5 5 5

4 4 4 4 4 4 4

3 3 3 3 3 3 3

2 2 2 2 2 2 2

1 1 1 1 1 1 1

0 0 0 0 0 0 0
RE 2

Re er

ZA

As

RE 2

Re er

ZA

As

RE 2

Re er

ZA

As

RE 2

Re er

ZA

As

RE 2

Re er

ZA

As

RE 2

Re er

ZA

As

RE 2

Re er

ZA

As
Vo

Vo

Vo

Vo

Vo

Vo

Vo
IN

IN

IN

IN

IN

IN

IN
tra

tra

tra

tra

tra

tra

tra
stT

stT

stT

stT

stT

stT

stT
AP

AP

AP

AP

AP

AP

AP
ST

ST

ST

ST

ST

ST

ST

P
ER n

ER n

ER n

ER n

ER n

ER n

ER n
es

es

es

es

es

es

es
I

I
l

l
tG

tG

tG

tG

tG

tG

tG
e

e
Figure 4: The vulnerabilities and their types uncovered by different tools on evaluation benchmarks .

Table 5: Compared with three RESTful API testing tools. #500 means the number of HTTP 500 errors found by tools. #Packet
means the number of all packets sent in testing. #Ratio is equal to total #500 divided by total #Packet.
Appwrite Casdoor Gitea Jellyfin Microcks Rbaskets GitLab
Compare #Ratio
#500 #Packet Time #500 #Packet Time #500 #Packet Time #500 #Packet Time #500 #Packet Time #500 #Packet Time #500 #Packet Time
RESTler 5 88,558 5h 0 165,470 5h 1 297,419 5h 54 2,175 12m07s 19 67,128 5h 0 20,435 5h 32 105,931 5h 0.015%
MINER 4 64,944 5h 0 104,352 5h 1 31,668 5h 68 23,002 5h 13 143,830 5h 0 17,422 5h 13 48,267 5h 0.023%
RestTestGen 1 9,030 19m49s 0 16,340 34m13s 2 58,200 4h37m 63 71,440 4h43m 15 7,340 8m5s 0 2,140 1m32s 41 57,550 1h52m 0.055%
RestTestGen+V 1 6,334 12m42s 0 8,920 13m09s 0 32,550 1h28m 55 48,740 3h57m 11 4,246 5m51s 0 1,550 1m03s 36 43,110 1h33m 0.071%
VOAPI2 1 2,123 1m51s 0 6,987 6m27s 0 9,137 7m05s 23 13,578 10m53s 2 509 24s 0 238 13s 3 1,558 5m03s 0.085%

14 bugs that elude detection by other tools. Based on our anal- positive rate.
ysis, this phenomenon can be attributed to twofold factors. For path traversal vulnerabilities, ZAP lacks further vali-
First, in the case of RESTful API testing tools (e.g., RESTler), dation. It solely bases its determination of a vulnerability’s
they often lack a comprehensive testing corpus required for existence on whether the test request returns a 2xx status code.
verifying various types of vulnerabilities (e.g., SSRF) within On the other hand, VOAPI2 goes a step further by analyzing
their corresponding endpoints. Thus, these tools often fail the response content, checking for the presence of content
to detect a vulnerability if it does not lead to a HTTP 500 er- corresponding to the test payload to confirm the existence
ror. Second, scanning tools (e.g., ZAP) face challenges in of the vulnerability. For instance, when the test payload is
constructing an appropriate sequence encompassing multi- "/etc/passwd", VOAPI2 will match the response content for
ple requests that align with the data dependencies existing characteristic strings (e.g., root) to validate the vulnerability.
among API endpoints. Thus, these tools fail to identify the re- In terms of VOAPI2 , false positives may occur when check-
spective vulnerabilities (e.g., XSS) concealed behind intricate ing a particular XSS (i.e., stored XSS) and the implicit unre-
interactions, as exemplified in §5.4. stricted upload vulnerability. In these scenarios, we need to
trigger vulnerability manually and check whether vulnerabil-
ity exists or not. Moreover, we thoroughly discuss the root
5.3 Accuracy (RQ1) causes and the corresponding improvement ways in §6.
We further analyzed the accuracy of VOAPI2 and vulnerabil-
ity scanners. All alerts were manually verified to determine if 5.4 Real-world Vulnerabilities (RQ1)
they were true vulnerabilities, thereby identifying false pos-
itives (FP). The false discovery rate (FDR = FP/(FP+TP)) We applied VOAPI2 to discover security vulnerabilities in
was calculated and presented in Table 7. It can be observed real API applications and found various vulnerabilities. As
that VOAPI2 has lower false positive rates compared to the shown in Table 6, we identified more types of vulnerabilities
scanners, which can be attributed to VOAPI2 ’s vulnerability- compared to scanning tools, especially on XSS and SSRF.
oriented strategy and more accurate vulnerability validation The main reason is that our method can generate effective
strategy. request sequences that access multiple endpoints in proper
For XSS vulnerabilities, ZAP and Astra conduct tests on order, allowing us to trigger deeper vulnerabilities.
all APIs and determine the existence of an XSS vulnera- Case Study: XSS. The XSS vulnerability (CVE-2022-
bility solely based on the presence of the XSS payload in 2925) was discovered in the Appwrite application. As
the response. However, a significant number of these APIs shown in Figure 5, this bug exists in five API end-
do not contain display functions. Consequently, the payload points (/teams, /users, /functions, /database/collections,
would not be displayed on any particular page, which re- /teams/{teamId}/memberships), and the vulnerable argu-
sults in a high incidence of false positives. Correspondingly, ments in these endpoints are "name" marked in blue color.
the vulnerability-oriented testing strategy helps VOAPI2 find In our experiments, both ZAP and Astra face significant
more API paths related to XSS, which leads to a lower false difficulties. While they may encounter problems in gen-

USENIX Association 33rd USENIX Security Symposium 749

Table 6: RESTful API vulnerabilities identified by all tools. For Producer, " indicates this API endpoint has a producer,
while % indicates it doesn’t have; For 0-day, " indicates that this vulnerability is a zero-day vulnerability while % indicates it is
not; For Vulnerability Identification Tools, " indicates that this tool can identify the vulnerability, " indicates only this tool
can identify the vulnerability, while % indicates it cannot.
Vulnerability Identification Tools
Application Version Path Parameter Type Producer 0-day Bug-IDs
VOAPI2 RESTler RestTestGen MINER ZAP Astra
0.9.3 /avatars/favicon url SSRF % " CVE-2023-27159 " % % % " "
0.9.3 /avatars/image url SSRF % " CVE-2023-27159 " % " % " "
0.9.3 /teams name XSS % % CVE-2022-2925 " % % % % "
Appwrite 0.9.3 /teams/{teamId}/memberships name XSS " % CVE-2022-2925 " % % % % %
0.9.3 /database/collections name XSS % % CVE-2022-2925 " % % % % %
0.9.3 /functions name XSS % % CVE-2022-2925 " % % % % %
0.9.3 /usrs name XSS % % CVE-2022-2925 " % % % % %
Rbaskets 1.2.3 /api/baskets/{name} forward_url SSRF % " CVE-2023-27163 " % % % % %
10.7.1 /Images/Remote imageUrl SSRF % % CVE-2021-29490 " " % " " "
10.7.1 /Items/RemoteSearch/Image imageUrl SSRF % % CVE-2021-29490 " % % % " "
10.7.1 /Items/{itemId}/RemoteImages/Download imageUrl SSRF " % CVE-2021-29490 " % % % % %
10.7.1 /Repositories Url SSRF % " CVE-2023-27161 " % % % % %
Jellyfin 10.7.1 /Playlists name XSS % % CVE-2023-23636 " % % % % %
10.7.1 /Repositories name XSS % % CVE-2022-35910 " % % % % %
10.7.1 /Collections name XSS % % CVE-2023-23635 " " % % % %
10.7.1 /Startup/User Name XSS % " 1 unassigned " % % % % %
1.13.0 /api/get-organizations field SQL Injection % % CVE-2022-24124 " % % % " "
Casdoor
1.13.0 /api/upload-resource fullFilePath Unrestricted Upload % % CVE-2022-38638 " % % % % %
1.17.1 /jobs repositoryUrl SSRF % " 1 unassigned " " " " % %
Microcks
1.17.1 /artifact/download url SSRF % " 1 unassigned " " % " " "
1.16.7 /repos/{owner}/{repo}/contents/{filepath} content Unrestricted Upload " % CVE-2022-1928 " % % % % %
Gitea
1.16.7 /repos/{owner}/{repo}/hooks url SSRF " % CVE-2018-15192 " % % % % %
8.17.0 /v3/hooks url SSRF % % CVE-2018-8801 " " % % % %
8.17.0 /v3/projects import_url SSRF % % CVE-2022-0249 " " % % % %
GitLab 8.17.0 /v3/projects/{id}/deploy_keys title XSS " % CVE-2022-2230 " % % % % %
8.17.0 /v3/projects/{id}/milestone title XSS " % CVE-2022-1190 " % " % % %

Table 7: FDR identified by different tools.

Appwrite Rbaskets Jellyfin Casdoor Microcks Gitea GitLab FP Rate 1. /teams: 16. /databases/collections:
2. POST: 18. POST:
VOAPI2 4/11 1/2 5/13 2/4 1/3 3/5 3/7 42.22%
3. RequestBody: 19. RequestBody:
ZAP 6/8 2/2 9/11 5/6 7/8 5/5 2/2 85.71% 4. name 20. name
Astra 5/8 0/0 4/6 3/4 2/3 1/1 1/1 69.57% 5. responses: 21. /users:
6. $id: 22. POST:
7. description: Team ID. 23. RequestBody:
8. /teams/{teamId}/memberships: 24. name
9. POST: 25. /functions:
erating appropriate parameter values for some APIs, the 10. parameters: 26. POST:
more critical issue is their inherent inability to request 11. description: Team unique ID. 27. RequestBody:
12. in: path 28. name
/teams/{teamId}/memberships (Line 8). Because the pres- 13. name: teamId
14. RequestBody:
ence of this XSS vulnerability on this endpoint requires ac- 15. name
cessing /teams (Line 1) with the POST method before, then
parsing the id value (Line 6) from the response, and assign- Figure 5: XSS vulnerabilities in Appwrite.
ing it to the path parameter teamId (Line 13). Obviously,
ZAP and Astra cannot infer this data dependency and con-
struct a suitable sequence that includes these requests. In for testing API endpoints, compared with state-of-the-art
contrast, VOAPI2 can extract the context in which the target RESTful testing methods (e.g., RestTestGen); (2) whether
API endpoint operates and generate similar request sequences VOAPI2 is more efficient than web scanners (e.g., ZAP) in
to discover such vulnerabilities. finding vulnerabilities.
Operation Coverage. In this experiment, we used VOAPI2 ’s
5.5 Efficiency (RQ2) Test Sequence Generation (TSG) module to generate a request
sequence for each endpoint. We selected three approaches to
We check the efficiency of VOAPI2 in two respects: (1) compare the API’s operation coverage with TSG. If a tool
whether VOAPI2 can efficiently generate request sequences is able to generate at least one valid request sequence for an

750 33rd USENIX Security Symposium USENIX Association

 100 100
100 97.5 96.7 96.7
RESTler
MINER
80.0 80.0 RestTestGen
80 75.0
77.3
TSG
Coverage %

61.1 62.7
59.1 58.8
60
52.9
50.0
45.3 46.6
44.0
40 37.4 37.3
32.1 30.1
27.4 27.1 28.9 28.6
24.9
21.1
20
11.7 9.8 11.5 11.5

0
Appwrite Casdoor Gitea Jellyfin Microcks Rbaskets GitLab Average

Figure 6: Compared with RESTful API testing tools on coverage.

(§3.2) module from VOAPI2 to ensure all API paths and

Table 8: Compared with scanners on testing time consume.
parameters are checked with equal priority. We refer to this
Appwrite Casdoor Gitea Jellyfin Microcks Rbaskets GitLab #Total
modified tool as VOAPI2 -V, which serves as a benchmark for
ASTRA 4:30m 18:07m 40:32m 103:28m 8:02m 4:34m 56:16m 235:19m
ZAP 10:23m 22:02m 51:27m 88:02m 10:22m 7:57m 182:41m 372:54m comparative analysis against VOAPI2 .
ZAP+V 6:39m 8:05m 16:33m 6:11m 1:31m 0:40m 24:07m 63:46m
VOAPI2 1:51m 6:27m 7:05m 10:53m 0:24m 0:13m 5:03m 31:05m In testing, no new vulnerabilities were found by VOAPI2 -
V. Table 9 shows the time VOAPI2 and VOAPI2 -V need
to expose each vulnerability. Obviously, VOAPI2 -V takes
API endpoint that results in a 2xx status code response, we several orders of magnitude more time, because VOAPI2 -V
consider that endpoint to be covered. must indiscriminately check every endpoint of API. When a
Figure 6 exhibits our experimental results, which evidently potentially vulnerable path is located near the end of the spec-
demonstrates that TSG performs better than the current state- ification document, VOAPI2 -V takes much longer to reach it.
of-the-art RESTful testing tools in terms of generating valid For example, in the case revealed in /Startup/User of Jellyfin,
request sequence for target API endpoint. More specifically, VOAPI2 found this vulnerability on this API in 10min30s,
TSG achieves the highest endpoint coverage on Appwrite, while VOAPI2 -V takes 495min28s. This is because VOAPI2
Gitea, Microcks, Rbaskets and GitLab. Specifically, the aver- can aim this endpoint directly from a large number of APIs,
age coverage of TSG on these products can reach 62.7%, while VOAPI2 -V must test all APIs sequentially. Further-
which is more than RESTler’s 37.3%, MINER’s 44.0%, more, VOAPI2 -V tests every parameter of the API using all
RestTestGen’s 58.8%. We attribute TSG’s outstanding per- available test payloads, which consumes a significant amount
formance to its superior ability to construct dependencies for of time. In contrast, VOAPI2 selects targeted payloads for
target API endpoints compared to other tools. For the sample testing the portion parameters marked with directed strategy,
API in §3.3.1, we found that none of the tools except VOAPI2 which saves a lot of time.
can construct the correct dependency relation for this API. Furthermore, we integrate the keyword shortlisting process
Compared With Web Scanners. Table 8 shows time con- into existing tools (i.e., RestTestGen and ZAP). Specifically,
sumed by the web vulnerability scanners and VOAPI2 . The these tools are now only applied to candidate APIs extracted
test time of VOAPI2 is much less than the other two tools. from VOAPI2 . The integrated models resulting from this
Specifically, it took VOAPI2 31min05s to test the seven prod- integration are named RestTestGen+V and ZAP+V. In order
ucts, while Astra took 235min19s, and ZAP took 372min54s. to compare the performance of these models, we conduct
This is because of VOAPI2 ’s vulnerability-oriented strategy, experiments focusing on efficiency and bug discovery.
which just tests candidate API endpoint within targeted pay- In terms of test efficiency, ZAP+V inspects solely the API
loads based on the feature of its path and parameters, greatly endpoints that are filtered by keywords. As a result, the testing
enhancing testing efficiency. In contrast, tools like ZAP and time is significantly reduced compared to the original ZAP,
Astra test parameters and paths indiscriminately within all as showcased in Table 8. This highlights the effectiveness of
payloads, resulting in significantly longer testing times. keyword lists when used with ZAP. On the other hand, for
RestTestGen+V, we expanded the testing scope to include not
5.6 Ablation Study (RQ3) only the API endpoints filtered by keywords but also those
that have data dependencies with them. The testing results
To investigate how the vulnerability-oriented strategy of presented in Table 5 demonstrate that these integrated mod-
VOAPI2 impacts the result of testing, we perform an ablation els exhibit lower time consumption and send fewer packets
study. Thus, we remove the Candidate Interface Extraction compared to their respective original tools.

USENIX Association 33rd USENIX Security Symposium 751

 Table 9: Compared with VOAPI2 -V on time consume.
Time-to-Exposure Can’t execute!
Application Bug-IDs Path
VOAPI2 VOAPI2 -V 1. /storage/files
2. post:
3. description: Create a new file.
4. operationId: storageCreateFile
CVE-2023-27159 /avatars/favicon 2.81s 1min31s 5. requestBody:
6. content:
webshell
CVE-2023-27159 /avatars/image 2.90s 1min39s 7. multipart/form-data:

CVE-2022-2925 /teams 53.42s 26min30s 8.

9.
schema:
properties: visit
10. file:
Appwrite CVE-2022-2925 .../memberships 1min03s 34min36s 11. description: Binary file.
12. type: string
CVE-2022-2925 /database/collections 15.28s 17min18s ------------------ Omitted -----------------

CVE-2022-2925 /functions 1min39s 60min16s post

CVE-2022-2925 /users 1min30s 48min8s
Rbaskets CVE-2023-27163 /api/baskets/{name} 2.08s 38s
CVE-2021-29490 /Images/Remote 5.79s 225min46s
CVE-2021-29490 /Items/.../Image 7.26s 246min29s
CVE-2021-29490 /Items/.../Download 6.68s 228min13s Figure 7: False positive of unrestricted upload.
CVE-2023-27161 /Repositories 10.24s 322min38s
Jellyfin
CVE-2023-23636 /Playlists 9min09s 368min04s
CVE-2022-35910 /Repositories 8min57s 322min47s
CVE-2023-23635 /Collections 8min25s 35min11s
fail to judge these vulnerabilities correctly and require manual
1 unassigned /Startup/User 10min30s 495min28s efforts. To address this issue, we plan to design an extensible
CVE-2022-24124 /api/get-organizations 5min04s 43min28s interface of the Feedback-based Vulnerability Verifier (§3.4.2)
Casdoor
CVE-2022-38638 /api/upload-resource 2min33s 75min50s for specific vulnerabilities. Furthermore, we can continuously
Microcks
1 unassigned /jobs 1.90s 11min19s extend vulnerability verification methods, for example, to
1 unassigned /artifact/download 2min06 90min55s
heuristically generate upload paths based on the context of
CVE-2022-1928 /repos/.../{filepath} 3.61s 94min41s
Gitea
CVE-2018-15192 /repos/.../hooks 2.49s 36min05s
implicit unrestricted upload of dangerous files, thereby en-
abling VOAPI2 to discover vulnerabilities more generally
CVE-2018-8801 /v3/hooks 1min12s 236min05s
CVE-2022-0249 /v3/projects 1min17s 20min17s and automatically.
GitLab
CVE-2022-2230 /v3/.../deploy_keys 1min49s 53min33s Manual Effort. In identifying and validating vulnerabilities,
CVE-2022-1190 /v3/.../milestone 2min36s 145min41s
VOAPI2 requires human intervention in two scenarios: im-
plicit unrestricted upload vulnerabilities and stored Cross-Site
Scripting (XSS).
Regarding bug discovery, RestTestGen+V identified fewer
HTTP 500 errors, as indicated in Table 5. This can be attributed In the case of unrestricted upload vulnerabilities, VOAPI2
to the fact that the number of candidate APIs is significantly submits a malicious file to check whether the bug exists or not.
smaller than the original APIs, and our vulnerability-oriented If the server’s response omits the file’s access path, VOAPI2
strategies are not specifically designed to detect HTTP 500 is unable to automatically retrieve this path, which hinders
errors. Furthermore, RestTestGen+V lacks a suitable test cor- further tests to assess whether the file can be executed. We
pus that is necessary to trigger potential vulnerabilities in present an example of a false positive of unrestricted upload in
the selected APIs. Therefore, it cannot benefit from keyword Figure 7. It can be seen that VOAPI2 automatically deduces
shortlisting. Similarly, ZAP+V does not discover more vulner- that the arbitrary file upload API interface permits unrestricted
abilities than ZAP because it is unable to construct a suitable uploads of malicious files. However, it lacks the prior knowl-
sequence comprising valid requests to satisfy dependencies edge necessary to automatically retrieve the unique access
among candidate APIs. paths of these uploaded files, such as identifying the location
In conclusion, although keyword selection can enhance of the file access interface or how to obtain random file IDs
testing efficiency, it does not lead to better effectiveness when like "651506441776a". This prior knowledge is not standard-
directly applied to traditional RESTful API testing tools and ized and tends to vary across different API services, making
vulnerability scanners. automatic retrieval of access paths a complex challenge. As
a result, manual intervention is currently necessary to visit
relevant pages, identify the page containing the file access
6 Discussion interface, and manually test whether the server can parse the
malicious file. If the server restricts the parsing of such ma-
In this section, we discuss the limitations of VOAPI2 and licious files, the upload vulnerability remains untriggered,
explore the improvement direction in the future. leading to a false positive.
Scale Ability of Bug Verification. The current state of In terms of stored XSS (also known as persistent XSS),
feedback-based vulnerability verification is limited by the VOAPI2 takes a POST request path as a potential XSS vul-
concise validation methods based on the testing corpus for nerability after getting a response packet with a 2xx status
different bugs. That brings false positives when a bug must be code, which also contains an XSS payload. However, similar
triggered interactively, like the stored XSS and implicit upload to the challenges unrestricted upload checking faces, VOAPI2
bug we mentioned before (§5.3). Consequently, VOAPI2 may cannot automatically determine which storage page stores

752 33rd USENIX Security Symposium USENIX Association

XSS payloads. Thus, this XSS payload can only be triggered lowing it to reach the "deep" states of the target API service.
by manually browsing the relevant pages guided by expert RestTestGen [16] utilizes data dependencies between oper-
knowledge. If the XSS behavior is not triggered even after ations to generate test cases, while RESTest [15] considers
browsing all relevant pages, it is considered a false positive. dependencies among parameters and generates test cases that
Support More Specifications. Currently, VOAPI2 only sup- satisfy specified dependency relationships using constraint
ports OpenAPI-formatted API specifications as input, while solving and random input generation. MINER [46] uses a
many applications do not provide such documentation and in- neural network model to predict critical request parameters.
stead offer API usage instructions in other files (e.g., HTML). However, the aforementioned black-box testing tools all fo-
In order to expand VOAPI2 ’s capabilities, we propose to use cus on better parsing API specifications, generating request
machine learning or natural language processing methods to sequences that comply with the API interface protocol state,
identify and extract API information from these usage instruc- and generating effective test cases that cover more API func-
tions. Finally, VOAPI2 can inspect more API applications. tionalities. They aim at the usability of API functions based
Machine Learning Method. VOAPI2 classifies API inter- on whether the API service returns HTTP 500 errors but do not
faces based on statistics of vulnerability characteristics and specifically target security vulnerabilities.
human expertise. Although it is a reliable method to sup- Penetration Techniques. Due to the close similarity in im-
port our prototype system in discovering real-world security plementation between the backend of RESTful APIs and tra-
vulnerabilities, we believe there is a trend for artificial intelli- ditional web services, researchers aim to expand web penetra-
gence technology to replace these expert experiences in the tion techniques to conduct vulnerability detection in RESTful
future. For example, advances in natural language processing APIs. Zed Attack Proxy (ZAP) [10] is primarily used for web
(e.g., large language model) have demonstrated powerful text vulnerability assessment and penetration testing. Its current
analysis capabilities, promising to audit API documents for OpenAPI extension supports parsing API specifications, en-
interface classification more effectively. abling web vulnerability testing for API services. Astra [45]
is a tool specifically designed to scan vulnerabilities in REST-
ful APIs. It verifies common web vulnerabilities by injecting
7 Related Work payloads into parameters, HTTP headers, and other locations.
However, these tools have significant limitations that they lack
RESTful API Security. The security operations teams the ability to interpret the state of the API protocol. They can
of popular API services can discover attacks against their only generate requests for individual API endpoints, limiting
services. The Facebook team [10] identified that attackers the scope to test each endpoint separately and preventing the
could exploit authorization vulnerabilities in a specific API detection of vulnerabilities in API services composed of mul-
endpoint, allowing third-party applications to access users’ tiple endpoints. NAUTILUS [50] incorporates comment poli-
private photos. The Twitter team [11] discovered that at- cies into API specifications to handle operation relationships
tackers could utilize a specific API endpoint that associates and parameter generation, resulting in meaningful sequences
phone numbers with account names to systematically launch of operations and the discovery of corresponding API security
widespread attacks and infer the target users’ phone numbers vulnerabilities. However, due to its oversight of vulnerabil-
and account names. Researchers note that there are typically ity characteristics, NAUTILUS focuses on a limited range of
two categories of bugs in RESTful APIs, including that result vulnerability types, mainly handling injection vulnerabilities.
in service unavailability and those related to web security [50].
The former is often caused by syntactically incorrect parame- 8 Conclusion
ters in API request data, leading to the backend being unable
to process them and resulting in HTTP 500 errors. Most of In summary, we propose VOAPI2 , a novel inspection frame-
these bugs typically indicate that the server is unable to han- work, to apply a vulnerability-oriented strategy to inspect
dle the current request properly. The latter category arises due RESTful APIs. Based on the insight that the type of vulnera-
to similarities between the backend processing of RESTful bility hidden in an API interface is strongly associated with
APIs and traditional web services, resulting in web bugs when its functionality, VOAPI2 can directly expose vulnerabilities
requests are mishandled. in RESTful APIs. We first track commonly used strings as
RESTful Service Testing Methods. Regarding the poten- keywords to identify APIs’ functionality. Then, we generate a
tial bugs in RESTful services, researchers have proposed stateful and suitable request sequence to inspect the candidate
numerous automated testing methods that can generate se- API function within a targeted payload. Finally, we verify
quence requests, assess various endpoints of the API, and whether vulnerabilities exist or not through feedback-based
determine if there are any issues based on the service’s re- testing. Our evaluation result shows that VOAPI2 demon-
sponses. RESTler [12] is a stateful API black-box testing tech- strates higher efficiency and effectiveness in bug discovery
nique that generates stateful test cases by inferring producer- than state-of-the-art tools, including RESTful API testing and
consumer dependencies between different API endpoints, al- penetration methods.

USENIX Association 33rd USENIX Security Symposium 753

Acknowledgments [14] Stefan Karlsson, Adnan Čaušević, and Daniel Sund-
mark. QuickREST: property-based test generation of
We thank the anonymous reviewers of this work for their OpenAPI-described RESTful APIs. In International
helpful feedback. This research was supported, in part, by Conference on Software Testing, Validation and Verifi-
National Natural Science Foundation of China under Grant cation (ICST), pages 131–141, 2020.
No. 62372297, Science and Technology Commission of
Shanghai Municipality Research Program under Grant No. [15] Alberto Martin-Lopez, Sergio Segura, and Antonio Ruiz-
20511102002, National Radio and Television Administration Cortés. RESTest: black-box constraint-based testing of
Laboratory Program (TXX20220001ZSB002). RESTful web APIs. In International Conference on
Service-Oriented Computing (ICSOC), pages 459–475,
2020.
References
[16] Emanuele Viglianisi, Michael Dallago, and Mariano
[1] Amazon. AWS. https://aws.amazon.com/. Ceccato. RestTestGen: automated black-box testing
of RESTful APIs. In International Conference on Soft-
[2] Microsoft. Azure. https://azure.microsoft.com/ ware Testing, Validation and Verification (ICST), pages
en-us/. 142–152, 2020.

[3] Jellyfin. https://jellyfin.org/. [17] Tobias Fertig and Peter Braun. Model-driven testing of
RESTful APIs. In International Conference on World
[4] Casdoor. https://casdoor.org/. Wide Web: Companion, pages 1497–1502, 2015.

[5] Appwrite. https://appwrite.io/. [18] Andrea Arcuri. RESTful API automated test case gener-
ation with EvoMaster. ACM Transactions on Software
[6] Microcks. https://microcks.io/. Engineering and Methodology (TOSEM), 28(1):1–37,
2019.
[7] CVE-2021-3044. https://nvd.nist.gov/vuln/de
tail/CVE-2021-3044. [19] Andrea Arcuri. Automated blackbox and whitebox test-
ing of RESTful APIs with EvoMaster. IEEE Software,
[8] CVE-2019-12643. https://nvd.nist.gov/vuln/de 2020.
tail/CVE-2019-12643.
[20] Andrea Arcuri and Juan P Galeotti. Handling SQL
[9] Representational state transfer. https://en.wikiped databases in automated system test generation. ACM
ia.org/wiki/Representational_state_transfe Transactions on Software Engineering and Methodology
r. (TOSEM), 29(4):1–31, 2020.

[10] Tomer Bar. Notifying our Developer Ecosystem about a [21] Andrea Arcuri and Juan P Galeotti. Enhancing search-
Photo API Bug. https://developers.facebook.co based testing with testability transformations for existing
m/blog/post/2018/12/14/notifying-our-devel APIs. ACM Transactions on Software Engineering and
oper-ecosystem-about-a-photo-api-bug/, 2018. Methodology (TOSEM), 31(1):1–34, 2021.

[11] Twitter. An Incident Impacting Your Account Identity. [22] Vaggelis Atlidakis, Roxana Geambasu, Patrice Gode-
https://privacy.twitter.com/en/blog/2020/a froid, Marina Polishchuk, and Baishakhi Ray. Pythia:
n-incident-impacting-your-account-identity, Grammar-Based Fuzzing of REST APIs with Coverage-
2020. guided Feedback and Learning-based Mutations. arXiv
preprint arXiv:2005.11498 [cs.SE], 2020.
[12] Vaggelis Atlidakis, Patrice Godefroid, and Marina Pol-
[23] S. T. Liu. Coverage guided fuzzing in python-based
ishchuk. RESTler: Stateful REST API Fuzzing. In
web server. Master’s thesis, Institute of Computer Sci-
2019 IEEE/ACM 41st International Conference on Soft-
ence and Engineering, National Chiao Tung University,
ware Engineering (ICSE), pages 748–758, Montreal,
Hsinchu, Taiwan, 2019.
QC, Canada, 2019. IEEE.
[24] Runtime application self-protection. https://en.wik
[13] Hamza Ed-Douibi, Javier Luis Cánovas Izquierdo, and ipedia.org/wiki/Runtime_application_self-p
Jordi Cabot. Automatic generation of test cases for rotection.
REST APIs: a specification-based approach. In Interna-
tional Enterprise Distributed Object Computing Confer- [25] Huayao Wu, Lixin Xu, Xintao Niu, and Changhai Nie.
ence, pages 181–190, 2018. Combinatorial testing of RESTful APIs. In Proceedings

754 33rd USENIX Security Symposium USENIX Association

 of the 44th International Conference on Software En- [42] Scrapy. https://scrapy.org/.
gineering (ICSE ’22), pages 426–437, New York, NY,
USA, 2022. Association for Computing Machinery. [43] Beautiful Soup. https://www.crummy.com/softwar
e/BeautifulSoup/.
[26] AppSpider. https://www.rapid7.com/products/
appspider. [44] OWASP Zed Attack Proxy. https://www.zaproxy.
org/.
[27] Qualys Web Application Scanning (WAS). https:
[45] ASTRA. https://github.com/flipkart-incubat
//www.qualys.com/apps/web-app-scanning/.
or/Astra.
[28] TnT-Fuzzer. https://github.com/Teebytes/Tn
[46] Chenyang Lyu, Jiacheng Xu, Shouling Ji, Xuhong
T-Fuzzer.
Zhang, Qinying Wang, Binbin Zhao, Gaoning Pan, Wei
[29] APIFuzzer. https://github.com/KissPeter/API Cao, Peng Chen, and Raheem Beyah. Miner: A hy-
Fuzzer. brid data-driven approach for rest api fuzzing. In 32th
USENIX Security Symposium (USENIX Security 23),
[30] Patrice Godefroid, Daniel Lehmann, and Marina Pol- 2023.
ishchuk. Differential regression testing for REST APIs.
In Proceedings of the 29th ACM SIGSOFT International [47] Docker Hub. https://hub.docker.com/.
Symposium on Software Testing and Analysis (ISSTA
[48] Rbaskets. https://rbaskets.in/.
2020), pages 312–323, New York, NY, USA, 2020. As-
sociation for Computing Machinery. [49] Gitea. https://gitea.io/.
[31] Y. Luo et al. RestSep: Towards a Test-Oriented Privi- [50] Deng Gelei, Zhang Zhiyi, Li Yuekang, Liu Yi, Zhang
lege Partitioning Approach for RESTful APIs. In 2017 Tianwei, Liu Yang, Yu Guo, and Wang Dongjin. Nau-
IEEE International Conference on Web Services (ICWS), tilus: Automated restful api vulnerability detection. In
pages 548–555, Honolulu, HI, USA, 2017. 32th USENIX Security Symposium (USENIX Security
23), 2023.
[32] Y. Luo, H. Zhou, Q. Shen, A. Ruan, and Z. Wu. RestPL:
Towards a Request-Oriented Policy Language for Ar-
bitrary RESTful APIs. In 2016 IEEE International
Conference on Web Services (ICWS), pages 666–671,
San Francisco, CA, USA, 2016.

[33] Swagger. https://swagger.io/.

[34] Vaggelis Atlidakis, Patrice Godefroid, and Marina Pol-

ishchuk. Checking Security Properties of Cloud Service
REST APIs. In 13th International Conference on Soft-
ware Testing, Validation and Verification (ICST), pages
387–397, Porto, Portugal, 2020. IEEE.

[35] CVE-2021-29490. https://nvd.nist.gov/vuln/de

tail/CVE-2021-29490.

[36] Gitlab. https://gitlab.com/.

[37] Bing Maps. https://www.bing.com/maps.

[38] Cybersecurity Vulnerability Landscape Report. https:

//www.h3c.com/cn/d_202303/1796824_30003_0.
htm.

[39] CVE. https://cve.mitre.org/.

[40] NVD. https://nvd.nist.gov/.

[41] Bernardo Damele A. G. and Miroslav Stampar. sqlmap.

https://github.com/sqlmapproject/sqlmap.

USENIX Association 33rd USENIX Security Symposium 755