See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/377219740

A Case Study with a Tool (Selenium) and Framework (Cucumber) for Testing
Web Environments

Conference Paper · December 2023

CITATIONS READS

0 526

2 authors:

Selcen Fethiye Mersinli Muhammet Sinan Başarslan

1 PUBLICATION 0 CITATIONS
Istanbul Medeniyet Universitesi
37 PUBLICATIONS 218 CITATIONS
SEE PROFILE
SEE PROFILE

All content following this page was uploaded by Muhammet Sinan Başarslan on 07 January 2024.

The user has requested enhancement of the downloaded file.

 Editors

Dr. Ali Etem Gürel

Dr. Ümit Ağbulut
Dr. Gökhan Yıldız

ISBN

978-605-71407-8-4

Düzce, Türkiye

vi
 11. Uluslararası Meslek Yüksekokulları Sempozyumu
07-09 Eylül 2023, Düzce/Türkiye

11th International Vocational Schools Symposium

www.umyos2023.duzce.edu.tr 07-09 September 2023, Düzce/Türkiye
ISBN: 978-605-71407-8-4

A Case Study with a Tool (Selenium) and Framework (Cucumber) for Testing Web
Environments
Selcen Fethiye Mersinlia, Muhammet Sinan Başarslana,*
aIstanbul Medeniyet University, Faculty of Engineering And Natural Sciences, Department of Computer Engineering,
Istanbul/Türkiye.
* Corresponding Author: muhammet.basarslan@medeniyet.edu.tr

ABSTRACT
Testing is a phase of the software development process that has existed since the beginning of the software
development process and has gained great importance today. In the early days of the software world, testing
was seen as a simple and undervalued step. As user experience declined, satisfaction decreased and
problems increased, testing became more important. As the variety of problems that arise day by day and
the expectations of users increase, software testing has been divided into different types. In order to
perform these tests, frameworks have started to be developed. Recently, the concept of "Quality Assurance"
has emerged. Within the scope of the study, the purpose of software testing, its benefits, commonly used
types, techniques, tools and frameworks used for its implementation are explained. In the application part,
a real estate site written by an undergraduate student and a real estate site that has been put into use were
automation tested using Selenium tool and Cucumber framework. As can be seen in the literature review,
there are almost no articles on software testing, especially in Turkish. In order to contribute to this point,
an automation test application has been developed by conducting a comprehensive literature review. In
future studies, a more comprehensive testing process from platform independence to security of a software
will be realized.
Keywords: Cucumber; Quality Assurance, Automation; Seleniu, Software Testing.

I. INTRODUCTION
Developers test and verify the system implementation early in the process to make sure it conforms to
the requirements and design. Since they are very familiar with the application, there is a high probability
that they will not notice bugs while testing. Problems often arise when applications that have been tested
by developers are released to the user. To overcome this problem, the concept of Quality Assurance has
emerged. Today, the concept of "Quality Assurance" has become very popular. It is an important factor in
preventing errors in the application that is released to the user, in detecting defects and in the rapid
progress of the operation. Early detection of defects saves us time and cost. There are different stages for
the control of quality assurance.
The most important phase is the "Test" phase. With the emergence of quality assurance, testing has taken
its place in the software cycle. Testing is primarily done to determine whether a software can adequately
meet the desired and expected features. In this way, bugs in the software can be detected and corrected. If
the objectives for the correction of errors are listed,
• Detection and prevention of errors,
• Checking that the application works as intended,
• Checking whether requirements are met,
• Ensuring compliance with standards,
• Building customer trust,
To increase customer satisfaction.
Testing is a wide-ranging process that can be done manually or automatically. The application is
subjected to multiple and different tests from the design phase until it is delivered to the user. The ideal
process is "analysis, design, test preparation, coding, dynamic testing, test finalization, product" [1].
Depending on the type of software to be tested, test types, techniques and the way these techniques are
applied vary.

250
 II. LITERATURE REVIEW
In this section, similar academic studies found in the last 10 years by searching indexes such as IEEE
digital Xplorer, WOS, Google Scholar are described. As a result of this search, not enough resources were
found in this field because quality assurance is actually a very new concept and software testing was not so
valuable in the past. There have been studies in this field in recent years. However, there are not enough
studies especially in Turkey and in Turkish.
In their study, Thummalapenta et al. explain what test automation is, why and how it is done [2]. Wang
and Du explained frameworks such as Selenium and JMeter and compared these two frameworks [3]. Jamil
et al. provide information about high quality assurance and testing techniques [4]. Jamil et al. also gave
information about Behavior-driven development (BDD) and Cucumber framework [4]. Laplante et al.
calculate the cost of software testing and explain why we do test automation and what we gain from it [5].
Bohnet and Meszaros answered the questions of what is test-driven migration and what are its effects [6]..
Lucca and Penta explained the techniques used to resolve inconsistencies in advanced web applications [7].
Freeman in his work has given his experiences about the types of software and some of them in the world
of information technology [8]. Garousi and Elberzhager describe test automation techniques used to enable
test engineers to work effectively and efficiently [9]. Polo et al. discuss the purpose of software testing, why
test automation is needed and the technologies required for test automation [10]. In their study, Weyuker
and Vokolos provide a case study for an application-based performance testing approach [11]. Sawant et al.
briefly summarize the types of tests and techniques that can be used in the testing phase of software [12].
Garoudı et al. conducted a survey on the importance of software testing and how much knowledge is
available on this subject and shared the results of this study [13]. Hooda and Chhılar, in their study, answer
the questions of how the software testing process proceeds, what kind of testing techniques exist and where
they are used [14]. Küçüksille et al. answer the questions of why we use a test-driven process, what are its
advantages and disadvantages, and which frameworks are used [15]. Sarı et al. shared surveys conducted
to answer questions such as whether public organizations are aware of the importance of software testing
and how much knowledge they have [16].
These studies answer questions such as what is the software testing process, what kind of testing
techniques are there. It was observed that application-supported studies were missing. In this study,
automation testing was performed.

III. MATERIALS AND METHOD

This section provides information about what software testing is; how the testing process proceeds; how
test cases are written; the most commonly used test types; test automation; Selenium; Cucumber.
A. Software Testing
Software testing is as important and almost as costly as the software phase of a project. Although this is a
minus for us, the benefits of testing are very high. There are pros such as predicting bugs, detecting bugs,
ensuring user satisfaction, proceeding with guarantees and saving time. Test codes are unique to each
application.
There are seven basic principles of software testing [17]:
1. The main purpose of testing is to detect defects.
2. One hundred percent testing is impossible. It is not possible to test all inputs separately when it
takes days to test even one input.
3. The testing process should start with the software. It adapts to the software more easily, saves
time, early detection of bugs reduces cost.
4. Bugs usually cover small parts. Major bugs are very rare.
5. Antibiotic Resistance. Testing the product on the same scenario all the time prevents us from
seeing bugs. By testing different scenarios, we are more likely to find the majority of bugs.
6. The scope of the test is independent. Each product has different characteristics. Therefore, a test
cannot be applied to more than one product. For example, the test of an e-commerce site is not
the same as the test of a banking site.
7. A successful test does not mean that the product is perfect. There may necessarily be overlooked
errors. The aim is to minimize the error.
An application is tested before it goes live. It can then be tested after it goes live, with features added and
updates made. Testing is done with little data. Testing with millions of data is both very difficult and not
logical. In addition, as data is added to a live site, there will be changes in the code, which requires updating
certain parts of the test each time. Especially in the testing of dynamic sites, it makes the most sense to test
with little data before starting to use it. What is more important in testing is that the sites provide basic

251
functions. Sites whose basic, that is, the most priority functions work correctly are actually suitable for use
in a way. Detailed testing increases quality assurance.
B. Test Process
This process consists of four sub-steps. The first step is the Test Preparation Process. In this process,
planning is done. A timeline is created and tasks are allocated. The scope of the test is determined, that is, it
is decided how much of the product the test will address. This value is called code coverage. Resources are
determined. A computer, a mobile device, an operator line, a browser that needs to be installed are all
examples of resources. Issues such as which language, which ide, which framework will be used are decided.
After such basic preparations are completed, the SRS document written by the analyst is received. Using the
SRS document, tests are designed and scenarios are written. The second step is the SRS Document. This
document is the requirements document prepared by analysts. In this step, the test environment is set up.
The project to be tested is installed in this environment, making sure that it is standing correctly. Data is set
up. Everything is brought up to date. The third step is the dynamic Test Process step. This is the stage where
the prepared test scenarios are run. The results are reported completely and clearly in plain language.
Errors are identified and resolved by the relevant people. Tests at this stage can be of different types. The
final step is the finalization of the test. If the product is successful, that is, if the criteria are successful at an
acceptable level, the test is terminated [17].
C. Test Scenarios Writing
Test cases are written by utilizing the SRS document. Expected and actual outputs are prepared according
to the content. Creativity is very important in scenario writing. It should not be too complicated, but it
should be looked at from different angles.
If we list what should be in a test case [17]:
• Scenario Name: Be clear and concise
• Scenario Type: Depends on what you are testing
• Scenario Number: Scenarios must be unique
• Scenario Description: The same thing should be understood by everyone who reads it
• Software Preparation: Software to be used while performing the test
• Hardware Preparation: Additional equipment required during the test
• Priority: Each scenario has a different importance and scenarios are run according to their
importance
• Preconditions: Conditions that must be fulfilled before the test is performed
• Test Steps: The actions to be taken must be clearly and concisely stated
• Test Data: Data to be used for inputs
• Expected Outcomes: Outputs that are expected to emerge with the realization of the test steps
• Corresponding SRS Clause in the Test Case: Information on which requirement it corresponds to in
the SRS document
• Status (Pass / Fail / Block): Comparison. Are expected and realized the same?
is in the form.
D. Types of Test
Test types are divided into functional and non-functional test types.
1)Functional ones
If functional test types are listed [17];
• API: Consists of unit testing, integration testing, end-to-end testing and UI testing.
• Unit: It is the testing of the smallest parts. It is done by software developers. The aim is early
detection of bugs. It is done throughout the software process.
• Smoke: The purpose of the test is to check whether the software can pass the small tests and
continue with the big tests. Testers do it.
• Sanity: It is done after a bug to check if the bug has been fixed.
• Integration: Tests how well different units work in harmony with each other. It is checked that
the system can work correctly as a whole.
• Regression: Tests the effect of changes on the system and also whether new errors occur.
• User Acceptance: There are two subtypes, Alpha and Beta. Alpha testing is done by team
members. It is the last test before the product is presented to the user. Beta testing is done by the
end user. The product is approved by the end user.
• Interface: It is actually a user test. The user interface is tested either manually or automated.
It is in the form.

252
 2) Non-Functional
If the non-functional test types are listed [17];
• Reliability Reliability requirements are tested.
• Usability: The interface is tested. Ease of use and understandability are tested.
• Portability: The transferability of the application to different environments is tested.
• Performance tests are divided into 4;
• Load: It aims to see the optimum performance of the system. It tests the system by giving the
system the load it will actually encounter.
• Stress: Actually, reliability is tested here. The system is tested to see how much it can withstand
by exceeding its capacity. Durability is measured. The aim of this test is to go beyond the limits.
• Volume: It is a data-oriented test method. Testing is done by increasing the data in the database
and possible data loss is prevented.
• Scalability: If the application grows, how it will meet this situation is tested. Demand may
increase, data input may increase. In other words, we look at what effect the increase in volume
and size will have.
is in the form.
Testing is basically divided into black box and white box. The aim is to distinguish between testing done
by the developer and testing done by the test engineer. Black box is the type of testing that deals only with
inputs and outputs without knowledge of the internal structure of the application. In the black box, the goal
is quality control. White box testing, in contrast to black box testing, is more concerned with content. In the
white box everything is open, everything is known about the code/software, so this type of testing is actually
done by developers, not testers. There are many more types of tests. However, in the scope of this study,
the most commonly used types are included.
E. Test Automation
We need to automate tests that are repetitive and difficult to do manually. In fact, the goal is to reduce the
burden and achieve quality in the long run. For this, we use software tools. The most popular ones are
Selenium for web and Appium for mobile. This activity is called test automation. Almost most companies
have test automation infrastructure. Those who develop test automation are actually referred to as test
developers [18].
Test automation plays a crucial role in increasing the success of software projects. But sometimes it may
not be necessary. If test automation is necessary, it is critical to choose the right tools for it. There are many
test automation tools available. First of all, if we list some items to understand that test automation is
necessary:
• Too many repetitive scenarios,
• Frequent need for regression testing,
• A wide variety of inputs, i.e. a large number of users,
• Manual testing is inadequate and unreliable,
• Interfaces are now stable.
There are many factors when choosing a test automation tool. One of the most important of these is the
requirements of the projects. The tools you have used before can help you in this regard. If some of the
criteria that are important during tool selection are listed (These criteria should be examined considering
that tests are written) [18]:
• It should be simple to operate for non-technical users and teammates when needed.
• Results should be reportable in different formats.
• Be suitable for BDD and Data Driven Testing (DDT)
• Must be suitable for parallel operation
• Support different browsers
• A screenshot should be taken in case of failure
• Should be easy to develop
• Must be easy to maintain
• Support multiple platforms
• If your project will use CI, your test tool must be compatible with the CI/CD tool
• Cost calculation should be done
• Locator structure must be appropriate.
Considering these factors according to their importance, the test tool can be selected. Thus, the most
suitable and most efficient tool for our project will be found.
If we list the ways to achieve success in test automation [18]:
• A well-thought-out test plan

253
 • Top management support
• Appointing people in the right way
• Prioritization of quality perception
• Software developers attach importance to this process
• Use of the TDD method
• Correct planning of the number of days
• The test environment is different from the software environment
• Selection of the appropriate test type according to the determination of the project type
• Proper risk management
• Scenarios are close to reality
F. Test Framework and Tool
Selenium, the tool used in the study, and cucumber, the framework, will be described in this section.
1) Selenium
Developed by Jason Huggins at Thought Works in 2004 as a tool for testing, Selenium is an open-source
project. Since it is open source, you do not have a contact person to get technical support. Therefore, tool
mastery and technical knowledge are very important. It is not a single tool, it consists of multiple packages.
It is also known as a software test automation library. It supports browser automation. It enables testing
without a scripting language. In addition to having its own id, testing can be done with Selenium on other
ids. It provides users with features such as WebDriver, Grid, IDE and RC [19].
WebDriver is a framework for writing and running test cases. It works with many browsers. Downloading
the WebDriver API is sufficient for testing, no additional work is required. Grid supports parallel execution.
Parallelism on different servers and different browsers. Development is still ongoing. Selenium runs its tests
using Hub and Node structure. The IDE is a plugin. It allows testing through the browser without the need
for anything additional. It allows the test to be automated. It has multiple language support. RC is a
framework that supports different programming languages.
The most important point of testing with Selenium is the path part. Locators are used to perform functions
such as clicking, selecting, typing to reach elements and objects. Locators are used to locate elements. The
most commonly used method is the XPath method [20]. There are also CSS Selector, class, id, name methods.
You can either write the path information of an element manually by reading its HTML code, or you can get
the path information with the method you want by finding the element through the html code of the page
and copying it with a right click. It is more useful to write it manually. The fastest way to access it is by ID
or class if it can be distinguished [20].
2) Cucumber
It is a BDD-oriented testing tool. Although it defaults to English, it supports many languages, including
Turkish. It uses the Gherkin language. Gherkin is a plain text language, designed to be understood by people
who are not software developers. It allows you to create tests without any software knowledge. It uses
expressions that almost everyone can understand [21].
If we examine the basic structure of Cucumber, the basic files that Cucumber test code should have are as
follows:POM.XML file. POM (Project Object Model) is actually an information file about the project. It
contains build, dependencies, packaging, instruction information.
1. Feature files. These files contain the scripts. Scenarios are written in a way that is completely
understandable to everyone in everyday language. It uses feature as its extension.
2. Step Definition files. This is where the scenarios we wrote in the feature files are defined.
Now we start to return to the software language.
3. Page files. Each page contains its own objects, web elements and functions.
4. Hook file. Contains preconditions and postconditions. It saves from code redundancy. Allows
to manage code flow. Always works. It is an invisible step.
5. Runner file. It allows tests to run. It contains the type of test, the starting point.
The biggest advantage of the Cucumber framework is its structure. Finding what you are looking for in
the code, making new additions and changes to the test becomes extra easy. Also, the use of Gherkin
language in the scenarios allows users or other team members to understand what we are doing.
G. Application
In this section, automated testing of a live real estate website and a real estate website written by an
undergraduate student using Selenium tool and Cucumber framework will be explained.

254
 Fig. 1 Automation Test Results in a Challenge with Selenium Tool and Cucumber Framework

As shown in Figure 1, the scenarios were run and successful results were obtained. Automation can be
performed either with Selenium alone or in addition to it by using frameworks such as Cucumber. The test
automation process is described in Section 3.5, the test tool Selenium and the cucumber framework in
Section 3.6.

IV. CONCLUSION AND DISCUSSION

In our study, it is mentioned that testing is as important as the development phase in the software cycle.
In fact, detailed information is given on topics such as what the ideal testing process should be and what we
gain when it is followed, what is the purpose of testing. As a conclusion drawn from the study, while the
main purpose of testing is to provide quality assurance, the ideal process for a successful project is to start
with the design phase. In addition, the stage at which testing starts is as important as the projects to which
it is applied. There are general rules for conducting tests. Testing should be done before projects go live.
Testing is run with little and sufficient input. Live projects can be dynamic or have millions of data. This
makes testing difficult or impossible. Of course, these alone are not enough. It is necessary to make use of
tools and frameworks to make testing more efficient and to benefit in every sense. The most common and
most important tool for web testing is Selenium and the framework is Cucumber.

REFERENCES
[1] A. Betül Karagöz and F. Molu, “Yazılım Test Sürecinde Hata Yönetimi ve Finans Sektöründen Örnek Uygulama.”
[2] S. Thummalapenta, S. Sinha, N. Singhania, and S. Chandra, “Automating test automation,” in 2012 34th International
Conference on Software Engineering (ICSE), IEEE, Jun. 2012, pp. 881–891. doi: 10.1109/ICSE.2012.6227131.
[3] F. Wang and W. Du, “A Test Automation Framework Based on WEB,” in 2012 IEEE/ACIS 11th International Conference on
Computer and Information Science, IEEE, May 2012, pp. 683–687. doi: 10.1109/ICIS.2012.21.
[4] M. A. Jamil, M. Arif, N. S. A. Abubakar, and A. Ahmad, “Software Testing Techniques: A Literature Review,” in 2016 6th
International Conference on Information and Communication Technology for The Muslim World (ICT4M), IEEE, Nov. 2016,
pp. 177–182. doi: 10.1109/ICT4M.2016.045.
[5] P. Laplante et al., “Software Test Automation,” Advances in Software Engineering, vol. 2010, pp. 1–2, Feb. 2010, doi:
10.1155/2010/163746.
[6] R. Bohnet and G. Meszaros, “Test-driven porting,” in Agile Development Conference (ADC’05), IEEE Comput. Soc, pp. 259–
266. doi: 10.1109/ADC.2005.46.
[7] G. A. Di Lucca and M. Di Penta, “Considering browser interaction in Web application testing,” in Fifth IEEE International
Workshop on Web Site Evolution, 2003. Theme: Architecture. Proceedings., IEEE Comput. Soc, pp. 74–81. doi:
10.1109/WSE.2003.1234011.
[8] H. Freeman, “Software testing,” IEEE Instrum Meas Mag, vol. 5, no. 3, pp. 48–50, Sep. 2002, doi: 10.1109/MIM.2002.1028373.
[9] V. Garousi and F. Elberzhager, “Test Automation: Not Just for Test Execution,” IEEE Softw, vol. 34, no. 2, pp. 90–96, Mar.
2017, doi: 10.1109/MS.2017.34.
[10] M. Polo, P. Reales, M. Piattini, and C. Ebert, “Test Automation,” IEEE Softw, vol. 30, no. 1, pp. 84–89, Jan. 2013, doi:
10.1109/MS.2013.15.
[11] E. J. Weyuker and F. I. Vokolos, “Experience with performance testing of software systems: issues, an approach, and case
study,” IEEE Transactions on Software Engineering, vol. 26, no. 12, pp. 1147–1156, 2000, doi: 10.1109/32.888628.
[12] A. A. Sawant, P. H. Bari, and P. M. Chawan, “Software Testing Techniques and Strategies,” vol. 2, pp. 980–986, [Online].
Available: www.ijera.com
[13] V. Garousi, A. Coşkunçay, and A. Betin-Can, “A survey of software testing practices in Turkey", (title in Turkish: "Türkiye’deki
yazılım test uygulamaları anketi) A Survey on Common Practices for Microservice-based Software Architectures View
project Systematic reviews in software engineering View project.” [Online]. Available:
https://www.researchgate.net/publication/288374442
[14] I. Hooda, R. Scholar, and R. Singh Chhillar, “Software Test Process, Testing Types and Techniques,” 2015.
[15] E. U. Küçüksđlle et al., “TEST GÜDÜMLÜ YAZILIM GELĐŞTĐRME SÜRECĐ VE KULLANILAN ÇERÇEVELER.”
[16] M. Sari, M. Özbek, and A. Kurt, “Türkiye’deki Kamu Kurumları Yazılım Test ve Kalite Faaliyetleri Anketi, Değerlendirilmesi
ve Öneriler.”
[17] B. Muhammed Maruf ÖZTÜRK, T. Danışmanı, and D. Ahmet ZENGİN, “Uzaktan Eğitimde Ölçme Değerlendirme Sistemi
Tasarimi Ve Yazilim Test Teknikleri İle Performans Analizi Yüksek Lisans Tezi Enstitü Anabilim Dalı : Bilgisayar Ve Bilişim
Mühendisliği Enstitü Bilim Dalı : Bilgisayar Mühendisliği,” 2012.
[18] U. Özkan, H. Sözer, and Ö. Üniversitesi, “Web Uygulamaları için Model Bazlı Test Süreci Otomasyonu.” [Online]. Available:
http://graphml.graphdrawing.org/
[19] A. Bruns, A. Kornstadt, and D. Wichmann, “Web Application Tests with Selenium,” IEEE Softw, vol. 26, no. 5, pp. 88–91, Sep.
2009, doi: 10.1109/MS.2009.144.

255
View publication stats

[20] T. Grust, “Accelerating XPath location steps,” in Proceedings of the 2002 ACM SIGMOD international conference on
Management of data, New York, NY, USA: ACM, Jun. 2002, pp. 109–120. doi: 10.1145/564691.564705.
[21] N. Li, A. Escalona, and T. Kamal, “Skyfire: Model-Based Testing with Cucumber,” in 2016 IEEE International Conference on
Software Testing, Verification and Validation (ICST), IEEE, Apr. 2016, pp. 393–400. doi: 10.1109/ICST.2016.41.

256