10 Test Automation

Frameworks for
Cross-Browser Testing
A Comparison Guide for 2017
Table of Contents

Introduction

Market overview
Rationale behind using JavaScript

How to select the correct test framework?

Organizational fitness
Technical fit

High-level framework descriptions

Comparison table

Summary

Appendix
Recommended resources

2
Introduction
The goal of this document is to offer web development
teams a set of selection criteria for choosing the correct
web testing frameworks for each project. The paper
provides a market overview of cross-browser testing
tools, including a summary of the ten leading web
testing frameworks available on the market.
Key Findings:
1. The open-source community is very active and offers great code
samples, innovation and support. New tools are introduced constantly
and it’s worth tracking tool adoption, contributions and features to
stay ahead of the market.
2. JavaScript is the standard development language in web testing
3. S elenium is the core API for the leading web test automation
frameworks
4. B
 DD is a clear trend in web test automation and is supported by the
majority of tools
5. Protractor, WebDriver.IO, and NightwatchJS are the leading E2E web
testing frameworks
6. For unit testing purposes, PhantomJS, CasperJS and JSDOM are the
most common frameworks
7. T
 o choose the right test framework, evaluate the six organizational
fit and six technical fit criteria.

3
JavaScript is The Leading
Web Development Language
Before looking at the landscape, it is important to understand that in the
desktop web market, JavaScript is the leading development language for
both developers and testers. Many leading web development frameworks
in the market are leveraging JavaScript. They include frameworks such as
AngularJS, React, Aurelia, Vue.JS, and others.

Selenium WebDriver is the market leader with respect to what traditional testing tools
in adoption for testing web applications; provide (e.g., object repository).
however, it is being used in conjunction
Non Selenium WebDriver frameworks are
with a number of frameworks designed to
out of scope.
boost productivity and it does have gaps

4
Market Overview
Selenium and Protractor are seeing the largest number of downloads (Fig. 1).
However, there are other parameters and solutions to consider as part of your
tool selection. Sometimes the best solution is to build a tool stack strategy
that includes a combination of frameworks, each serving a different purpose.

The tools that you select for your testing activities should
support specific practices such as Behavior Driven Development
(BDD), or back-end services, or other JavaScript frameworks
(such as Node.JS). What we see in the market is that various tools
best fit a specific practice, such as BDD, which can be technically
supported through different capabilities. The goal is choosing
tools matching both organizational and technical criteria.

Figure 1: Cross-browser testing tools download stats (Source: NPM Trends)

5
Here are the top 10 test frameworks (in no particular order). There are at least a dozen more
frameworks outside of these top ten. We include Selenium in this list but do not compare it to
the rest, simply because Selenium is the basis for most of these solutions.

As can be seen in Figure 1, Protractor, from an E2E testing perspective, has the largest number of
downloads, while JSDOM, from a unit testing perspective, is the most downloaded. These figures
alone are not sufficient to drive a concrete decision but they can help to understand trends in the
market. As we will see in the criteria section, a good decision needs to be based on a mixture of
technical considerations and organizational fit.

Buster.JS Casper.JS Chimp

Often, teams find greatest

benefit using a set of test
frameworks rather than
CodeceptJS JSDom
using only one.

Nightwatch.JS PhantomJS Protractor

Robot Webdriver.IO

6
JavaScript is King
JavaScript is, by far, the leading dev language
for web. Most web development frameworks
are based on JavaScript; it’s not surprising that
the associated test tools are also written in
and support this language.

In a recently conducted survey, JavaScript was

leading the preferred programming language
of 62% of respondents.

Among the key benefits of using JavaScript

for testing, these are the clearest:
1. O
 pen-source is free
2. It’s modular
3. I t’s backed by an active and
vibrant community
4. Y
 our client-server is written in JavaScript
why not the tests? Figure 2: Preferred programing language survey (Source: Stack Overflow)
a. T
 his assumes that the test engineer
has proper test automation skills

7
How to Select the Correct Test
Framework for Your Needs?
When choosing a testing framework, consider both current organizational
and technical fit.
The following are 6 organizational fit –– What are the budget constraints? Even Do you need to mimic different locations,
considerations and 6 technical-fit-related though most web testing frameworks are etc?
considerations to help make the best choice open-source and free, people aren’t; these
–– What are the platform coverage
for your project tools test against real platforms like desktop
requirements for the project? How many
browsers and mobile devices, and setting up
Organizational Fit a lab requires budget space.
browsers? Is mobile also part of the mix?
5. Test Types / Practices
As part of the framework selection process, –– What is the time frame for the project
each team needs to assess the following 6 –– What are the quality criteria for the
and its release cadence? This will impact
considerations and bake them in to the overall project, e.g., which testing types are
tool selection due to fast ramp-up, available
decision. The overall SDLC of a project is driven required for the project? This will impact
support and documentation, and execution
by existing and defined practices (BDD, TDD, the decision-making for the various tools
speed (e.g. going with headless browser vs.
waterfall etc.) and skills; therefore, the tool that (plugins, integrations)
real browser, etc.) Also, how is reusability
is selected should suit them. being leveraged? –– How good is the tool at building a robust
1. Project Complexity object repository and identifying all object
3. E xisting Tech Stack (Devops, COE, etc.)
locators (Angular objects, React etc.)?
–– How complex is the website under test? –– What kind of development framework is
The more complex the tool is, the more my web site built on? What other tools does –– Is the tool able to support things like visual
dependent your test automation architect the web site integrate with? What current CI validation and testing of responsive web?
will be upon APIs and extensions to be able tools and test case/defect management –– Which execution engine and test runners
to cover all functionalities and to manage tools are being used within R&D? are supported with the test framework?
test assets.
4. T
 est Environment 6. Reporting
2. Resources –– What are the testing environment –– What kinds of reports and dashboards
–– Do you have within the team the right requirements? Do you need to cover should be provided to developers, testers,
skills and resources to accomplish your performance metrics? Do you need to and management?
objectives? Do you need to hire and train capture network traffic data per execution?
new resources for the selected framework?

8
Technical Fit
When the organizational fit items are well defined, it is time to match them against the
relevant frameworks and assess their technical fit to the teams involved. Below, we define
6 technical fit considerations.

1. Community Size, Support 2. SDLC Process Fit (Integration, teams when selecting a framework. Lastly, test
and Documentation Plugins, Dev Methodology fit) automation is an ongoing project and, as with
code, it requires continuous support; therefore,
The tool should be well documented, have The tool should support the SDLC practices
it is important to build the foundation on a
an active community for resolving bugs, for your web project, whether BDD or
framework that can grow with your project and
and commercial support. other methods are utilized. In addition, it
add more capabilities as your project evolves.
While there are various frameworks in the should be easily extended and integrate
open-source community, not all of them into various other tools (CI, IDE’s, Reporting, 3. Feedback Loop and Reporting
are equal with respect to level of support, Defect Management, etc.)
The tools should be able to provide
continuous contribution, documentation, and What we’ve learned during tool research is that actionable reports with sufficient artifacts
commercial support – can you call someone most tools have some level of BDD support. for fast resolution of defects.
when something doesn’t work? The number Some have better support than others. There
Dealing with a large amount of test data across
of contributors, their skills and availability to are tools which allow running BDD-based
many platforms is a challenge for most of
support are important criteria, as well as how test automation as part of their integration,
the testing tools we evaluated. Some of these
recently they contributed new code to the while others implement their own unique
tools offer plugins to support robust reporters,
project. We have seen that frameworks such BDD syntax. In addition, a tool should play
such as Allure and others, and are also
as Protractor and WebDriver.IO have, by far, nicely in the overall DevOps code pipeline.
integrated into a variety of test management
a larger number of contributors, branches Whether the team uses CircleCI, Jenkins, or
systems. Being able to get fast quality analysis
and support than tools such as Nightwatch.JS another continuous integration server, the test
and reports through the test framework is a
and Chimp; however, as part of your decision frameworks should support it seamlessly. Also,
critical requirement for fast release cadence,
making, you need to look closely at what kinds having the ability to extend the framework and
test management, data driven decision making,
of contributions have been made and when build more capabilities on top of it through
and efficiency.
they were made. supported APIs is another consideration for

9
4. Cloud and Automation at Scale 5. Automation Coverage
The tool should be able to support automation at scale The tool should provide enhanced test automation capabilities
across browsers and mobile devices through integrations for your web site, including network monitoring, memory and
with cloud providers. performance profiling, visual navigation testing, and others.
Most web projects today are not only about desktop browsers. End As part of your test automation considerations, you need to make
users consume web content from multiple screens, locations, and in sure that the selected tool or tools support your JavaScript framework
varying conditions. This reality requires a lab scaleable to support that and can support additional types of testing other than functional test
can provide on-demand access to the latest browser version, beta, automation: tests such as performance, responsive design validations,
legacy web, mobile platforms, and various OS versions. These targets internationalization and localization, and more. In many cases,
should be easily tested in parallel through CI, execution engines such commercial tools that wrap the underlying open-source framework can
as TestNG, or another grid configuration. Some of the frameworks complement it with these additional capabilities which will also include
that were evaluated have OOB integrations to cloud providers, such future enhancements to the framework.
as Perfecto and others, and can be used to address these criteria.
As a nice example of how to leverage from the browser vendors’ built-
in capabilities, Figure 3 shows how Google implemented extended
testing capabilities within its browser which can also be used through
WebDriver for your automation scripts. Such examples can add network
monitoring, performance and code coverage capabilities to your testing.

Figure 3: Google Chrome Dev Tools Options

10
6. Automation Robustness and Maintainability
The tool should support automation best practices such as page object
model (POM), sync testing, and be easy to set up and maintain.
SYNC-based (e.g. Protractor, Codecept) From a maintenance perspective, tools
tools that have an object repository that provide an automated wizard
allow for faster maintenance and easier to quickly setup your environment,
to read scripts; in addition, having a generate a config.js file for you, and
synchronized test suite means that, more, obviously save time in the overall
from a testing perspective, you don't test automation development process,
need to care about callbacks, or promises and reduce manual configuration errors.
— no “wait states” make for more robust Lastly, tools that support different
tests. A synchronized framework syncs approaches — data-driven, keyword-
the test steps with your application so driven, etc. — simplify the testing
that they will be executed properly,
and at the right time, only when the
application is ready and has processed
the previous step. There is less overhead
associated with waits, elements not
found, and broken CI/automation builds
when having this capability.

This does not mean that unsynchronized

frameworks won’t deliver the same
outcome, however; for that to happen,
teams will need to implement better
mechanisms based on implicit/explicit
waits, assertions, and more.

11
Test Framework Summaries
Once both the organizational fit items and the technical fit are clearly set, let’s examine
these 10 frameworks at a high level and look at how they stack up against each other from
a technical standpoint.

The table below represents the current state; however, with the market moving fast, keep
in mind that even if your team has built a working strategy for their web product, it is always
recommended to revise and validate that it is still the best choice considering the progress
of the various communities and commercial tools.

Buster.JS Casper.JS Chimp

CodeceptJS JSDom

Nightwatch.JS PhantomJS Protractor

Robot Webdriver.IO

12
 CATEGORIES

Casper.JS Robot Buster.JS Nightwatch.JS WebDriver.IO Protractor Codecept.JS Phantom.JS JSDom Chimp

Visual navigation With external Using external Using external

No No No No No No No
testing libraries tools tools
Take screenshots Yes Yes Yes Yes Yes Yes Yes Yes No Yes
Network monitoring, Yes (through Google Yes (through Google Yes (through Google Yes (through Google Yes (through Google
No No No Yes No
Har File Chrome Driver) Chrome Driver) Chrome Driver) Chrome Driver) Chrome Driver)
Automation
Coverage Memory
Yes (through Google Yes (through Google Yes (through Google Yes (through Google Yes (through Google
and Performance No No No No No
Chrome Driver) Chrome Driver) Chrome Driver) Chrome Driver), Chrome Driver)
Profiling
Code Coverage
No No No Jacoco Jacoco Jacoco Jacoco No No Jacoco
Analysis
Testing Types Unit testing/Headless E2E/Acceptance Unit Testing E2E/Acceptance E2E/Acceptance E2E/Acceptance E2E/Acceptance Unit, Headless Unit, Headless E2E/Acceptance
Various community
Cucumber, IntelliJ, Jasmine, Qunit, Mocha, Jasmine,
plugins, Jenkins CI,
Jira, Junit, Grunt, Jenkins, Travis CI, Jenkins, Cucumber, Meteor,
IntelliJ, Maven, eMacs, TextMate, Community Plugins Jasmine, Mocha, Mocha for test
Plugins/Integrations Cucumber, Gradle, TeamCity, TeamCity, Buster.JS Karma Most CI solutions,
Jenkins, Other xUnit available Cucumber, Visual execution
Maven, Ant Hudson, Junit XML is built-in. This FW Simian, Chai,
Studio, WebStorm,
reporting built-in uses QtWebKit WebDriver.IO
SDLC Grunt
Process Fit
No, Phantom tests
Yes (Jasmine, Mocha, Yes (Jasmine, Mocha, Yes (Support Jasmine, Mocha JavaScript
BDD/ATDD Friendly Yes Yes, KDT Yes can be triggered from No Yes
Cucumber) Cucumber) Mocha, Cucumber) DSL language
BDD frameworks
Dev Language JavaScript,
JavaScript Java, Python JavaScript JavaScript JavaScript JavaScript JavaScript JavaScript JavaScript
Support TypeScript
Config File
No No No No Yes No No No No No
Generation

Automation Page Object Model

No No No No Yes Yes Yes NA Yes No
Robustness and Creation
Maintainability Slower than Slower than Slower than Slower than Slower than Slower than Slower than
Execution Speed Fast Fast Fast
headless headless headless headless headless headless headless
Sync No No No No Yes Yes Yes No No Yes (via wrappers)
HTML, Built-in Relies on integrated
Feedback Loop HTML, Allure plugin, Allure, Junit, HTML, Jasmine2HTML, Jasmine reporters, Built in console for
Reporters CMD, XUnit-XML Rebot (XML) reports, API for CLI, XML, HTML FW reports like
and Reporting Junit XML XML, Perfecto JUnit, Allure Karma reports reporting
custom reporter Mocha and others
Documentation Casper.JS Robot Buster.JS Nightwatch.JS WebDriver.IO Protractor Codecept.JS Phantom.JS JSDom Chimp
Community
Strength 20+ Contributors
Contributors 176 contributors 50 Contributors 59 contributors 250 contributors 234 contributors 81 contributors 147 contributors 202 contributors 40 Contributors
(not that active)
Cloud and
Yes Yes Yes Yes Yes
Automation Cloud Support No No No No Yes
(web and mobile) (web and mobile) (web and mobile) (web and mobile) (web and mobile)
at Scale

13
 Casper.JS is A navigation scripting & testing utility for PhantomJS and SlimerJS Automation Coverage
headless browser tools. CasperJS allows you to build full navigation scenarios
using high-level functions and a straightforward interface – this makes the solution Visual navigation testing No

very appealing and easy to start writing tests for folks who don’t have a strong Take screenshots Yes
technical background. You can get started with this tool through this URL:
Network monitoring, Har File No
http://docs.casperjs.org/en/latest/quickstart.html
Memory and Performance Profiling No
Pro’s Con’s Code Coverage Analysis No
• Automation Coverage: Designed for unit • Automation Coverage: Not the best fit for E2E
Testing Types Unit testing/Headless
testing activities testing scenarios
• SDLC Process Fit: Uses older headless SDLC Process Fit
• Automation Robustness: Fast feedback due
to fast execution time browser technologies, compared to recent Jira, Junit, Cucumber,
Plugins/Integrations
Chrome Blink- based headless browser Gradle, Maven, Ant
• Automation Coverage: Seamlessly works with
Phantom and SlimerJS headless browsers • Automation at Scale: Doesn’t cover BDD/ATDD Friendly Yes
real desktop browsers OOB, needs to be Dev Language Support JavaScript
• SDLC Process Fit: Easy ramp-up from a skillset
complemented by an E2E solution
perspective, BDD-based scripting Automation Robustness and Maintainability

Config File Generation No

Page Object Model Creation No

Execution Speed Fast

Sync No

Feedback Loop and Reporting

Reporters CMD, XUnit-XML

Community Strength

Documentation Casper.JS

Contributors 176 contributors

Cloud and Automation at scale

Cloud Support No

14
 Robot Framework is a test automation framework for acceptance testing and Automation Coverage
acceptance test-driven development (ATDD). It has easy-to-use tabular test data
syntax and utilizes a keyword-driven testing approach. Its testing capabilities can Visual navigation testing No
be extended with test libraries implemented either in Python or Java, and users can
Take screenshots Yes
create new higher-level keywords from existing ones using the same syntax that is
used for creating test cases. Network monitoring, Har File No

Memory and Performance Profiling No

Pro’s Con’s
Code Coverage Analysis No
• Automation Coverage: Designed for • SDLC Process Fit: Not fully embedded into
acceptance testing activities (ATDD) dev workflows, suitable more to QA Testing Types E2E/Acceptance

• SDLC Process Fit: Leverages keyword-driven • SDLC Process Fit: Mostly based on Python, SDLC Process Fit
testing method making it easy to develop less JavaScript focused – may be a skillset issue IntelliJ, Maven,
readable scripts Plugins/Integrations
• Automation Robustness: Uses WebDriver Jenkins, Other

• Community and Support: Large community as an external test library rather than fully BDD/ATDD Friendly Yes, KDT
behind it, plenty of plugins and extensions designed around it
Dev Language Support Java, Python
• Automation at Scale: Cross platform support • Feedback Loop and Reporting:
Automation Robustness and Maintainability
— supports testing of Android, MongoDB Limited reporting
and more. Config File Generation No
• SDLC Process Fit: KDT is a less common
practice than BDD (source) Page Object Model Creation No

Execution Speed Slower than headless

Sync No

Feedback Loop and Reporting

Reporters Rebot (XML)

Community Strength

Documentation Robot

Contributors 50 Contributors

Cloud and Automation at scale

Cloud Support Yes (web and mobile)

15
 Buster.JS is a Node.JS testing toolkit that is highly extensible for reporting and Automation Coverage
other purposes. This toolkit is built in to headless browser tools such as Phantom.
JS and JSDom. This tool fits testing of browsers and Node apps. Visual navigation testing No

Take screenshots Yes

It is still being positioned as Beta — unclear if this will evolve. That means
less community behind it, perhaps technical limitations, no IDE plugins and Network monitoring, Har File No

potentially some integration issues etc. Memory and Performance Profiling No

Buster.JS comes with built-in assertions and DSL support for adding app-specific Code Coverage Analysis No
custom assertions. Testing Types Unit Testing

In addition, Buster.JS comes with a few front-end plugins such as XUnit, as well as a SDLC Process Fit
BDD plugin that enables automation engineers to write scenarios. eMacs, TextMate,
Plugins/Integrations
xUnit
Pro’s Con’s BDD/ATDD Friendly Yes
• Automation Coverage: Supports headless • Automation Robustness/Community and
Dev Language Support JavaScript
browser testing and unit testing for Node Support: Still growing, currently in Beta –
and browsers might mean less experience, less stability, Automation Robustness and Maintainability
less functionality
• SDLC Process Fit: Friendly DSL-based scripting Config File Generation No
technology that supports assertions • Automation Robustness and
Maintainability: Async testing tool Page Object Model Creation No
• SDLC Process Fit: Extensible and integrates
with many CI and reporting tools • SDLC Process Fit: Requires setting up a proxy Execution Speed Slower than headless
server to cover testing against an application Sync No
server from your web tests
Feedback Loop and Reporting
• Automation Coverage: Doesn’t fit a complete
E2E functional testing objective HTML, Built-in
Reporters reports, API for
custom reporter

Community Strength

Documentation Buster.JS

20+ Contributors
Contributors
(not that active)

Cloud and Automation at scale

Cloud Support No

16
 Nightwatch.js is an easy to use Node.js-based End-to-End (E2E) testing solution Automation Coverage
for browser-based apps and websites. It uses the powerful W3C WebDriver API
to perform commands and assertions on DOM elements. The tool has a few Visual navigation testing With external libraries
built-in plugins for Junit XML reporting that make it easy to send steps reports
Take screenshots Yes
to CI servers such as Jenkins, TeamCity and Hudson, as well as a Grunt plugin for
simple task execution. You can execute your tests against a local Selenium server Yes (through Google
Network monitoring, Har File
Chrome Driver)
or against a cloud server such as Perfecto and others. This solution is one of the
main competitors to Protractor. Memory and Performance Profiling
Yes (through Google
Chrome Driver)

Code Coverage Analysis Jacoco

Pro’s Con’s
• Automation Coverage: Good for E2E • Community and Support: Community of Testing Types E2E/Acceptance
functional testing contributors is relatively small compared to SDLC Process Fit
Protractor and WebDriver.IO
• Feedback Loop and Reporting: Good Cucumber, IntelliJ,
reporting plugins such as Allure and JUnit • Automation Robustness and Grunt, Jenkins,
Maintainability: Async testing tool compared Plugins/Integrations TeamCity, Hudson,
• Automation at Scale and Cloud: Easily Junit XML reporting
to Protractor which is a sync-based framework
integrated into cloud testing solutions built-in
(built-in waits)
(e.g. Perfecto) Yes (Jasmine, Mocha,
BDD/ATDD Friendly
–– Uses Magellan.json for scaled browser • SDLC Process Fit: Complex setup with a lot of Cucumber)
automation pre-requisites
Dev Language Support JavaScript
• SDLC Process Fit: BDD friendly – comes with • SDLC Process Fit: Nonstandard BDD
Automation Robustness and Maintainability
a built-in BDD FW based on Chai, and also assertions compared to WebDriver.IO, which
supports Jasmine, Mocha and Cucumber supports Jasmine, Mocha etc. assertions that Config File Generation No
are more common.
Page Object Model Creation No

Execution Speed Slower than headless

Sync No

Feedback Loop and Reporting

HTML, Allure
Reporters
plugin, Junit XML

Community Strength

Documentation Nightwatch.JS

Contributors 59 contributors

Cloud and Automation at scale

Cloud Support Yes (web and mobile)

Figure 4: Nightwatch.JS sample code snippet & execution console side by side (source: nightwatch.js)

17
 WebDriver.IO is the leading Pro’s
Automation Coverage
WebDriver binding for Node.JS. • Automation Coverage: Good for E2E functional
The framework basically sends testing Visual navigation testing Using external tools
requests to the Selenium server
via the WebDriver protocol and • Fast Feedback and Reporting: Good reporting Take screenshots Yes
manages the responses. Requests plugins like Allure, Junit, Perfecto DigitalZoom™
Yes (through Google
are wrapped in useful commands for ease of Network monitoring, Har File
• Automation at Scale and Cloud: Easily integrated Chrome Driver)
development and reuse for multiple test scenarios into cloud testing solutions (e.g. Perfecto) for Memory and Performance Yes (through Google
of your web site and more. parallel testing Profiling Chrome Driver)
The integrated test runner lets you write • Community and Support: Strong community Code Coverage Analysis Jacoco
asynchronous commands in a synchronous way so backing the technology — integrations, plugins,
that you don’t need to worry about how to handle support, documentation Testing Types E2E/Acceptance
a promise to avoid race conditions. Additionally,
• Automation Robustness and Maintainability: SDLC Process Fit
it takes away all the cumbersome setup work and
Sync-based testing supported
manages the Selenium session for you. Plugins/Integrations
Community Plugins
• SDLC Process Fit: BDD-friendly through Jasmine, available
Working with elements on a page is very easy due Mocha, Cucumber and others — more standard Yes (Jasmine, Mocha,
to its synchronous nature. When fetching or looping BDD/ATDD Friendly
FW to choose vs. proprietary Cucumber)
over elements you can use just native JavaScript
functions. With the $ and $$ functions, WebdriverIO • SDLC Process Fit: Adopted by new emerging tools Dev Language Support JavaScript
provides useful shortcuts which can also be chained such as Chimp.JS
Automation Robustness and Maintainability
to move deeper in the DOM tree without using • Automation Robustness and Maintenance:
complex xPath selectors. Config file generation wizard supported, speeds Config File Generation Yes
up the testing setup (see image below)
The WDIO framework is easily integrated into many Page Object Model Creation Yes
tools; therefore, during the config file setup, test
automation engineers can specify their tool Con’s Execution Speed Slower than headless
stack (Cucumber, Mocha, Jasmine, local vs. cloud, • SDLC Process Fit: Less fit for angular-specific
Sync Yes
Selenium grid, and more). web sites; angular-specific object locating can be
challenging Feedback Loop and Reporting
• SDLC Process Fit: No TypeScript support Allure, Junit, HTML,
Reporters
compared to Protractor, if this is a relevant XML, Perfecto
requirement — worth mentioning
Community Strength

Documentation WebDriver.IO

Contributors 250 contributors

Cloud and Automation at scale

Cloud Support Yes (web and mobile)

Fig 5: WebDriver.IO built-in command-line configuration wizard

18
 Protractor is an end-to-end test framework for Angular and AngularJS Automation Coverage
applications. Protractor runs tests against your application running in a real
browser, interacting with it as a user would. This framework is highly supported Visual navigation testing Using external tools

by contributors from Google, the main leader behind AngularJS. This framework Take screenshots Yes
that is built on top of Selenium and is the most widely adopted framework, Yes (through Google
Network monitoring, Har File
especially when testing Angular/AngularJS websites. In addition to many Chrome Driver)
extensions and plugins that Protractor has for things such as visual testing Yes (through Google
Memory and Performance Profiling
and more, Protractor has a unique synchronization mechanism that can Chrome Driver)

automatically execute the next step in your test the moment the webpage Code Coverage Analysis Jacoco
finishes pending tasks so you don’t have to worry about waiting for your test and web page
Testing Types E2E/Acceptance
to sync.
SDLC Process Fit
Another great thing about Protractor is its ability to support BDD and Cucumber scripting. If Various community
you’re leveraging BDD as part of your Angular/AngularJS website SDLC, then this framework plugins, Jenkins CI,
should be a great fit for you. Jasmine, Mocha,
Plugins/Integrations
Cucumber, Visual
Studio, WebStorm,
Pro’s • Community and Support: Strong community Grunt
• Automation Coverage: Good for E2E backing the technology — integrations, plugins, Yes (Support Jasmine,
support, documentation BDD/ATDD Friendly
functional testing of Angular-based websites Mocha, Cucumber)

• Fast Feedback and Reporting: Good • Automation Robustness and Dev Language Support JavaScript, TypeScript
reporting plugins such as Allure, Junit, Maintainability: Sync-based testing supported
Automation Robustness and Maintainability
Perfecto DigitalZoom™ • SDLC Process Fit: BDD-friendly through
Config File Generation No
• Automation at Scale and Cloud: Easily Jasmine, Mocha, Cucumber and others —
integrated with cloud testing solutions more standard FW to choose vs. proprietary Page Object Model Creation Yes
(e.g. Perfecto) for parallel testing • SDLC Process Fit: Supports Execution Speed Slower than headless
TypeScript development
Sync Yes

Con’s Feedback Loop and Reporting

• Automation Robustness and Jasmine2HTML, JUnit,
Maintainability: If there is an issue with Reporters
Allure
WebdriverJs layer (between Selenium and
Community Strength
Protractor), the Protractor team should wait
for the WebDriverJs team to fix that issue. Documentation Protractor

Contributors 234 contributors

Cloud and Automation at scale

Cloud Support Yes (web and mobile)

19
 Codecept.IO is a JavaScript acceptance testing solution for Node.JS. In this solution, Automation Coverage
testing can be authored from an end-user perspective. Every command is
described as an action of a user visiting a site. Visual navigation testing No

Take screenshots Yes

In addition, Codecept supports various helpers like WebDriverIO, Protractor,
Yes (through Google
Nightmare and Selenium WebDriver, enabling teams to extend their testing Network monitoring, Har File
Chrome Driver)
use cases.
Yes (through Google
Memory and Performance Profiling
Chrome Driver),
Pro’s Con’s
Code Coverage Analysis Jacoco
• Automation Coverage: Suits acceptance • SDLC Process Fit: Not fully designed for
testing in a DSL/BDD language mobile app testing even though Appium Testing Types E2E/Acceptance

• Automation Robustness and helper is available.

SDLC Process Fit
Maintainability: Synchronous test API’s for • SDLC Process Fit: Requires command
Jasmine, Qunit,
more stable and linear tests line and/or wrappers to execute scripts Travis CI, Jenkins,
• SLDC Process Fit: Easy to write and in opposed to seamless IDE plugin Plugins/Integrations TeamCity, Buster.JS is
understand each test due to the unique • Automation Robustness and built-in. This FW uses
QtWebKit
syntax the tests are written in Maintainability: Async testing solution
Mocha JavaScript
• SDLC Process Fit: Backend agnostic to • Community and Support: Relatively small BDD/ATDD Friendly
DSL language
multiple WebDrivers that are used number of contributors
Dev Language Support JavaScript
• Automation Robustness and • Automation Coverage: Cannot perform
Maintainability: Built-in dependency enables visual navigation testing (OCR based testing) Automation Robustness and Maintainability
creation of a POM
Config File Generation No
• Automation Robustness and
Maintainability/SDLC Process Fit: Uses Page Object Model Creation Yes
JavaScript DSL for the BDD-based test Execution Speed Slower than headless
authoring with common predefined functions
Sync Yes
• Automation at Scale and Cloud: Among
the large set of helpers that Codecept uses, Feedback Loop and Reporting
there is also Appium to enable mobile
testing in addition to Web. Reporters CLI, XML, HTML

Community Strength

Documentation Codecept.JS

Contributors 81 contributors

Cloud and Automation at scale

Cloud Support Yes (web and mobile)

20
 PhantomJS is a headless WebKit scriptable with a JavaScript API. It has fast and Automation Coverage
native support for various web standards: DOM handling, CSS selector, JSON,
Canvas, and SVG. Among its key benefits are screen capturing, network monitoring Visual navigation testing No

that captures an HAR file, as well as friendly script execution capabilities through Take screenshots Yes
Jasmine, Mocha and other tools. It’s important to understand that PhantomJS itself
Network monitoring, Har File Yes
is not a test framework, it is only used to launch the tests via a suitable test runner
(e.g. Buster.JS has Phantom.JS as a supported tool in its solution). Memory and Performance Profiling No

Code Coverage Analysis No

Since this is a headless (and local) testing solution, it is compatible with various web development
frameworks like jQuery, Bootstrap, CodeMirror, and others. Testing Types Unit, Headless

SDLC Process Fit

This tool is mostly beneficial for fast unit testing driven through CI or command line post code
commits for fast feedback, and less for larger UI functional web testing. Google recently launched Plugins/Integrations
Mocha for test
execution
a Chrome headless browser that may be a suitable replacement for Phantom and which uses the
No, Phantom tests
Blink rendering engine, compared to the WebKit one used by Phantom.
BDD/ATDD Friendly can be triggered from
BDD frameworks
Pro’s Con’s
Dev Language Support JavaScript
• Automation Coverage: Suits unit • Automation Coverage: Cannot fit an E2E
testing, mostly testing objective Automation Robustness and Maintainability

• Automation Coverage: Additional automation • Automation at Scale and Cloud: Doesn’t Config File Generation No
artifacts such as HAR file and more run on real browsers, automation at scale
Page Object Model Creation NA
• Automation Coverage: Screen capture is an issue
capabilities • SDLC Process Fit: Doesn’t support BDD as Execution Speed Fast

• Community and Support: Great a built-in capability, tests can be triggered

Sync No
documentation and community support from BDD frameworks
• SDLC Process Fit: Outdated rendering engine Feedback Loop and Reporting
• Automation Coverage: Strong APIs for
various testing capabilities (filesystem, compared to Google’s headless solution Jasmine reporters,
Reporters
cookies, page size etc.) Karma reports

Community Strength

Documentation Phantom.JS

Contributors 147 contributors

Cloud and Automation at scale

Cloud Support No

21
 JSDom is an in-JavaScript implementation of the DOM to be used with node.js. The Automation Coverage
DOM is the document object model, which is the tree of nodes that make up the
UI for documents shown in web browsers. Among the leading contributors for this Visual navigation testing No

open-source you’ll find developers from Google and whatwg.org. Because jsdom Take screenshots No
is implemented in JavaScript, we can have a DOM-like API to work with or without
Network monitoring, Har File No
needing a browser. That means that we can run our tests in environments without
browsers, such as in Node or in continuous integration environments. Memory and Performance Profiling No

Code Coverage Analysis No

Pro’s Con’s
Testing Types Unit, Headless
• SDLC Process Fit: Good for unit testing and • Automation Coverage: Cannot fit an E2E
for fast feedback on specified subset of your testing objective SDLC Process Fit
website implementation
• Automation Coverage: Does not target real Plugins/Integrations Karma
• Community and Support: Large community browsers, doesn’t test at scale or in a cloud
BDD/ATDD Friendly No
of contributors based environment
• SDLC Process Fit: Implements the standard • Automation Robustness and Dev Language Support JavaScript
WHATWG DOM Maintainability: Asynchronous script loading. Automation Robustness and Maintainability
There is no way with JSDom to tell the user
• SDLC Process Fit: Supports Mocha for Config File Generation No
when it's a good time to run your code and
scripting in an easy and readable syntax
inspect the resulting DOM structure since it Page Object Model Creation Yes
has no real way to know when the entire page
was loaded Execution Speed Fast

• Automation Coverage: No screenshots or Sync No

visual testing support
Feedback Loop and Reporting
• Fast Feedback Loop and Reporting: Limited Built in console for
reporting capabilities, mostly leverages built-in Reporters
reporting
console for reports
Community Strength

Documentation JSDom

Contributors 202 contributors

Cloud and Automation at scale

Cloud Support No

22
 Chimp makes it super easy for developers to write automated tests by taking Automation Coverage
away all the pain associated with setting up tools and allowing developers to
focus on building in quality. It does so by integrating and sprinkling magic over Visual navigation testing No

the following tools: Take screenshots Yes

• Mocha, Jasmine or Cucumber.js Network monitoring, Har File

Yes (through Google
Chrome Driver)
• Selenium and WebdriverIO
Yes (through Google
• Chai or Jasmine assertion libraries inside your steps Memory and Performance Profiling
Chrome Driver)

• Built in Node.js, works for any web application (with special Meteor support) Code Coverage Analysis Jacoco

Testing Types E2E/Acceptance

Pro’s Con’s
SDLC Process Fit
• Automation Coverage: E2E/Acceptance • Automation Coverage: Doesn’t support
Mocha, Jasmine,
solution, BDD friendly, Mix of supported tools visual navigation testing
Cucumber, Meteor,
• Automation Robustness and • Community and Support: Community Plugins/Integrations Most CI solutions,
Simian, Chai,
Maintainability: Synchronized is ramping up, 40 contributors WebDriver.IO
testing solution
• Fast Feedback Loop and Reporting: BDD/ATDD Friendly Yes
• SDLC Process Fit: Large set of integrations Basic reporting, doesn’t provide a cross-
to leading standard tools like Mocha, Jasmine, platform reporting dashboard Dev Language Support JavaScript
Cucumber, and most CI tools
• SDLC Process Fit: Setup and configuration Automation Robustness and Maintainability
• SDLC Process Fit: Innovative approach to seems complex
Config File Generation No
agile/fast feedback loop through tools like
Simian and Meteor Page Object Model Creation No

• Automation at Scale and Cloud: integrations Execution Speed Slower than headless
to cloud supported for automation at scale
Sync Yes (via wrappers)
• Automation Coverage: Support for taking
Feedback Loop and Reporting
web screenshots
Relies on integrated
• Fast Feedback Loop and Reporting: Reporters FW reports like
Good assertions mechanism through Chai Mocha and others
and others
Community Strength
• Fast Feedback Loop and Reporting: Supports
Documentation Chimp
debug mode for inspecting nodes through
breakpoints and more. Contributors 40 Contributors

• Automation at Scale: Mobile testing support Cloud and Automation at scale

through Appium
Cloud Support Yes

23
 Summary
We have explored the web testing landscape as well as smartphones and tablets in the cloud
with a special focus on the Selenium WebDriver that can be targeted from your Nightwatch,
API. Many open-source projects were WebDriverIO, Selenium, Protractor, Robot or
reviewed, illustrating unique benefits, pros other framework that you choose, in order to
and cons. Choosing test frameworks should achieve automation at scale with maximum
be determined by the combination of applying availability. At the end of execution, quality
organizational and technical fit criteria. analysis is fast, with digital reporting that
offers side-by-side execution dashboard with
While we mostly covered tools that offer screenshots, testing artifacts such as HAR files,
E2E functional testing and unit testing, it is performance measurements, logs, and more.
important to mention that there obviously All tests can be orchestrated via CI engine
are also tools for API testing, such as SoapUI such as Jenkins, CircleCI, TeamCity, Bamboo
and others, that were not in the scope of and others.
this research.
You should continuously evaluate the market
With the key considerations of technical fit and adjust your tool stack according to both
and organizational fit in mind, Perfecto’s market evolution and your own product’s
Continuous Quality Lab in the cloud is growth and its required capabilities; also, make
architected to provide support for teams sure you have the correct match for your web
regardless of their open-source framework of developers and testers.
choice or their development practice method.
Perfecto’s lab offers a large set of desktop VMs

24
Appendix
Recommended resources
The Top 8 Essential JavaScript Automation Frameworks — Joe Colantonio
Differences between nightwatch.js and webdriver.io — StackOverflow
Test frameworks comparison — Slant
Protractor vs. Webdriver.io vs. Nightwatch.JS — WebDriverJS.com
Top 5 most rated node.js frameworks for E2E web testing — Adrian Lewis on Medium
An overview of JavaScript testing in 2017 — Vitalik Zaidman on Medium
How to automate web testing using open-source frameworks — Perfecto’s slide share
Recommended JavaScript unit testing tools — Slant

25
 Eran Kinsbruner
Director, Mobile Evangelist at Perfecto Mobile

About Perfecto
Perfecto enables exceptional digital experiences. We help you transform your business and strengthen
every digital interaction with a quality—first approach to creating web and native apps, through a cloud—
based test environment called the Continuous Quality Lab™. The CQ Lab is comprised of real devices and
real end—user conditions, giving you the truest test environment available.

More than 1,500 customers, including 50% of the Fortune 500 across the banking, insurance, retail,
telecommunications and media industries rely on Perfecto to deliver optimal mobile app functionality
and end user experiences, ensuring their brand’s reputation, establishing loyal customers, and continually
attracting new users. For more information about Perfecto, visit www.perfectomobile.com, join our
community follow us on Twitter at @PerfectoMobile.

Get content just like this delivered to your inbox!