Introduction

Testing is a very important phase in the development process of all WikiMedia's repositories. However, unit tests are not sufficient for all situations needed to be tested. End-to-End tests need to be done. We can't navigate through webpages with unit tests, Wikimedia uses a browser Automation framework(WebdriverIO) and selenium for its End-to-End tests.
WebdriverIO has been around for a very long while and has a large community behind it, however, there are some useful alternatives out there that can be used as a replacement framework while still giving a better performance than WebdriverIO. Some of such replacement framework is puppeteer, a high-level browser automation API built atop Chrome DevTools Protocols.

Why should puppeteer be considered as a replacement for WebdriverIO?

Puppeteer has quite a number of advantages over WebdriverIO and a few of them are highlighted in bullet points below.

• Easier to setup
• Fewer Dependencies
• Easy to Upgrade to a newer version
• Handles device emulation properly
• Access to Chrome Dev tools
• Simpler syntax
• Simpler JavaScript Execution
• Easier to run across multiple browsers(Chrome and Firefox)

The above highlights some of the advantages puppeteer has over WebdriverIO but doesn't give in-depth or technical details about how this highlighted points make puppeteer a better choice. Some of the above listed have been grouped or categorized into sections in which a more in-depth detail is discussed below.

Numbers

WebdriverIO has been around for a long time, it has a lot of community support and engagement when compared to puppeteer but over the years, the number of active users of puppeteer has grown considerably way beyond that of WebdriverIO. According to npm trends, Puppeteer is currently used by 70.5k repositories on Github andWebdriverIO is used in 21.7k repositories on Github.

Setup and Installation

Puppeteer requires no extra setup or tedious installation procedures, it can be installed via just one command line code(npm install puppeteer) unlike WebdriverIO.
WebdriverIO uses selenium as its browser automation framework, currently, the following WebdriverIO dependencies are being used on the MediaWiki/core repository

• "@wdio/cli": "5.13.2"
• "@wdio/devtools-service": "5.13.2"
• "@wdio/dot-reporter": "5.13.2"
• "@wdio/junit-reporter": "5.13.2"
• "@wdio/local-runner": "5.13.2"
• "@wdio/mocha-framework": "5.13.2"
• "@wdio/sauce-service": "5.13.2"
• "@wdio/sync": "5.13.2"
• "wdio-chromedriver-service": "5.0.2"
• "wdio-mediawiki": "file:tests/selenium/wdio-mediawiki"
• "webdriverio": "5.13.2"

Depending on the browser that we want to use for our test, we would also need to install the dependencies of that browser's driver and the drivers themselves separately. Chrome for examples uses the chromium driver and Firefox uses the gecko driver which means we would still install dependencies such as the ones highlighted below

• "karma": "3.1.4"
• "karma-chrome-launcher": "2.2.0"
• "karma-firefox-launcher": "1.1.0"
• "chromedriver": "73.0.0"

Installing the above would take quite some time and it's just too many dependencies to work with.

In puppeteer's case, just the following dependencies are required to have it up and running;

• "puppeteer": "^2.1.1"
• "jest-puppeteer": "^4.4.0"

As highlighted above, puppeteer requires just a testing framework and a single dependency to be installed, this makes it very easy to set up and use. Also, no additional browser drivers need to be installed.

Device Emulation

Puppeteer handles device emulation way better than WebdriverIO. The table below shows a comparison of both.

We would notice that Puppeteer supports 3 OS while Puppeteer supports 2. Also, WebdriverIO does not emulate devices individually but rather generalizes its emulations to a particular OS which isn't so great as each device has a separate User-Agent, even if they are of the same Operating System.

Emulating devices with Puppeteer also has a cleaner syntax than Emulating with WebdriverIO

Access to Chrome Dev tools

Puppeteer by default has full access and control over all of the chrome Devtools functionality which makes it very useful when writing tests, It's gives access to functionalities like Network throttling, code coverage, browser's console, etc. However, WebdriverIO needs to use Puppeteer under the hood to access chrome dev tools.

Simple code examples

Here, I would be showing code samples to perform a very basic test on MediaWiki/core. The basic test performed will be to test for the account creation feature of MediaWiki/core. This is used to show the difference between the syntax of WebdriverIO and Puppeteer.

WebdriverIO

tests/selenium/wdio-mediawiki/Page.js

const querystring = require( 'querystring' );

/**
 * Based on http://webdriver.io/guide/testrunner/pageobjects.html
 */
class Page {
	openTitle( title, query = {}, fragment = '' ) {
		query.title = title;
		browser.url(
			browser.config.baseUrl + '/index.php?' +
			querystring.stringify( query ) +
			( fragment ? ( '#' + fragment ) : '' )
		);
	}
}

module.exports = Page;

tests/selenium/pageobjects/createaccount.page.js

const Page = require( 'wdio-mediawiki/Page' );

class CreateAccountPage extends Page {
	get username() { return $( '#wpName2' ); }
	get password() { return $( '#wpPassword2' ); }
	get confirmPassword() { return $( '#wpRetype' ); }
	get create() { return $( '#wpCreateaccount' ); }
	get heading() { return $( '.firstHeading' ); }

	open() {
		super.openTitle( 'Special:CreateAccount' );
	}

	createAccount( username, password ) {
		this.open();
		this.username.setValue( username );
		this.password.setValue( password );
		this.confirmPassword.setValue( password );
		this.create.click();
	}
}

module.exports = new CreateAccountPage();

tests/selenium/specs/user.js

const assert = require( 'assert' );
const CreateAccountPage = require( '../pageobjects/createaccount.page' );
const Util = require( 'wdio-mediawiki/Util' );

describe( 'User', function () {
	let password, username
        beforeEach( function () {
		browser.deleteAllCookies();
		username = Util.getTestString( 'User-' );
		password = Util.getTestString();
	} );
        it( 'should be able to create account', function () {
		// create
		CreateAccountPage.createAccount( username, password );

		// check
		assert.strictEqual( CreateAccountPage.heading.getText(), `Welcome, ${username}!` );
	} );
} );

Puppeteer

tests/puppeteer/pageObjects/createAccount.js

class CreateAccountPage{
	constructor(page){
		this.page = page;
		this.url = `${process.env.MW_SERVER || 'http://127.0.0.1:8080'}/index.php?title=Special:CreateAccount&returnto=Main+Page`
	} 

	async createAccount(username, password){
		try{
			await this.page.goto(this.url);
			await this.page.type('#wpName2', username);
			await this.page.type('#wpPassword2', password);
			await this.page.type('#wpRetype', password);
			await this.page.click('#wpCreateaccount');
			await this.page.waitFor('#firstHeading');
		}
		catch(err){
			console.log("The error is ", err);
		}
	}

	getHeadingText(){
		return this.page.$$eval('#firstHeading', h => h[0].textContent);
	}

}

module.exports = (browser) => new CreateAccountPage(browser);

tests/puppeteer/__tests__/createAccount.test.js

let CreateAccountPage = require('../pageObjects/createAccount');
let reporterObj = require('../Utils/reporter');
let credentials = require('../Utils/credentials');


describe('User should be able to create an account', () => {
	let page, credential;
	beforeAll(async () => {
        page = await global.__BROWSER__.newPage()
        credential = credentials.generateCredential('User');
        CreateAccountPage = await CreateAccountPage(page);
    });

    it("It should be able to create an account", async () => {
        await CreateAccountPage.createAccount(credential.username, credential.password);
        let headingText = await CreateAccountPage.getHeadingText();
        expect(headingText).toBe(`Welcome, ${credential.username}!`);
    });
});

Conclusion

Puppeteer has a whole lot of useful advantages over WebdriverIO, has a clean syntax, easier to work with and comes with more functionality. It would, therefore, be very beneficial if WebdriverIO is replaced with Puppeteer as the automated browser framework for Wikimedia