Synopsis

Mediawiki is an important software component of Wikimedia that powers a large number of companies and organisations, including Wikipedia. Because of the various organisations and websites that depend upon it, it is necessary to evaluate software components under various expected and unexpected conditions so as to ensure delivery of optimal quality code.

To facilitate frequent and repetitive regression and E2E testing of a web application, a robust test approach would involve automating checks at the browser level. This reduces the manual testing effort and allows for detection of defects at an early stage. Wikimedia does this using a browser automation framework called WebdriverIO.

In this task, through experimentation, we compare one of the two most popular browser automation frameworks, focusing on primarily two points(speed and stability)

• WebdriverIO: WebdriverIO has been around for quite some time now, and has established itself as one of the popular web browser automation tool. It supports a wide range of browsers (Chrome, Firefox, Safari …), and uses Javascript. It uses Selenium under the hood.
• Puppeteer: Puppeteer is much newer (first released in 2017). It was developed by Google for the purpose of automating and simplifying front-end tests and development. It can be used with Chrome or Chromium (which forms the basis of Chrome). It supports Javascript (Node.js).

Why look for an alternative?

A quick glance through various automation frameworks in trend at present times:

Though highly reliable, WebdriverIO comes with its own set of cons. The major one being:

• Difficulty in setup. A brief glance through Mediawiki's "package.json" file highlights the dependency tree for WebdriverIO

"chromedriver": "73.0.0"
"karma": "3.1.4"
"karma-chrome-launcher": "2.2.0"
"karma-firefox-launcher": "1.1.0"
"@wdio/cli": "5.13.2"
"@wdio/devtools-service": "5.13.2"
"@wdio/dot-reporter": "5.13.2"
"@wdio/junit-reporter": "5.13.2"
"@wdio/local-runner": "5.13.2"
"@wdio/mocha-framework": "5.13.2"
"@wdio/sauce-service": "5.13.2"
"@wdio/sync": "5.13.2"
"wdio-chromedriver-service": "5.0.2"
"wdio-mediawiki": "file:tests/selenium/wdio-mediawiki"
"webdriverio": "5.13.2

• Flaky tests: Flaky tests lead to false negatives which can confuse the team and increase the time and effort needed in debugging.
• No consistent device emulation features
• Difficulty upgrading: Since WebdriverIO does not update in sync with Selenium, it makes it very difficult to upgrade WebDriverIO from one major version to another without breaking the existing code.
• Scattered documentation

Setup

Assuming Node.js is installed, setting up Puppeteer is straight forward: just run npm install puppeteer. This will install Puppeteer, and download a recent version of Chromium that is guaranteed to work with Puppeteer.

In addition to Puppeteer, we also installed:

• Jest: npm install --save-dev jest
• Jest-puppeteer: npm install --save-dev jest-puppeteer

Test Implementation

While migrating test cases from WebdriverIO to Puppeteer, the primary goal I had in mind was to implement this migration while making as little structural changes to existing code as possible.
Though this part needed more efforts than just simply copy pasting code from documentation, it was worth it!! The end result was a code migrated to a much richer browser automation framework without sacrificing the existing code structure. This is a plus for developers who have previously worked on WebdriverIO with Mediawiki to still be able to understand the code without knowing Puppeteer.

Below sample code tests for an Admin to be able to login
Here we try to implement the following scenario:

• Open the browser and access "localhost:8080/index.php'
• Enter Admin username
• Admin password
• Check if the admin has logged in successfully
• Close the browser

WebdriverIO

tests/selenium/wdio-mediawiki/Page.js

const querystring = require( 'querystring' );

/**
 * Based on http://webdriver.io/guide/testrunner/pageobjects.html
 */
class Page {
	openTitle( title, query = {}, fragment = '' ) {
		query.title = title;
		browser.url(
			browser.config.baseUrl + '/index.php?' +
			querystring.stringify( query ) +
			( fragment ? ( '#' + fragment ) : '' )
		);
	}
}

module.exports = Page;

tests/selenium/wdio-mediawiki/LoginPage.js

const Page = require( './Page' );

class LoginPage extends Page {
	get username() { return $( '#wpName1' ); }
	get password() { return $( '#wpPassword1' ); }
	get loginButton() { return $( '#wpLoginAttempt' ); }
	get userPage() { return $( '#pt-userpage' ); }

	open() {
		super.openTitle( 'Special:UserLogin' );
	}

	login( username, password ) {
		this.open();
		this.username.setValue( username );
		this.password.setValue( password );
		this.loginButton.click();
	}

	loginAdmin() {
		this.login( browser.config.mwUser, browser.config.mwPwd );
	}
}

module.exports = new LoginPage();

tests/selenium/specs/user.js

const assert = require( 'assert' );
const UserLoginPage = require( 'wdio-mediawiki/LoginPage' );
const Util = require( 'wdio-mediawiki/Util' );

describe( 'User', function () {
        beforeEach( function () {
		browser.deleteAllCookies();
	} );

        it( 'should be able to log Admin in @daily', function () {
		// log in
		UserLoginPage.loginAdmin();

		// check
		const actualUsername = browser.execute( () => {
			return mw.config.get( 'wgUserName' );
		} );
		assert.strictEqual( actualUsername, browser.config.mwUser );
	} );
} );

Puppeteer

tests/pupeeteer/puppeteer-mediawiki/Page.js

var querystring = require( 'querystring' );

/**
 * Based on http://webdriver.io/guide/testrunner/pageobjects.html
 */
class Page {
	/**
	 * Navigate the browser to a given page.
	 *
	 * @param {string} page Page | Puppeteer Page Object
	 * @param {Object} [title] |  Query parameter
	 * @param {Object} [query] | Query String Object
	 * @param {string} [fragment] Fragment parameter
	 * @return {void} This method runs a browser command.
	 */
	async openTitle( page, title, query = {}, fragment = '' ) {
		const baseUrl = ( process.env.MW_SERVER || 'http://127.0.0.1:8080' ) + (
			process.env.MW_SCRIPT_PATH || '/w'
		);

		query.title = title;
		await page.goto(
			baseUrl + '/index.php?' +
			querystring.stringify( query ) +
			( fragment ? ( '#' + fragment ) : '' )
		);
	}
}

module.exports = Page;

tests/puppeteer/puppeteer-mediawiki/LoginPage.js

var querystring = require( 'querystring' );

class Page {
	/**
	 * Navigate the browser to a given page.
	 *
	 * @param {string} page Page | Puppeteer Page Object
	 * @param {Object} [title] |  Query parameter
	 * @param {Object} [query] | Query String Object
	 * @param {string} [fragment] Fragment parameter
	 * @return {void} This method runs a browser command.
	 */
	async openTitle( page, title, query = {}, fragment = '' ) {
		const baseUrl = ( process.env.MW_SERVER || 'http://127.0.0.1:8080' ) + (
			process.env.MW_SCRIPT_PATH || '/w'
		);

		query.title = title;
		await page.goto(
			baseUrl + '/index.php?' +
			querystring.stringify( query ) +
			( fragment ? ( '#' + fragment ) : '' )
		);
	}
}

module.exports = Page;

tests/puppeteer/specs/user.spec.js

var assert = require( 'assert' );
var UserLoginPage = require( '../puppeteer-mediawiki/LoginPage' );
var puppeteer = require( 'puppeteer' ),
	timeout = process.env.SLOWMO ? 30000 : 20000,
	browser, page;

beforeAll( async () => {
	browser = await puppeteer.launch();
	page = await browser.newPage();
} );

describe( 'should be able to log in admin to @daily', () => {
	test( 'should login admin to @daily', async () => {
		await UserLoginPage.loginAdmin( page );
		await page.waitForSelector( UserLoginPage.userPage );
		const userPageText = await page.$eval( UserLoginPage.userPage, el => el.innerText );
		assert.strictEqual( userPageText, global.MEDIAWIKI_USER );
	}, timeout );
} );

Conclusion

Why Puppeteer?

There are a tonne of features that Puppeteer comes included with that make it desirable over WebdriverIO:

• Super easy setup. Just a

npm i --save-dev puppeteer jest jest-puppeteer

• It comes bundled with the browser, and it doesn’t need a browser driver.
• Tests at an average were way faster than that of WebdriverIO.
• Test showed great stability.
• There was no need to configure an explicit sleeping time.
• Because it is backed by Chromium, it generally supports most of latest Javascript features as and when they get supported by Chrome
• Superior device emulation features.
• Easy to configure.

Experiences:

• Setting up Puppeteer was super simple as compared to setting up WebdriverIO on my local machine.
• Async syntax of Puppeteer is a bit less readable at first but at least there are no callback hells.
• I tried running Puppeteer both headless and in-browser and both were extremely fast.
• The documentation are really good but they are scattered so a bit harder to find.

To summarize, it will be extremely useful to evaluate Puppeteer as a possible replacement for WebdriverIO as a browser automation framework and discuss the scope of migrating it to eventually all of Wikimedia's front-end repos