[go: up one dir, main page]
Page MenuHomePhabricator
Create Task
Maniphest T307914

Potential time based race condition with webdriver.io xunit filenames
Closed, ResolvedPublic
Actions

Description

When building the job quibble-daily-InterwikiSorting-vendor-mysql-php72-docker I have encountered an UNSTABLE build.

Build kept for comparison:

Unstablehttps://integration.wikimedia.org/ci/job/quibble-daily-InterwikiSorting-vendor-mysql-php72-docker/259/
Good rebuildhttps://integration.wikimedia.org/ci/job/quibble-daily-InterwikiSorting-vendor-mysql-php72-docker/260/

The webdriver.io failure comes from mediawiki/core tests. Both builds have five tests running.

The bad one only has four xunit reports:

-rw-r--r-- 1 hashar hashar 3556 9 mai 12:19 bad/archive/log/WDIO.xunit-2022-05-09T12-19-28-801Z.xml
-rw-r--r-- 1 hashar hashar 7583 9 mai 12:19 bad/archive/log/WDIO.xunit-2022-05-09T12-19-28-806Z.xml
-rw-r--r-- 1 hashar hashar 2178 9 mai 12:19 bad/archive/log/WDIO.xunit-2022-05-09T12-19-28-840Z.xml
-rw-r--r-- 1 hashar hashar 1023 9 mai 12:19 bad/archive/log/WDIO.xunit-2022-05-09T12-19-31-236Z.xml

The good one has five:

-rw-r--r-- 1 hashar hashar 7585 9 mai 12:40 rebuild/archive/log/WDIO.xunit-2022-05-09T12-40-54-041Z.xml
-rw-r--r-- 1 hashar hashar 1502 9 mai 12:40 rebuild/archive/log/WDIO.xunit-2022-05-09T12-40-54-066Z.xml
-rw-r--r-- 1 hashar hashar 3659 9 mai 12:41 rebuild/archive/log/WDIO.xunit-2022-05-09T12-40-54-116Z.xml
-rw-r--r-- 1 hashar hashar 2174 9 mai 12:40 rebuild/archive/log/WDIO.xunit-2022-05-09T12-40-54-160Z.xml
-rw-r--r-- 1 hashar hashar 1023 9 mai 12:40 rebuild/archive/log/WDIO.xunit-2022-05-09T12-40-56-808Z.xml

I suspect a time based race condition in the wdio xunit reporter which forge two files with the same name and write two set of test result in the same file.

Details

SubjectRepoBranchLines +/-
wdio-mediawiki: Add random element to junit-reporter file namemediawiki/coremaster+2 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

hashar created this task.May 9 2022, 12:58 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 9 2022, 12:58 PM
hashar added a comment.May 9 2022, 1:12 PM

The bad XML file is WDIO.xunit-2022-05-09T12-19-28-806Z.xml it has <testsuite name="Special RecentChanges"> which is the MediaWiki core test suite tests/selenium/specs/recentchanges.js.

Then the file has at some point the lines:

</testsuites>ut>
    </testcase>
    <testcase classname="chrome.90_0_4430_212.linux.Page" name="should be creatable" time="0">

Which comes from tests/selenium/specs/page.js.

What I suspect is the reporter generated two identical file names. The result for the Page suite got written then the one from the Recentchange suite. End result is borked :-\

Looks like the issue comes from our wdio-mediawiki NodeJS module, specially:

tests/selenium/wdio-mediawiki/wdio-defaults.conf.js
// See also: https://webdriver.io/docs/dot-reporter.html
reporters: [
    // See also: https://webdriver.io/docs/spec-reporter/
    'spec',
    // See also: https://webdriver.io/docs/junit-reporter/
    [ 'junit', {
        outputDir: logPath,
        outputFileFormat: function () {
            return `WDIO.xunit-${makeFilenameDate()}.xml`;
        }
    } ]
],

Which I guess is subject to time based race condition when the function call to new Date().toISOString() is done in the same millisecond?

zeljkofilipin added a project: User-zeljkofilipin.May 9 2022, 1:40 PM
Jrbranaa moved this task from Inbox to Test Infrastructure on the Quality-and-Test-Engineering-Team board.May 31 2022, 10:07 PM
hashar mentioned this in T317878: selenium-daily Jenkins jobs fail sometimes with: "Failed to read test report file".Sep 15 2022, 9:00 PM
hashar merged a task: T317878: selenium-daily Jenkins jobs fail sometimes with: "Failed to read test report file".
hashar added a subscriber: Masumrezarock100.
zeljkofilipin moved this task from Backlog 🪒 to Q1 👔 on the User-zeljkofilipin board.Sep 16 2022, 9:58 AM
zeljkofilipin moved this task from Inbox to Selenium on the MediaWiki-Core-Tests board.
zeljkofilipin claimed this task.Oct 25 2022, 1:01 PM
zeljkofilipin moved this task from Q1 👔 to Deep work 🌊 on the User-zeljkofilipin board.
zeljkofilipin changed the task status from Open to In Progress.Oct 25 2022, 1:11 PM
zeljkofilipin triaged this task as High priority.
zeljkofilipin removed zeljkofilipin as the assignee of this task.Oct 28 2022, 8:59 AM
zeljkofilipin moved this task from Deep work 🌊 to Q1 👔 on the User-zeljkofilipin board.
zeljkofilipin claimed this task.Nov 9 2022, 4:21 PM
zeljkofilipin moved this task from Q1 👔 to Deep work 🌊 on the User-zeljkofilipin board.
zeljkofilipin removed zeljkofilipin as the assignee of this task.Nov 11 2022, 11:30 AM
zeljkofilipin moved this task from Deep work 🌊 to Q1 👔 on the User-zeljkofilipin board.
zeljkofilipin added a comment.Nov 23 2022, 5:28 PM

I guess we could use crypto.randomBytes() to generate a more random file name.

zeljkofilipin claimed this task.Nov 23 2022, 5:29 PM
zeljkofilipin moved this task from Q1 👔 to Deep work 🌊 on the User-zeljkofilipin board.
zeljkofilipin changed the task status from In Progress to Open.Dec 9 2022, 11:35 AM
zeljkofilipin removed zeljkofilipin as the assignee of this task.
zeljkofilipin moved this task from Deep work 🌊 to Q1 👔 on the User-zeljkofilipin board.
zeljkofilipin added a parent task: T210726: Selenium framework maintenance.Feb 22 2023, 5:55 PM
zeljkofilipin mentioned this in T210726: Selenium framework maintenance.
zeljkofilipin moved this task from Q1 👔 to Backlog 🪒 on the User-zeljkofilipin board.
TAdeleye_WMF edited projects, added Quality-and-Test-Engineering-Team (Test Infrastructure); removed Quality-and-Test-Engineering-Team.Mar 1 2023, 2:56 PM
zeljkofilipin raised the priority of this task from High to Needs Triage.Oct 31 2023, 4:15 PM
WMDE-leszek subscribed.Fri, Sep 6, 11:09 AM
gerritbot added a comment.Fri, Sep 6, 11:19 AM

Change #1071170 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/core@master] wdio-mediawiki: Add random element to junit-reporter file name

https://gerrit.wikimedia.org/r/1071170

gerritbot added a project: Patch-For-Review.Fri, Sep 6, 11:19 AM
Lucas_Werkmeister_WMDE claimed this task.Fri, Sep 6, 11:20 AM
Lucas_Werkmeister_WMDE edited projects, added Wikidata Dev Team (Wikidata.org Slice); removed Patch-For-Review.
Lucas_Werkmeister_WMDE subscribed.

Happened again in InterwikiSorting build #155, so here’s a suggested fix (similar to what @zeljkofilipin suggested above, but without an async function).

Lucas_Werkmeister_WMDE added a project: Patch-For-Review.Fri, Sep 6, 11:20 AM
Lucas_Werkmeister_WMDE moved this task from In Task Breakdown to Ready for Peer Review on the Wikidata Dev Team (Wikidata.org Slice) board.
Maintenance_bot added a project: Wikidata.Fri, Sep 6, 11:29 AM
zeljkofilipin moved this task from Backlog 🪒 to Deep work 🌊 on the User-zeljkofilipin board.Fri, Sep 6, 3:34 PM
ArthurTaylor removed Lucas_Werkmeister_WMDE as the assignee of this task.Mon, Sep 9, 8:40 AM
ArthurTaylor moved this task from Ready for Peer Review to In Peer Review on the Wikidata Dev Team (Wikidata.org Slice) board.
ArthurTaylor moved this task from In Peer Review to Ready for Tech Verification on the Wikidata Dev Team (Wikidata.org Slice) board.
gerritbot added a comment.Mon, Sep 9, 9:09 AM

Change #1071170 merged by jenkins-bot:

[mediawiki/core@master] wdio-mediawiki: Add random element to junit-reporter file name

https://gerrit.wikimedia.org/r/1071170

ReleaseTaggerBot added a project: MW-1.43-notes (1.43.0-wmf.22; 2024-09-10).Mon, Sep 9, 10:00 AM
hashar closed this task as Resolved.Mon, Sep 9, 2:56 PM
hashar assigned this task to Lucas_Werkmeister_WMDE.

There is a similar issue with video recording due to parallel instances sharing the same Xvfb buffer T344754

Adding some randomization is a great idea. Well done @Lucas_Werkmeister_WMDE

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits