[EPIC] Encourage developers to increase code coverage
Closed, ResolvedPublic
Actions

Assigned To

None

Authored By

	phuedx
	May 25 2015, 12:52 PM

Description

As a developer I want to develop on software that doesn't break in unexpected ways so that I can feel confident in the changes I am making to the codebase.

Possible parts of a solution

Generate a code coverage report, i.e. increase/decrease in code coverage, in the pre-review hook and warn the developer if their patch decreases code coverage (or doesn't increase it, maybe)
- T71685: Generate PHP code coverage reports for extensions
Tooling:
- T111396: [EPIC] Wikimedia should use a standard set of tools for managing code quality
Dashboards:
- T108768: Create QA Health scoreboard
- Do something with https://integration.wikimedia.org/cover/mediawiki-core/master/php/index.dashboard.html
  - expand to cover extensions (see also: https://tools.wmflabs.org/coverage/MobileFrontend/extensions.html )

Details

	Subject	Repo	Branch	Lines +/-
	Provide command to adjust phpunit.xml for code coverage	mediawiki/core	master	+65 -2
	Add thresholds for QUnit coverage to ensure we don't get lazy	mediawiki/extensions/MobileFrontend	master	+4 -0

Customize query in gerrit

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved		• Jhernandez	T104318 [GOAL]: Testing practices & expose test coverage
Resolved		None	T100294 [EPIC] Encourage developers to increase code coverage
Resolved		Legoktm	T71685 Generate PHP code coverage reports for extensions
Declined		None	T88434 generate Wikibase.git code coverage on Jenkins
Declined		None	T74318 [Task] add possibility to generate proper phpunit coverage reports of Wikibase.git
Duplicate		None	T88435 [Task] generate patch code coverage on gerrit patch-set upload for wikibase.git
Resolved		Jdlrobson	T102006 Add @covers annotations to MobileFrontend PHPUnit tests
Duplicate		None	T151333 Proof of concept to generate coverage report of some Wikimedia mediawiki extensions
Resolved		hashar	T185211 Generate coverage report for Wikidata extensions
Open		None	T288396 [REPO][CLIENT][SW] Re-start Wikibase test coverage reporting
Resolved		Legoktm	T101544 Provide (pre-merge) code coverage reports on patchsets
Declined		Jdlrobson	T103440 Write tests for modules that are in stable and have low code coverage
Declined		Jdlrobson	T112568 `grunt qunit` tool no longer works
Declined		None	T103578 Improve Gather unit test coverage
Resolved		Legoktm	T111396 [EPIC] Wikimedia should use a standard set of tools for managing code quality
Declined		None	T86587 [jsduck] Various custom tags should be easily shareable between projects
Declined		None	T107588 EPIC: Detect and prevent UI regressions
Resolved	Spike	ovasileva	T236722 [Spike, 10hrs] Investigate automated visual regression tools
Declined		None	T184657 Generate JavaScript code coverage reports for extensions
Declined		None	T116808 Provide a CI job to generate JS code coverage reports for extensions by using karma-coverage instead of just karma

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Tgr added a project: Epic.Jun 4 2015, 10:40 PM

greg triaged this task as Medium priority.Jun 5 2015, 6:44 PM

hashar mentioned this in T101545: Provide infrastructure to store files by project/branch post-merge to compare with pre-merge.Jun 5 2015, 9:54 PM

hashar added a subtask: T101544: Provide (pre-merge) code coverage reports on patchsets.

• KLans_WMF removed a project: Mobile-Web-Sprint-48-Voyage-of-the-Damned.Jun 8 2015, 9:20 PM

• ggellerman subscribed.Jun 9 2015, 9:14 PM

Jdlrobson moved this task from Product triage to Developer backlog on the Gather board.Jun 10 2015, 7:25 PM

Change 220025 had a related patch set uploaded (by Jdlrobson):
Add thresholds for QUnit coverage to ensure we don't get lazy

https://gerrit.wikimedia.org/r/220025

gerritbot added a project: Patch-For-Review.Jun 22 2015, 11:55 PM

It seems a cheap way to do this would be to use the existing grunt qunit:cov command and then reject patches that push test coverage below a certain threshold (see patch above). This can be tweaked to not require any kind of storage and possibly plugged into Jenkins - thoughts @hashar ?

Jdlrobson added a subtask: T103578: Improve Gather unit test coverage.Jun 23 2015, 6:43 PM

Change 220025 merged by jenkins-bot:
Add thresholds for QUnit coverage to ensure we don't get lazy

https://gerrit.wikimedia.org/r/220025

• Konfused-Kitten mentioned this in rEMFR192f04807fd5: Add thresholds for QUnit coverage to ensure we don't get lazy.Jun 23 2015, 8:08 PM

Diffusion mentioned this in rMEXT22e469c338db: Updated mediawiki/extensions Project: mediawiki/extensions/MobileFrontend….Jun 23 2015, 8:09 PM

https://gerrit.wikimedia.org/r/220025 (branch master): WMF-deploy-2015-06-30_(1.26wmf12)

It seems a cheap way to do this would be to use the existing grunt qunit:cov command and then reject patches that push test coverage below a certain threshold (see patch above). This can be tweaked to not require any kind of storage and possibly plugged into Jenkins - thoughts @hashar ?

I'm not in favour of straight up rejection – you'll note that I said "warn the developer". I think that this patch jumps over a lot of discussion that we should be having as a team. We've only just started to have those discussions around the Q1 planning.

Legoktm removed a subscriber: • Forrestbot.Jun 29 2015, 5:49 PM

• Jhernandez edited projects, added Reading-Web-Planning; removed Web-Team-Backlog.Jul 2 2015, 12:04 PM

• Jhernandez moved this task from To classify to Q1 Goals on the Reading-Web-Planning board.

• Jhernandez added a parent task: T104318: [GOAL]: Testing practices & expose test coverage.

• Jhernandez moved this task from Q1 Goals to 2015-16 Q4 on the Reading-Web-Planning board.Jul 8 2015, 11:05 AM

• Jhernandez moved this task from 2015-16 Q4 to To classify on the Reading-Web-Planning board.Jul 8 2015, 11:07 AM

Jdlrobson moved this task from Backlog to Epics on the MobileFrontend board.Aug 4 2015, 6:37 PM

Jdlrobson added a parent task: T102006: Add @covers annotations to MobileFrontend PHPUnit tests.Aug 12 2015, 1:35 AM

Jdlrobson removed a parent task: T102006: Add @covers annotations to MobileFrontend PHPUnit tests.

Jdlrobson added a subtask: T102006: Add @covers annotations to MobileFrontend PHPUnit tests.

• ggellerman added a subscriber: dduvall.Aug 12 2015, 3:52 PM

Should we use this task as the main tracking task across all projects? I made T111546: [EPIC] Add a voting test code coverage report to gate pipeline to track that, but the more I think about it/look at the blockers of this task already, I think *this* task is it.

Is it OK for me/RelEng to "take" ownership of this task (basically: removing the reading-team related projects, and have them create any reading-specific tasks as blockers of this)? I'll just merge that other task (T111546) into this one, if that's OK.

Better to merge / decline T111546: [EPIC] Add a voting test code coverage report to gate pipeline imho. Until we get a coverage build that is fast enough, I don't see it being added to gate (it takes 2 hours currently).

greg merged a task: T111546: [EPIC] Add a voting test code coverage report to gate pipeline.Sep 4 2015, 6:59 PM

Jdlrobson awarded a token.Sep 4 2015, 10:58 PM

Is it OK for me/RelEng to "take" ownership of this task (basically: removing the reading-team related projects, and have them create any reading-specific tasks as blockers of this)? I'll just merge that other task (T111546) into this one, if that's OK.

Fine by me. There'll be reading-web-specific tasks that describe/track the work that we'll be doing so the beginning and end states will be almost the same, except that we'll have Release-Engineering-Team pushing us to do better too (!!!).

phuedx awarded a token.Sep 7 2015, 11:16 AM

greg updated the task description. (Show Details)Sep 8 2015, 9:06 PM

greg removed projects: Reading-Web-Planning, Patch-For-Review, MobileFrontend, Gather.

Jdlrobson added a parent task: T111396: [EPIC] Wikimedia should use a standard set of tools for managing code quality.Sep 16 2015, 6:34 PM

greg added a project: Release-Engineering-Epics.Sep 24 2015, 1:04 AM

greg removed a project: Release-Engineering-Team.

greg removed a parent task: T111396: [EPIC] Wikimedia should use a standard set of tools for managing code quality.Sep 24 2015, 1:13 AM

greg added a subtask: T111396: [EPIC] Wikimedia should use a standard set of tools for managing code quality.

greg updated the task description. (Show Details)

hashar changed the status of subtask T101544: Provide (pre-merge) code coverage reports on patchsets from Open to Stalled.Oct 6 2015, 12:35 PM

jayvdb subscribed.Oct 15 2015, 9:05 AM

Ricordisamoa subscribed.Jan 4 2016, 4:01 AM

Jdlrobson closed subtask T103440: Write tests for modules that are in stable and have low code coverage as Declined.Feb 2 2016, 1:02 AM

greg edited projects, added Release-Engineering-Team; removed Release-Engineering-Epics.Mar 11 2016, 10:08 PM

greg moved this task from INBOX to Epics (ARCHIVED) on the Release-Engineering-Team board.Mar 11 2016, 10:09 PM

Jdlrobson closed subtask T102006: Add @covers annotations to MobileFrontend PHPUnit tests as Resolved.Mar 30 2016, 6:58 PM

MBinder_WMF closed subtask T103578: Improve Gather unit test coverage as Declined.Jun 1 2016, 9:59 PM

hashar mentioned this in T147093: Run a phpunit coverage job pre-merge for libraries.Sep 30 2016, 10:23 PM

hashar created subtask T147778: Switch MediaWiki coverage job from PHP 5 to PHP 7.Oct 10 2016, 8:21 AM

Legoktm closed subtask T111396: [EPIC] Wikimedia should use a standard set of tools for managing code quality as Resolved.Jun 29 2017, 10:58 PM

• ggellerman unsubscribed.Nov 21 2017, 10:56 PM

Legoktm closed subtask T71685: Generate PHP code coverage reports for extensions as Resolved.Jan 23 2018, 11:42 PM

Legoktm changed the status of subtask T101544: Provide (pre-merge) code coverage reports on patchsets from Stalled to Open.Feb 2 2018, 8:43 AM

Legoktm closed subtask T101544: Provide (pre-merge) code coverage reports on patchsets as Resolved.Apr 3 2018, 1:43 AM

Legoktm removed a subtask: T147778: Switch MediaWiki coverage job from PHP 5 to PHP 7.Apr 14 2018, 7:32 AM

xSavitar subscribed.Nov 5 2018, 6:55 PM

Some ideas I have been playing with: T220939: Display extension test coverage in infobox, T220938: Display per-line test coverage during code review.

Aklapper added a project: Code-Health.May 4 2019, 10:13 PM

hashar unsubscribed.May 6 2019, 7:57 AM

• Phabricator_maintenance edited projects, added Release-Engineering-Team-TODO; removed Release-Engineering-Team.Jun 12 2019, 11:40 PM

• Phabricator_maintenance moved this task from Should be empty (use Release-Engineering-Team) to Epics on the Release-Engineering-Team-TODO board.Jun 12 2019, 11:41 PM

greg added a project: Release-Engineering-Team.Jun 21 2019, 10:35 PM

greg edited projects, added Release-Engineering-Team (Code Health); removed Release-Engineering-Team.Jul 6 2019, 5:59 AM

Change 526421 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[mediawiki/core@master] (wip) Provide command to adjust phpunit.xml for code coverage

https://gerrit.wikimedia.org/r/526421

gerritbot added a project: Patch-For-Review.Jul 30 2019, 1:42 PM

Change 526421 merged by jenkins-bot:
[mediawiki/core@master] Provide command to adjust phpunit.xml for code coverage

https://gerrit.wikimedia.org/r/526421

ReleaseTaggerBot added a project: MW-1.34-notes (1.34.0-wmf.22; 2019-09-10).Sep 9 2019, 8:00 PM

MobileFrontend now publishes JS code coverage reports: https://doc.wikimedia.org/MobileFrontend/master/js/lcov-report

Jdlrobson unsubscribed.Sep 10 2019, 11:59 PM

thcipriani removed a project: Release-Engineering-Team (Code Health).Apr 20 2021, 12:00 AM

thcipriani edited projects, added Release-Engineering-Team (thcipriani-workboard-fiddling); removed Release-Engineering-Team-TODO.Apr 20 2021, 3:44 AM

thcipriani moved this task from thcipriani-workboard-fiddling to Seen (ARCHIVE) on the Release-Engineering-Team board.Apr 20 2021, 4:05 AM

thcipriani edited projects, added Release-Engineering-Team; removed Release-Engineering-Team (thcipriani-workboard-fiddling).

thcipriani edited projects, added Release-Engineering-Team (Seen); removed Release-Engineering-Team.Apr 20 2021, 3:23 PM

Pppery removed a project: Patch-For-Review.Apr 1 2023, 8:02 PM

Should we close this epic? Or assign some specific target that we want to reach?

• Jrbranaa moved this task from Inbox to SonarCloud Admin on the Quality-and-Test-Engineering-Team board.Jul 12 2024, 9:25 PM

• Jrbranaa edited projects, added Quality-and-Test-Engineering-Team (SonarCloud Admin); removed Quality-and-Test-Engineering-Team.

hashar closed subtask T184657: Generate JavaScript code coverage reports for extensions as Declined.Thu, Oct 24, 2:25 PM

hashar mentioned this in T184657: Generate JavaScript code coverage reports for extensions.

I am claiming this one to have resolved years ago. The aim was to increase code coverage and we have plenty of coverage at https://doc.wikimedia.org/cover-extensions/ , a CI job which reports coverage improvement on patchset etc.

This was a program from ten years ago to improve testing practices at the foundation which resulted in many improvement. It has not been active in years as far as I know, but we still benefits from its improvements.

There are surely improvements to be made, but this task/goal/Epic has long been considered completed and decided to not pursue further.

hashar removed hashar as the assignee of this task.Thu, Oct 24, 2:40 PM

hashar subscribed.

[EPIC] Encourage developers to increase code coverageClosed, ResolvedPublicActions