[go: up one dir, main page]

Page MenuHomePhabricator

CI PerformanceBudgetTest fails on GrowthExperiments master branch with 0.5kB difference
Closed, ResolvedPublic

Description

On an empty change on top of the master branch ( https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/1064336 ), CI now fails for GrowthExperiments:

00:33:13.607 There was 1 failure:
00:33:13.607 
00:33:13.607 1) MediaWiki\Skins\Vector\Tests\Structure\PerformanceBudgetTest::testTotalModulesSize with data set "vector-2022" ('vector-2022', array('50KB', '21KB'))
00:33:13.608 T346813: Performance budget for styles in skin vector-2022 on main article namespace has been exceeded. Total size of styles modules is 23.5 Kbs is greater than the current budget size of 23 Kbs (see Vector/bundlesize.config.json).
00:33:13.608 If you are adding code on page load, please reduce styles that you are loading on page load or talk to the web team about increasing the budget.
00:33:13.608 If you are not adding code, and this seems to be an error, it is possible that something running without CI has bypassed this check and we can address this separately.Please reach out to the web team to discuss this via Phabricator or #talk-to-web.[PLEASE DO NOT SKIP THIS TEST. If this is blocking a deploy this might signal a potential performance regression with the desktop site.Instead of skipping the test you can increase the value in bundlesize.config.json and create a ticket to investigate this error. If the error is > 1kb please tag this as a train blocker.Please tag the ticket #Web-Team-Backlog.
00:33:13.608 
00:33:13.608 The following modules are enabled on page load:
00:33:13.608 skins.vector.search.codex.styles
00:33:13.609 skins.vector.styles
00:33:13.609 skins.vector.icons
00:33:13.609 ext.flaggedRevs.basic
00:33:13.609 ext.uls.pt
00:33:13.609 ext.visualEditor.desktopArticleTarget.noscript
00:33:13.609 wikibase.client.init
00:33:13.609 wikibase.client.data-bridge.externalModifiers
00:33:13.609 ext.wikimediaBadges
00:33:13.609 ext.wikimediamessages.styles
00:33:13.609 site.styles
00:33:13.609 noscript
00:33:13.609 user.styles
00:33:13.609 Failed asserting that 23974 is equal to 23504.0 or is less than 23504.0.
00:33:13.609 
00:33:13.609 /workspace/src/skins/Vector/tests/phpunit/structure/PerformanceBudgetTest.php:212

It would be nice to actually identify the source of this change in the style modules, since it is obviously not this change to GrowthExperiments.

Event Timeline

It's probably just the new release of Codex that used up the 'free' space in the limits?

It's probably just the new release of Codex that used up the 'free' space in the limits?

Maybe? But shouldn't the change that touched this Codex module then not have failed this test? CI being suddenly read on master hints at there still being something wrong somewhere in the process that got us here.

But also:
Why is this failing on all the quibble-vendor-mysql-php*-noselenium tests, but passing on wmf-quibble-vendor-mysql-php74?

It's probably just the new release of Codex that used up the 'free' space in the limits?

Maybe? But shouldn't the change that touched this Codex module then not have failed this test? CI being suddenly read on master hints at there still being something wrong somewhere in the process that got us here.

But also:
Why is this failing on all the quibble-vendor-mysql-php*-noselenium tests, but passing on wmf-quibble-vendor-mysql-php74?

The clone lists differ. Look for INFO:quibble.cmd:Projects: for the list:

https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php74-noselenium/34058/consoleFull

mediawiki/core,
mediawiki/extensions/AbuseFilter,
mediawiki/extensions/AntiSpoof,
mediawiki/extensions/ArticlePlaceholder,
mediawiki/extensions/BetaFeatures,
mediawiki/extensions/CentralAuth,
mediawiki/extensions/CheckUser,
mediawiki/extensions/CirrusSearch,
mediawiki/extensions/Cite,
mediawiki/extensions/CodeEditor,
mediawiki/extensions/CommunityConfiguration,
mediawiki/extensions/ConfirmEdit,
mediawiki/extensions/DiscussionTools,
mediawiki/extensions/Echo,
mediawiki/extensions/Elastica,
mediawiki/extensions/EventBus,
mediawiki/extensions/EventLogging,
mediawiki/extensions/EventStreamConfig,
mediawiki/extensions/FlaggedRevs,
mediawiki/extensions/Flow,
mediawiki/extensions/Gadgets,
mediawiki/extensions/GeoData,
mediawiki/extensions/GlobalBlocking,
mediawiki/extensions/Graph,
mediawiki/extensions/GrowthExperiments,
mediawiki/extensions/GuidedTour,
mediawiki/extensions/JsonConfig,
mediawiki/extensions/Kartographer,
mediawiki/extensions/Linter,
mediawiki/extensions/MobileApp,
mediawiki/extensions/MobileFrontend,
mediawiki/extensions/PageImages,
mediawiki/extensions/PageViewInfo,
mediawiki/extensions/ParserFunctions,
mediawiki/extensions/PdfHandler,
mediawiki/extensions/Popups,
mediawiki/extensions/PropertySuggester,
mediawiki/extensions/Renameuser,
mediawiki/extensions/Scribunto,
mediawiki/extensions/SecurePoll,
mediawiki/extensions/SiteMatrix,
mediawiki/extensions/SyntaxHighlight_GeSHi,
mediawiki/extensions/TemplateData,
mediawiki/extensions/TextExtracts,
mediawiki/extensions/Thanks,
mediawiki/extensions/TimedMediaHandler,
mediawiki/extensions/TorBlock,
mediawiki/extensions/UniversalLanguageSelector,
mediawiki/extensions/VisualEditor,
mediawiki/extensions/WikiEditor,
mediawiki/extensions/Wikibase,
mediawiki/extensions/WikibaseCirrusSearch,
mediawiki/extensions/WikibaseLexeme,
mediawiki/extensions/WikibaseLexemeCirrusSearch,
mediawiki/extensions/WikibaseMediaInfo,
mediawiki/extensions/WikibaseQualityConstraints,
mediawiki/extensions/WikimediaBadges,
mediawiki/extensions/WikimediaEvents,
mediawiki/extensions/WikimediaMessages,
mediawiki/extensions/cldr,
mediawiki/skins/MinervaNeue,
mediawiki/skins/Vector,
mediawiki/vendor

… whereas for https://integration.wikimedia.org/ci/job/wmf-quibble-vendor-mysql-php74/15105/consoleFull

mediawiki/core,
mediawiki/extensions/AbuseFilter,
mediawiki/extensions/AntiSpoof,
mediawiki/extensions/Babel,
mediawiki/extensions/BetaFeatures,
mediawiki/extensions/CheckUser,
mediawiki/extensions/CirrusSearch,
mediawiki/extensions/Cite,
mediawiki/extensions/CiteThisPage,
mediawiki/extensions/CodeEditor,
mediawiki/extensions/CommunityConfiguration,
mediawiki/extensions/ConfirmEdit,
mediawiki/extensions/ContentTranslation,
mediawiki/extensions/Disambiguator,
mediawiki/extensions/Echo,
mediawiki/extensions/Elastica,
mediawiki/extensions/EventBus,
mediawiki/extensions/EventLogging,
mediawiki/extensions/EventStreamConfig,
mediawiki/extensions/FileImporter,
mediawiki/extensions/Gadgets,
mediawiki/extensions/GeoData,
mediawiki/extensions/GlobalCssJs,
mediawiki/extensions/GlobalPreferences,
mediawiki/extensions/Graph,
mediawiki/extensions/GrowthExperiments,
mediawiki/extensions/GuidedTour,
mediawiki/extensions/IPInfo,
mediawiki/extensions/ImageMap,
mediawiki/extensions/InputBox,
mediawiki/extensions/Interwiki,
mediawiki/extensions/JsonConfig,
mediawiki/extensions/Kartographer,
mediawiki/extensions/Math,
mediawiki/extensions/MediaModeration,
mediawiki/extensions/MobileApp,
mediawiki/extensions/MobileFrontend,
mediawiki/extensions/NavigationTiming,
mediawiki/extensions/PageImages,
mediawiki/extensions/PageTriage,
mediawiki/extensions/PageViewInfo,
mediawiki/extensions/ParserFunctions,
mediawiki/extensions/PdfHandler,
mediawiki/extensions/Poem,
mediawiki/extensions/ProofreadPage,
mediawiki/extensions/SandboxLink,
mediawiki/extensions/Scribunto,
mediawiki/extensions/SiteMatrix,
mediawiki/extensions/SpamBlacklist,
mediawiki/extensions/TemplateData,
mediawiki/extensions/Thanks,
mediawiki/extensions/TimedMediaHandler,
mediawiki/extensions/Translate,
mediawiki/extensions/UniversalLanguageSelector,
mediawiki/extensions/VisualEditor,
mediawiki/extensions/WikiEditor,
mediawiki/extensions/WikiLove,
mediawiki/extensions/Wikibase,
mediawiki/extensions/WikibaseCirrusSearch,
mediawiki/extensions/WikibaseMediaInfo,
mediawiki/extensions/WikimediaMessages,
mediawiki/extensions/cldr,
mediawiki/skins/MinervaNeue,
mediawiki/skins/Vector,
mediawiki/vendor

This is also blocking merges in Quibble at
https://gerrit.wikimedia.org/r/c/integration/quibble/+/1063912

A check in a skin repo should not run be running with extension hooks enabled, because that's unenforcable in practice, as demonstrated by this CI failure, failing to raise the issue in CI on the actual commit that indirectly causes the excess, and instead until hours/days later starts blocking random unrelated merges, leaving no obviously attributable cause. This is inherently the case, because you're going to have an interplay between 1) a change in core adding unused code, 2) an extension conditionally making use of a thing, 3) another core change or skin enabling said condition without depending on the extension, 4) CI is now failing for repos that do depend on that extension.

Not to mention that the check is non-determinstic, and to a first approximation meaningless since it doesn't actually represent production. This is what synthetic testing is for: https://wikitech.wikimedia.org/wiki/Performance/Synthetic_testing. Instead, this budget is not measuring the software, or production, but instead measuring an undefined subset of extensions, contextual to what repository a Git changeset is being committed, and what its currently installed CI dependencies are. This means a repo like Wikibase with more dependencies is going to exceed the budget, when the actual repo that owns the check (Vector) is passing. Wikibase may very well be testing with more dependencies that in production (e.g. running tests for both Wikidata/WikibaseRepo and Wikipedia/WikibaseClient one after the other).

We have MediaWikiIntegationTestCase->clearHooks for this purpose.

By updating the Vector test to clear extension hooks, it will reliably assert what modules load by default on a page given core + Vector skin; which is the scope these repos can and should be enforcing their budgets on, and is a meaningful baseline for the software.

Fixed with Bump vector-2022 bundlesize by @Catrope

I'm still leaving this task open for now, so that it maybe can be used as a data point for investigating the underlying issue of what caused this and why it was not caught when that was merged. What @Krinkle writes makes sense to me.

I did that because it was also blocking merges in the Charts extension, e.g. https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Chart/+/1063857 . FWIW the Chart extension doesn't contain any ResourceLoader modules and doesn't affect how much code is loaded on page views, so I found this bundlesize failure blocking the CI there unexpected to say the least.

Change #1069137 had a related patch set uploaded (by Michael Große; author: Michael Große):

[mediawiki/skins/Vector@master] tests: Bump general styles bundle size

https://gerrit.wikimedia.org/r/1069137

Change #1069137 merged by jenkins-bot:

[mediawiki/skins/Vector@master] tests: Bump general styles bundle size

https://gerrit.wikimedia.org/r/1069137

Change #1069278 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/skins/Vector@master] Adjustments to bundle size tests

https://gerrit.wikimedia.org/r/1069278

Change #1071984 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/extensions/WikimediaMessages@master] Add bundlesize test to WikimediaMessages

https://gerrit.wikimedia.org/r/1071984

Change #1071988 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/extensions/UniversalLanguageSelector@master] Add bundlesize tests to UniversalLanguageSelector extension

https://gerrit.wikimedia.org/r/1071988

Change #1071991 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/extensions/FlaggedRevs@master] Add bundlesize test for modules loading on page load

https://gerrit.wikimedia.org/r/1071991

Change #1071992 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/extensions/VisualEditor@master] Add bundlesize test for modules loading on page load

https://gerrit.wikimedia.org/r/1071992

Change #1071993 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/extensions/GrowthExperiments@master] Add bundlesize test for modules loading on page load

https://gerrit.wikimedia.org/r/1071993

Change #1071994 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/skins/Vector@master] POC: Rethink how bundlesize tests work

https://gerrit.wikimedia.org/r/1071994

Jdlrobson lowered the priority of this task from High to Medium.Sep 11 2024, 9:24 PM

Change #1071984 merged by jenkins-bot:

[mediawiki/extensions/WikimediaMessages@master] Add bundlesize test to WikimediaMessages

https://gerrit.wikimedia.org/r/1071984

Change #1073561 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/extensions/CentralNotice@master] Track bundlesizes for modules loaded on page load

https://gerrit.wikimedia.org/r/1073561

Change #1071988 merged by jenkins-bot:

[mediawiki/extensions/UniversalLanguageSelector@master] Add bundlesize tests to UniversalLanguageSelector extension

https://gerrit.wikimedia.org/r/1071988

Change #1071993 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Add bundlesize test for modules loading on page load

https://gerrit.wikimedia.org/r/1071993

Change #1075536 had a related patch set uploaded (by Sergio Gimeno; author: Sergio Gimeno):

[mediawiki/skins/Vector@master] PerformanceBugetTest: bump vector-2022 bundle size budget

https://gerrit.wikimedia.org/r/1075536

Change #1075536 merged by jenkins-bot:

[mediawiki/skins/Vector@master] PerformanceBugetTest: increase budget for GrowthExperiments styles

https://gerrit.wikimedia.org/r/1075536

Change #1075648 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/skins/Vector@master] Remove PerformanceBudgetTest

https://gerrit.wikimedia.org/r/1075648

Change #1075659 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/core@master] POC: Enforce bundle size requirement for modules loaded on page load

https://gerrit.wikimedia.org/r/1075659

Change #1071994 abandoned by Jdlrobson:

[mediawiki/skins/Vector@master] POC: Rethink how bundlesize tests work

Reason:

Moved to https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1075659

https://gerrit.wikimedia.org/r/1071994

Change #1075648 merged by jenkins-bot:

[mediawiki/skins/Vector@master] Remove PerformanceBudgetTest

https://gerrit.wikimedia.org/r/1075648

Change #1075663 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/core@master] POC: Enforce bundle size requirement for modules loaded on page load

https://gerrit.wikimedia.org/r/1075663

Change #1075659 abandoned by Jdlrobson:

[mediawiki/core@master] POC: Enforce bundle size requirement for modules loaded on page load

Reason:

https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1075663

https://gerrit.wikimedia.org/r/1075659

Jdlrobson claimed this task.
Jdlrobson subscribed.

I chatted to James today and we decided to disable the test. Please subscribe to T360590 (and scroll to the latest comment T360590#10178103) to see what we're proposing will replace this.

Change #1069278 abandoned by Jdlrobson:

[mediawiki/skins/Vector@master] Adjustments to bundle size tests

Reason:

Continued at https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1075663?usp=search

https://gerrit.wikimedia.org/r/1069278

Change #1071991 merged by jenkins-bot:

[mediawiki/extensions/FlaggedRevs@master] Add bundlesize test for modules loading on page load

https://gerrit.wikimedia.org/r/1071991

Change #1073561 merged by jenkins-bot:

[mediawiki/extensions/CentralNotice@master] Track bundlesizes for modules loaded on page load

https://gerrit.wikimedia.org/r/1073561