Run HTML dump scraper (June 2024)
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	awight
	May 29 2024, 9:11 AM

Description

We need to do the full run again, due to a mistake in how we were detecting template-generated content.

This task will be linked to other subtasks which should be completed before the run.

Enable production Prometheus configuration.
Re-run on dewiki for 2024-04-01 and verify that the affected columns are updated.
- (Output to a sibling directory named "20240401-v0.3.1")
Run on all wikis, on the 2024-05-01 snapshot.
Update the changelog to explain the data quality issues which we fixed, and mark up results spreadsheets so that deprecations are clear.
Upload results to results spreadsheet
Publish the metadata and link to public data from Figshare
- https://figshare.com/articles/dataset/Reference_usage_and_in-page_reuse_all_Wikimedia_wikis_snapshot_2024-05-01/26003965
- https://doi.org/10.6084/m9.figshare.26003965.v1

Details

	Subject	Repo	Branch	Lines +/-
	Revert "Temporary monitoring for scraper"	operations/puppet	production	+1 -19
	Temporary monitoring for scraper	operations/puppet	production	+19 -1

Customize query in gerrit

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved		awight	T366144 Run HTML dump scraper (June 2024)
Resolved		Tobi_WMDE_SW	T364981 Scraper: Fix how template output is collected
Resolved	BUG REPORT	ssastry	T364925 Scraper: Parsoid attaches mw:Transclusion data in <ref> tag to unexpected DOM node
Resolved		Tobi_WMDE_SW	T364471 How common is cite_error_references_duplicate_key?
Resolved		awight	T363675 Publish scraper results on figshare
Resolved		awight	T366436 Scraper: truncated_json error in CiteRefs
Invalid		None	T366457 [draft] Corrupt Enterprise HTML dump for arwiki, 2024-05-01 snapshot

Event Timeline

awight created this task.May 29 2024, 9:11 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 29 2024, 9:11 AM

awight added a subtask: T364981: Scraper: Fix how template output is collected.May 29 2024, 9:11 AM

awight added a subtask: T364471: How common is cite_error_references_duplicate_key?.

awight updated the task description. (Show Details)

awight updated the task description. (Show Details)May 29 2024, 9:19 AM

awight added a subtask: T363675: Publish scraper results on figshare.May 29 2024, 9:24 AM

WMDE-Fisch moved this task from Incoming to Ready for pickup on the WMDE-References-FocusArea board.May 29 2024, 9:57 AM

Tobi_WMDE_SW closed subtask T364471: How common is cite_error_references_duplicate_key? as Resolved.May 29 2024, 12:38 PM

Tobi_WMDE_SW closed subtask T364981: Scraper: Fix how template output is collected as Resolved.

thiemowmde added a project: WMDE-TechWish-Sprint-2024-05-29.May 29 2024, 1:47 PM

Change #1037437 had a related patch set uploaded (by Awight; author: Awight):

[operations/puppet@production] Temporary monitoring for scraper

https://gerrit.wikimedia.org/r/1037437

gerritbot added a project: Patch-For-Review.May 30 2024, 8:50 AM

Change #1037437 merged by Filippo Giunchedi:

[operations/puppet@production] Temporary monitoring for scraper

https://gerrit.wikimedia.org/r/1037437

Maintenance_bot removed a project: Patch-For-Review.May 30 2024, 9:30 AM

awight moved this task from Sprint Backlog to Doing on the WMDE-TechWish-Sprint-2024-05-29 board.May 30 2024, 10:29 AM

awight updated the task description. (Show Details)

awight claimed this task.May 30 2024, 1:24 PM

Lina_Farid_WMDE moved this task from Ready for pickup to In progress on the WMDE-References-FocusArea board.May 31 2024, 11:03 AM

Verification is slightly harder than expected, because one of the code changes makes it possible to time out on individual pages. This caused the page count to decrease by 224 pages, presumably due to length and complexity.

Generally, columns we expected to stay the same are nearly the same, and the ones we expected to change are very different. For example, ref_by_transclusion_count increased by 47%, and refs_with_solely_transclusion_count increased by 177 *fold*. proportion_of_refs_from_transclusion increased by 48%. Lists of templates producing refs have increased with lots of templates that we were missing due to their structure.

Full details are in the worksheet v0.3.1 verification

I'll go ahead and start the full run.

awight updated the task description. (Show Details)May 31 2024, 11:41 AM

The timeout issue is interesting. Our new run reported 569 timed-out pages, but only 224 fewer pages were processed than in the previous run using the older software version. Maybe the unexplained 345 pages failed silently in the older version, and the errors weren't logged as nicely?

We could test this theory locally by running a few of the timed-out pages through the older version of the software.

awight moved this task from In progress to Watching on the WMDE-References-FocusArea board.May 31 2024, 11:57 AM

awight moved this task from Watching to In progress on the WMDE-References-FocusArea board.

awight moved this task from Doing to Watching / Epic / Stalled on the WMDE-TechWish-Sprint-2024-05-29 board.

awight updated the task description. (Show Details)Jun 7 2024, 12:22 PM

Change #1040075 had a related patch set uploaded (by Awight; author: Awight):

[operations/puppet@production] [DNM] Revert "Temporary monitoring for scraper"

https://gerrit.wikimedia.org/r/1040075

gerritbot added a project: Patch-For-Review.Jun 7 2024, 12:47 PM

awight moved this task from Watching / Epic / Stalled to Demo on the WMDE-TechWish-Sprint-2024-05-29 board.Jun 10 2024, 12:18 PM

awight removed awight as the assignee of this task.Jun 10 2024, 12:21 PM

awight updated the task description. (Show Details)