Maniphest T364406

Migrate IP masking dashboard data pipeline to airflow
Open, Stalled, MediumPublic
Actions

Assigned To

Authored By

	jwang
	May 7 2024, 5:00 PM

Tags

Referenced Files

None

Subscribers

Description

Goal

The IP Masking (Temp Accounts) guardrail metrics dashboard currently is maintained by the product analytics team. The staging tables are fetched monthly by a cron job into a database under jwang's account. We plan to migrate the queries to Airflow, so that it can be fully automated and easily maintained.

Dependency

Add cu_log to the data lake. T364398

To Do

Write updated SQL queries for table creation and updating
Write Airflow DAG
Build new version of tables
Backfill old data to new tables
Update metric queries to use new tables
Update Superset dashboard to use new tables

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Stalled		jwang	T364406 Migrate IP masking dashboard data pipeline to airflow
		Open		jwang	T376081 Measure number of blocks on temp accounts

Event Timeline

jwang created this task.May 7 2024, 5:00 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 7 2024, 5:00 PM

mpopov mentioned this in T364398: Add MW table 'cu_log' to data lake.May 14 2024, 4:12 PM

kostajh added a project: Temporary accounts.May 21 2024, 1:21 PM

kostajh subscribed.

jwang moved this task from Next 2 weeks to Doing on the Product-Analytics (Kanban) board.May 29 2024, 5:24 PM

Tchanders moved this task from Inbox to Figure out Analytics/Instrumentation for Temp Accounts rollout on the Temporary accounts board.Jun 4 2024, 12:58 PM

Tchanders edited projects, added Temporary accounts (Figure out Analytics/Instrumentation for Temp Accounts rollout); removed Temporary accounts.

Write updated SQL queries for table creation and updating

Queries have been checked in to Gitlab: link

jwang updated the task description. (Show Details)Jun 5 2024, 5:32 PM

DAG files for trust_safety_admin_action_daily have been submitted to gitlab. A request for review and merge has been made. link

mpopov triaged this task as Medium priority.Jun 11 2024, 1:00 PM

Pausing this work in favor of T371402: [WE4.4] Additional monitoring capabilities for rollout of Temporary Account

mpopov mentioned this in T346466: Investigate how temporary accounts are logged and categorized.Jul 30 2024, 3:18 PM

bearloga opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/795

analytics_product: T&S admin actions: fix MW sensor

bearloga merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/795

analytics_product: T&S admin actions: fix MW sensor

Maintenance_bot removed a project: Patch-For-Review.Aug 13 2024, 3:30 PM

Mayakp.wiki subscribed.Sep 9 2024, 7:02 PM