Using statistical analysis, Bayesian inference, machine learning, and software/data engineering to solve problems and inform decisions in Product Analytics
User Details
- User Since
- Jul 27 2015, 4:15 PM (476 w, 5 h)
- Availability
- Available
- IRC Nick
- bearloga
- LDAP User
- Bearloga
- MediaWiki User
- MPopov (WMF) [ Global Accounts ]
Fri, Aug 23
Approved
Mon, Aug 19
not applicable, data scientist will check if tracking works
Instrumentation QA is two-step process. First, the engineer (whether the software engineer or a QTE) needs to QA the instrumentation as much as they can to make sure that it is producing the desired events (see https://wikitech.wikimedia.org/wiki/Metrics_Platform/How_to/Validate_Events, including that there aren't any validation errors).
Fri, Aug 16
Will sample consistently if given the same starting value (e.g. if we're sampling on page ID, the same page ID will always return the same assignment).
Thu, Aug 15
Mon, Aug 12
Questions for @JTannerWMF & @SNowick_WMF:
Aug 8 2024
Hi @ifried! @Iflorez and I discussed your question yesterday and it's going to come down to how the feature actually works. Either:
- Scenario A: User can switch between the tabs seamlessly (powered by JS) without needing to reload the page.
- In this scenario there is only one pageview of this special page no matter how many times the user goes back and forth between the tabs.
- Scenario B: When page loads it checks for ?tab=Events (default) or ?tab=Communities and switching between them causes the page to reload with different URI query parameter.
- In this scenario there is one pageview for each time the user switches tabs.
Aug 7 2024
@mforns: I just discussed this with @KCVelaga_WMF and confirmed that multiple pieces of information will be "all manifest at the same time, atomically"
Aug 1 2024
Apologies @Tchanders – I created the placeholder ticket based on misunderstanding/misinterpreting the term "rollback" when Niharika's request came in to have a rollback metric (no other details given) so I read "rollback" as rolling back Temp Accounts, and I see now that a much different thing was intended.
@larissagaulia: Can you please check with your team if T52655: Allow adding canonical names for custom namespaces is indeed still a blocker for this or if it has been resolved, just not linked to the ticket?
Jul 31 2024
@cjming @Ottomata: Another negative to document (and think about): event sanitization. We can configure sanitization/retention policies on a per-instrument basis since they are different streams/tables, but with the monostream/monotable we would lose that flexibility. Without changing how the current sanitization pipeline works, we would have a single entry in the allowlist for the monotable. We would have to reconsider how we evaluate risk when it comes to retaining sanitized data longer than 90 days.
Jul 30 2024
How do we measure this in an automated/pipeline-able way without requiring manual data input?
@Niharika: Is this still needed for any decision making with regards to Temp Accounts implementation or can I decline this?
Pausing this work in favor of T371402: [WE4.4] Additional monitoring capabilities for rollout of Temporary Account
Jul 29 2024
Please add the following to Negative Consequences:
- Vast majority of interaction data would be going into one massive table which will create significant limitations for how much data we would be able to query with Presto – potentially only an hour at a time as opposed to multiple days or weeks or even months that is possible now with the smaller, per-instrument tables. Depending on how powerful our Presto cluster is, we may would likely have to switch to working with interaction data exclusively outside of Superset's SQL Lab since Presto and Spark SQL differ substantially and require a high degree to effort to translate queries between those two SQL dialects.
- This will also negatively affect our ability to create Superset dashboards with Presto based on the un-aggregated interaction data, which has become a common practice among Product Analysts. We accept this consequence because the metrics we measure and make available in those dashboards and other reports should be pre-computed with data pipelines (that have access to more powerful and robust Spark SQL) rather than calculated on-the-fly with Presto. We can still use Presto but mainly for working with pre-computed measurements of interaction metrics rather than with raw interaction data.
Jul 26 2024
Jul 25 2024
Rescinding after checking in with @acooper and clarifying that this was created out of a misunderstanding.
Jul 24 2024
(The file attachment – a diagram I suspect? – is missing.)
Jul 19 2024
select normalized_host.project_class, ip, count(1) as view_count from pageview_actor where year = 2024 and month = 6 and day = 28 and http_status = '301' and agent_type = 'user' and uri_path = '/w/index.php' and regexp_like(uri_query, 'title=Special%3AGlobalUsage') and (is_redirect_to_pageview or is_pageview) group by 1, 2 order by view_count desc limit 1000
It also means there's nobody to ask to fix the behavior. I believe this requires engineering help from DPE.
@Mayakp.wiki: Special:GlobalUsage comes from Extension:GlobalUsage (GlobalUsage), which is a volunteer-authored extension.
Jul 18 2024
Thanks for checking in! No objection from me.
Jul 17 2024
@phuedx @VirginiaPoundstone: Okie dokie, I updated the task description with the data model specification and updated the requirements based on a guess of what is involved.
Jul 15 2024
@sbassett: I have folks on my team very eager to use this again for reporting. (I've asked them to pause until it's marked as Approved in ITS' Software Catalog.) Is there an ETA for this review?
Jul 12 2024
Morten completed this as part of SDS 2.5.7. Here is his report: https://docs.google.com/document/d/1HzEuDzEkwBtBexTlL5b4JiRjlhOP-dCBfAbckwMj4vc/edit#
Jul 11 2024
Status update: Product Analytics is going to own a hypothesis under WE 4.2, likely in Q2 (since we are fully booked for Q1), to the effect of:
If we develop a metric for measuring how many event participants are prevented from participating in events due to IP blocks, we will be able to measure baselines across different dimensions (such as geographic regions and languages) to understand the severity and distribution of the problem.
(To be refined by the hypothesis owner (TBD) with @kostajh as KR owner)
Jul 3 2024
Collected various discussions & points into this decision brief (viewable by public). Will follow up here when the decision is made.
Jul 2 2024
Jun 28 2024
I believe mediawiki.web_ui_actions succeeds eventlogging_DesktopWebUIActionsTrackingand eventlogging_MobileWebUIActionsTracking
and mediawiki.web_ui_scroll_migrated succeeds mediawiki.web_ui_scroll
@KSarabia-WMF: The default is in the stream config (pasted above) and it's currently set to 20% of sessions on all projects. And what's happening with the migrated instrumentation is that it's being double-sampled – the instrument initializes on 20% of pageviews (1% on English projects) and when it produces events to the mediawiki.web_ui_actions stream, only 20% of sessions will actually submit an event.
Ensure that the sampling for this instrument is per session not per-pageview.
Jun 27 2024
@phuedx: I reviewed the modeling approaches in PA team sharing (notes w/ link to transcript & recording) and Morten brought up a great point which is the computational cost of using virtual table generation (LATERAL VIEW EXPLODE() in Spark SQL and CROSS JOIN UNNEST() in Presto) for version 1 which reminded him of evaluating custom data approach of monoschema MP.
Jun 25 2024
@phuedx: I see in the Figma designs that instruments/experiments will have user-provided unique names with machine-readable versions auto-generated. Will those instruments/experiments have numeric IDs in the database that also uniquely identify them?
Andrew brought up an excellent point (in the same thread) which is that https://wikitech.wikimedia.org/wiki/Event_Platform/Schemas/Guidelines#Complex_array_element_and_map_value_type_evolution_is_not_well_supported so if we did go the array-of-objects (version 1) route, we MAY want array<map<string,string>> not array<struct<id:int,name:string,group:string>> in case there is anything we would need to add later. Like, if we needed to include a new field for each experiment it'd be easy with the former and impossible with the latter.
That's a great point!
Jun 24 2024
@phuedx @VirginiaPoundstone: T368326: Update Metrics Platform Client Libraries to accept experiment membership and just to confirm: yes I have bandwidth to consult on this
Sounds good, yeah! Thanks for closing the loop on this.
Jun 21 2024
Oh cool! Yeah I'd love to squeeze that into scope. Should I make a separate task?
@Milimetric Once this is finalized can you please document this change at https://wikitech.wikimedia.org/wiki/Data_Platform_Engineering/Decision_log?
Jun 13 2024
As an Experiment Owner, when I select a location(wiki) I need to see the default sample rate for that location(wiki), so I know whether or not I need to modify it according to my experimentation plan.
As Experiment Owner, I need to override the default sample rate at a per wiki level, so I can collect data at different rates based on the various wikis that are part of the experiment.
As an Experiment Owner, I need to set a default sample rate, so that rate will be applied across all locations(wikis) except the ones that I need to override with a different sampling rate.Should there be standard sampling rates per wiki? Or a low conservative default sampling rate (.1%) for all?
I think that providing defaults for these values is part of the product. I also think that it would be better for our users (and for the platform!) if we provided a conservative default value that the user than can increase (with relatively little friction because MPIC is gonna be _awesome_!).
Jun 11 2024
@MNeisler just caught me up on this. I just want to share some thoughts about
Jun 5 2024
May 31 2024
Kind of? Categories are…a giant unstructured, manually-maintained mess and inconsistent across languages. (In general, not just specific to templates.) Categories can also have layers upon layers of sub-categories nested within each other, and there's not an easy way to get them all.
All good! Thanks so much!
First, I want to acknowledge that I appreciate wanting to incorporate data into the design stage of product development cycle. Unfortunately we do not have capacity to take this request (or any request, really) for data analysis for the remainder of Q4.
May 30 2024
@JWheeler-WMF Is this for FY24–25 PES 1.2.3?
May 29 2024
Thank you for this analysis, KC!
May 24 2024
May 21 2024
@ngkountas @KCVelaga_WMF: Perhaps this will be useful: https://wikitech.wikimedia.org/wiki/Metrics_Platform/How_to/Create_An_Instrument#In_JavaScript
May 17 2024
May 16 2024
@CDanis: Can you please paste the Spark code / Spark SQL query you used for reproducibility?
May 15 2024
@Krinkle: Thank you for sharing the results of your queries in a manner consistent with the Data Publication Guidelines.