Change Details

This task is about conducting an A/B test to help us understand what impact the New Discussion Tool is having on Junior Contributors' likelihood to start (activation) and continue (retention) participating on Wikipedia talk pages. === Decision to be made //The decision this analysis is intended to help us make:// **Should the New Discussion Tool be offered to all people, at all wikis, as an opt-out user preference?** === Hypotheses //To help evaluate the impact of the New Discussion Tool, we will analyze whether adding a more intuitive workflow for starting new discussions on Wikipedia talk pages:// |ID|Hypothesis|Metric(s) for evaluation |---|---|--- |KPI|**...causes** a greater percentage of Junior Contributors to publish the new discussions they start without a significant increase in disruption. (see "Guardrail" below) | New discussion completion rate as defined by the number of people who click the `Add topic` / `New section` link (action = `init`), what % of people successfully publish the discussion topics they were drafting (action = `saveSuccess`). | Guardrail|**...does not cause** a significant increase in the number of disruptive edits being made to talk pages|The number of edits made to talk pages that are reverted within 48 hours. The number of editors who are blocked after making an edit to a talk page. |Curiosity #1|**...causes** a greater percentage of Junior Contributors to //start// participating productively on talk pages.| The number of distinct Junior Contributors who make at least one edit to a page in a talk namespace that is not reverted within 48 hours. |Curiosity #2|**...causes** a greater percentage of Junior Contributors continue participating productively on talk pages.| The percentage of Junior Contributors who who make at least one edit to a page in a talk namespace that is not reverted within 48 hours in each of the following time intervals: 2 to 7 days after making their edit (read: within the first week), 8 to 14 days after making their first edit (read: within the second week), and 15 to 30 days after making their first edit (read: within the third or fourth weeks). === Decision matrix |ID|Scenario|Plan of action |---|---|--- |1.|People are **more likely** to publish edits using the New Discussion Tool than they are using the existing `section=`new` interface|Continue with plans to make the New Discussion Tool available at all Wikipedias, by default. //See T275256 for more detail.// |2.|People are **less likely** to publish edits using the New Discussion Tool than they are using full-page editing.|Investigate where within the New Discussion Tool workflow people are dropping off and hypothesize what could be contributing to this drop-off. In parallel, we will pause plans to make the New Discussion Tool available at all Wikipedias, by default. |3.|People are **as likely** to publish edits using the New Discussion Tool as they are using the existing `section=new` workflow.|Continue with plans to offer the New Discussion Tool as an opt-out preference at all Wikipedias considering qualitative feedback suggests the tool is leading people to find participating on talk pages easier / more efficient. === Done - [ ] Finalize `===Hypotheses` and `===Decision matrix` - [ ] Complete analysis - [ ] Publish findings on mediawiki.org: [Talk pages project/New discussion](https://www.mediawiki.org/wiki/Talk_pages_project/New_discussion)

This task is about conducting an impact analysis of the New Discussion Tool's availability at the wikis where it is available to everyone by defaultA/B test to help us understand what impact the New Discussion Tool is having on Junior Contributors' likelihood to start (activation) and continue (retention) participating on Wikipedia talk pages. T=== Decision to be made //The decision this analysis willis intended to help us decide whethermake:// **Should the New Discussion Tool can be made availablebe offered to all people, at all wikis, as an opt-out user preference?** === Hypotheses //To help evaluate the impact of the New Discussion Tool, by default.we will analyze whether adding a more intuitive workflow for starting new discussions on Wikipedia talk pages:// === Meta|ID|Hypothesis|Metric(s) for evaluation **Research questions**|---|---|--- //This analysis will help us answer the following questions://|KPI|**...causes** a greater percentage of Junior Contributors to publish the new discussions they start without a significant increase in disruption. (see "Guardrail" below) | New discussion completion rate as defined by the number of people who click the `Add topic` / `New section` link (action = `init`), what % of people successfully publish the discussion topics they were drafting (action = `saveSuccess`). - How – if at all – did peoples' likelihood to successfully publish a new discussion topic to a talk page change after the New Discussion Tool was made available?| Guardrail|**...does not cause** a significant increase in the number of disruptive edits being made to talk pages|The number of edits made to talk pages that are reverted within 48 hours. The number of editors who are blocked after making an edit to a talk page. |Curiosity #1|**...causes** a greater percentage of Junior Contributors to //start// participating productively on talk pages.| The number of distinct Junior Contributors who make at least one edit to a page in a talk namespace that is not reverted within 48 hours. - How – if at all – did the amount of disruptive behavior on talk pages change after the New Discussion Tool was made available?|Curiosity #2|**...causes** a greater percentage of Junior Contributors continue participating productively on talk pages.| The percentage of Junior Contributors who who make at least one edit to a page in a talk namespace that is not reverted within 48 hours in each of the following time intervals: 2 to 7 days after making their edit (read: within the first week), 8 to 14 days after making their first edit (read: within the second week), and 15 to 30 days after making their first edit (read: within the third or fourth weeks). **=== Decision(s) to be made** matrix - [ ] Does the impact|ID|Scenario|Plan of action |---|---|--- |1.|People are **more likely** to publish edits using the New Discussion Tool than they are using the existing `section=`new` interface|Continue with plans to make the New Discussion Tool is having on who uses talk pages and how they use talk pages warrant the tool being made available to everyoneavailable at all Wikipedias, by default. //See T275256 for more detail.// |2.|People are **less likely** to publish edits using the New Discussion Tool than they are using full-page editing.|Investigate where within the New Discussion Tool workflow people are dropping off and hypothesize what could be contributing to this drop-off. In parallel, we will pause plans to make the New Discussion Tool available at all wikisWikipedias, by default? === Metrics. |3.|People are **as likely** to publish edits using the New Discussion Tool as they are using the existing `section=new` workflow.|Continue with plans to offer the New Discussion Tool as an opt-out preference at all Wikipedias considering qualitative feedback suggests the tool is leading people to find participating on talk pages easier / more efficient. === Done - [ ] "Metrics" are definedFinalize `===Hypotheses` and `===Decision matrix` - [ ] Complete analysis - [ ] "Metrics" are analyzedPublish findings on mediawiki.org: [Talk pages project/New discussion](https://www.mediawiki.org/wiki/Talk_pages_project/New_discussion)