Background/Goal
Develop and deliver an alpha version of our instrument configuration system (MPIC). This system will enable product teams to set up, manage, and configure various instruments for data collection and experimentation. The alpha release will include core functionalities and serve as a foundation for further development and refinement.
We are targeting usage on this release by at least 1 product team. We are collaborating with the Growth team to configure their basic split A/B test for the Community Updates module (SDS 2.1.3). Our goal is to help product teams reduce their time to data collection Time to Data Collection (TTDC). Baseline Report on of TTDC Estimates – Q4 FY24.
👀 Please Note:
This work is happening concurrently with T368163: [EPIC] FY 24/25 SDS 2.1.1 POC Integration tests of 3rd Party experimentation engine solutions . We are working in parallel so we can continue to deliver incremental value and cross team learning while we navigate the decision making process to custom build an experimentation platform or install a 3rd party solution. We produced a initial Decision Brief - Experimentation Platform - Are Licenses of the Open Core… and were asked to come with a specific choice in software. In order to make a confident decision about which software we would recommend if we are not to custom build, we needed deeper investigations to be done in terms of technical feasibility, vendor risk, privacy & security risk, stakeholder acceptance, and return on investment. Advancing this work also allows us to better estimate the amount of effort required to custom build a solution if the 3rd party choice is not viable after stakeholder review.
KR/Hypothesis(Initiative)
FY24-25 SDS 2.1 Experimentation Platform
[[[ https://wikimedia.slack.com/archives/C073UMUFPBQ/p1721853487830429 | Drafting in process ]]] If we XX, then XX goal will be achieved.
Success metrics
- 1 product team successfully uses the system to configure an experiment
- target date: Early October, 2024 (precise data TDB)
- Using it reduces their time to data collection by [TBD] 50%
- System produces less than [5] critical errors per week
- User satisfaction rate is positive
- Zero data integrity issues
- [X% of calibrations are successful]
Requirements
Target Outcomes
- Enable basic instrument used for A/B testing setup and management
- Implement essential configuration and calibration features
- Enable data analysis of data collected
- Gather early user feedback for future improvements
Key Features
- Basic configuration template creation
- Instrument calibration tools
- Configuration version control
- Simple instrument health monitoring
- Basic reporting and logging
- Access to raw data
- Instrument catalog
User Stories
- As an Experiment Owner, I want to create an A/B test to compare different feature variations, so I can learn if the new feature moves our success metric while not having a negative effect on our guardrail metrics.
- As an Experiment Owner, I want to capture click through rate metrics in an A/B test so I can learn how many users exposed to the new feature initiate the call to action.
- As a Product Engineer, I want to assign traffic weights to the two variations so I can control the percentage of users who see each variation during the test.
- As a data engineer, I need to create a pipeline that processes collected data using the CTR metric definition, so data analysts can create a superset dashboard for stakeholder review.
- As an experiment owner, I can register new experiment instruments in the system
- As a experiment owner, I can create and modify configuration templates for experiments
- As a data analysts, I can calibrate experiment instruments using the system's tools, so I can ensure data collection accuracy and quality
- As a experiment owner, I can view the configuration history of an experiment
- As an Experiment Owner, I want to view and explore the analysis of the click through rate data collected so I can decide whether or not to roll out the feature.
- As a data analyst, I can access the raw data collected, so I can conduct analysis
- As a data analyst, I want to access the raw data produced by the test, so I can create superset dashboards that I can share with stakeholders.
Tasks
- Design system architecture
- Develop instrument registration and management interface
- Implement configuration template creation and editing tools
- Create basic calibration functionality
- Develop version control for instrument configurations
- Build simple instrument health monitoring dashboard
- Implement basic reporting and logging features
- Conduct internal testing and bug fixes
- Prepare documentation and user guides
- Set up system monitoring and error handling
- Plan and execute alpha release to select users
- Plan to gather feedback
Acceptance Criteria
- Users can successfully register and manage instruments
- System allows creation and modification of configuration templates
- Basic calibration tools function correctly
- Version control accurately tracks configuration changes
- Simple health monitoring provides basic instrument status
- System handles multiple instruments and users without conflicts
- Data Analysts can access the raw data collected
In scope
- Still emergent as part of the example mapping process
Out of Scope
- Still emergent as part of the example mapping process
Artifacts & Resources
One Pager
Architecture work has been done in Miro.
Metrics Platform / Experimentation System User Stories
Instrument Configurator – Design Document
Example Mapping