Opportunities for Adaptive Experiments to Enable Continuous Improvement in Computer Science Education
Authors: 
Ilya Musabirov, 
Angela Zavaleta Bernuy, Pan Chen, 
Michael Liut, 
Joseph WilliamsAuthors Info & Claims
Ilya Musabirov, Angela Zavaleta Bernuy, Pan Chen, Michael Liut, Joseph WilliamsAuthors Info & ClaimsArticle No.: 4, Pages 1 - 7
Abstract
Randomized A/B comparisons of alternative pedagogical strategies or other course improvements could provide useful empirical evidence for instructor decision-making. However, traditional experiments do not provide a straightforward pathway to rapidly utilize data, increasing the chances that students in an experiment experience the best conditions. Drawing inspiration from the use of machine learning and experimentation in product development at leading technology companies, we explore how adaptive experimentation might aid continuous course improvement. In adaptive experiments, data is analyzed and utilized as different conditions are deployed to students. This can be achieved using machine learning algorithms to identify which actions are more beneficial in improving students’ learning experiences and outcomes. These algorithms can then dynamically deploy the most effective conditions in subsequent interactions with students, resulting in better support for students’ needs. We illustrate this approach with a case study that provides a side-by-side comparison of traditional and adaptive experiments on adding self-explanation prompts in online homework problems in a CS1 course. This work paves the way for exploring the importance of adaptive experiments in bridging research and practice to achieve continuous improvement in educational settings.
References
[1]
Alekh Agarwal, Sarah Bird, Markus Cozowicz, Luong Hoang, John Langford, Stephen Lee, Jiaji Li, Dan Melamed, Gal Oshri, Oswaldo Ribas, Siddhartha Sen, and Alex Slivkins. 2016. Making Contextual Decisions with Low Technical Debt. (June 2016). https://doi.org/10.48550/arXiv.1606.03966
[2]
Shipra Agrawal and Navin Goyal. 2012. Analysis of Thompson Sampling for the Multi-armed Bandit Problem. In Conference on learning theory. JMLR Workshop and Conference Proceedings, 39–1.
[3]
Ahmed Al-Zubidy, Jeffrey C. Carver, Sarah Heckman, and Mark Sherriff. 2016. A (Updated) Review of Empiricism at the SIGCSE Technical Symposium. In Proceedings of the 47th ACM Technical Symposium on Computing Science Education. ACM, Memphis Tennessee USA, 120–125. https://doi.org/10.1145/2839509.2844601
[4]
Vincent Aleven, Ido Roll, Bruce M McLaren, and Kenneth R Koedinger. 2016. Help helps, but only so much: Research on help seeking with intelligent tutoring systems. International Journal of Artificial Intelligence in Education 26 (2016), 205–223.
[5]
Yuya Asano, Madhurima Dutta, Trisha Thakur, Jaemarie Solyst, Stephanie Cristea, Helena Jovic, Andrew Petersen, and Joseph Jay Williams. 2021. Exploring Additional Personalized Support While Attempting Exercise Problems in Online Learning Platforms. In Proceedings of the Eighth ACM Conference on Learning@ Scale. 235–238.
[6]
Eduardo M. Azevedo, Alex Deng, Jose Luis Montiel Olea, Justin Rao, and E. Glen Weyl. 2018. The A/B Testing Problem. In Proceedings of the 2018 ACM Conference on Economics and Computation. ACM, Ithaca NY USA, 461–462. https://doi.org/10.1145/3219166.3219204
[7]
Eytan Bakshy, Lili Dworkin, Brian Karrer, Konstantin Kashin, Benjamin Letham, Ashwin Murthy, and Shaun Singh. 2018. AE: A domain-agnostic platform for adaptive experimentation. In Conference on Neural Information Processing Systems. 1–8.
[8]
Lecia Barker, Christopher Lynnly Hovey, and Jane Gruning. 2015. What Influences CS Faculty to Adopt Teaching Practices?. In Proceedings of the 46th ACM Technical Symposium on Computer Science Education. ACM, Kansas City Missouri USA, 604–609. https://doi.org/10.1145/2676723.2677282
[9]
Adi Berliner Senderey, Tamar Kornitzer, Gabriella Lawrence, Hilla Zysman, Yael Hallak, Dan Ariely, and Ran Balicer. 2020. It’s how you say it: systematic A/B testing of digital messaging cut hospital no-show rates. PloS one 15, 6 (2020), e0234817. Publisher: Public Library of Science San Francisco, CA USA.
[10]
Anthony S. Bryk, Louis M. Gomez, and Alicia Grunow. 2011. Getting Ideas into Action: Building Networked Improvement Communities in Education. In Frontiers in Sociology of Education, Maureen T. Hallinan (Ed.). Springer Netherlands, Dordrecht, 127–162. https://doi.org/10.1007/978-94-007-1576-9_7
[11]
Olivier Chapelle and Lihong Li. 2011. An Empirical Evaluation of Thompson Sampling. In Advances in Neural Information Processing Systems, J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K. Q. Weinberger (Eds.). Vol. 24. Curran Associates, Inc.https://proceedings.neurips.cc/paper/2011/file/e53a0a2978c28872a4505bdb51db06dc-Paper.pdf
[12]
John M. Clement. 2004. A Call for Action (Research): Applying Science Education Research to Computer Science Instruction. Computer Science Education 14, 4 (Dec. 2004), 343–364. https://doi.org/10.1080/0899340042000303474
[13]
Nina Deliu, Joseph J. Williams, and Sofia S. Villar. 2021. Efficient Inference Without Trading-off Regret in Bandits: An Allocation Probability Test for Thompson Sampling. http://arxiv.org/abs/2111.00137 arXiv:2111.00137 [cs, stat].
[14]
Alex Deng, Pavel Dmitriev, Somit Gupta, Ron Kohavi, Paul Raff, and Lukas Vermeer. 2017. A/B Testing at Scale: Accelerating Software Innovation. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval(SIGIR ’17). Association for Computing Machinery, New York, NY, USA, 1395–1397. https://doi.org/10.1145/3077136.3082060
[15]
Xu Du, Juan Yang, Brett E. Shelton, Jui-Long Hung, and Mingyan Zhang. 2021. A systematic meta-Review and analysis of learning analytics research. Behaviour & Information Technology 40, 1 (Jan. 2021), 49–62. https://doi.org/10.1080/0144929X.2019.1669712 Publisher: Taylor & Francis _eprint: https://doi.org/10.1080/0144929X.2019.1669712.
[16]
Stephen Fancsali, April Murphy, and Steven Ritter. 2022. "Closing the Loop" in Educational Data Science with an Open Source Architecture for Large-Scale Field Trials. In Proceedings of the 15th International Conference on Educational Data Mining, Antonija Mitrovic and Nigel Bosch (Eds.). International Educational Data Mining Society, Durham, United Kingdom, 834–838. https://doi.org/10.5281/zenodo.6852930
[17]
Vitor Hadad, David A. Hirshberg, Ruohan Zhan, Stefan Wager, and Susan Athey. 2021. Confidence intervals for policy evaluation in adaptive experiments. Proceedings of the National Academy of Sciences 118, 15 (April 2021), e2014602118. https://doi.org/10.1073/pnas.2014602118
[18]
Martin S. Hagger, Linda D. Cameron, Kyra Hamilton, Nelli Hankonen, and Taru Lintunen (Eds.). 2020. The Handbook of Behavior Change (1 ed.). Cambridge University Press. https://doi.org/10.1017/9781108677318
[19]
Sarah Heckman, Jeffrey C. Carver, Mark Sherriff, and Ahmed Al-zubidy. 2022. A Systematic Literature Review of Empiricism and Norms of Reporting in Computing Education Research Literature. ACM Transactions on Computing Education 22, 1 (March 2022), 1–46. https://doi.org/10.1145/3470652
[20]
Neil T. Heffernan and Cristina Lindquist Heffernan. 2014. The ASSISTments ecosystem: Building a platform that brings scientists and teachers together for minimally invasive research on human learning and teaching. International Journal of Artificial Intelligence in Education 24, 4 (2014), 470–497. Publisher: Springer.
[21]
Isabel Hilliger, Camila Aguirre, Constanza Miranda, Sergio Celis, and Mar Pérez-Sanagustín. 2020. Design of a curriculum analytics tool to support continuous improvement processes in higher education. In Proceedings of the tenth international conference on learning analytics & knowledge. 181–186.
[22]
Christopher Lynnly Hovey, Lecia Barker, and Vaughan Nagy. 2019. Survey Results on Why CS Faculty Adopt New Teaching Practices. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education. ACM, Minneapolis MN USA, 483–489. https://doi.org/10.1145/3287324.3287420
[23]
Ron Kohavi, Diane Tang, and Ya Xu. 2020. Trustworthy online controlled experiments: a practical guide to A/B testing. Cambridge University Press, Cambridge, United Kingdom ; New York, NY.
[24]
Ron Kohavi, Diane Tang, Ya Xu, Lars G. Hemkens, and John Ioannidis. 2020. Online randomized controlled experiments at scale: lessons and extensions to medicine. Trials 21, 1 (2020), 1–9. Publisher: Springer.
[25]
Juho Leinonen, Paul Denny, and Jacqueline Whalley. 2021. Exploring the Effects of Contextualized Problem Descriptions on Problem Solving. In Australasian Computing Education Conference. ACM, Virtual SA Australia, 30–39. https://doi.org/10.1145/3441636.3442302
[26]
Tong Li, Jacob Nogas, Haochen Song, Harsh Kumar, Audrey Durand, Anna Rafferty, Nina Deliu, Sofia S. Villar, and Joseph J. Williams. 2022. Algorithms for Adaptive Experiments that Trade-off Statistical Analysis with Reward: Combining Uniform Random Assignment and Reward Maximization. arxiv:2112.08507 [cs.LG]
[27]
David Liben-Nowell and Anna N. Rafferty. 2022. Student Motivations and Goals for CS1: Themes and Variations. In Proceedings of the 53rd ACM Technical Symposium on Computer Science Education - Volume 1 (Providence, RI, USA) (SIGCSE 2022). Association for Computing Machinery, New York, NY, USA, 237–243. https://doi.org/10.1145/3478431.3499358
[28]
Alex Lishinski, Jon Good, Phil Sands, and Aman Yadav. 2016. Methodological Rigor and Theoretical Foundations of CS Education Research. In Proceedings of the 2016 ACM Conference on International Computing Education Research. ACM, Melbourne VIC Australia, 161–169. https://doi.org/10.1145/2960310.2960328
[29]
Alexia Charis Martin, Kimberly Michelle Ying, Fernando J. Rodríguez, Christina Suzanne Kahn, and Kristy Elizabeth Boyer. 2022. Intelligent Support for All? A Literature Review of the (In)Equitable Design & Evaluation of Adaptive Pedagogical Systems for CS Education. In Proceedings of the 53rd ACM Technical Symposium on Computer Science Education - Volume 1(SIGCSE 2022). Association for Computing Machinery, New York, NY, USA, 996–1002. https://doi.org/10.1145/3478431.3499418
[30]
Samiha Marwan, Ge Gao, Susan Fisk, Thomas W. Price, and Tiffany Barnes. 2020. Adaptive Immediate Feedback Can Improve Novice Programming Engagement and Intention to Persist in Computer Science. In Proceedings of the 2020 ACM Conference on International Computing Education Research(ICER ’20). Association for Computing Machinery, New York, NY, USA, 194–203. https://doi.org/10.1145/3372782.3406264 event-place: Virtual Event, New Zealand.
[31]
Karen D. Mattingly, Margaret C. Rice, and Zane L. Berge. 2012. Learning analytics as a tool for closing the assessment loop in higher education. Knowledge management & e-learning: An international journal 4, 3 (2012), 236–247.
[32]
Susan McKenney and Thomas C. Reeves. 2018. Conducting Educational Design Research (2 ed.). Routledge, Second edition. | New York : Routledge, 2019. | “[First edition. https://doi.org/10.4324/9781315105642
[33]
Benjamin A. Motz, Paulo F. Carvalho, Joshua R. De Leeuw, and Robert L. Goldstone. 2018. Embedding Experiments: Staking Causal Inference in Authentic Educational Contexts. Journal of Learning Analytics 5, 2 (Aug. 2018). https://doi.org/10.18608/jla.2018.52.4
[34]
Claudia Ott, Anthony Robins, and Kerry Shephard. 2016. Translating Principles of Effective Feedback for Students into the CS1 Context. ACM Trans. Comput. Educ. 16, 1, Article 1 (jan 2016), 27 pages. https://doi.org/10.1145/2737596
[35]
William R. Penuel, Anna-Ruth Allen, Cynthia E. Coburn, and Caitlin Farrell. 2015. Conceptualizing Research–Practice Partnerships as Joint Work at Boundaries. Journal of Education for Students Placed at Risk (JESPAR) 20, 1-2 (April 2015), 182–197. https://doi.org/10.1080/10824669.2014.988334
[36]
Anna Rafferty, Huiji Ying, and Joseph Williams. 2019. Statistical Consequences of using Multi-armed Bandits to Conduct Adaptive Educational Experiments. (June 2019). https://doi.org/10.5281/ZENODO.3554749 Publisher: Zenodo Version Number: 1.0.0.
[37]
Mohi Reza, Juho Kim, Ananya Bhattacharjee, Anna N. Rafferty, and Joseph Jay Williams. 2021. The MOOClet Framework: Unifying Experimentation, Dynamic Improvement, and Personalization in Online Courses. In Proceedings of the Eighth ACM Conference on Learning@ Scale. 15–26.
[38]
Steven Ritter, Neil Heffernan, Joseph Jay Williams, Derek Lomas, Ben Motz, Debshila Basu Mallick, Klinton Bicknell, Danielle McNamara, Rene F. Kizilcec, and Jeremy Roschelle. 2022. Third Annual Workshop on A/B Testing and Platform-Enabled Learning Research. In Proceedings of the Ninth ACM Conference on Learning@ Scale. 252–254.
[39]
J.C. Stamper, D. Lomas, D. Ching, S. Ritter, Koedinger, and J. K.R., Steinhart. 2012. The rise of the super experiment. In Proceedings of the 5th International Conference on Educational Data Mining. International Educational Data Mining Society, Chania, Greece, 196–199.
[40]
Allan Wigfield and Alison C. Koenka. 2020. Where do we go from here in academic motivation theory and research? Some reflections and recommendations for future work. Contemporary Educational Psychology 61 (2020), 101872. Publisher: Elsevier.
[41]
Joseph Jay Williams, Juho Kim, Anna Rafferty, Samuel Maldonado, Krzysztof Z Gajos, Walter S Lasecki, and Neil Heffernan. 2016. Axis: Generating explanations at scale with learnersourcing and machine learning. In Proceedings of the Third (2016) ACM Conference on Learning@ Scale. 379–388.
[42]
Joseph Jay Williams, Jacob Nogas, Nina Deliu, Hammad Shaikh, Sofia S. Villar, Audrey Durand, and Anna Rafferty. 2021. Challenges in Statistical Analysis of Data Collected by a Bandit Algorithm: An Empirical Exploration in Applications to Adaptively Randomized Experiments. http://arxiv.org/abs/2103.12198
[43]
Joseph Jay Williams, Anna N. Rafferty, Dustin Tingley, Andrew Ang, Walter S. Lasecki, and Juho Kim. 2018. Enhancing Online Problems Through Instructor-Centered Tools for Randomized Experiments. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, Montreal QC Canada, 1–12. https://doi.org/10.1145/3173574.3173781
[44]
Fernando J. Yanez, Angela Zavaleta-Bernuy, Ziwen Han, Michael Liut, Anna Rafferty, and Joseph Jay Williams. 2022. Increasing Students’ Engagement to Reminder Emails Through Multi-Armed Bandits. https://doi.org/10.48550/arXiv.2208.05090
[45]
Angela Zavaleta-Bernuy, Qi Yin Zheng, Hammad Shaikh, Jacob Nogas, Anna Rafferty, Andrew Petersen, and Joseph Jay Williams. 2021. Using Adaptive Experiments to Rapidly Help Students. In Artificial Intelligence in Education(Lecture Notes in Computer Science). Springer International Publishing, Cham, 422–426. https://doi.org/10.1007/978-3-030-78270-2_75
Index Terms
- Opportunities for Adaptive Experiments to Enable Continuous Improvement in Computer Science Education
Recommendations
A Case Study in Opportunities for Adaptive Experiments to Enable Rapid Continuous Improvement
SIGCSE 2023: Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 2Drawing inspiration from machine learning and experimentation in product development at leading technology companies, we explore how adaptive experimentation might help in continuous course improvement. In adaptive experiments, as different arms/...
Creating a Sustainable Continuous Feedback and Development Ecosystem in Higher Education
IC4E '17: Proceedings of the 8th International Conference on E-Education, E-Business, E-Management and E-LearningLinking development activities to productivity is a management dilemma as higher education instructions and learning involving multiples dimensions is becoming more complex. Learning process has advanced from the foreordained and very organized learning ...
Motivational active learning for computer science education (abstract only)
SIGCSE '14: Proceedings of the 45th ACM technical symposium on Computer science educationMotivational Active Learning (MAL) is an innovative pedagogical approach based on MIT's teaching format TEAL (Technology-Enabled Active Learning) combined with advanced motivational strategies based on gamification design aspects. The main idea of MAL ...
Comments
Information & Contributors
Information
Published In
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 20 June 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
Conference
WCCCE '24
WCCCE '24: The 26th Western Canadian Conference on Computing Education
May 2 - 3, 2024
BC, Kelowna, Canada
Acceptance Rates
Overall Acceptance Rate 78 of 117 submissions, 67%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 16Total Downloads
- Downloads (Last 12 months)16
- Downloads (Last 6 weeks)6
Reflects downloads up to 15 Oct 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format