Abstract
Researchers have used various methods to evaluate the fine-grained interactions of intelligent tutors with their students. We present a case study comparing three such methods on the same data set, logged by Project LISTEN’s Reading Tutor from usage by 174 children in grades 2-4 (typically 7-10 years) over the course of the 2005-2006 school year. The Reading Tutor chooses randomly between two different types of reading practice. In assisted oral reading, the child reads aloud and the tutor helps. In “Word Swap,” the tutor reads aloud and the child identifies misread words. One method we use here to evaluate reading practice is conventional analysis of randomized controlled trials (RCTs), where the outcome is performance on the same words when encountered again later. The second method is learning decomposition, which estimates the impact of each practice type as a parameter in an exponential learning curve. The third method is knowledge tracing, which estimates the impact of practice as a probability in a dynamic Bayes net. The comparison shows qualitative agreement among the three methods, which is evidence for their validity.
This is a preview of subscription content, log in via an institution to check access.
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Mostow, J., Aist, G.: Evaluating tutors that listen: An overview of Project LISTEN. In: Forbus, K., Feltovich, P. (eds.) Smart Machines in Education, pp. 169–234. MIT/AAAI Press, Menlo Park (2001)
Beck, I.: Reading Today and Tomorrow (Teachers’ Editions for Grades 1 and 2). Holt and Co., Austin (1989)
Beck, I., Hamilton, R.: Beginning reading module. American Federation of Teachers, Washington (2000) (Original work published 1996)
Cunningham, P.M., Cunningham, J.W.: Making Words: Enhancing the invented spelling-decoding connection. The Reading Teacher 46(2), 106–114 (1992)
McCandliss, B., Beck, I.L., Sandak, R., Perfetti, C.: Focusing attention on decoding for children with poor reading skills: Design and preliminary tests of the word building intervention. Scientific Studies of Reading 7(1), 75–104 (2003)
Razzaq, L., Heffernan, N.T., Lindeman, R.W.: What level of tutor feedback is best? In: Proceedings of the 13th Conference on Artificial Intelligence in Education, IOS Press (2007)
Mostow, J., Beck, J.: Some useful tactics to modify, map, and mine data from intelligent tutors. Natural Language Engineering (Special Issue on Educational Applications) 12(2), 195–208 (2006)
Zhang, X., Mostow, J., Beck, J.E.: All in the (word) family: Using learning decomposition to estimate transfer between skills in a Reading Tutor that listens. In: AIED 2007 Educational Data Mining Workshop, Marina del Rey (2007)
Mostow, J., Aist, G.: The sounds of silence: Towards automated evaluation of student learning in a Reading Tutor that listens. In: Proceedings of the Fourteenth National Conference on Artificial Intelligence (AAAI 1997), pp. 355–361. American Association for Artificial Intelligence, Providence (1997)
Beck, J.: Using learning decomposition to analyze student fluency development. In: Ikeda, M., Ashley, K.D., Chan, T.-W. (eds.) ITS 2006. LNCS, vol. 4053. Springer, Heidelberg (2006)
Cohen, P.R.: Empirical Methods for Artificial Intelligence, p. 405. MIT Press, Cambridge (1995)
Corbett, A., Anderson, J.: Knowledge tracing: Modeling the acquisition of procedural knowledge. User modeling and user-adapted interaction 4, 253–278 (1995)
Beck, J.E., Chang, K.: Identifiability: A Fundamental Problem of Student Modeling. In: Conati, C., McCoy, K., Paliouras, G. (eds.) UM 2007. LNCS (LNAI), vol. 4511. Springer, Heidelberg (2007)
Beck, J.E.: Difficulties in inferring student knowledge from observations (and why you should care). In: Proceedings of the AIED 2007 Workshop on Educational Data Mining, Marina del Rey, pp. 21–30 (2007)
Chang, K.-m., Beck, J., Mostow, J., Corbett, A.: A Bayes Net Toolkit for Student Modeling in Intelligent Tutoring Systems. In: Proceedings of the 8th International Conference on Intelligent Tutoring Systems, Jhongli, Taiwan (2006)
Jonsson, A., Johns, J., Mehranian, H., Arroyo, I., Woolf, B., Barto, A., Fisher, D., Mahadevan, S.: Evaluating the Feasibility of Learning Student Models from Data. In: Educational Data Mining: Papers from the AAAI Workshop, pp. 1–6. AAAI Press, Pittsburgh (2005)
Chang, K.-m., Beck, J.E., Mostow, J., Corbett, A.: Does Help Help? A Bayes Net Approach to Modeling Tutor Interventions. In: AAAI 2006 Workshop on Educational Data Mining, Boston (2006)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2008 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Zhang, X., Mostow, J., Beck, J.E. (2008). A Case Study Empirical Comparison of Three Methods to Evaluate Tutorial Behaviors. In: Woolf, B.P., Aïmeur, E., Nkambou, R., Lajoie, S. (eds) Intelligent Tutoring Systems. ITS 2008. Lecture Notes in Computer Science, vol 5091. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-69132-7_17
Download citation
DOI: https://doi.org/10.1007/978-3-540-69132-7_17
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-69130-3
Online ISBN: 978-3-540-69132-7
eBook Packages: Computer ScienceComputer Science (R0)