Abstract
Teachers’ mathematical content knowledge (CK) is of vital importance and need in teacher professional development (PD) programs. Computer-aided asynchronous PD systems are the most recent proposed PD techniques. However, current automatic CK identification methods face challenges such as diversity of user responses and scarcity of high-quality annotated data. To tackle these challenges, we propose a Multi-Agent LLMs-based framework, LLMAgent-CK, to assess the user responses’ coverage of identified CK learning goals without human annotations. Leveraging multi-agent LLMs with strong generalization ability and human-like discussions, our proposed LLMAgent-CK presents promising CK identifying performance on a real-world mathematical CK dataset MaCKT.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
A formal definition of content knowledge identification problem is provided in Sect. 3.
- 2.
Note that the selected question has the lowest Question-level Recall scores with GPT-4 Turbo under LLMAgent-CK-Single. We made such a selection to check the effects of multi-agent frameworks on the hard problems for LLMs.
References
Burns, M., et al.: Barriers and supports for technology integration: views from teachers. UNESCO Global Monitoring Report (2023)
Chan, C.M., et al.: ChatEval: towards better LLM-based evaluators through multi-agent debate. arXiv preprint arXiv:2308.07201 (2023)
Copur-Gencturk, Y., Li, J., Cohen, A.S., Orrill, C.H.: The impact of an interactive, personalized computer-based teacher professional development program on student performance: a randomized controlled trial. Comput. Educ. (2024)
Copur-Gencturk, Y., Orrill, C.H.: A promising approach to scaling up professional development: intelligent, interactive, virtual professional development with just-in-time feedback. J. Math. Teach. Educ. (2023)
Copur-Gencturk, Y., Papakonstantinou, A.: Sustainable changes in teacher practices: a longitudinal analysis of the classroom practices of high school mathematics teachers. J. Math. Teach. Educ. (2016)
Deng, Y., et al.: Rephrase and respond: let large language models ask better questions for themselves. arXiv preprint arXiv:2311.04205 (2023)
Devlin, J., et al.: BERT: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
Du, Y., et al.: Improving factuality and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305.14325 (2023)
Esquibel, J.S., Darwin, T.: The teacher talent pipelines: a systematic literature review of rural teacher education in the virtual age. In: Handbook of Research on Advancing Teaching and Teacher Education in the Context of a Virtual Age (2023)
Gomaa, W.H., Fahmy, A.A.: Ans2vec: a scoring system for short answers. In: Hassanien, A.E., Azar, A.T., Gaber, T., Bhatnagar, R., F. Tolba, M. (eds.) AMLTA 2019. AISC, vol. 921, pp. 586–595. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-14118-9_59
Jordan, S.: Short-answer e-assessment questions: five years on. In: International Computer Assisted Assessment Conference (2012)
Li, G., et al.: CAMEL: communicative agents for “mind” exploration of large scale language model society. arXiv preprint arXiv:2303.17760 (2023)
Liang, T., et al.: Encouraging divergent thinking in large language models through multi-agent debate. arXiv preprint arXiv:2305.19118 (2023)
Mikolov, T., et al.: Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013)
Nicula, B., et al.: Automated assessment of comprehension strategies from self-explanations using LLMs. Information (2023)
Park, J.S., et al.: Generative agents: interactive simulacra of human behavior. In: ACM UIST (2023)
Reimers, N., Gurevych, I.: Sentence-BERT: sentence embeddings using Siamese BERT-networks. arXiv preprint arXiv:1908.10084 (2019)
Schneider, J., et al.: Towards LLM-based autograding for short textual answers. arXiv preprint arXiv:2309.11508 (2023)
Siddiqi, R., Harrison, C.: A systematic approach to the automated marking of short-answer questions. In: IEEE INMIC. IEEE (2008)
Wang, H.C., Chang, C.Y., Li, T.Y.: Assessing creative problem-solving with automated text grading. Comput. Educ. (2008)
Wu, Q., et al.: AutoGen: enabling next-gen LLM applications via multi-agent conversation framework. arXiv preprint arXiv:2308.08155 (2023)
Zhang, Y., Lin, C., Chi, M.: Going deeper: automatic short-answer grading by combining student and question models. UMUAI 30 (2020)
Acknowledgments
This material is based upon work supported by the National Science Foundation under Grant No. (NSF 2234015) and Institute of Education Sciences (R305A180392). Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation and Institute of Education Sciences.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Yang, K. et al. (2024). Content Knowledge Identification with Multi-agent Large Language Models (LLMs). In: Olney, A.M., Chounta, IA., Liu, Z., Santos, O.C., Bittencourt, I.I. (eds) Artificial Intelligence in Education. AIED 2024. Lecture Notes in Computer Science(), vol 14830. Springer, Cham. https://doi.org/10.1007/978-3-031-64299-9_23
Download citation
DOI: https://doi.org/10.1007/978-3-031-64299-9_23
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-64298-2
Online ISBN: 978-3-031-64299-9
eBook Packages: Computer ScienceComputer Science (R0)