Abstract
The passage of the No Child Left Behind Act has increased an emphasis on developing K-12 curricula around existing and emergent state and national standards. The ever-growing volume of readily available K-12 digital content has increased the need for aligning learning and assessment content to relevant educational standards at scale. However, manual alignment is labor-intensive and time-consuming. Inspired by prior works on automated content alignment systems that leveraged recent advances in deep learning and NLP, this study explores a scalable solution for automatically aligning assessment items to multiple state and national standards. Results indicate the Transformer encoder-decoder model trained from scratch shows decent performance, reaching 34.3 BLEU score and 0.4 averaged ROUGE score on a holdout set. To investigate the limitation of the conventional evaluation metrics and gain deeper insights into the many-to-many relationships observed in the data, a series of metrics are utilized to evaluate the matches between the source and target sequences. In-depth error analysis identifies major error categories and explains the discrepancies in performances observed between the training and test set. Finally, this study discusses the potential for a production-level system and the future direction in extending the current approach to facilitate the development of a general skill taxonomy as a “crosswalk” for mapping educational content to standards.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
Slightly increased from the original 0.1 dropout rate to improve overfitting.
References
Common Core State Standards Initiative. http://www.corestandards.org. Accessed 25 May 2022
Nelson, G.D.: AAAS Web page (1997). http://www.project2061.org/publications/articles/nelson/nelson1.htm. Accessed 25 May 2022
Diekema, A.R.: Implications and challenges of educational standards metadata. J. Libr. Metadata 9(3–4), 239–251 (2009). https://doi.org/10.1080/19386380903405157
Kendall, J.S.: The use of metadata for the identification and retrieval of resources for K–12 education. In: Proceedings of the 2003 International Conference on Dublin Core and Metadata Applications: Supporting Communities of Discourse and Practice—Metadata Research & Applications. Seattle, Washington (2003)
Purpose of this Work. http://www.mcrel.org/standards-benchmarks/docs/purpose.asp. Accessed 25 May 2022
Yilmazel, O., Ingersoll, G., Liddy, E.D.: Finding questions to your answers. In: IEEE 23rd International Conference on Data Engineering, pp. 755–759. IEEE, New York (2007)
Reitsma, R.F., Diekema, A.R.: Comparison of human and machine-based educational standard assignment networks. Int. J. Digit. Libr. 11, 209–223 (2010). https://doi.org/10.1007/s00799-011-0074-8
Khan, S.M., Rosaler, J., Hamer, J., Almeida, T.: Catalog: an educational content tagging system. In: Proceedings of the 14th International Conference on Educational Data Mining. Virtual (2021)
Jay, M., Longdon, D.: Death, taxes and correlations: a primer on the state of correlation in the K-12 education. Upgrade, SIIA, 20–21 (2003)
Reitsma, R., Marshall, B., Chart, T.: Can intermediary-based science standards crosswalking work? Some evidence from mining the standard alignment tool (SAT). J. Am. Soc. Inf. Sci. Technol. 63(9), 1843–1858 (2012)
Diekema, A.R., Yilmazel, O., Bailey, J., Harwell, S.C., Liddy, E.D.: Standards alignment for metadata assignment. In: Proceedings of the Joint Conference of Digital Libraries. Vancouver, BC (2007)
Diekema, A.R., Chen, J.: Experimenting with the automatic assignment of educational standards of digital library content. In: Proceedings of the 5th ACM/IEE-CS Joint Conference on Digital Libraries, pp. 223–224. Association for Computing Machinery, New York, NY (2005). https://doi.org/10.1145/1065385.1065436
Devaul, H., Diekema, A.R., Ostwald, J.: Computer-assisted assignment of educational standards using natural language processing. J. Am. Soc. Inf. Sci. Technol. 62, 395–405 (2011)
Reitsma, R., Marshall, B., Dalton, M., Cyr, M.: Exploring educational standard alignment: in search of ‘relevance’. In: Proceedings of the 8th ACM/IEEE-CS Joint Conference on Digital libraries, pp. 57–65. Association for Computing Machinery New York, NY (2008). https://doi.org/10.1145/1378889.1378901
Sutton, S., Golder, D.: Achievement Standards Network (ASN): an application profile for mapping K–12 educational resources to achievement standards. In: Proceedings of the International Conference on Dublin Core and Metadata Applications. Berlin, Germany (2008)
Yilmazel, O., Balasubramanian, N., Harwell, S.C., Bailey, J., Diekema, A.R., Liddy, E.D.: Text categorization for aligning educational standards. In: Proceedings of the 40th Hawaii International Conference of Systems Sciences. IEEE, New York (2007)
Ainsworth, L.: “Unwrapping” the Standards: A Simple Process to Make Standards Manageable. Advanced Learning Press, Denver, CO (2003)
Sutton, S.A.: Metadata quality, utility and the Semantic Web: the case of learning resources and achievement standards. Cat. Classif. Q. 46(1), 81–107 (2008)
Vaswani, A., et al.: Attention is all you need. In: Proceedings of the 31st Conference on Neural Information Processing Systems. Long Beach, CA (2017)
Ruder, S.: Tracking progress in natural language processing. NLP-progress (2022). http://nlpprogress.com/
Yu, R., Das, S., Gurajada, S., Varshney, K., Raghavan, H., Lastra-Anadon, C.: A research framework for understanding education-occupation alignment with NLP techniques. In: Proceedings of the 1st Workshop on NLP for Positive Impact, pp. 100–106 (2021)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1810.04805 (2018)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
National Governors Association Center for Best Practices, Council of Chief State School Officers: Common Core State Standards for English Language Arts. National Governors Association Center for Best Practices, Council of Chief State School Officers, Washington, DC (2010)
Cizek, G.J., Kosh, A.E., Toutkoushian, E.: Gathering and evaluating validity evidence: the generalized assessment alignment tool. J. Educ. Meas. 55(4), 477–512 (2018)
Martone, A., Sireci, S.: Evaluating alignment between curriculum, assessment, and instruction. Rev. Educ. Res. 79(4), 1332–1361 (2009)
Honnibal, M., Montani, I.: spaCy 2: natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing (2017). https://spacy.io
Wolf, T., et al.: Transformers: state-of-the-art natural language processing. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38–45 (2020)
Trevett, B.: pytorch-seq2seq [Source code] (2022). https://github.com/bentrevett/pytorch-seq2seq
Pennington, J., Socher, R., Manning, C.D.: Glove: global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pp. 1532–1543. A meeting of SIGDAT, a Special Interest Group of the ACL, Doha, Qatar (2014)
Rehurek, R., Sojka, P.: Gensim–Python framework for vector space modelling. NLP Centre, Faculty of Informatics, Masaryk University, Brno, Czech Republic 3(2), 2 (2011)
Reimers, N., Gurevych, I.: Sentence-BERT: sentence embeddings using siamese BERT-networks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pp. 3982–3992. Association for Computational Linguistics. Hong Kong, China (2019)
Lin, C.Y.: ROUGE: a package for automatic evaluation of summaries. In: Text summarization branches out, pp. 74–81 (2004)
Tatman, R.: Evaluating text output in NLP: BLEU at your own risk. Towards Data Science. (2019). https://towardsdatascience.com/evaluating-text-output-in-nlp-bleu-at-your-own-risk-e8609665a213
Qi, Y., Sachan, D.S., Felix, M., Padmanabhan, S.J., Neubig, G.: When and why are pre-trained word embeddings useful for neural machine translation? arXiv preprint arXiv: 1804.06323 (2018)
Rothe, S., Narayan, S., Severyn, A.: Leveraging pre-trained checkpoints for sequence generation tasks. Trans. Assoc. Comput. Linguist. 8, 264–280 (2020)
Von Platten, P.: Leveraging re-trained language model checkpoints for encoder-decoder models. Hugging Face (2020). https://huggingface.co/blog/warm-starting-encoder-decoder
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Zhou, Z., Ostrow, K.S. (2022). Transformer-Based Automated Content-Standards Alignment: A Pilot Study. In: Meiselwitz, G., et al. HCI International 2022 - Late Breaking Papers. Interaction in New Media, Learning and Games. HCII 2022. Lecture Notes in Computer Science, vol 13517. Springer, Cham. https://doi.org/10.1007/978-3-031-22131-6_39
Download citation
DOI: https://doi.org/10.1007/978-3-031-22131-6_39
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-22130-9
Online ISBN: 978-3-031-22131-6
eBook Packages: Computer ScienceComputer Science (R0)