[go: up one dir, main page]
Skip to main content

Advertisement

Springer Nature Link
Log in

Pilot study on the digitalization of the national qualification exam for Korean engineers

  • Published:
Education and Information Technologies Aims and scope Submit manuscript

Abstract

The Human Resources Development Service of Korea developed a digital exam for five representative engineering categories and conducted a pilot study comparing the findings with the paper-and-pencil exam results from the last three years. This study aimed to compare the test efficiency between digital and paper-and-pencil examinations. A digital examination was conducted with 93 participants through two rounds of test-status measurements. We collected the following data: early check-out rate, pass rate, and the average total response time per grade. The composition ratio of each question type in the digital exam was determined per the “Exam Operation Regulation” and selected from a databank spanning five years by experienced instructors. The early check-out rate of the digital exam at the Engineer level was 90% (percentage points), approximately 15%p higher than that for the paper-and-pencil exam, but the pass rate was low (9.8%p); the response time per question for the digital exam was approximately 0.2 min shorter. The results for both exams were similar in the Industrial Engineer category. The early check-out rate did not positively affect the exam pass rate, but the time taken depended on the number and characteristics of the subjects. It is necessary to improve the simplification and convenience of digital examinations, optimize test environments, and establish operation and management systems in line with the change in the Korean Technical Qualification Framework. We also suggest a gradual shift to digital examination by developing questions and tools using technologies to accurately measure individual abilities.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data, materials and/or code availability

The datasets generated and/or analyzed during the current study are not publicly available to privacy and ethical concerns, neither the data nor the source of the data can be made available.

Notes

  1. %p (Percentage Point, pp, %P) is the arithmetical difference between two percentages. In contrast, % is the increase from one value to another as a percentage of the first value.

References

  • Alek, A., Marzuki, A. G., Farkhan, M., Surahman, D., Daryanto, D., & Febrianto, S. (2020). Computer based testing in senior high school on national examination. Indonesian Journal of Learning Education and Counseling, 2(2), 204–210. https://doi.org/10.31960/ijolec.v2i2.340.

    Article  Google Scholar 

  • Allen, J., Cruce, T., & Dingler, C. (2023). School-Day Administration of the ACT® Test: Removing Barriers and Opening Doors for All Students. ACT Research-Issue Brief. Retrieved July 8, 2023. https://www.act.org/content/dam/act/unsecured/documents/R2310-School-Day-ACT-Administration-Removing-Barriers-for-All-Students-06-2023.pdf

  • Arner, D. W., Barberis, J. N., Walker, J., Buckley, R. P., Dahdal, A. M., & Zetzsche, D. A. (2020). Digital finance & the COVID-19 crisis (University of Hong Kong Faculty of Law Research Paper, No. 2020/017). SSRN. https://doi.org/10.2139/ssrn.3558889.

  • Australian Government (2007). Australian Government: My GovID My GovID. Retrieved November 8, 2022. https://www.mygovid.gov.au.

  • Baker, F. B. (1985). The basics of Item Response Theory. Heinemann.

  • Becker, G. S. (1964). Human capital: A theoretical and empirical analysis, with special reference to education. University of Chicago Press.

  • Benton, T. (2021). Item response theory, computer adaptive testing and the risk of self-deception. Research Matters, 32, 82–100.

    Google Scholar 

  • Boevé, A. J., Meijer, R. R., Albers, C. J., Beetsma, Y., & Bosker, R. J. (2015). Introducing computer-based testing in high-stakes exams in higher education: Results of a field experiment. PloS One, 10(12), e0143616. https://doi.org/10.1371/journal.pone.0143616.

    Article  Google Scholar 

  • Branscombe, M. (2020, April 14). The network impact of the global COVID-19 pandemic The News Stack. Retrieved November 17, 2022. https://thenewstack.io/the-network-impact-of-the-global-covid-19-pandemic.

  • Bridgeman, B., & Lennon, M. L. (2018). E-exams in higher education: A review of the literature. Educational Research Review, 25, 19–36.

    Google Scholar 

  • Bridgeman, B., McCamley-Jenkins, L., & Ervin, N. E. (2000). The relation of computer-based test scores with background variables, test-taking strategies, and examinee attitudes. Journal of Educational Measurement, 37(3), 213–235.

    Google Scholar 

  • Bridgeman, B., Lennon, M. L., & Jackenthal, A. (2003). Effects of screen size, screen resolution, and display rate on computer-based test performance. Applied Measurement in Education, 16(3), 191–205. https://doi.org/10.1207/S15324818AME1603_2.

    Article  Google Scholar 

  • Brüggemann, T., Ludewig, U., Lorenz, R., & McElvany, N. (2023). Effects of mode and medium in reading comprehension tests on cognitive load. Computers and Education, 192, 104649. https://doi.org/10.1016/j.compedu.2022.104649.

    Article  Google Scholar 

  • Bunderson, C. V., Inouye, D. K., & Olsen, J. B. (1988). The four generations of computerized educational measurement. ETS Research Report Series, 1988(1), i–148. https://doi.org/10.1002/j.2330-8516.1988.tb00291.x.

    Article  Google Scholar 

  • Burton-Jones, A., & Spender, J. C. (Eds.). (2011). The Oxford Handbook of Human Capital. Oxford University Press.

  • Care, E., Griffin, P., & Wilson, M. (2016). Technology and assessment design: Appropriate and inappropriate testing scenarios. In M. Wilson, E. Care, & B. Schneider (Eds.), Assessment and teaching of 21st century skills: Methods and approach (pp. 55–79). Springer.

  • Cedefop (European Centre for the Development of Vocational Training). (2014). The role of qualification systems in lifelong learning: A literature review. Publications Office of the European Union.

  • Cedefop (European Centre for the Development of Vocational Training). (2017). Qualification systems and the labour market: A literature review. Publications Office of the European Union.

  • Cedefop (European Centre for the Development of Vocational Training). (2018). The future of qualification systems: A literature review. Publications Office of the European Union.

  • Chaplin, S., & O’Hagan, A. (Eds.). (2014). Assessment methods in statistical education: An international perspective. John Wiley & Sons.

  • Cho, S., Jang, S., & Kang, J. (2021). Research on the linkage between SQF and KQF. Human resource service of Korea.

  • Choi, Y. (2016 Dec). Development of the Korean Qualification Framework and Future Plan. Paper presented at the conference of vocation and qualification, Seoul.

  • Chung, H., Kim, S., Kim, Y., & Kang, I. (2019). How to apply the Korean qualification Framework to South Korea’s Education System. Korea Research Institute for Vocational Education and Training.

  • Daniels, L. M., & Gierl, M. J. (2017). The impact of immediate test score reporting on university students’ achievement emotions in the context of computer-based multiple-choice exams. Learning and Instruction, 52, 27–35. https://doi.org/10.1016/j.learninstruc.2017.04.001.

    Article  Google Scholar 

  • de Vries, J., van der Vleuten, C. P. M., & Schuwirth, L. W. T. (2018). Digital exams in vocational qualification systems: Current state and future perspectives. Studies in Educational Evaluation, 57, 73–85.

    Google Scholar 

  • Denvir, B., & Brown, G. T. (2013). Computer-based assessment: Potential, pitfalls, and quality considerations. Educational Measurement: Issues and Practice, 32(2), 16–25.

    Google Scholar 

  • Doyle, R., & Conboy, K. (2020). The role of IS in the COVID-19 pandemic: A liquid-modern perspective. International Journal of Information Management, 55, 102184. https://doi.org/10.1016/j.ijinfomgt.2020.102184.

    Article  Google Scholar 

  • Educational Testing Service (2022). The GRE tests. ETS. Retrieved June 15, 2022. http://www.ets.org/gre.

  • Ehrenberg, R. G., & Smith, R. S. (2014). Modern labor economics: Theory and public policy (12th ed.). Routledge.

  • Eleje, L., Onah, F. E., & Abanobi, C. C. (2018). Comparative study of classical test theory and item response theory using diagnostic quantitative economics skill test item analysis results. European Journal of Educational and Social Sciences, 3(1), 57–75. Retrieved July (2023). 9, https://dergipark.org.tr/en/pub/ejees/issue/40156/477675

    Google Scholar 

  • Eltahir, M. E., Alsalhi, N. R., & Al-Qatawneh, S. S. (2022). Implementation of e-exams during the COVID-19 pandemic: A quantitative study in higher education. PloS One, 17(5), e0266940. https://doi.org/10.1371/journal.pone.0266940.

    Article  Google Scholar 

  • European Training Foundation. (2019). Digital exam in vocational education and training: A review of the evidence. European Training Foundation.

  • Fulcher, G. (2003). Interface design in computer-based language testing. Language Testing, 20(4), 384–408. https://doi.org/10.1191/0265532203lt265oa.

    Article  Google Scholar 

  • Greenhalgh, S. P., & Rogers, R. L. (2010). Digital assessment: State of the art and future directions. In T. J. Cochrane, & L. P. Cook (Eds.), Assessing and evaluating adult learning in career and technical education (pp. 191–213). Information Age Publishing.

  • Hardwick, R. M., Forrence, A. D., Costello, M. G., Zackowski, K. M., & Haith, A. M. (2022). Age-related increases in reaction time result from slower preparation, not delayed initiation. Journal of Neurophysiology, 128(3), 582–592. https://doi.org/10.1152/jn.00072.2022

    Article  Google Scholar 

  • Huff, K. L., & Sireci, S. G. (2001). Validity issues in computer-based testing. Educational Measurement: Issues and Practice, 20(3), 16–25. https://doi.org/10.1111/j.1745-3992.2001.tb00066.x.

    Article  Google Scholar 

  • Huh, S. (2014). Computer-based testing and construction of an item bank database for medical education in Korea. Korean Medical Education Review, 16(1), 11–15. https://doi.org/10.17496/kmer.2014.16.1.011.

    Article  Google Scholar 

  • Human Resources Development Service of Korea. (2016). Cost analysis research service for Korean Technical Qualification test questions. Internal document.

  • Jaskova, J. (2021). Digital testing during the pandemic crisis: University students’ opinions on computer-based tests. International Journal for Innovation Education and Research, 9(1), 36–53. https://doi.org/10.31686/ijier.vol9.iss1.2833.

    Article  Google Scholar 

  • Javidanmehr, Z., & Anani Sarab, M. (2017). Cognitive diagnostic assessment: Issues and considerations. International Journal of Language Testing, 7(2), 73–98.

    Google Scholar 

  • Kane, M. T. (2006). Validation. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 17–64). Praeger Publishers.

  • Kano, Y., & Watanabe, Y. (2023). Development and evaluation of the CAT system to reduce mathematics test anxiety in Japanese high schools [Conference presentation abstract]. 17th International Technology Education and Development Conference Valencia Spain. https://doi.org/10.21125/inted.2023.1733.

    Article  Google Scholar 

  • Kim, M., & Han, C. H. (2021). A survey on the perception and satisfaction of Korean traditional medical students on computer-based test. Journal of Society of Preventive Korean Medicine, 25(3), 57–71. https://doi.org/10.25153/SPKOM.2021.25.3.005.

    Article  Google Scholar 

  • Kuzmina, I. P. (2010). Computer-based testing: Advantages and disadvantages. NBUV. http://nbuv.gov.ua/UJRN/VKPI_fpp_2010_1_35

  • Lakner, C., Yonzan, N., Mahlerr, D. G., Castaneda, A., Aguilar, R. A. C., Wu, H., & Fleury, M. (2020, October 7). Updated estimates of the impact of COVID-19 on global poverty: The effect of new data World Bank Blogs. Retrieved November 24, 2022. https://blogs.worldbank.org/opendata/updated-estimates-impact-covid-19-global-poverty-effect-new-data.

  • Lane, S., Raymond, M. R., & Haladyna, T. M. (2013). Handbook of Test Development. Routledge.

  • Lee, S. (1998). Interface design with focus on icons, color, and text. Multimedia-Assisted Language Learning, 1(1), 51–62.

    Google Scholar 

  • Lee, Y., & Chen, H. (2011). A review of recent response-time analyses in educational testing. Psychological Test and Assessment Modeling, 53(3), 359–379.

    Google Scholar 

  • Lee, S., & Park, J. (2012). A cross-national comparative study of National Qualification frameworks and the policy implications for Korea. Korea Social Policy Review, 19(4), 9–47.

    Article  Google Scholar 

  • Lee, D., Chung, J., Min, S., Hyun, J., & Kim, H. (2018). Linkage between the Korean qualification Framework and labor market. Korea Research Institute for Vocational Education and Training.

  • Mazzeo, J., & Harvey, A. L. (1988). The equivalence of scores from automated and conventional educational and psychological tests: A review of the literature. ETS Research Report Series, 1988(1), i–27. https://doi.org/10.1002/j.2330-8516.1988.tb00277.x.

    Article  Google Scholar 

  • Middleton, H. (2002). Complex problem solving in a workplace setting. International Journal of Educational Research, 37(1), 67–84. https://doi.org/10.1016/S0883-0355(02)00022-8.

    Article  Google Scholar 

  • Ministry of Employment and Labor (2020, September 18). National technical qualification digital innovation! Industrial engineer written test method, full conversion to computer-based test (CBT). MOEL. Retrieved May 10, 2022. https://www.moel.go.kr/news/enews/report/enewsView.do?news_seq=11410

  • Nam, J. (2017). Revisions of university entrance exams of mathematics in the UK, Australia, and Japan. Journal of Educational Research in Mathematics, 27(4), 679–700. https://www.koreascience.or.kr/article/JAKO201721464985575.pdf.

    Google Scholar 

  • Omorede, A. (2023). Business in the digital age: Digital innovation outcome, exit and the founder’s start-up experience role. Journal of the International Council for Small Business, 4(1), 68–78. https://doi.org/10.1080/26437015.2022.2125355.

    Article  Google Scholar 

  • Organization for Economic Cooperation and Development (2022). Paying the price of war. OECD economic outlook, interim report. OECD. Retrieved May 15, 2022. https://www.oecd-ilibrary.org/sites/ae8c39ec-en/index.html?itemId=/content/publication/ae8c39ec-en

  • Organization for Economic Cooperation and Development (2021b). OECD economic outlook: Volume 2021, issue 1. OECD. Retrieved May 15, 2022. https://www.oecd-ilibrary.org/economics/oecd-economic-outlook/volume-2021/issue-1_edfbca02-en

  • Organization for Economic Cooperation and Development. (2021a). Risks that matter 2020: The long reach of COVID-19. OECD. https://www.oecd.org/coronavirus/policy-responses/risks-that-matter-2020-the-long-reach-of-covid-19-44932654/

  • Petrillo, J., Cano, S. J., McLeod, L. D., & Coon, C. D. (2015). Using classical test theory, item response theory, and rasch measurement theory to evaluate patient-reported outcome measures: A comparison of worked examples. Value in Health, 18(1), 25–34. https://doi.org/10.1016/j.jval.2014.10.00510.1016/j.jval.2014.10.005.

  • Pokhrel, S., & Chhetri, R. (2021). A literature review on impact of COVID-19 pandemic on teaching and learning. Higher Education for the Future, 8(1), 133–141. https://doi.org/10.1177/2347631120983481.

  • Robinson, A. (2023). SAT Changes 2023–2024: What You Need To Know. SAT General Info. Retrieved July 15, 2023. https://blog.prepscholar.com/sat-changes-2023-2024.

  • Roy, S., Singh, A., & Choudhary, C. (2021). Artificial Intelligence in Healthcare. In M. S. Kaiser, J. Xie, & V. S. Rathore (Eds.), Information and Communication Technology for competitive strategies (ICTCS 2020). Lecture notes in networks and systems (190 vol.). Springer. https://doi.org/10.1007/978-981-16-0882-7_24.

  • Seoul National University (n,d,). Test of English Proficiency by Seoul National University (TEPS)www.ielts.org.

  • Services Australia (2019). 2018–19 Annual Report. Services Australia. Retrieved May 15, 2022. https://www.servicesaustralia.gov.au/sites/default/files/annual-report-191019-v2.pdf

  • Sheedy, J. E., Subbaram, M. V., Zimmerman, A. B., & Hayes, J. R. (2005). Text legibility and the letter superiority effect. Human Factors, 47(4), 797–815. https://doi.org/10.1518/001872005775570998.

    Article  Google Scholar 

  • Singh, A. K., Satapathy, S. C., Roy, A., & Gutub, A. A. (2022). AI-Based Mobile Edge Computing for IoT: Applications, challenges, and future scope. Arabian Journal for Science and Engineering, 1–31.

  • Sireci, S. G., & Zenisky, A. L. (2006). Gender and computer-based testing: Differences in examinees’ persistence and performance. Journal of Educational Computing Research, 34(1), 67–82.

    Google Scholar 

  • Spence, M. (1973). Job market signaling. The Quarterly Journal of Economics, 87(3), 355–374. https://doi.org/10.2307/1882010.

    Article  Google Scholar 

  • Thathsarani, H., Ariyananda, D. K., Jayakody, C., Manoharan, K., Munasinghe, A. A. S. N., & Rathnayake, N. (2023). How successful the online assessment techniques in distance learning have been, in contributing to academic achievements of management undergraduates? Education and Information Technologies, 1–25. https://doi.org/10.1007/s10639-023-11715-7

  • Thurow, L. C. (1975). Generating inequality: Mechanisms of distribution in the US economy. Basic Books.

  • Ting, D. S. W., Carin, L., Dzau, V., & Wong, T. Y. (2020). Digital technology and COVID-19. Nature Medicine, 26(4), 459–461. https://doi.org/10.1038/s41591-020-0824-5.

    Article  Google Scholar 

  • Uh, S., Lee, S., & Chun, S. (2020). Past, Present and Future of National Competency standards & Korean Qualification Framework. Journal of Vocation and Qualification, 9(2), 129–149.

    Google Scholar 

  • World Bank (2022). World development report: Finance for an equitable recovery World Bank. Retrieved May 15, 2022. https://openknowledge.worldbank.org/bitstream/handle/10986/36883/9781464817304.pdf

  • Xu, X., & Bejar, I. I. (2009). Design considerations for computer-based test accommodations: A validity-based approach. Journal of Educational Measurement, 46(2), 187–206.

    Google Scholar 

  • Yonzn, N., Cojocaru, A., Lakner, C., Mahler, D. G., & Narayan, A. (2022). The impact of COVID-19 on poverty and inequality: Evidence from phone surveys World Bank. Retrieved May 15, 2022. https://blogs.worldbank.org/opendata/impact-covid-19-poverty-and-inequality-evidence-phone-surveys#comments.

  • Ziefle, M. (1998). Effects of display resolution on visual performance. Human Factors, 40(4), 554–568. https://doi.org/10.1518/001872098779649355.

    Article  Google Scholar 

Download references

Funding

The authors did not receive support from any organization for the submitted work.

Author information

Authors and Affiliations

  1. Evaluation and Accreditation Team, Korean Council for University Education, Gasan-dong, Gumcheon-Gu, Seoul, Republic of Korea

    Mi-Hyun Bang

  2. Dept. of Public Administration, Sookmyung Women’s University, 04310 Chungpa-dong, Yongsan-gu, Seoul, Republic of Korea

    Young-Min Lee

Authors
  1. Mi-Hyun Bang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Young-Min Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

The study conception and design, material preparation, data collection and analysis were performed by Mi-Hyun Bang and Young-Min Lee. Young-Min Lee contributed new methods or models. Mi-Hyun Bang and Young-Min Lee wrote the paper.

Corresponding author

Correspondence to Young-Min Lee.

Ethics declarations

Ethical approval and consent

All study participants provided informed consent, and the study design was approved by the ethics review board of the Human Resource Development Service of Korea where main author worked.

Competing interests

The authors have no relevant financial or non-financial interests to disclose.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bang, MH., Lee, YM. Pilot study on the digitalization of the national qualification exam for Korean engineers. Educ Inf Technol 29, 21–50 (2024). https://doi.org/10.1007/s10639-023-12279-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10639-023-12279-2

Keywords