Abstract
Machine learning and artificial intelligence (AI) are ubiquitous, although accessibility and application are often misunderstood and obscure. Automatic short answer grading (ASAG), leveraging natural language processing (NLP) and machine learning, has received notable attention as a method of providing instantaneous, corrective feedback to learners without the time and energy demands of human graders. However, ASAG systems are only as valid as the reference answers, or training sets, they are compared against. We introduce an AI-based, machine learning method of autograding online tutor lessons that is easily accessible and user friendly. We present two methods of training set creation using: a subset of learnersourced, human-graded tutor responses from the lessons; and a surrogate model using the recently released AI-chatbot, ChatGPT. Findings indicate human-created training sets perform considerably better than AI-generated training sets (F1 = 0.84 and 0.67, respectively). Our straightforward approach, although not accurate enough for wide use, demonstrates application of directly available machine learning based NLP methods and highlights a constructive use of ChatGPT for pedagogical purposes that is not without limitations.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
ChatGPT Dec 15 version. Retrieved December 22, 2022. https://openai.com
References
Altman, S.: [@sama]. ChatGPT launched on Wednesday. Today it crossed 1 million users! Twitter, 5 December 2022. https://twitter.com/sama/status/1599668808285028353
Burrows, S., Gurevych, I., Stein, B.: The eras and trends of automatic short answer grading. Int. J. Artif. Intell. Educ. 25(1), 60–117 (2015)
Grand View Research: Artificial Intelligence Market Size, Share & Trends Analysis Report By Solution, By Technology (Deep Learning, Machine Learning, Natural Language Processing, Machine Vision), by End Use, By Region, and Segment Forecasts, 2022 – 2030. Market Analysis Report (2022)
Lan, A. S., Vats, D., Waters, A.E., Baraniuk, R.G.: Mathematical language processing: automatic grading and feedback for open response mathematical questions. In: Proceedings of the Second (2015) ACM Conference on Learning@ Scale, pp. 167–176 (2015)
Leacock, C., Chodorow, M.: C-rater: automated scoring of short-answer questions. Comput. Hum. 37(4), 389–405 (2003)
Mayfield, E., Rosé, C.P.: LightSIDE: open source machine learning for text. In Handbook of Automated Essay Evaluation, pp. 146–157. Routledge (2013)
Mello, R.F., et al.: Enhancing instructors’ capability to assess open-response using natural language processing. In: Hilliger, I., Muñoz-Merino, P.J., De Laet, T., Ortega-Arranz, A., Farrell, T. (eds.) Educating for a New Future: Making Sense of Technology-Enhanced Learning Adoption. EC-TEL 2022. Lecture Notes in Computer Science, vol. 13450, pp. 102–115. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-16290-9_8
Thomas, D.R., Yang, X., Gupta, S., Adeniran, A., McLaughlin, E.A., Koedinger, K.R.: When the tutor becomes the student: design and evaluation of efficient scenario-based lessons for tutors. In LAK23: 13th International Learning Analytics and Knowledge Conference (LAK 2023), 13–17 March 2023, Arlington, TX, USA. ACM, USA (2023)
Zhang, M., Baral, S., Heffernan, N., Lan, A.: Automatic short math answer grading via in-context meta-learning. arXiv preprint arXiv:2205.15219 (2022)
Acknowledgements
This work is supported with funding from the Chan Zuckerberg Initiative (Grant #2018-193694), Richard King Mellon Foundation (Grant #10851), Bill and Melinda Gates Foundation, and the Heinz Endowments (E6291).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Thomas, D.R., Gupta, S., Koedinger, K.R. (2023). Comparative Analysis of Learnersourced Human-Graded and AI-Generated Responses for Autograding Online Tutor Lessons. In: Wang, N., Rebolledo-Mendez, G., Dimitrova, V., Matsuda, N., Santos, O.C. (eds) Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners, Doctoral Consortium and Blue Sky. AIED 2023. Communications in Computer and Information Science, vol 1831. Springer, Cham. https://doi.org/10.1007/978-3-031-36336-8_110
Download citation
DOI: https://doi.org/10.1007/978-3-031-36336-8_110
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-36335-1
Online ISBN: 978-3-031-36336-8
eBook Packages: Computer ScienceComputer Science (R0)