Challenges and approaches related to AI- driven grading of open exam

questions in higher education: Human in the loop

(Jordan & Mitchell, 2015)- Although AI is capable of grading multiple-choice

questions, The grading of open exam questions still has room for

Improvement before being implemented in higher education. That means

these tools are currently in the progression stage. One of these tools is
“EasyGrader”, a tool based on AI which Focuses on grading open questions
from exams in higher Education. Tools like this need a high amount of
training data From past exams to learn to create a proper assessment for the
Questions. Algorithms are used to focus on words and sentences Regarding a
certain topic. If these words or sentences appear in The answer to the open
questions, the algorithm identifies them And accordingly grades the open
question. The principle behind it is machine learning, which can be Described
as the action of automatic improvement of computers Through experience,
lying at the intersection of computer Science and statistics and the core of
the AI and data science . A crucial point to take into consideration when
speaking of the Implementation of AI-based grading tools is the perception of
Students. Fairness and transparency need to be present. It is Important to
know what the students think of AI-based grading Of open questions and
where exactly in the grading process they Can imagine AI doing the work and
where they prefer leaving It to the teachers. The opinion of students is
crucial since they Are the subjects affected by the possible implementation
of AI-Based grading in the future. Further, acceptance to change is better
reached once trust is Established, which can be done by being transparent
and fair in Designing, developing, and implementing the tool. Also, it can Be
done by educating and communicating to the students about AI-based
grading tools, especially in the given situation where There might be a lack
of information and knowledge. Once Students are persuaded and resistance
is overcome, one possible Outcome might be that they will even help
implement the tool In the future (Kotter & Schlesinger, 1989).

THEORIES

1. Artificial Intelligence

Firstly, the term Artificial Intelligence (AI) needs to be defined.

Numerous definitions exist, among the most popular are the Following:
 • Artificial intelligence is the mechanical simulation of Collecting
knowledge and information and Processing intelligence of the universe:
(collating and Interpreting) and disseminating it to the election in The
form of an actionable intelligence (Grewal, 2014)

• Artificial Intelligence is the simulation of human Intelligence

processes by machines, especially Computer systems. Specific
applications of AI Include expert systems, natural language processing,
Speech recognition, and machine vision (Burns, Laskowski, & Tucci,
2022)

• Computers that perform cognitive tasks are usually Associated with

human minds, particularly in the Learning and problem-solving (Baker
& Smith, 2019) Since this research deals with the AI-based grading tool
EasyGrader, which is able to execute tasks previously done by
Humans, the best fitting definition of AI for this paper is the one From
Baker & Smith. The AI is executing the cognitive tasks Needed, so the
definition matches the given situation.

1.2 The grading process

Secondly, the framework of the different steps in grading needs To
be specified. At the University level, grading mostly consists Of the
following steps: The first examiner checks every Question and adds
all points together. Afterward, the second Examiner goes through
the same process before both examiners Compare both of their
scores and determine the final score.
1.3 Open questions in exams
Thirdly, the concept of “open questions in exams in higher
Education” needs to be consolidated. “Open questions” in time
limited exams are questions that ask students to explain a Certain
process, model, or concept in their own words and Written form.
Open questions can be so-called fill-in-the-blank Questions, short
answers without a bank, or long answers.

Faculty member’s use of artificial intelligence to grade student papers: a

Case of implications

Robert Coles (1989) stated that artifcial intelligence (AI) in post secondary
education (PSE) are being her-Alded as opportunities (Martiniello et al. 2020)
and equally vociferously are the calls to Be cautious (Barnett, 2023; Wingard
2023). In this instance, a plausible hypothetical case Can shed light on
educational integrity issues and ethical concerns.using an AI bot to grade
student papers was To save time and restore a healthier work-life balance an
argument that non human Aids can save time have been repeated since
Pressey (1926) to more recently by Holmes Et al. (2019). Potential benefts
of the AI grading service.

1. Discretion

In its communication, the AI service provider assured Dr. Case that any
personal information collected about Case himself, the details of student
papers that he wants to Grade, and student records would not be shared
with anyone.

2. Time saving

Te most signifcant reason Case considered using an AI bot to grade

student papers was To save time and restore a healthier work-life balance
– an argument that non-human Aids can save time have been repeated
since Pressey (1926) to more recently by Holmes Et al. (2019).

3. Convenience

Te entire system was convenient too, ofering Dr. Case three options: First,
he could Upload student papers from within the comfort of his home; all
he needed was a computer and a stable internet connection. Potential
pitfalls of the AI grading service.

1. Cost

After flling out the AI grading service’s requested information form, Dr.
Case received a price quote, which was not insignifcant.

2. Privacy concerns

Two problematic privacy issues emerged in Case’s contemplation of using

the AI gradIng service: his own and his students. Case was concerned
about how the information Collected about him would be used: Would the
company list his name on its website? Would his employers be notifed
that he was using the company’s service? Would stuDents be told that he
uses AI technology to grade their papers? What happens to his
Information if he terminates the contract with the AI company? Would his
use of the AI Grading service be disclosed to other universities where
Case might apply in the future.
 Do teachers spot AI? Evaluating the detectability of AI generated texts
among student essays.

Generative AI is a broad term for any type of artificial intelligence (AI) that
can produce new texts, images, videos, or computer code. Based on massive
amounts of training data, generative AI can use prompts written in natural
language to generate new output. Large language models (LLM), such as the
Generative Pre-trained Transformer 3 (GPT-3), can generate human-like text
and complete other language- related tasks such as translation, question
answering, and coding. The use of LLM like GPT-3 in educational contexts has
been a recent matter of discussion (Cotton et al., 2023; Kasneci et al., 2023).
Sailer et al. (2023) showed that AI-generated adaptive feedback facilitates
pre-service teachers’ quality of justifications in written assignments as a part
of their diagnostic competence. Jansen et al. (2024) showed that automatic
feedback including goal-setting support is in particular helpful when students
had to revise their texts. In one of the first experimental studies on the use
of LLMs for feedback generation, Meyer et al. (2024) showed that AI-
generated feedback increased students’ revisions, motivation, and positive
emotions. On the other hand, there are certain caveats regarding the role of
generative AI. Cotton et al. (2023) discussed the challenges concerning the
use of ChatGPT in higher education. These include cheating and deception,
causing issues for assessment and learning. Students who use tools like
ChatGPT in written assignments have an unfair advantage when it comes to
assessment and grading. At the same time, they might miss out on learning
opportunities. Therefore, it is vital for teachers to know if and when AI tools
are used, especially for the formative and summative assessment of
students’ writing performance. If students’ writing skills suffer because they
use AI tools for their texts very frequently, teachers could counteract, for
example, by creating assignment or exam conditions which ensure that
students produce their texts without using AI.

THEORIES

1. Large Language Models (LLMs) and Generative AI

The study is based on the development and capabilities of Generative
AI models, particularly Large Language Models (LLMs) like ChatGPT
(GPT-3, GPT-4). LLMs use deep learning and natural language
processing (NLP) to generate coherent, structured, and contextually
relevant texts.

1.2. AI and Writing Assessment

AI-generated texts can mimic human writing but might lack originality,
critical thinking, and personal voice. Prior studies in news articles
(Graefe et al., 2018), scientific abstracts (Gao et al., 2023), and poetry
(Günser et al., 2022) suggest that AI-generated texts are often
indistinguishable from human writing.

2. Theories on Human Perception and AI Detection

2.1. Overconfidence Bias
People often overestimate their ability to detect AI-generated
content.Studies by Köbis & Mossink (2021) and Gunser et al. (2022)
found that readers believe they can identify AI-generated texts but
perform poorly in actual detection tasks.
2.2. Fluency Heuristic
People associate well-structured and grammatically correct writing with
human authorship, while they assume errors and inconsistencies
indicate human writing. Teachers may falsely classify high-quality AI
texts as student-written and low-quality AI texts as human work.
2.3. Source Bias
The perception of text quality is influenced by its assumed origin (AI vs.
human). Gunser et al. (2022) found that participants rated AI-
generated poems less inspiring and less creative even when they could
not reliably identify the source.
3. Theories on AI and Education
3.1. AI in Learning and Assessment
AI-generated feedback can improve writing quality, revision practices,
and learning outcomes (Meyer et al., 2024). However, if students rely
too much on AI-generated writing, it may affect their cognitive
engagement and skill development.
3.2. Academic Integrity and AI

The study connects to the discourse on academic integrity (Cotton et

al., 2023), which warns that: AI enables cheating by generating
complete assignments. Current detection methods are unreliable in
distinguishing AI-generated texts.
 4. Theories on Teacher Expertise and Text Evaluation
4.1. Teacher Judgment Theory

Experienced teachers are assumed to be better at identifying AI-

generated texts due to their familiarity with common student writing
errors. However, research indicates that even experienced teachers
struggle to identify AI-generated essays accurately (Jansen et al.,
2024).

4.2.Automated Essay Scoring and Human Assessment

AI-based essay scoring models (e.g., TOEFL e-Rater®) are trained to

evaluate writing quality based on syntax, grammar, coherence, and
structure. However, these models lack subjective evaluation aspects,
such as critical thinking, argument depth, and originality.

Challenges and approaches related to AI- driven grading of open exam

questions in higher education: Human in the loop

(Jordan & Mitchell, 2015)- Although AI is capable of grading multiple-choice

questions,

The grading of open exam questions still has room for

Improvement before being implemented in higher education.

That means these tools are currently in the progression stage.

One of these tools is “EasyGrader”, a tool based on AI which

Focuses on grading open questions from exams in higher

Education. Tools like this need a high amount of training data

From past exams to learn to create a proper assessment for the

Questions. Algorithms are used to focus on words and sentences

Regarding a certain topic. If these words or sentences appear in

The answer to the open questions, the algorithm identifies them

And accordingly grades the open question.

The principle behind it is machine learning, which can be

Described as the action of automatic improvement of computers

Through experience, lying at the intersection of computer

Science and statistics and the core of the AI and data science .

A crucial point to take into consideration when speaking of the

Implementation of AI-based grading tools is the perception of

Students. Fairness and transparency need to be present. It is

Important to know what the students think of AI-based grading

Of open questions and where exactly in the grading process they

Can imagine AI doing the work and where they prefer leaving It to the
teachers. The opinion of students is crucial since they

Are the subjects affected by the possible implementation of AI-

Based grading in the future.

Further, acceptance to change is better reached once trust is

Established, which can be done by being transparent and fair in

Designing, developing, and implementing the tool. Also, it can

Be done by educating and communicating to the students about

AI-based grading tools, especially in the given situation where

There might be a lack of information and knowledge. Once

Students are persuaded and resistance is overcome, one possible

Outcome might be that they will even help implement the tool

In the future (Kotter & Schlesinger, 1989).

THEORIES

1. Artificial Intelligence
Firstly, the term Artificial Intelligence (AI) needs to be defined.

Numerous definitions exist, among the most popular are the

Following:

• Artificial intelligence is the mechanical simulation of

Collecting knowledge and information and

Processing intelligence of the universe: (collating and

Interpreting) and disseminating it to the election in

The form of an actionable intelligence (Grewal, 2014)

• Artificial Intelligence is the simulation of human

Intelligence processes by machines, especially

Computer systems. Specific applications of AI

Include expert systems, natural language processing,

Speech recognition, and machine vision (Burns,

Laskowski, & Tucci, 2022)

• Computers that perform cognitive tasks are usually

Associated with human minds, particularly in the

Learning and problem-solving (Baker & Smith, 2019)

Since this research deals with the AI-based grading tool

EasyGrader, which is able to execute tasks previously done by

Humans, the best fitting definition of AI for this paper is the one

From Baker & Smith. The AI is executing the cognitive tasks

 Needed, so the definition matches the given situation.

1.2 The grading process

Secondly, the framework of the different steps in grading needs

To be specified. At the University level, grading mostly consists

Of the following steps: The first examiner checks every

Question and adds all points together. Afterward, the second

Examiner goes through the same process before both examiners

Compare both of their scores and determine the final score.

1.4 Open questions in exams

1.5

Thirdly, the concept of “open questions in exams in higher

Education” needs to be consolidated. “Open questions” in time limited

exams are questions that ask students to explain a

Certain process, model, or concept in their own words and

Written form. Open questions can be so-called fill-in-the-blank

Questions, short answers without a bank, or long answers.

Faculty member’s use of artificial intelligence to grade student papers: a

Case of implications

Robert Coles (1989) stated that artifcial intelligence (AI) in post-secondary

education (PSE) are being her-

Alded as opportunities (Martiniello et al. 2020) and equally vociferously are

the calls to
Be cautious (Barnett, 2023; Wingard 2023). In this instance, a plausible
hypothetical case

Can shed light on educational integrity issues and ethical concerns.using an

AI bot to grade student papers was

To save time and restore a healthier work-life balance – an argument that

non-human

Aids can save time have been repeated since Pressey (1926) to more
recently by Holmes

Et al. (2019).

Potential benefts of the AI grading service

1. Discretion

In its communication, the AI service provider assured Dr. Case that any
personal information collected about Case himself, the details of student
papers that he wants to

Grade, and student records would not be shared with anyone.

2. Time saving

Te most signifcant reason Case considered using an AI bot to grade student

papers was

To save time and restore a healthier work-life balance – an argument that

non-human

Aids can save time have been repeated since Pressey (1926) to more
recently by Holmes

Et al. (2019).

3. Convenience

Te entire system was convenient too, ofering Dr. Case three options: First, he
could
Upload student papers from within the comfort of his home; all he needed
was a computer and a stable internet connection.

Potential pitfalls of the AI grading service

1. Cost

After flling out the AI grading service’s requested information form, Dr. Case
received a price quote, which was not insignifcant.

2. Privacy concerns

Two problematic privacy issues emerged in Case’s contemplation of using the

AI grad-

Ing service: his own and his students. Case was concerned about how the
information

Collected about him would be used: Would the company list his name on its
website?

Would his employers be notifed that he was using the company’s service?
Would stu-

Dents be told that he uses AI technology to grade their papers? What

happens to his

Information if he terminates the contract with the AI company? Would his use
of the AI

Grading service be disclosed to other universities where Case might apply in

the future.
Do teachers spot AI? Evaluating the detectability of AI generated texts
among student essays.

Generative AI is a broad term for any type of artificial intelligence (AI) that
can produce new texts, images, videos, or computer code. Based on massive
amounts of training data, generative AI can use prompts written in natural
language to generate new output. Large language models (LLM), such as the
Generative Pre-trained Transformer 3 (GPT-3), can generate human-like text
and complete other language- related tasks such as translation, question
answering, and coding. The use of LLM like GPT-3 in educational contexts has
been a recent matter of discussion (Cotton et al., 2023; Kasneci et al., 2023).
Sailer et al. (2023) showed that AI-generated adaptive feedback facilitates
pre-service teachers’ quality of justifications in written assignments as a part
of their diagnostic competence. Jansen et al. (2024) showed that automatic
feedback including goal-setting support is in particular helpful when students
had to revise their texts. In one of the first experimental studies on the use
of LLMs for feedback generation, Meyer et al. (2024) showed that AI-
generated feedback increased students’ revisions, motivation, and positive
emotions. On the other hand, there are certain caveats regarding the role of
generative AI. Cotton et al. (2023) discussed the challenges concerning the
use of ChatGPT in higher education. These include cheating and deception,
causing issues for assessment and learning. Students who use tools like
ChatGPT in written assignments have an unfair advantage when it comes to
assessment and grading. At the same time, they might miss out on learning
opportunities. Therefore, it is vital for teachers to know if and when AI tools
are used, especially for the formative and summative assessment of
students’ writing performance. If students’ writing skills suffer because they
use AI tools for their texts very frequently, teachers could counteract, for
example, by creating assignment or exam conditions which ensure that
students produce their texts without using AI.