Journal of Science Education and Technology

https://doi.org/10.1007/s10956-024-10182-0

Exploring Students’ Multimodal Representations of Ideas About

Epistemic Reading of Scientific Texts in Generative AI Tools
Kason Ka Ching Cheung1 · Jack Pun2 · Wangyin Kenneth‑Li2 · Jiayi Mai2

Accepted: 3 November 2024

© The Author(s) 2024

Abstract
As students read scientific texts created in generative artificial intelligence (GenAI) tools, they need to draw on their epistemic
knowledge of GenAI as well as that of science. However, only a few research discussed multimodality as a methodological
approach in characterising students’ ideas of GenAI-science epistemic reading. This study qualitatively explored 44 eighth
and ninth graders’ multimodal representations of ideas about GenAI-science epistemic reading and developed an analyti-
cal framework based on Lemke’s (1998) typology of representational meaning, namely presentational, organisational, and
orientational meanings. Under each representational meaning, several categories were inductively generated while students
expressed preferences in using drawn, written, or both drawn and written mode to express certain categories. Findings indi-
cate that a multimodal approach is fruitful in characterising students’ semiotic resources in meaning-making of ideas about
GenAI-science epistemic reading. We suggested implications regarding future intervention studies on tracking students’
ideas about GenAI-science epistemic reading using the analytical framework developed in this study.

Keywords Generative artificial intelligence · Nature of science · Nature of GenAI · Multimodality

Introduction students’ ideas about epistemic reading of scientific texts.

When students interact with GenAI, the first thing students
Generative artificial intelligence (GenAI) tools, such as need to do is to read the scientific texts created in GenAI. AI,
GPT-4 and DALL.E2, have gained an increasing popular- particularly GenAI, has significantly influenced scientific
ity in science education (Alasadi & Baiz, 2024; Ng et al., practices (Erduran, 2023; Erduran & Levrini, 2024), includ-
2023a), while research instruments that measure students’ ing anomaly detection in data (Nigam et al., 2021). Despite
competence of using AI in disciplinary learning are mostly its enormous impact on scientific discovery, students con-
in the form of Likert scale (e.g., Ng et al., 2023a, b). Consid- flated its role with that of scientists, with some holding the
ering this, this paper seeks to develop a new methodological misconception that GenAI performed manipulative experi-
approach to the field of science education, by characteris- ments to generate scientific texts (Authors 1 and 2, 2024).
ing students’ multimodal representations, to understand Hence, it is important for teachers and students to explicitly
address how students read scientific outputs as GenAI by
* Jack Pun drawing on their epistemic knowledge.
jack.pun@cityu.edu.hk Although science education literature conceptualises
Kason Ka Ching Cheung epistemic reading of scientific texts (Tang et al., 2022a,
cheungkac@eduhk.hk b), it is necessary for students to explicitly communicate
Wangyin Kenneth‑Li and represent how they read such texts in GenAI with mul-
wangyili@cityu.edu.hk timodal representations. When reading scientific texts cre-
Jiayi Mai ated by GenAI, however, students need to comprehend
maggimai@cityu.edu.hk epistemic underpinnings of both science and GenAI in
order to foster critical thinking regarding these texts. We
1
Department of Curriculum and Instruction, The Education argue that the construct of epistemic reading subsumes
University of Hong Kong, Hong Kong SAR, China
two components: epistemic trust and epistemic knowl-
2
Department of English, City University of Hong Kong, edge. Epistemic trust refers to the recognition of valid
Hong Kong SAR, China

Vol.:(0123456789)
 Journal of Science Education and Technology

knowledge (Ford & Forman, 2006), specifically in terms Literature Review

of perceiving the authority of scientific claims (Tang &
Cooper, 2024) in relation to the nature of science and Ideas About GenAI‑Science Epistemic Reading
GenAI; epistemic knowledge pertains to the origin of
scientific information in GenAI outputs (Cheung et al., Epistemic reading entails the understanding of a disci-
2024a), such as whether the claims in GenAI outputs have pline’s underlying epistemology to interpret its special-
been supported by empirical evidence and were reliable ized texts (Tang et al., 2022a, b). When reading scientific
under large language models. Students commonly equate texts created by GenAI, students need to comprehend the
GenAI tools with human intelligence and struggle to pin- epistemic underpinnings of two disciplines, namely GenAI
point how GenAI creates scientific texts (Cheung et al., and science. To make such interdisciplinary knowledge
2024b). While the cognitive, affective, behavioral, and “insightful,” teachers need to connect learning experi-
ethical domains of literacy are crucial for students to uti- ences of these two disciplines to avoid compartmentaliza-
lise GenAI in science learning, we argue that the epistemic tion (Barelli et al., 2024; Billingsley & Hardman, 2017).
domain of GenAI literacy is a prerequisite for critically Emerging research has demonstrated the need for explicit
interpreting claims generated by GenAI. The existing sci- discussion of the epistemic interaction between AI and
ence education literature lacks a detailed methodological science. As shown in classroom discourse, students’
account of characterising students’ ideas about GenAI- epistemic knowledge of generic AI applications remains
science epistemic reading. uninformed, as they perceive AI-generated knowledge as
This study aims to fill methodological and theoretical consistently reliable and trustworthy (Kim et al., 2023;
gaps in characterising students’ ideas on GenAI-science Kim, 2022). According to Park et al. (2023), when teach-
epistemic reading. Methodologically, it proposes a multi- ers incorporated generic AI applications into science les-
modal approach to characterise these ideas. Theoretically, sons, teachers could discuss the contrasting epistemic
it contributes to the science and educational technology aims of science (generating knowledge claims) and AI
literature by developing a coding scheme for diverse (predicting and forming solutions). Regarding epistemic
meanings of students’ GenAI-science epistemic reading. processes, students who harnessed machine learning algo-
Likert scale measurements can offer a statistical overview rithms to predict future weather need to discuss how AI
of whole-class response changes but are limited to self- technologies help scientists construct models (Park et al.,
reported data (Paap et al., 2024; Subedi, 2016) and only 2023). In these generic AI applications, students used AI
address students’ understanding in the linguistic modal- technologies as a tool to facilitate their scientific practices
ity. Researchers in science education and technology have (Erduran, 2023; Erduran & Levrini, 2024).
explored the use of drawings to characterise students’ While there is an emerging line of research on foster-
understanding of technologies (Chang & Tsai, 2023; Yeh ing students’ AI-science epistemic knowledge, it has not
et al., 2019). Social semiotics can be a potential field for specifically focused on GenAI. GenAI, such as ChatGPT,
exploring students’ representations of ideas on GenAI- Sora, and Google Bard, mimics human intelligence to gen-
science epistemic reading. The field of social semiotics erate content based on user input (Fui-Hoon Nah et al.,
concerns how individuals use signs, texts, and discourse, 2023; Jeon et al., 2023). Compared to generic AI appli-
to communicate their meaning within a particular social cations, GenAI positions itself as an epistemic authority.
context (Peirce, 1931; van Leeuwen, 2005). From a social According to Cooper (2023), GenAI tools often provide
semiotic perspective, students may produce diverse signs claims without properly grounding them with sufficient
to symbolise their presentation, orientation, and organisa- scientific evidence. Some GenAI tools, such as GPT-3.5,
tion of ideas about GenAI’s interaction with scientific texts were trained on vast textual data to generate human-like
(Lemke, 1998). Therefore, it is important to explore how responses that answer queries and perform language-
students use diverse modes to characterise their meaning- related tasks with accurate predictions and estimations
making of ideas about GenAI-science epistemic reading. (Floridi & Chiriatti, 2020). Moreover, as a large language
To address the methodological gap, we explored the model pre-trained with Western-centric data, GenAI may
potential of a multimodal approach to characterise stu- often provide biased evidence (Cooper & Tang, 2024). For
dents’ ideas regarding GenAI-science epistemic reading example, ChatGPT lacks contextual awareness in generat-
in a classroom-based case study. The present study was ing claims about climate change (Gursesli et al., 2023).
guided by the following research question: Given the challenges posed by GenAI, it is crucial for
What are students’ multimodal representations of ideas science educators to explore how to characterise students’
about GenAI-science epistemic reading in terms of pres- ideas about GenAI-science epistemic reading. This may
entational, organisational, and orientational meanings? offer insights into how an epistemic-focused teaching
Journal of Science Education and Technology

approach can enrich students’ semiotic resources. Ideas education technology, researchers have been employing
about GenAI-science epistemic reading specifically refer drawing to explore technology-mediated learning (Chang
to students’ understanding of the interplay between the & Tsai, 2023; Yeh et al., 2019). Through the analysis of
epistemology of GenAI and science when they engage children’s drawings, researchers have identified catego-
with GenAI-generated scientific texts. We propose two ries related to students’ conception of technology-assisted
elements of GenAI-science epistemic reading: epistemic learning, including among others, the location of learning,
trust and epistemic knowledge. GenAI-science epistemic types of technologies, content of learning, affordances of
trust encompasses vigilant trust which is “a complex of technologies, and types of learning activities (Yeh et al.,
cognitive mechanisms, emotional depositions, inherited 2019). Drawing also enables researchers to uncover sub-
norms, reputational cues” (Origgi, 2012, p. 224). Regard- categories within these main categories. For example,
ing the term “epistemic,” individuals employ heuristics to under the category of “technology,” students can depict
evaluate claim truth (Origgi, 2012; Wilholt, 2013). When hardware, experimental instruments, teaching equipment,
interpreting GenAI outputs, these heuristics can include and innovative and future technology products. However,
an individual’s assessment of GenAI’s representation of drawing is just one form of representation, as different
scientists’ authority in forming claims. On the other hand, modes offer various affordances for meaning-making
GenAI-science epistemic knowledge refers to how knowl- (Prain & Tytler, 2012; Tang et al., 2022a, b). For example,
edge of science and knowledge of GenAI interact (Author while drawing can illustrate the connection between two
1, 2024). For example, it includes whether GenAI gener- atoms, written mode can specify chemical symbols (Tang
ates claims through scientific practices (Erduran, 2023; et al., 2019).
Erduran & Levrini, 2024) or how GenAI conveys scientific Drawing on systemic functional linguistics, we explored
claims through large language models. three types of meaning in students’ multimodal represen-
tations of ideas about GenAI-science epistemic reading:
A Multimodal Approach Towards Characterising presentational, orientational, and organisational mean-
Ideas About GenAI‑Science Epistemic Reading ings (Lemke, 1998). In Lemke’s (1998) seminal work, he
argued that multimedia texts in science involved different
In characterising general epistemic belief within a discipline, modalities. Although two modes (e.g., the visual and writ-
researchers commonly employ Likert scale questionnaires ten modes) can complement each other to make meaning,
(Bråten et al., 2011; Conley et al., 2004; Schraw et al., 2012). one mode cannot replace another because different modes
For instance, in a study on students’ epistemic knowledge of have various affordances, for example, no written texts can
the science-theology relationship, Konnemann et al. (2018) replace a picture (Lemke, 1998). The three types of mean-
used a Likert scale item; “evolutionary theory is supported ings can be realised by individual modalities or co-deploy-
by factual, historical and laboratory data” to measure stu- ment of various modalities:
dents’ acceptance of evolutionary theory based on scientific
data. Likert scale items are straightforward and adaptable, • Presentational meaning refers to the content of the texts,
enabling the analysis of collective responses to opinions and such as processes, relations, events, participants, and cir-
attitudes (Johns, 2010). However, Likert scale items inher- cumstances.
ently embody predefined notions (Mellor & Moore, 2014), • Orientational meaning refers to individuals’ orientation
which may fail to capture the diverse range of ideas that in terms of feelings, attitudes, and perspectives.
students hold regarding the interaction between GenAI and • Organisational meaning refers to the way of how two or
science when reading GenAI-generated scientific texts. more elements of representation join together to com-
A multimodal approach can potentially characterise stu- municate meaning.
dents’ ideas about GenAI-science epistemic reading. The
study of multimodality entails an examination of languages The three types of meanings have been applied to differ-
in combination of other mode (e.g., images) (O'Halloran, ent empirical studies in science education which character-
2011). Specifically, it explores how students integrate dif- ise students’ multimodal resources in meaning-making from
ferent modes of representation, including visual and written a social semiotic perspective (Tang et al., 2019; Wanselin
modes, to create meaning (Kress, 2010). Students’ visual et al., 2021). Also, previous works in science education also
and written representations serve as semiotic resources for demonstrated that both drawing and writing can illustrate
meaning-making in teaching and learning (Kress, 2010). different aspects of epistemology of science (e.g., Barak
Semiotic resources are the accumulated meaning poten- et al., 2022). Hence, Lemke’s meanings can be a potential
tial of various modalities (e.g., visual or linguistic) used fruitful approach to character students’ ideas about GenAI-
in communication within a social community (Airey & science epistemic reading. In the context of our study,
Linder, 2017; Kress, 2010). In the field of science and presentational meaning refers to students’ ideas about the
 Journal of Science Education and Technology

interaction between GenAI and science; orientational mean- Instrument

ing pertains to how students position themselves in relation
to their interaction with GenAI in acquiring scientific knowl- The multimodal instrument targets on how students repre-
edge; organisational meaning involves how parts of meaning sent ideas about GenAI-science epistemic reading from an
(e.g., visual structures or cohesive devices) can be organised object and a character viewpoint (Quinto‐Pozos & Parrill,
into a coherence whole. 2015). Regarding object viewpoint, students were prompted
by “Based on your understanding and experience, please
draw a picture to describe how ChatGPT/Poe generates texts
on a science topic.” This prompt situates students to create
Methods
a third-person narrative on whether they pose vigilant trust
(Origgi, 2012, p. 224) when ChatGPT/Poe generates scien-
Participants
tific texts. Also, such a prompt also explores characteristics
of science and characteristics of GenAI as well as their inter-
This study adopts a collective case study approach (Stake,
action. For character viewpoint, students were prompted
2006) to explore the affordances of a multimodal approach
by “Based on your understanding and experience, please
in characterising students’ ideas about GenAI-science epis-
draw a picture to describe how you read scientific texts on
temic reading. Data for the study are drawn from a large-
ChatGPT/Poe.” This prompt would like to delve into stu-
scale project aimed at improving Hong Kong students’ ideas
dents’ epistemologies in practice (Grapin et al., 2023). As a
about epistemic reading of scientific texts during extracur-
practice, the prompt explores how students read by drawing
ricular lessons. The second author gained ethical approval
on their epistemology of science/GenAI. We anticipate that
from their host institution, and all students and parents pro-
through analysing students’ drawing and writing, research-
vided consent to participate in the study. This collective case
ers can examine whether students create more signs related
study comprised 23 Grade 8 secondary-school students (13
to the science domain or the GenAI domain, or if they were
females and 10 males) ranging in age from 12 to 14 years
inclined to draw on the nature of knowledge or trust in a
old (M = 13.26, SD = 0.54), as well as 21 Grade 9 secondary-
particular discipline. Under each prompt, some spaces were
school students (12 females and 9 males) aged between 14
left approximately 8 cm × 12 cm to allow students to draw
and 16 years old (M = 14.05, SD = 0.49). A total of 44 stu-
diagrams to represent their ideas about GenAI-science epis-
dents took part in this study.
temic reading, while under the drawing space, they were
also prompted by the sentence “Please write down three
Data Collection to five sentences to describe your drawing.” This writing
prompt was referenced from Tu and Hwang (2023) which
We collected responses before a teaching-and-learning also explores students’ conception of ChatGPT. The prompts
activity in the two classrooms with explicit-reflective were reviewed by the two participating teachers who were
instruction on ideas about GenAI-science epistemic responsible for communicating the logistics of the study
reading. This teaching activity is part of an intervention with our research team.
aimed at improving students’ understanding of epistemic
reading of scientific texts from various sources including Data Analysis and Reliability
GenAI and online media. The teaching was conducted
by a Cantonese-English bilingual teacher with a Mas- To analyse students’ multimodal representations of ideas
ter of Arts in English Studies and has several years of about GenAI-science epistemic reading, the first and forth
teaching experience. The teacher obtained his teaching authors examined the data together. The presentational,
qualification through a Postgraduate Diploma in Educa- orientational, and organisational meanings of text-image
tion (PGDE) awarded by a higher education institution in from Lemke (1998) formed the overarching semiotic
Hong Kong. Prior to the data collection, the teacher par- meaning. After the first and the fourth authors examined
ticipated in online training workshops on how to imple- the responses, they developed categories and codes under
ment the teaching approach. Before the explicit-reflective the three types of representational meanings. After several
instruction, students were given 10 min to complete the rounds of discussion, some codes were removed because of
multimodal instrument. Based on their responses, the overlapping meaning between the two codes. For example,
teacher and students were engaged in explicit-reflective under the category of epistemic trust, the code “fact check”
discussion regarding epistemic reading of scientific texts was removed because it has a similar meaning with “critical
in GenAI outputs. In the discussion, students did not thinking.” Both authors coded 11 multimodal representa-
engage in interaction with GenAI, as the teacher spent tions on question 1 (more than 20% of the data) and yielded
more time to discussing epistemic aspects. a range of intercoder reliability of 63.6 to 100% (Appendix
Journal of Science Education and Technology

S1). Cheung and Tai (2023) suggest that the important con- search, large language models, human-like, general func-
sideration of intercoder reliability for qualitative educational tions, and simple conversation. For Google search, students
research was to document the percentage agreement of each expressed that GenAI equates to Google search; for large
category and to explain why there was inconsistency among language models, students expressed the use of pre-trained
the two coders. For epistemic knowledge, there was a signifi- data or transformer architecture behind GenAI; for general
cant discrepancy between two coders agreeing on whether function, students simply expressed input and outputs of
a representation showed Google search or general function. texts; for simple conversation, students represented two-way
We then arrived at a consensus that if there was an indication dialogue between human and GenAI.
that ChatGPT/Poe searched or organised information online, Question 1 and question 2 prompt students to express how
students’ visual or written representations were classified GenAI creates scientific texts and how they read scientific
as the code “Google search” instead of “general function.” texts in GenAI, respectively. The first question situates them-
Representation of “general function” should not imply any selves in an object/third-person viewpoint while the second
mechanism behind GenAI. question situates students in a character/first-person view-
Codes were put in an excel spreadsheet under each cat- point, to elicit diverse perspectives of how students make
egory for each student’s responses. Under the same category, meaning of the epistemic aspects while reading scientific texts
codes were mutually exclusive. However, two codes from in GenAI. Most students (54.5% for Q1 and 43.5% for Q2)
different categories can be applied to the same representa- did not express any stance regarding epistemic trust in their
tion. A “D” was put down if only students’ drawing inherited written and visual modes both questions (refer to Fig. 1). In
the code of meaning; a “W” was put down if only students’ contrast, a significant proportion of students (22.7% for Q1
writing inherited the code of meaning; a “D/W” was put and 11.4% for Q2) used the written mode to express their
down if students’ both writing and drawing inherited the trust, such as expressing their belief that ChatGPT can clas-
code of meaning. sify true or false claims (“First, science topic be found. And
ChatGPT find what scientist said. Then, it find some. It sum
up, Then it the science topic is True of false,” Student 01, Q1).
Results For epistemic knowledge represented in Q1, approxi-
mately one-fifth students initially drew simple conversation
Presentational Meaning (31.8%) or expressed Google search in drawn/written mode
(22.7%); for epistemic knowledge represented in Q2, one-
Presentational meaning refers to students’ ideas about the fourth students (25%) represented simple conversation in
world (Lemke, 1998; Tang et al., 2019). In this study, pres- the written mode while 18.2% of the students represented
entational meaning specifically refers to students’ ideas of general function in the written mode.
how GenAI creates scientific texts as well as how they read
scientific texts. As justified in the literature review, there are Organisational Meaning
two overarching categories, epistemic trust and epistemic
knowledge. Epistemic trust concerns how students draw on Organisational meaning refers to how elements in drawn or
their epistemic knowledge of GenAI and scientific claims written modes were arranged into a coherent piece (Lemke,
to determine if scientific information present in GenAI is 1998). For drawing, two codes were inductively gener-
true. Under epistemic trust, four codes were inductively ated, namely flowchart and fictional (Table 1). Flowchart
generated, namely trust, critically thinking, confusion, and expressed workflow in the forms of arrows connecting steps
non-evaluation (Table 1). For trust, students expressed their or stages in a sequential manner; fictional expressed ideas
belief that scientific information portrayed in GenAI is reli- about GenAI-science epistemic reading in the form of car-
able, such as GenAI can distinguish right or wrong scientific toon. For writing, three codes were inductively generated,
information; for critical thinking, students expressed that namely time marker, point form, and narrative. Time marker
they would not trust what GenAI said about science, while connects sentences by connectives such as “first” and “then”;
they might need to fact check in other sources of informa- point form expressed phrases preceding bullet points or
tion; for confusion, students did not express whether they dashes; narrative expressed their ideas about GenAI-science
believed the scientific information created in GenAI, rather epistemic reading in the form of a story.
they expressed puzzlement; for non-evaluation, students did Most students represented ideas about GenAI-science epis-
not express stance regarding trusting scientific information temic reading in the form of fictional drawing (Q1, 70.5%;
in GenAI or confusion. Q2, 52.3%) and writing (Q1, 65.9%; Q2, 47.7%) in narrative
Epistemic knowledge concerns how GenAI creates sci- (Fig. 2). Fictional drawing and narrative are the most apt forms
entific information. Under epistemic knowledge, five codes for students to express (1) how GenAI creates scientific texts
were inductively generated from the data, namely Google and (2) how students read scientific texts created by GenAI.
 Journal of Science Education and Technology

Table 1 Categories, codes, definitions, and examples of presentational meaning of ideas about GenAI-science epistemic reading
Categories Codes Definitions Examples
Drawing Writing

Epistemic Trust Express firm First, science topic be found. And ChatGPT
Trust belief on find what scientist said. Then, it find some.
GenAI can It sum up, Then it the science topic is True
distinguish of false.
true or false (Student 01, Q1)
claims

(Student 01, Q1)

Critical thinking Express critical They know nothing, sometime so convenient,
stance on and choose the main point. Don’t copy ALL.
GenAI- (Student 40, Q2)
scientific
information

(Student 40, Q2)

Confusion Express uncer- Everytime I asked that ChatGPT to generate a
tainty and text for my homework or anything else, all it
puzzlement done is giving me the doubt. After it loaded,
when reading it says “I don’t understand, Everytime I read
GenAI- that, I was wondering why in there.
scientific (Student 08, Q1)
information

(Student 08, Q1)

Non-evaluation Did not express First, ChatGPT get sources from big data.
any stance Next, find some useful topic and organise to a
regarding text. Last, generates texts to users.
authority (Student 04, Q1)
of GenAI-
scientific texts
(e.g. only
describe the
mechanism of
GenAI with- (Student 04, Q1)
out evaluative
elements)

Orientational Meaning Such meaning was classified as orientational meaning

instead of presentational meaning because represen-
Orientational meaning refers to individuals’ interac- tation of GenAI/science can be shaped by the viewer
tion with other people and the world (Lemke, 1998; through various modalities. Under nature of science,
Tang et al., 2019). In the context of ideas about GenAI- two categories were inductively generated, explicit and
science epistemic reading, orientational meaning implicit. For explicit, students formally use different
describes how people explicitly or implicitly draw on elements that relate epistemic practices or epistemol-
their understanding of the nature of GenAI and sci- ogy of science to represent their reading of scientific
ence while interacting with GenAI tools (Table 1). texts created in GenAI. In visual mode, students can
Journal of Science Education and Technology

undefined(continued)
Categories Codes Definitions Examples
Drawing Writing

Epistemic Google search Equates GenAI 1. get what is the aim of reader
knowledge as the Inter- 2. search information from its knowledge
net search 3. write texts
engine such 4. check the texts
as checking 5. send to the reader
information (Student 14, Q1)

(Student 14, Q1)

Large language Indicates that 1. Know the instruction
model GenAI is 2. use large language model to write essay
based on 3.
pre-trained 4. check
data and can 5. Send the text
reorganise (Student 22, Q1)
such data

(Student 22, Q1)

Human-like Equates GenAI The woman is the virtual image of ChatGPT,
with human and she is a writer, and she can write many
texts. Writer.
(Student 12, Q1)

(Student 12, Q1)

General function Describes input Text what do you want to see
and output of Load for a few sec
GenAI with- Have a story
out explaining (Student 13, Q1)
the mecha- (Student 13, Q1)
nisms
Simple Conversa- Expresses two- yes, literally the same picture I drew earlier
tion way conversa- (Student 08, Q2)
tion between
two parties

(Student 08, Q2)

draw how scientists discover knowledge and generate of science. For example, Student 42 (Q1) drew a planet
claims, to formally represent nature of science. On the and wrote “science topic” without indicating how sci-
other hand, in written mode, metalanguages in science entists carry out investigations in relation to the planet.
encompass words or phrases of how science works, such Similarly, for nature of GenAI, explicit refers to how
as “carrying out scientific investigation” (Student 17, students formally use elements related to epistemology of
Q1); for implicit, students did not represent formal ele- GenAI to represent their reading of GenAI scientific texts.
ments related to epistemic practices of epistemology For example, students can visualise how large language
 Journal of Science Education and Technology

Fig. 1  Presentational meaning

of students’ multimodal repre-
sentations of how ChatGPT/Poe
generated scientific texts (Q1)
and their ideas about epistemic
reading of scientific texts in
ChatGPT/Poe (Q2) (N = 44)

model can use fine-tuning strategies to get scientific infor- Discussion

mation needed by the users (Student 23, Post Q1). Also,
students might articulate using terminologies specific to Using a Likert scale questionnaire to measure student’s
the nature of GenAI, such as “LLM” (Student 10, Post epistemic knowledge of GenAI tools in creating scientific
Q1). On the other side, implicit refers to that students rep- texts is challenging. In contrast to the previous approach
resented their epistemic knowledge of GenAI in a subtle of using a Likert scale questionnaire on AI literacy, this
manner without articulating how GenAI works. For exam- study revealed the effectiveness of a multimodal approach
ple, students can just draw two speech bubbles represent- in characterising students’ multimodal representations of
ing human-GenAI interaction without accounting for how ideas about GenAI-science epistemic reading. Specifically,
GenAI works (Student 10, Q1), as well as giving a general we drew on representational meanings from Lemke (1998)
written account of function of GenAI (Student 10, Q1). to interpret the diverse semiotic resources through which
Figure 3 shows the orientational meaning of stu- students made meaning of ideas about GenAI-science
dents’ multimodal representations of ideas about epistemic reading from both a character and objective
GenAI-science epistemic reading. In students’ viewpoint.
responses in both third-person (Q1) and first-person The key contribution of this study was to offer an ana-
viewpoint (Q2), a high percentage of students used lytical framework that characterises students’ multimodal
both visual and written modes to represent implicit representations of ideas about GenAI-science epistemic
nature of science. On the other hand, more students reading. This analytical framework, which encompasses
represented explicit nature of GenAI in pre-instruction six categories and 18 codes, offers teachers a theoretical
in both drawn and written modes in both Q1 (52.3%) tool to diagnose students’ epistemic knowledge of how
and Q2 (36.4%). In contrast, more students tended to GenAI creates scientific texts. Such a framework chal-
give implicit nature of science, indicated by 81.8% of lenges the use of Likert scale items to elicit students’
drawn/written expression in Q1 and 61.4% in drawn/ epistemic knowledge regarding the interaction between
written expression in Q2. two domains, such as science and religion (Konnemann
Journal of Science Education and Technology

Table 2 Categories, codes, definitions, and examples of organisational meaning of ideas about GenAI-science epistemic reading
Categories Codes Definitions Examples

Drawing Flowchart Express workflow and

processes in form of a flow
diagram (e.g., arrows)

(Student 28, Q2)

Fictional Depict the reading process
with figures or imaginary
characters (e.g., cartoon
figures)

(Student 24, Q1)

Writing Time marker Indicate sequence by connec- First , ChatGPT searches some information with science in the
tives such as first, then browser, then turn it into useful information. After that, collect and
tidy it. Finally, generates it text through software.
(Student 27, Q1)
Point Form Express in simple phrases 1. Receive the text
with dash/bullets in the front 2. Read it
(e.g., with numbering) 3. Try to know what it means
4. Pass to the brain
5. Remember the details
(Student 22, Q2)
Narrative Express with a lengthy I think ChatGPT will search many information from internet and
description without point then that ChatGPT will make a long article for people who need.
form or two or more time (Student 29, Q1)
markers

et al., 2018). Students’ diverse ideas about GenAI-science (Tang et al., 2019) to discuss and refine students’ ideas
epistemic reading cannot be fully explored if teachers limit about GenAI-science epistemic reading.
them to written expressions only. For example, as indi- Compared to other studies on GenAI-science epis-
cated in Fig. 1, students tended to use dialogue bubbles temic reading, the results indicate students’ diverse
to depict simple human-GenAI conversations instead of semiotic resources in meaning-making of these ideas.
relying solely on written texts, as drawing was considered For example, students expressed confusion and critical
a more appropriate form (Kress, 2010) to represent such thinking under epistemic trust. These semiotic resources
interactions. Therefore, teachers should provide explicit cannot be indicated by Likert scale questions that ask
opportunities for students to reflect on the epistemology of students to endorse options ranging from strongly agree
science and GenAI by using multimodal representations. to strongly disagree. More importantly, the ways of how
This analytical framework can serve as a metalanguage students represent their ideas about GenAI-science epis-
temic reading can be shown in different organisational
 Journal of Science Education and Technology

Fig. 2  Organisational meaning

of students’ multimodal repre-
sentations of how ChatGPT/Poe
generated scientific texts (Q1)
and their ideas about epistemic
reading of scientific texts in
ChatGPT/Poe (Q2) (N = 44)

meanings, such as point form and time marker in the There are a few limitations needed to be acknowl-
written mode. Such an analytical framework can help edged. Firstly, only two classes of students’ multi-
teachers scaffold and guide students to express their modal representations were collected. We envisage
ideas about GenAI-science epistemic reading in stu- that future studies can compare students’ multimodal
dent’s interaction with GenAI. representations on their ideas about GenAI-science
In terms of methodological considerations, we have epistemic reading across grade levels. Secondly,
ensured the reliability of the coding process through cal- this paper does not compare the differences of stu-
culating intercoder reliability for each category, as well dents’ multimodal representations on their ideas
as provided examples for the definition of each code. We about GenAI-science epistemic reading before and
envisage that future works will modify and refine our ana- after explicit-instruction. The study does not engage
lytical framework of students’ ideas about GenAI-science students in interacting with GenAI. Future research
epistemic reading when it is applied to pre/post interven- studies can compare students’ such multimodal rep-
tion or diverse population of students. resentations under explicit-reflective in experimental
Journal of Science Education and Technology

Table 3 Categories, codes, definitions, and examples of orientational meaning of ideas about GenAI-science epistemic reading
Categories Codes Definitions Examples
Drawing Writing

Nature of Science Explicit Formally relate outputs of ChatGPD provides some scientific
ChatGPT to epistemic prac- essay and information to the
tices/epistemology of science reader base on the internet
information. GDP not only
carried out scientific investiga-
tion to generate texts, it also
suggest the useful knowledge to
the reader.
(Student 17, Q1)
(Student 01, Q1)
Implicit Mention elements of science After asking ChatGPT/Poe ques-
(e.g., planets) but do not con- tions about a science topic, it
nect to epistemic practices/ will search through its data and
epistemology of science find the answer needed to pro-
vide answer. (Student 02, Q1)

(Student 42, Q1)

Nature of GenAI Explicit Formally relate outputs of First, ChatGPT searches some
ChatGPT to how GenAI information with science in the
generates information (e.g., browser, then turn it into useful
large language models) information. After that, collect
and tidy it. Finally, generates it
to text through software. (Stu-
dent 27, Q1)

(Student 27, Q1)

Implicit Mention elements of GenAI Give information correctly.
but do not connect to epis- (Student 10, Q1)
temic practices/epistemology
of GenAI (e.g. large language
models)

’
(Student 10, Q1)

and controlled groups. Thirdly, we engaged students Conclusion

to create multimodal representations on their ideas
about GenAI-science epistemic reading because our In conclusion, we developed an analytical framework that
project targets on language and literacy in extracur- characterises students’ multimodal representations on
ricular lessons. Though it is not a common practice ideas about GenAI-science epistemic reading. Drawing on
in science classrooms, we advocate that students and Lemke’s (1998) presentational, orientational, and organisa-
teachers should explicitly talk about their ideas about tional meanings, we characterised diverse semiotic resources
GenAI-science epistemic reading. Otherwise, students in students’ expressing these ideas. Epistemic reading is cru-
will just blindly trust the outputs created in GenAI, cial for students’ reading of scientific texts from all sources
though they just know verbal terminologies like “large such as GenAI, as they need to draw on their epistemic
language models.” Hence, there could be future efforts knowledge of science and epistemic knowledge of another
in engaging students’ expressions of ideas about discipline. We anticipate these categories will serve as a
GenAI-science epistemic reading using multimodal metalanguage for teachers to negotiate with students about
representations. their ideas when they read scientific texts created in GenAI.
 Journal of Science Education and Technology

Fig. 3  Orientational meaning of students’ multimodal representations of how ChatGPT/Poe generated scientific texts (Q1) and their ideas about
epistemic reading of scientific texts in ChatGPT/Poe (Q2) (N = 44)

Author Contribution All authors contributed to the study conception Competing Interests The authors declare no competing interests.
and design. Material preparation, data collection, and analysis were
performed by KC and JM. The first draft of the manuscript was writ-
Open Access This article is licensed under a Creative Commons Attri-
ten by KC, and all authors commented on versions of the manuscript.
bution 4.0 International License, which permits use, sharing, adapta-
All authors read and approved the final manuscript.
tion, distribution and reproduction in any medium or format, as long
as you give appropriate credit to the original author(s) and the source,
Funding Open access publishing enabled by City University of
provide a link to the Creative Commons licence, and indicate if changes
Hong Kong Library’s agreement with Springer Nature. The study
were made. The images or other third party material in this article are
reported in this manuscript is based upon work supported by the
included in the article’s Creative Commons licence, unless indicated
Quality Education Fund, Hong Kong SAR Government (Grant Num-
otherwise in a credit line to the material. If material is not included in
ber: 9420033).
the article’s Creative Commons licence and your intended use is not
permitted by statutory regulation or exceeds the permitted use, you will
Data Availability The data is not available publicly owing to restric-
need to obtain permission directly from the copyright holder. To view a
tions of the ethical approval.
copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Declarations
Ethics Approval The second author obtained ethical approval from his insti-
tution. This project was approved by the City University of Hong Kong. References
Consent to Participate Informed consent was obtained from all indi-
Airey, J., & Linder, C. (2017). Social semiotics in university physics
vidual participants included in the study.
education. Multiple representations in physics education, 95–122.
https://​doi.​org/​10.​1007/​978-3-​319-​58914-5_5
Consent for Publication Informed consent was obtained from all indi-
vidual participants included in the study.
Journal of Science Education and Technology

Alasadi, E. A., & Baiz, C. R. (2024). Multimodal generative artifi- novel narratives on climate change through ChatGPT. Paper
cial intelligence tackles visual problems in chemistry. Journal of presented at the International Conference on Interactive Digital
Chemical Education., 101, 2716–2729. Storytelling.
Barak, M., Yachin, T., & Erduran, S. (2022). Tracing preservice Jeon, J., Lee, S., & Choe, H. (2023). Beyond ChatGPT: A conceptual
teachers’ understanding of nature of science through their draw- framework and systematic review of speech-recognition chatbots
ings and writing. Research in Science Education. https://​doi.​ for language learning. Computers & Education, 206, 104898.
org/​10.​1007/​s11165-​022-​10069-3 Johns, R. (2010). Likert items and scales. Survey Question Bank:
Barelli, E., Lodi, M., Branchetti, L., & Levrini, O. (2024). Epistemic Methods Fact Sheet, 1(1), 11–28.
insights as design principles for a teaching-learning module on Kim, W. J. (2022). AI-integrated science teaching through facilitating
artificial intelligence. Science & Education, 1–36. https://​doi.​ epistemic discourse in the classroom. Asia-Pacific Science Educa-
org/​10.​1007/​s11191-​024-​00504-4 tion, 8(1), 9–42. https://​doi.​org/​10.​1163/​23641​177-​bja10​041
Billingsley, B., & Hardman, M. (2017). Epistemic insight and the Kim, K., Kwon, K., Ottenbreit-Leftwich, A., Bae, H., & Glazewski,
power and limitations of science in multidisciplinary arenas. K. (2023). Exploring middle school students’ common naive
School Science Review, 99, 367. conceptions of artificial intelligence concepts, and the evolution
Bråten, I., Britt, M. A., Strømsø, H. I., & Rouet, J.-F. (2011). The role of these ideas. Education and Information Technologies. https://​
of epistemic beliefs in the comprehension of multiple expository doi.​org/​10.​1007/​s10639-​023-​11600-3
texts: Toward an integrated model. Educational Psychologist, Konnemann, C., Höger, C., Asshoff, R., Hammann, M., & Rieß,
46(1), 48–70. W. (2018). A role for epistemic insight in attitude and belief
Chang, H.-Y., & Tsai, C.-C. (2023). Epistemic network analysis change? Lessons from a cross-curricular course on evolution
of students’ drawings to investigate their conceptions of sci- and creation. Research in Science Education, 48(6), 1187–1204.
ence learning with technology. Journal of Science Education https://​doi.​org/​10.​1007/​s11165-​018-​9783-y
and Technology, 32(2), 267–283. https://​d oi.​o rg/​1 0.​1 007/​ Kress, G. (2010). Multimodality: A social semiotic approach to
s10956-​022-​10026-9 contemporary communication. Routledge.
Cheung, K. K. C., & Tai, K. W. (2023). The use of intercoder reli- Lemke, J. L. (1998). Multiplying meaning: Visual and verbal semi-
ability in qualitative interview data analysis in science educa- otics in scientific text. In J. Martin & R. Veel (Eds.), Reading
tion. Research in Science & Technological Education, 41(3), science. Routeledge.
1155–1175 Mellor, D., & Moore, K. A. (2014). The use of Likert scales with
Cheung, K. K. C., Long, Y., Liu, Q., & Chan, H. Y. (2024a). Unpack- children. Journal of pediatric psychology, 39(3), 369–379
ing epistemic insights of Artificial Intelligence (AI) in Science Ng, D. T. K., Wu, W., Leung, J. K. L., Chiu, T. K. F., & Chu, S. K.
Education: A Systematic Review. Science & Education, 1–31. W. (2023a). Design and validation of the AI literacy question-
Cheung, K. K. C., Pun, J. K., & Li, W. (2024b). Students’ holistic naire: The affective, behavioural, cognitive and ethical approach.
reading of socio-scientific texts on climate change in a ChatGPT British Journal of Educational Technology. https://​doi.​org/​10.​
scenario. Research in Science Education, 54(5), 957–976. 1111/​bjet.​13411
Conley, A. M., Pintrich, P. R., Vekiri, I., & Harrison, D. (2004). Ng, D. T. K., Wu, W., Leung, J. K. L., & Chu, S. K. W. (2023b). Arti-
Changes in epistemological beliefs in elementary science stu- ficial Intelligence (AI) literacy questionnaire with confirmatory
dents. Contemporary Educational Psychology, 29(2), 186–204. factor analysis. Paper presented at the 2023 IEEE International
Cooper, G. (2023). Examining science education in chatgpt: An Conference on Advanced Learning Technologies (ICALT).
exploratory study of generative artificial intelligence. Journal Nigam, A., Pollice, R., Hurley, M. F., Hickman, R. J., Aldeghi, M.,
of Science Education and Technology, 32(3), 444–452. Yoshikawa, N., …, Aspuru-Guzik, A. (2021). Assigning confi-
Cooper, G., & Tang, K.-S. (2024). Pixels and pedagogy: Examining dence to molecular property prediction. Expert opinion on drug
science education imagery by generative artificial intelligence. discovery, 16(9), 1009–1023.
Journal of Science Education and Technology, 1–13. https://​doi.​ O’Halloran, K. L. (2011). Multimodal discourse analysis. The
org/​10.​1007/​s10956-​024-​10104-0 Bloomsbury handbook of discourse analysis (pp. 249–282).
Erduran, S., & Levrini, O. (2024). The impact of artificial intelligence Origgi, G. (2012). Epistemic injustice and epistemic trust. Social
on scientific practices: An emergent area of research for science Epistemology, 26(2), 221–235.
education. International Journal of Science Education, 1–8. Paap, K. R., Anders-Jefferson, R. T., Balakrishnan, N., & Majoubi, J.
https://​doi.​org/​10.​1080/​09500​693.​2024.​23066​04 B. (2024). The many foibles of Likert scales challenge claims that
Erduran, S. (2023). AI is transforming how science is done. Science self-report measures of self-control are better than performance-
education must reflect this change. American Association for the based measures. Behavior Research Methods, 56(2), 908–933.
Advancement of Science, 382, eadm9788. Park, J., Teo, T. W., Teo, A., Chang, J., Huang, J. S., & Koo, S.
Floridi, L., & Chiriatti, M. (2020). GPT-3: Its nature, scope, limits, (2023). Integrating artificial intelligence into science les-
and consequences. Minds and Machines, 30, 681–694. sons: Teachers’ experiences and views. International Jour-
Ford, M. J., & Forman, E. A. (2006). Chapter 1: Redefining disci- nal of STEM Education, 10(1), 61. https://​d oi.​o rg/​1 0.​1 186/​
plinary learning in classroom contexts. Review of research in s40594-​023-​00454-3
education, 30(1), 1–32. Peirce, C. S. (1931). Logic as semiotic: The theory of signs. In B.
Fui-Hoon Nah, F., Zheng, R., Cai, J., Siau, K., & Chen, L. (2023). Gen- Justus (Ed.), Philosophical writings of Peirce (1893–1910) (pp.
erative AI and ChatGPT: Applications, challenges, and AI-human 98–119). Dover.
collaboration. In (Vol. 25, pp. 277–304): Taylor & Francis. Prain, V., & Tytler, R. (2012). Learning through constructing repre-
Grapin, S. E., Haas, A., Llosa, L., Wendel, D., Pierson, A., & Lee, sentations in science: A framework of representational construc-
O. (2023). Multilingual learners’ epistemologies in practice in tion affordances. International Journal of Science Education,
the context of computational modeling in an elementary science 34(17), 2751–2773.
classroom. Journal of Research in Science Teaching. https://​doi.​ Quinto-Pozos, D., & Parrill, F. (2015). Signers and co-speech ges-
org/​10.​1002/​tea.​21850 turers adopt similar strategies for portraying viewpoint in nar-
Gursesli, M. C., Taveekitworachai, P., Abdullah, F., Dewantoro, ratives. Topics in Cognitive Science, 7(1), 12–35.
M. F., Lanata, A., Guazzini, A., …, Thawonmas, R. (2023).
The chronicles of ChatGPT: Generating and evaluating visual
 Journal of Science Education and Technology

Schraw, G., Bendixen, L. D., & Dunkle, M. E. (2012). Development Tu, Y.-F., & Hwang, G.-J. (2023). University students’ conceptions of
and validation of the Epistemic Belief Inventory (EBI). In Per- ChatGPT-supported learning: A drawing and epistemic network
sonal epistemology (pp. 261–275): Routledge. analysis. Interactive Learning Environments, 1–25. https://d​ oi.o​ rg/​
Stake, R. E. (2006). Multiple case study analysis. Guilford Press. 10.​1080/​10494​820.​2023.​22863​70
Subedi, B. P. (2016). Using Likert type data in social science van Leeuwen, T. (2005). Introducing social semiotics. Routledge.
research: Confusion, issues and challenges. International Jour- Wanselin, H., Danielsson, K., & Wikman, S. (2021). Analysing multi-
nal of Contemporary Applied Sciences, 3(2), 36–49. modal texts in science—A social semiotic perspective. Research
Tang, K.-S., Won, M., & Treagust, D. (2019). Analytical framework in Science Education.
for student-generated drawings. International Journal of Science Wilholt, T. (2013). Epistemic trust in science. The British Journal for
Education, 41(16), 2296–2322. the Philosophy of Science. https://​doi.​org/​10.​1093/​bjps/​axs007
Tang, K.-S., Jeppsson, F., Danielsson, K., & Bergh Nestlog, E. (2022a). Yeh, H.-Y., Tsai, Y.-H., Tsai, C.-C., & Chang, H.-Y. (2019). Investigat-
Affordances of physical objects as a material mode of representa- ing students’ conceptions of technology-assisted science learning:
tion: A social semiotics perspective of hands-on meaning-making. A drawing analysis. Journal of Science Education and Technol-
International Journal of Science Education, 44(2), 179–200. https://​ ogy, 28(4), 329–340. https://​doi.​org/​10.​1007/​s10956-​019-​9769-1
doi.​org/​10.​1080/​09500​693.​2021.​20213​13
Tang, K.-S., Lin, S.-W., & Kaur, B. (2022b). Mapping and extending Publisher's Note Springer Nature remains neutral with regard to
the theoretical perspectives of reading in science and mathemat- jurisdictional claims in published maps and institutional affiliations.
ics education research. International Journal of Science and
Mathematics Education, 20(Suppl 1), 1–15.
Tang, K.-S., & Cooper, G. (2024). The role of materiality in an era
of generative artificial intelligence. Science & Education, 1–16.