MULTIMODAL ANALYSIS

TEXTS AND ARTEFACTS

INTERNATIONAL MULTIMODAL COMMUNICATION I

2024-2025
what is a multimodal text?
● Many texts are multimodal, in which all the information is expressed
by integrating different or even more modes.

● In a multimodal text, each mode has its own role and function

(Kress, 2010) in the meaning-making process, and usually only

carries a portion of the message.

● Written language, spoken language, and visual, auditory, gestural,

tactile, and spatial, patterns of meaning are all modes.

● Different kinds of modes so that we take in information from other

people (Mamiko, 2010)

 "Meaning is made in all modes which are available in a culture. These

modes come together in the multimodal ensemble of a communicative event"

(Kress, 2010).
mode versus medium
● A mode is a communication ● The route or system through
method. which communications are
● Written language, spoken transmitted is referred to as a
language, and visual, auditory, medium.
gestural, tactile, and spatial ● Media is the plural variant of
patterns of menaing are all medium.
modes. ● Usually, it refers to objects
involved in transmitting the
message such as telephones, TV,
radio, newspaper, etc.
different modes in communication
WRITTEN MEANING ● Conveyed through written language via
handwriting, the printed page, and the
SPOKEN MEANING screen.
● Choices of words, phrases, and sentences
VISUAL MEANING
are organized through linguistic grammar

AUDIO MEANING conventions, register (where language is

varied according to context), and genre
SPATIAL MEANING (knowledge of how a text type is
organized ans staged to meet a specific
GESTURE MEANING purpose).
different modes in communication
WRITTEN MEANING ● Conveyed through spoken language via live or
recorded speech and can be monologic or
SPOKEN MEANING dialogic.
● Composing oral meaning includes choices
VISUAL MEANING
around mood, emotion, emphasis, fluency,

AUDIO MEANING speed, volume, tempo, pitch, rhythm,

pronunciation, intonation, and dialect.
SPATIAL MEANING

GESTURE MEANING
different modes in communication
WRITTEN MEANING ● Conveyed through choices of visual resources
and includes both still image and moving
SPOKEN MEANING images.
● Images may include diverse cultural
VISUAL MEANING connotations, symbolism and portray
different people, cultures and practices.
AUDIO MEANING ● Visual resources include: framing, vectors,
symbols, perspective, gaze, point of view,
SPATIAL MEANING color, texture, line, shape, casting, saliency,
distance, angles, form, power,
involvement/detachment, contrast, lighting,
GESTURE MEANING naturalistic/non-naturalistic, camera movement,
and subject movement.
different modes in communication
WRITTEN MEANING ● Conveyed through sound, including choices of
music representing different cultures,
SPOKEN MEANING ambient sounds, noises, alerts, silence,
natural/unnatural sound, and use of volume,
VISUAL MEANING beat, tempo, pitch, and rhythm.
● Lyrics in a song may also include multiple
AUDIO MEANING
languages.

SPATIAL MEANING

GESTURE MEANING
different modes in communication
WRITTEN MEANING ● Conveyed through design of spaces, using
choice sof spatial, resources including:
SPOKEN MEANING scale, proximity, boundaries, direction,
layout, and organization of objects in the
VISUAL MEANING space.
● Space extends from design of the page in a
AUDIO MEANING
book, a page in a graphic novel, or comic, a
webpage on a screen, framing of shots in
SPATIAL MEANING
moving images, to the design of a room,
architecture, streetscapes, and landscapes.
GESTURE MEANING
different modes in communication
WRITTEN MEANING ● Conveyed through choices of body movement,
facial expression, eye movements and gaze,
SPOKEN MEANING demeanor, gait, dance, acting, action
sequences.
VISUAL MEANING ● It also includes rhythm, speed, stillness
and angles, including ‘timing, frequency,
AUDIO MEANING
ceremony and ritual’ (Cope & Kalantzis,
2009).
SPATIAL MEANING
● Gestures and body language may have diverse
cultural connotations.
GESTURE MEANING
multimodal communication in practice
● Multimodality does not necessarily mean the use of technology, and
multimodal texts can be paper-based, live, or digital.
● Paper-based multimodal texts include picture books, text books,
graphic novels, comics, and posters.
● Live multimodal texts, por example, dance, performance, and oral
storytelling, convey meaning through combinations of various modes
such as gestural, spatial, audio, and oral language.
● Digital multimodal texts include film, animation, slide shows, e-
posters, digital stories, podcasts, and web pages that may include
hyperlinks to external pronunciation guides or translations.
so …
KEY CONCEPTS IN MULTIMODAL ANALYSIS

MODES

MULTIMODAL ENSEMBLE

ARTEFACTS
so …
KEY CONCEPTS IN MULTIMODAL ANALYSIS

MODES

● These are the different forms of communication, such as

linguistic, visual, audio, gestural, and spatial.
● Modes are not limited to written and spoken language but include
any resource used for communication.
so …
KEY CONCEPTS IN MULTIMODAL ANALYSIS

MULTIMODAL ENSEMBLE

● This refers to the combination of different modes in a

communicative act.
● The interplay between these modes can enhance, complement, or
even contradict each other to produce complex meanings.
so …
KEY CONCEPTS IN MULTIMODAL ANALYSIS

ARTEFACTS

● These are physical objects or digital items that carry

communicative potential, such as books, websites, advertisements,
tools, and clothing.
● Artefacts can be analyzed for their semiotic resources and how
they contribute to communication.
MULTIMODAL COMMUNICATION
In a print advertisement for a
car, the text may highlight the
features and benefits, while the
image shows the car in a
desirable setting, and the layout
guides the viewer's attention.

Kress and van Leeuwen (2006) state, "In multimodal texts, meaning is realized
through the co-deployment of semiotic resources whose relationships are complex and
interdependent" (p. 177).
MULTIMODAL COMMUNICATION
A science teacher might use a
combination of spoken
explanations, diagrams on a
whiteboard, and digital
simulations to explain a concept.

According to Jewitt (2008), "The educational environment is rich in multimodal

resources. Understanding how these resources are used can provide insights into the
learning process and the communication of knowledge" (p. 246).
MULTIMODAL COMMUNICATION
On Instagram, a post may combine
a photograph with hashtags,
emojis, and text captions to
convey a message.

Zhao and Zappavigna (2018) argue that "social media platforms are inherently
multimodal, leveraging text, image, and other modes to create and share meaning in
ways that are interactive and dynamic" (p. 392).
analytical approach … let’s consider three…

Systemic Functional Multimodal Discourse Analysis (SF-MDA)

Integration of modes

Functional approach ideotional metafunction

interpersonal metafunction

textual metafunction

Contextual consideration
applications
 Systemic Functional Multimodal Discourse Analysis (SF-MDA)

● Systemic Functional Multimodal Discourse Analysis offers a robust

framework for exploring how different modes combine to perform
specific functions in multimodal texts.
● By extending the principles of systemic functional linguistics, SF-
MDA enables a nuanced and detailed understanding of the multifaceted
nature of communication in contemporary media landscapes.
● This approach is invaluable for researchers, educators, and
professionals seeking to unpack the layers of meaning in complex
multimodal interactions.
analytical approach … let’s consider three…

Social Semiotics

Semiotic resources

Social meaning

Contextual analysis

Multimodality
applications
 Social Semiotics

● Social Semiotics offers a rich framework for understanding the

social meanings of semiotic resources and their use in different
contexts.
● By emphasizing the dynamic and contextual nature of meaning-making,
it provides valuable insights into how communication functions
within social practices.
● This approach is particularly powerful for analyzing complex,
multimodal texts and for exploring the intricate ways in which signs
and symbols shape and are shaped by social interactions.
analytical approach … let’s consider three…

Visual Analysis

Semiotic resources in images

Interaction with text

Multimodal integration

Contextual interpretation
applications
 Visual Analysis

● Visual Analysis is a vital tool in multimodal analysis, offering a

detailed understanding of how images contribute to meaning-making in
conjunction with text and other modes.
● By examining visual elements and their interactions within specific
contexts, this approach provides valuable insights into the complex
processes of communication and interpretation in various media.
● Whether in advertisements, social media, or educational materials,
visual analysis helps uncover the nuanced ways in which visual and
textual elements work together to convey messages and engage
audiences.
example on media literacy

● What does the text say?

○ How does the text relate to the images?

● What does the image say?

○ How do the images relate to the text?

● Who is the
magazine/newspaper/billboard’s target
market?
○ What are the values of the ad contains?
example on media literacy
questions to reflect visual and
textual
integration

1. How do the visual elements (e.g., imagery of athletes, use of

color, composition) in the "Dream Crazy" campaign work together
with the textual elements (e.g., slogans, captions) to convey
the campaign's message?
2. What role does the use of black and white imagery play in the
overall impact of the campaign?
narrative and
3. What stories or narratives are being told through the ideational
metafunctions

combination of visuals and text in the campaign?

4. How do these narratives relate to broader social and cultural
contexts, such as the themes of perseverance, equality, and
social justice?
questions to reflect
interpersona
l
metafunction

1. How does the campaign establish a connection with its audience

through its multimodal elements? Consider aspects like tone,
language, and the portrayal of athletes.
2. In what ways does the campaign's use of high-profile athletes and
public figures, such as Colin Kaepernick, influence the audience's
perception and engagement with the message?
3. How does the layout and sequence of the visuals and text in the textual
metafunctions
and structure
campaign guide the viewer’s attention and create a cohesive
narrative?
4. What is the significance of the tagline “Believe in something. Even
if it means sacrificing everything.” in the context of the
campaign’s visual and textual presentation?
questions to reflect contextual
considerations

1. How do the social, cultural, and historical contexts influence the

interpretation of the "Dream Crazy" campaign’s multimodal elements?
2. What are the potential cultural differences in interpreting the
campaign’s message globally, considering Nike’s diverse audience?
3. How do the multimodal elements of the campaign contribute to its
ability to spark social movements and conversations around topics impact and
social
dynamics
such as racial equality, social justice, and activism?
4. In what ways has the campaign succeeded or failed in creating a
socially dynamic impact, and how do the multimodal aspects play a
role in this?
questions to reflect critical
perspectives

1. What are some potential criticisms of the "Dream Crazy" campaign,

and how might these criticisms relate to its use of multimodal
texts?
2. How does the campaign navigate the balance between commercial
interests and social advocacy through its multimodal strategies?
3. How do different audience demographics (e.g., age, race, audience
engagement
and reception

socioeconomic status) perceive and respond to the multimodal

elements of the campaign?
4. What role do social media platforms play in the dissemination and
reception of the "Dream Crazy" campaign’s multimodal message?
questions to reflect ethical
considerations

1. What ethical considerations arise from the use of multimodal texts

in the "Dream Crazy" campaign, particularly concerning the portrayal
of social issues and the use of athletes' personal narratives?
2. How does Nike’s corporate identity and history influence the ethical
reception of the campaign’s multimodal messages?
3. How might the "Dream Crazy" campaign influence future advertising
future
strategies in terms of using multimodal texts to address social implications

issues?
4. What lessons can be learned from the campaign regarding the power
and responsibility of brands in shaping social narratives through
multimodal communication?
references
● Kress, G. (2010). Multimodality: A Social Semiotic Approach to
Contemporary Communication. Routledge.
● Jewitt, C. (2009). The Routledge Handbook of Multimodal Analysis.
Routledge.
● van Leeuwen, T. (2005). Introducing Social Semiotics. Routledge.
● Zhao, S., & Zappavigna, M. (2018). Social Media and the New Writing.
Cambridge University Press.